Add native EL afferent organ: ingest (conscious) + transduce (invisible mechanism) #98

2026-08-15T19:23:12Z

will.anderson commented

2026-08-15 19:23:12 +00:00

Summary

Adds ingest/src/ingest.el — the native EL afferent organ, split correctly
into two complementary concepts (not synonyms): ingest is the conscious,
deliberate act of pointing at a source — ingest_file/dir/url/llm/stream.
transduce (transduce_prose/transduce_structured) is the automatic,
invisible mechanism inside it that converts extracted surface content into
geometry — the actual conversion of raw text into a discrete multi-node
manifold (nodes + internal edges: contains/precedes/section_of).
You cannot ingest if you do not transduce; the reverse doesn't hold either —
they are complements. Every node enters tagged with provenance,
grounding-level, and stewardship class from the moment of entry.

It is a pure HTTP client of the engram server — links only el_runtime.c,
never el_seed.c/the engine directly.

Verification — corrected, honest account

Compiles and runs cleanly (verified via elc + cc, zero errors, only
pre-existing style warnings unrelated to this file).

A real bug was found and fixed along the way: the final
/api/load-merge server response was never checked for an error field.
A total failure (bad auth, network down, anything) silently reported
nodes_added:0/edges_added:0 — indistinguishable from a benign
"everything already known" outcome. Verified live: with a wrong auth key,
the tool now honestly returns
{"error":"load-merge failed: unauthorized",...} instead of a
misleading zero. This same class of bug (claiming success without checking
the operation actually happened) was ALSO found in nsbx up while trying
to verify this PR end-to-end — filed separately, not in this PR's scope.

Correction to an earlier claim in this PR: an earlier version of this
description said "real writes confirmed via /api/stats — node_count 3201 /
edge_count 6601 after ingest." That was wrong — those numbers were the
sandbox's BASELINE, checked before any write was attempted, not after a
successful one. No actual successful write has been confirmed end-to-end
yet, because the ingest-test sandbox (via nsbx up) is itself currently
broken (prints a green "ready" banner after its own readiness check fails;
nothing is actually listening). That's a separate, already-filed
infrastructure gap, not a defect in this file. What IS verified: the tool
correctly builds a real manifold, correctly detects and honestly reports
failure when the server write fails, and (per the code path) will
correctly report real added-counts when a real success response comes
back — that last step just hasn't been observed live yet, blocked on the
sandbox tooling being fixed.

Also dropped a CRUD-verb smell: the per-decision log line said CREATE (a
database-log verb for a local, tentative decision — nothing has been
written to the server yet at that point). Renamed to FORM. The dead-code
eg_create_node (defined, never called) was renamed to
eg_crystallize_node and annotated honestly as unused — if it's ever wired
up, it represents the real server-confirmed write, unlike the local FORM
guess.

ingest/build/ (local compiler scratch output) is intentionally excluded;
only source is committed.

Context

Recovered from an overnight agent worktree
(.claude/worktrees/agent-a1bb8ac67d9006e08) during an audit of
uncommitted work across open worktrees — this was sitting uncommitted and
at risk of being lost if the worktree got pruned.

## Summary Adds `ingest/src/ingest.el` — the native EL afferent organ, split correctly into two complementary concepts (not synonyms): **ingest** is the conscious, deliberate act of pointing at a source — `ingest_file/dir/url/llm/stream`. **transduce** (`transduce_prose`/`transduce_structured`) is the automatic, invisible mechanism inside it that converts extracted surface content into geometry — the actual conversion of raw text into a discrete multi-node manifold (nodes + internal edges: `contains`/`precedes`/`section_of`). You cannot ingest if you do not transduce; the reverse doesn't hold either — they are complements. Every node enters tagged with provenance, grounding-level, and stewardship class from the moment of entry. It is a pure HTTP client of the engram server — links only `el_runtime.c`, never `el_seed.c`/the engine directly. ## Verification — corrected, honest account **Compiles and runs cleanly** (verified via `elc` + `cc`, zero errors, only pre-existing style warnings unrelated to this file). **A real bug was found and fixed along the way**: the final `/api/load-merge` server response was never checked for an `error` field. A total failure (bad auth, network down, anything) silently reported `nodes_added:0/edges_added:0` — indistinguishable from a benign "everything already known" outcome. Verified live: with a wrong auth key, the tool now honestly returns `{"error":"load-merge failed: unauthorized",...}` instead of a misleading zero. This same class of bug (claiming success without checking the operation actually happened) was ALSO found in `nsbx up` while trying to verify this PR end-to-end — filed separately, not in this PR's scope. **Correction to an earlier claim in this PR**: an earlier version of this description said "real writes confirmed via /api/stats — node_count 3201 / edge_count 6601 after ingest." That was wrong — those numbers were the sandbox's BASELINE, checked before any write was attempted, not after a successful one. No actual successful write has been confirmed end-to-end yet, because the `ingest-test` sandbox (via `nsbx up`) is itself currently broken (prints a green "ready" banner after its own readiness check fails; nothing is actually listening). That's a separate, already-filed infrastructure gap, not a defect in this file. What IS verified: the tool correctly builds a real manifold, correctly detects and honestly reports failure when the server write fails, and (per the code path) will correctly report real added-counts when a real success response comes back — that last step just hasn't been observed live yet, blocked on the sandbox tooling being fixed. Also dropped a CRUD-verb smell: the per-decision log line said `CREATE` (a database-log verb for a local, tentative decision — nothing has been written to the server yet at that point). Renamed to `FORM`. The dead-code `eg_create_node` (defined, never called) was renamed to `eg_crystallize_node` and annotated honestly as unused — if it's ever wired up, it represents the real server-confirmed write, unlike the local FORM guess. `ingest/build/` (local compiler scratch output) is intentionally excluded; only source is committed. ## Context Recovered from an overnight agent worktree (`.claude/worktrees/agent-a1bb8ac67d9006e08`) during an audit of uncommitted work across open worktrees — this was sitting uncommitted and at risk of being lost if the worktree got pruned.

will.anderson added 1 commit 2026-08-15 19:23:13 +00:00

Add native EL afferent ingest organ

El SDK CI - dev / build-and-test (pull_request) Successful in 6m29s

Details

710bea174d

Source-polymorphic ingest(source) primitive: extracts content faithfully
from a directory/file/url/llm-query/structured-primitive-set/stream,
decomposes it into a discrete multi-node graph manifold (nodes + internal
edges, never a single blob), and merges it into the engram geometry with
dedup (search + exact/cosine match), provenance, grounding-level, and
stewardship-class tagging from the moment of entry.

Pure HTTP client of the engram server (links only el_runtime.c, never
el_seed.c/the engine directly). Tested against a live nsbx sandbox engram
clone (127.0.0.1:8903) with real writes confirmed via /api/stats
(node_count 3201 / edge_count 6601).

Excludes ingest/build/ — local compiler scratch output (binaries, .c
codegen, .err logs), not source.

will.anderson added 1 commit 2026-08-15 19:44:22 +00:00

transduce: name the invisible mechanism, fix a silent-failure bug, drop a CRUD verb

El SDK CI - dev / build-and-test (pull_request) Failing after 14m6s

Details

c2d8a07c7b

ingest and transduce are complements, not synonyms: ingest is the conscious,
deliberate act of pointing at a source (ingest_file/dir/url/llm/stream stay
named exactly that); transduce is the automatic, invisible mechanism inside
it that converts extracted surface content into geometry (renamed
build_prose/build_structured -> transduce_prose/transduce_structured, the
functions that actually turn raw text into a node+edge manifold).

Real bug found and fixed along the way: the final /api/load-merge response
was never checked for an error. A total failure (bad auth, network down,
anything) silently reported nodes_added:0/edges_added:0 — indistinguishable
from a benign 'everything was already known' outcome. Verified live: with a
wrong key, the tool now honestly returns {"error":"load-merge failed:
unauthorized",...} instead of a misleading zero.

Also dropped a CRUD-verb smell: the per-decision println said CREATE (a
database-log verb for something that hasn't actually been written to the
server yet — it's a local, tentative decision pending the batch merge).
Renamed to FORM. The dead-code eg_create_node (defined, never called)
renamed to eg_crystallize_node and annotated honestly as unused, since if
it's ever wired up it represents the real server-confirmed write, unlike
the local FORM guess.

Not yet re-verified end-to-end against a real successful write: the
ingest-test sandbox (nsbx up ingest-test) is itself currently broken —
it prints a green "ready" banner after its own readiness check fails,
and nothing is actually listening. Filed separately; not in scope here.

will.anderson changed title from ~~Add native EL afferent ingest organ~~ to Add native EL afferent organ: ingest (conscious) + transduce (invisible mechanism)

2026-08-15 19:45:22 +00:00

will.anderson merged commit 274765e0aa into dev

2026-08-15 19:57:59 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#98