ingest: unify transduce_prose/transduce_structured into one transduce() #117

2026-08-15T22:28:47Z

will.anderson commented

2026-08-15 22:28:47 +00:00

Summary

transduce() is now the single mechanism: one function, no content-type branch. It splits on "\n\n" as a universal boundary check, falls back to fixed 4096-char windows if none is found, and wires the same generic edges (contains/precedes/heading section_of) regardless of what's inside a chunk. Dedup is the existing find_existing_by_content path, applied uniformly.
The old transduce_structured JSON dataset/records/feature-node interpretation is deleted outright — a JSON file is now chunked and deduped like anything else, no pre-computed structure.
This unlocks raw/opaque content (audio, or anything with no natural text/JSON shape) with no DSP, no LLM call, no external API: transduce() chunks it exactly like it chunks anything else. Zero semantic understanding of audio (or any payload) is claimed or built here — meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time.
Two small C builtins added (fs_size, fs_read_b64_chunk in el_runtime.c/.h) because El strings are NUL-unsafe under strlen-based ops and fs_read() silently truncates at the first embedded NUL — routine in real binary/audio bytes. ingest_file now compares fs_read()'s string length against a real fs_size() byte count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim), joined with the same "\n\n" marker transduce()'s boundary scan already looks for. This is a mechanical fidelity fix, not content interpretation — transduce() never learns a fallback happened.
INGEST_KIND keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not a content-type flag. The redundant "structured" value is removed. ingest_dir drops its file-extension filter for the same reason.
All five ingest_* entry points are unchanged in name and role. ingest_stream was never wired to either old function (it builds its own turn-nodes directly), so it's untouched.

Verification

Compiles clean via the real elc + the modified el_runtime.c/engram_*.c — built and booted an actual sandbox engram off this exact source (nsbx create --branch origin/dev).
Local manifold construction confirmed correct against a real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte real prefix slice): exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice — matches ceil(bytes/3072)+1 nodes, 2n-1 edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes.

Known gap (disclosed, not papered over)

End-to-end server-confirmed persistence (a real before/after /api/stats delta, and a fetched node by id) for the audio/prose/JSON cases was not obtained this session. Every local nsbx sandbox tried (two stock pre-#109 binaries hitting the known O(N·D) brute-force scan bug, then a fresh #109/HNSW binary built from current dev) took minutes-to-indefinitely-long on the final /api/load-merge write's embedding step and hit the client's 60s HTTP timeout, even for a 5-node write. Confirmed as real (if slow) forward progress via growing WAL file size, not a hang. The code's own pre-existing honesty gate correctly refused to report success in every case ("load-merge failed: ..." + "nothing below this manifold was confirmed persisted by the server") rather than silently claiming a false success. This is an environment/infrastructure characteristic tonight, not a defect introduced by this change — the engram server binary itself is untouched here.

Test plan

Confirm a fast/idle sandbox engram, then re-run INGEST_KIND=file INGEST_ARG=<audio file> ingest/build/ingest and capture a real before/after /api/stats delta
Fetch one newly-created node by id via /api/nodes/<id> and confirm real, non-empty base64 content
Re-run the same for a plain-text file and a {"dataset":...,"records":[...]}-shaped JSON fixture to confirm no regression on either case now flowing through the same unified path

## Summary - `transduce()` is now the single mechanism: one function, no content-type branch. It splits on `"\n\n"` as a universal boundary check, falls back to fixed 4096-char windows if none is found, and wires the same generic edges (`contains`/`precedes`/heading `section_of`) regardless of what's inside a chunk. Dedup is the existing `find_existing_by_content` path, applied uniformly. - The old `transduce_structured` JSON dataset/records/feature-node interpretation is deleted outright — a JSON file is now chunked and deduped like anything else, no pre-computed structure. - This unlocks raw/opaque content (audio, or anything with no natural text/JSON shape) with **no DSP, no LLM call, no external API**: `transduce()` chunks it exactly like it chunks anything else. Zero semantic understanding of audio (or any payload) is claimed or built here — meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time. - Two small C builtins added (`fs_size`, `fs_read_b64_chunk` in `el_runtime.c`/`.h`) because El strings are NUL-unsafe under strlen-based ops and `fs_read()` silently truncates at the first embedded NUL — routine in real binary/audio bytes. `ingest_file` now compares `fs_read()`'s string length against a real `fs_size()` byte count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim), joined with the same `"\n\n"` marker `transduce()`'s boundary scan already looks for. This is a mechanical fidelity fix, not content interpretation — `transduce()` never learns a fallback happened. - `INGEST_KIND` keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not a content-type flag. The redundant `"structured"` value is removed. `ingest_dir` drops its file-extension filter for the same reason. - All five `ingest_*` entry points are unchanged in name and role. `ingest_stream` was never wired to either old function (it builds its own turn-nodes directly), so it's untouched. ## Verification - Compiles clean via the real `elc` + the modified `el_runtime.c`/`engram_*.c` — built and booted an actual sandbox engram off this exact source (`nsbx create --branch origin/dev`). - Local manifold construction confirmed correct against a real captured audio file (`will_clean.wav`, 304288 bytes, and a 12288-byte real prefix slice): exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice — matches `ceil(bytes/3072)+1` nodes, `2n-1` edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes. ## Known gap (disclosed, not papered over) End-to-end **server-confirmed** persistence (a real before/after `/api/stats` delta, and a fetched node by id) for the audio/prose/JSON cases was **not** obtained this session. Every local nsbx sandbox tried (two stock pre-#109 binaries hitting the known O(N·D) brute-force scan bug, then a fresh #109/HNSW binary built from current `dev`) took minutes-to-indefinitely-long on the final `/api/load-merge` write's embedding step and hit the client's 60s HTTP timeout, even for a 5-node write. Confirmed as real (if slow) forward progress via growing WAL file size, not a hang. The code's own pre-existing honesty gate correctly refused to report success in every case (`"load-merge failed: ..."` + `"nothing below this manifold was confirmed persisted by the server"`) rather than silently claiming a false success. This is an environment/infrastructure characteristic tonight, not a defect introduced by this change — the engram server binary itself is untouched here. ## Test plan - [ ] Confirm a fast/idle sandbox engram, then re-run `INGEST_KIND=file INGEST_ARG=<audio file> ingest/build/ingest` and capture a real before/after `/api/stats` delta - [ ] Fetch one newly-created node by id via `/api/nodes/<id>` and confirm real, non-empty base64 content - [ ] Re-run the same for a plain-text file and a `{"dataset":...,"records":[...]}`-shaped JSON fixture to confirm no regression on either case now flowing through the same unified path

will.anderson added 1 commit 2026-08-15 22:28:48 +00:00

ingest: unify transduce_prose/transduce_structured into one transduce()

El SDK CI - dev / build-and-test (pull_request) Failing after 3m42s

Details

e29fe4fd0b

transduce() is now THE single mechanism: one function, no content-type
branch inside it. It never asks whether `source` is prose, JSON, or
raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally:
split on "\n\n" as a universal boundary-marker check, and if that finds
no boundary, fall back to fixed 4096-char windows. Same node/edge wiring
(root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets
a heading/section_of link) regardless of what's inside a chunk. Dedup is
the existing find_existing_by_content path via merge_manifold, applied
uniformly. The old transduce_structured JSON dataset/records/feature-node
interpretation is deleted outright, not just unused — a JSON file now
gets chunked and deduped like anything else, with no pre-computed
structure. All five ingest_* entry points still exist unchanged in name
and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one
transduce() (ingest_stream builds its own turn-nodes directly and never
called either old function, so it's untouched).

This unlocks raw/opaque content (audio, or anything else with no natural
text/JSON shape) without any DSP, LLM call, or external API: transduce()
chunks it exactly like it chunks anything else. There is zero semantic
understanding of audio (or any payload) claimed or built here — any
meaning is expected to emerge later from Neuron's own existing mechanisms
(embedding, spreading activation, dedup) acting on this real geometry
over time.

Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk)
because El strings are NUL-unsafe under strlen-based ops and fs_read()'s
result silently truncates at the first embedded NUL, which is routine in
real binary/audio bytes. ingest_file compares fs_read()'s string length
against a real fs_size() stat() count; on mismatch it rebuilds the
payload as base64-encoded fixed 3072-byte windows read directly off disk
(binary-safe in C, verbatim, no invention), joined with the same "\n\n"
marker transduce()'s boundary scan already looks for. This is a
mechanical fidelity fix, not interpretation of content — transduce()
never learns a fallback happened. Registered both builtins' arity in
codegen.el; did not rebuild the elc compiler binary itself (unrelated,
pre-existing gap: self-hosting elc via el_seed.c fails on this worktree
independent of this change, reproduced with codegen.el reverted) — the
existing elc binary compiles calls to unregistered builtins via its
already-existing arity=-1 passthrough, confirmed by an actual clean
`elc ingest.el` + `cc` build against the modified el_runtime.c.

INGEST_KIND keeps existing only as an acquisition-mechanism selector
(dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a
content-type flag; the redundant "structured" value (an alias for "file"
that hinted the now-deleted JSON branch) is removed. ingest_dir drops its
file-extension filter for the same reason: transduce() takes anything now.

Verification: local manifold construction confirmed correct against a
real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte
real prefix slice) — exact expected node/edge counts both times
(101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching
ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64
content confirmed decoding back to the actual WAV header bytes. Compiles
clean via the real elc + the modified el_runtime.c/engram_*.c (built and
booted an actual sandbox engram off this exact source with `nsbx create
--branch`).

NOT verified this session, disclosed rather than papered over: end-to-end
server-confirmed persistence (a real before/after /api/stats delta, and a
fetched node by id) for the audio, prose, and JSON-fixture cases. Every
local nsbx sandbox engram tried tonight (two stock pre-#109 binaries
hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW
binary built from current dev) took minutes-to indefinitely long on the
final /api/load-merge write's embedding step and hit the client's 60s
HTTP timeout before responding, even for a 5-node write. This is
confirmed as real (if slow) forward progress, not a hang: the sandbox's
WAL file was observed growing steadily across every attempt. The code's
own pre-existing HONESTY GATE correctly refused to report success in
every case, returning "load-merge failed: ..." with a
"nothing below this manifold was confirmed persisted by the server" note
instead — exactly as designed. This is an environment/infrastructure
limitation, not a defect introduced by this change: the engram server
binary itself is untouched by this commit.

will.anderson merged commit ee39aa5f17 into dev

2026-08-15 22:36:13 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#117