ingest: unify transduce_prose/transduce_structured into one transduce()

transduce() is now THE single mechanism: one function, no content-type
branch inside it. It never asks whether `source` is prose, JSON, or
raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally:
split on "\n\n" as a universal boundary-marker check, and if that finds
no boundary, fall back to fixed 4096-char windows. Same node/edge wiring
(root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets
a heading/section_of link) regardless of what's inside a chunk. Dedup is
the existing find_existing_by_content path via merge_manifold, applied
uniformly. The old transduce_structured JSON dataset/records/feature-node
interpretation is deleted outright, not just unused — a JSON file now
gets chunked and deduped like anything else, with no pre-computed
structure. All five ingest_* entry points still exist unchanged in name
and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one
transduce() (ingest_stream builds its own turn-nodes directly and never
called either old function, so it's untouched).

This unlocks raw/opaque content (audio, or anything else with no natural
text/JSON shape) without any DSP, LLM call, or external API: transduce()
chunks it exactly like it chunks anything else. There is zero semantic
understanding of audio (or any payload) claimed or built here — any
meaning is expected to emerge later from Neuron's own existing mechanisms
(embedding, spreading activation, dedup) acting on this real geometry
over time.

Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk)
because El strings are NUL-unsafe under strlen-based ops and fs_read()'s
result silently truncates at the first embedded NUL, which is routine in
real binary/audio bytes. ingest_file compares fs_read()'s string length
against a real fs_size() stat() count; on mismatch it rebuilds the
payload as base64-encoded fixed 3072-byte windows read directly off disk
(binary-safe in C, verbatim, no invention), joined with the same "\n\n"
marker transduce()'s boundary scan already looks for. This is a
mechanical fidelity fix, not interpretation of content — transduce()
never learns a fallback happened. Registered both builtins' arity in
codegen.el; did not rebuild the elc compiler binary itself (unrelated,
pre-existing gap: self-hosting elc via el_seed.c fails on this worktree
independent of this change, reproduced with codegen.el reverted) — the
existing elc binary compiles calls to unregistered builtins via its
already-existing arity=-1 passthrough, confirmed by an actual clean
`elc ingest.el` + `cc` build against the modified el_runtime.c.

INGEST_KIND keeps existing only as an acquisition-mechanism selector
(dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a
content-type flag; the redundant "structured" value (an alias for "file"
that hinted the now-deleted JSON branch) is removed. ingest_dir drops its
file-extension filter for the same reason: transduce() takes anything now.

Verification: local manifold construction confirmed correct against a
real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte
real prefix slice) — exact expected node/edge counts both times
(101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching
ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64
content confirmed decoding back to the actual WAV header bytes. Compiles
clean via the real elc + the modified el_runtime.c/engram_*.c (built and
booted an actual sandbox engram off this exact source with `nsbx create
--branch`).

NOT verified this session, disclosed rather than papered over: end-to-end
server-confirmed persistence (a real before/after /api/stats delta, and a
fetched node by id) for the audio, prose, and JSON-fixture cases. Every
local nsbx sandbox engram tried tonight (two stock pre-#109 binaries
hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW
binary built from current dev) took minutes-to indefinitely long on the
final /api/load-merge write's embedding step and hit the client's 60s
HTTP timeout before responding, even for a 5-node write. This is
confirmed as real (if slow) forward progress, not a hang: the sandbox's
WAL file was observed growing steadily across every attempt. The code's
own pre-existing HONESTY GATE correctly refused to report success in
every case, returning "load-merge failed: ..." with a
"nothing below this manifold was confirmed persisted by the server" note
instead — exactly as designed. This is an environment/infrastructure
limitation, not a defect introduced by this change: the engram server
binary itself is untouched by this commit.
This commit is contained in:
bigmerge
2026-08-15 17:28:19 -05:00
parent 90d3f0bc76
commit 9d40f87926
5 changed files with 283 additions and 262 deletions
+51
View File
@@ -2226,6 +2226,22 @@ el_val_t fs_exists(el_val_t pathv) {
return (el_val_t)(stat(path, &st) == 0 ? 1 : 0);
}
/* fs_size — real on-disk byte count of a file, via stat() (not strlen).
* Needed alongside fs_read_b64_chunk() below: fs_read()'s el_val_t string
* result is NUL-terminated and length-unsafe for arbitrary binary content
* (str_len/str_slice fall back to strlen when the buffer isn't a tagged
* binary value see el_input_len), so any caller that needs to walk a
* binary file in fixed-size windows (e.g. transduce()'s raw/opaque chunking
* of audio bytes) must learn the true length here instead of from the
* decoded string. Returns -1 if the path doesn't exist or isn't stat-able. */
el_val_t fs_size(el_val_t pathv) {
const char* path = EL_CSTR(pathv);
if (!path || !*path) return -1;
struct stat st;
if (stat(path, &st) != 0) return -1;
return (el_val_t)st.st_size;
}
/* fs_mkdir — create directory at path with mode 0755, mkdir -p semantics.
* Returns 1 if path exists or was created (incl. all parents); 0 on failure.
* Walks the path component-by-component so missing intermediate dirs are
@@ -16469,6 +16485,41 @@ el_val_t el_base64_encode_n(const unsigned char* data, size_t len, int url_safe)
return el_wrap_str(out);
}
/* fs_read_b64_chunk — binary-safe windowed file read: read up to `length`
* raw bytes starting at byte `offset` from `path` and return them base64-
* encoded (RFC 4648, standard alphabet, padded). The raw bytes are read into
* a local C buffer and base64-encoded directly here they never pass
* through an el_val_t string as raw bytes, so embedded NUL bytes (common in
* real PCM audio) never hit a strlen()-based code path. This mirrors the
* existing llm_vision() image-attachment path (read file -> base64 in C ->
* hand back a plain-ASCII string) and the http_*_to_file() rationale above:
* bypass the string wrapper entirely for the part that must stay binary.
*
* Returns "" if the path can't be opened, offset is negative or past EOF,
* or length <= 0. A short final chunk (less than `length` bytes remaining)
* is returned truncated to what's actually on disk never padded/invented. */
el_val_t fs_read_b64_chunk(el_val_t pathv, el_val_t offsetv, el_val_t lengthv) {
const char* path = EL_CSTR(pathv);
int64_t offset = (int64_t)offsetv;
int64_t length = (int64_t)lengthv;
if (!path || !*path || offset < 0 || length <= 0) return el_wrap_str(el_strdup(""));
FILE* f = fopen(path, "rb");
if (!f) return el_wrap_str(el_strdup(""));
fseek(f, 0, SEEK_END);
long sz = ftell(f);
if (sz < 0 || offset >= sz) { fclose(f); return el_wrap_str(el_strdup("")); }
fseek(f, (long)offset, SEEK_SET);
long remain = sz - (long)offset;
size_t want = (size_t)(((int64_t)remain < length) ? remain : length);
unsigned char* buf = malloc(want > 0 ? want : 1);
if (!buf) { fclose(f); return el_wrap_str(el_strdup("")); }
size_t got = fread(buf, 1, want, f);
fclose(f);
el_val_t out = el_base64_encode_n(buf, got, /*url_safe=*/0);
free(buf);
return out;
}
/* Decode either alphabet — accepts both '+/' and '-_' transparently, and
* tolerates missing padding (which JWTs typically omit). Whitespace is
* skipped for robustness. Invalid characters cause the decode to stop and