bigmerge
e29fe4fd0b
El SDK CI - dev / build-and-test (pull_request) Failing after 3m42s
transduce() is now THE single mechanism: one function, no content-type branch inside it. It never asks whether `source` is prose, JSON, or raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally: split on "\n\n" as a universal boundary-marker check, and if that finds no boundary, fall back to fixed 4096-char windows. Same node/edge wiring (root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets a heading/section_of link) regardless of what's inside a chunk. Dedup is the existing find_existing_by_content path via merge_manifold, applied uniformly. The old transduce_structured JSON dataset/records/feature-node interpretation is deleted outright, not just unused — a JSON file now gets chunked and deduped like anything else, with no pre-computed structure. All five ingest_* entry points still exist unchanged in name and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one transduce() (ingest_stream builds its own turn-nodes directly and never called either old function, so it's untouched). This unlocks raw/opaque content (audio, or anything else with no natural text/JSON shape) without any DSP, LLM call, or external API: transduce() chunks it exactly like it chunks anything else. There is zero semantic understanding of audio (or any payload) claimed or built here — any meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time. Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk) because El strings are NUL-unsafe under strlen-based ops and fs_read()'s result silently truncates at the first embedded NUL, which is routine in real binary/audio bytes. ingest_file compares fs_read()'s string length against a real fs_size() stat() count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim, no invention), joined with the same "\n\n" marker transduce()'s boundary scan already looks for. This is a mechanical fidelity fix, not interpretation of content — transduce() never learns a fallback happened. Registered both builtins' arity in codegen.el; did not rebuild the elc compiler binary itself (unrelated, pre-existing gap: self-hosting elc via el_seed.c fails on this worktree independent of this change, reproduced with codegen.el reverted) — the existing elc binary compiles calls to unregistered builtins via its already-existing arity=-1 passthrough, confirmed by an actual clean `elc ingest.el` + `cc` build against the modified el_runtime.c. INGEST_KIND keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a content-type flag; the redundant "structured" value (an alias for "file" that hinted the now-deleted JSON branch) is removed. ingest_dir drops its file-extension filter for the same reason: transduce() takes anything now. Verification: local manifold construction confirmed correct against a real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte real prefix slice) — exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes. Compiles clean via the real elc + the modified el_runtime.c/engram_*.c (built and booted an actual sandbox engram off this exact source with `nsbx create --branch`). NOT verified this session, disclosed rather than papered over: end-to-end server-confirmed persistence (a real before/after /api/stats delta, and a fetched node by id) for the audio, prose, and JSON-fixture cases. Every local nsbx sandbox engram tried tonight (two stock pre-#109 binaries hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW binary built from current dev) took minutes-to indefinitely long on the final /api/load-merge write's embedding step and hit the client's 60s HTTP timeout before responding, even for a 5-node write. This is confirmed as real (if slow) forward progress, not a hang: the sandbox's WAL file was observed growing steadily across every attempt. The code's own pre-existing HONESTY GATE correctly refused to report success in every case, returning "load-merge failed: ..." with a "nothing below this manifold was confirmed persisted by the server" note instead — exactly as designed. This is an environment/infrastructure limitation, not a defect introduced by this change: the engram server binary itself is untouched by this commit.