3 Commits

Author	SHA1	Message	Date
bigmerge	e29fe4fd0b	ingest: unify transduce_prose/transduce_structured into one transduce() ... transduce() is now THE single mechanism: one function, no content-type branch inside it. It never asks whether `source` is prose, JSON, or raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally: split on "\n\n" as a universal boundary-marker check, and if that finds no boundary, fall back to fixed 4096-char windows. Same node/edge wiring (root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets a heading/section_of link) regardless of what's inside a chunk. Dedup is the existing find_existing_by_content path via merge_manifold, applied uniformly. The old transduce_structured JSON dataset/records/feature-node interpretation is deleted outright, not just unused — a JSON file now gets chunked and deduped like anything else, with no pre-computed structure. All five ingest_* entry points still exist unchanged in name and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one transduce() (ingest_stream builds its own turn-nodes directly and never called either old function, so it's untouched). This unlocks raw/opaque content (audio, or anything else with no natural text/JSON shape) without any DSP, LLM call, or external API: transduce() chunks it exactly like it chunks anything else. There is zero semantic understanding of audio (or any payload) claimed or built here — any meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time. Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk) because El strings are NUL-unsafe under strlen-based ops and fs_read()'s result silently truncates at the first embedded NUL, which is routine in real binary/audio bytes. ingest_file compares fs_read()'s string length against a real fs_size() stat() count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim, no invention), joined with the same "\n\n" marker transduce()'s boundary scan already looks for. This is a mechanical fidelity fix, not interpretation of content — transduce() never learns a fallback happened. Registered both builtins' arity in codegen.el; did not rebuild the elc compiler binary itself (unrelated, pre-existing gap: self-hosting elc via el_seed.c fails on this worktree independent of this change, reproduced with codegen.el reverted) — the existing elc binary compiles calls to unregistered builtins via its already-existing arity=-1 passthrough, confirmed by an actual clean `elc ingest.el` + `cc` build against the modified el_runtime.c. INGEST_KIND keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a content-type flag; the redundant "structured" value (an alias for "file" that hinted the now-deleted JSON branch) is removed. ingest_dir drops its file-extension filter for the same reason: transduce() takes anything now. Verification: local manifold construction confirmed correct against a real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte real prefix slice) — exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes. Compiles clean via the real elc + the modified el_runtime.c/engram_*.c (built and booted an actual sandbox engram off this exact source with `nsbx create --branch`). NOT verified this session, disclosed rather than papered over: end-to-end server-confirmed persistence (a real before/after /api/stats delta, and a fetched node by id) for the audio, prose, and JSON-fixture cases. Every local nsbx sandbox engram tried tonight (two stock pre-#109 binaries hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW binary built from current dev) took minutes-to indefinitely long on the final /api/load-merge write's embedding step and hit the client's 60s HTTP timeout before responding, even for a 5-node write. This is confirmed as real (if slow) forward progress, not a hang: the sandbox's WAL file was observed growing steadily across every attempt. The code's own pre-existing HONESTY GATE correctly refused to report success in every case, returning "load-merge failed: ..." with a "nothing below this manifold was confirmed persisted by the server" note instead — exactly as designed. This is an environment/infrastructure limitation, not a defect introduced by this change: the engram server binary itself is untouched by this commit.	2026-08-15 17:28:19 -05:00
will.anderson	c2d8a07c7b	transduce: name the invisible mechanism, fix a silent-failure bug, drop a CRUD verb ... ingest and transduce are complements, not synonyms: ingest is the conscious, deliberate act of pointing at a source (ingest_file/dir/url/llm/stream stay named exactly that); transduce is the automatic, invisible mechanism inside it that converts extracted surface content into geometry (renamed build_prose/build_structured -> transduce_prose/transduce_structured, the functions that actually turn raw text into a node+edge manifold). Real bug found and fixed along the way: the final /api/load-merge response was never checked for an error. A total failure (bad auth, network down, anything) silently reported nodes_added:0/edges_added:0 — indistinguishable from a benign 'everything was already known' outcome. Verified live: with a wrong key, the tool now honestly returns {"error":"load-merge failed: unauthorized",...} instead of a misleading zero. Also dropped a CRUD-verb smell: the per-decision println said CREATE (a database-log verb for something that hasn't actually been written to the server yet — it's a local, tentative decision pending the batch merge). Renamed to FORM. The dead-code eg_create_node (defined, never called) renamed to eg_crystallize_node and annotated honestly as unused, since if it's ever wired up it represents the real server-confirmed write, unlike the local FORM guess. Not yet re-verified end-to-end against a real successful write: the ingest-test sandbox (nsbx up ingest-test) is itself currently broken — it prints a green "ready" banner after its own readiness check fails, and nothing is actually listening. Filed separately; not in scope here.	2026-08-15 14:44:18 -05:00
bigmerge	710bea174d	Add native EL afferent ingest organ ... Source-polymorphic ingest(source) primitive: extracts content faithfully from a directory/file/url/llm-query/structured-primitive-set/stream, decomposes it into a discrete multi-node graph manifold (nodes + internal edges, never a single blob), and merges it into the engram geometry with dedup (search + exact/cosine match), provenance, grounding-level, and stewardship-class tagging from the moment of entry. Pure HTTP client of the engram server (links only el_runtime.c, never el_seed.c/the engine directly). Tested against a live nsbx sandbox engram clone (127.0.0.1:8903) with real writes confirmed via /api/stats (node_count 3201 / edge_count 6601). Excludes ingest/build/ — local compiler scratch output (binaries, .c codegen, .err logs), not source.	2026-08-15 14:22:07 -05:00