ingest: unify transduce_prose/transduce_structured into one transduce()
El SDK CI - dev / build-and-test (pull_request) Failing after 3m42s
El SDK CI - dev / build-and-test (pull_request) Failing after 3m42s
transduce() is now THE single mechanism: one function, no content-type branch inside it. It never asks whether `source` is prose, JSON, or raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally: split on "\n\n" as a universal boundary-marker check, and if that finds no boundary, fall back to fixed 4096-char windows. Same node/edge wiring (root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets a heading/section_of link) regardless of what's inside a chunk. Dedup is the existing find_existing_by_content path via merge_manifold, applied uniformly. The old transduce_structured JSON dataset/records/feature-node interpretation is deleted outright, not just unused — a JSON file now gets chunked and deduped like anything else, with no pre-computed structure. All five ingest_* entry points still exist unchanged in name and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one transduce() (ingest_stream builds its own turn-nodes directly and never called either old function, so it's untouched). This unlocks raw/opaque content (audio, or anything else with no natural text/JSON shape) without any DSP, LLM call, or external API: transduce() chunks it exactly like it chunks anything else. There is zero semantic understanding of audio (or any payload) claimed or built here — any meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time. Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk) because El strings are NUL-unsafe under strlen-based ops and fs_read()'s result silently truncates at the first embedded NUL, which is routine in real binary/audio bytes. ingest_file compares fs_read()'s string length against a real fs_size() stat() count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim, no invention), joined with the same "\n\n" marker transduce()'s boundary scan already looks for. This is a mechanical fidelity fix, not interpretation of content — transduce() never learns a fallback happened. Registered both builtins' arity in codegen.el; did not rebuild the elc compiler binary itself (unrelated, pre-existing gap: self-hosting elc via el_seed.c fails on this worktree independent of this change, reproduced with codegen.el reverted) — the existing elc binary compiles calls to unregistered builtins via its already-existing arity=-1 passthrough, confirmed by an actual clean `elc ingest.el` + `cc` build against the modified el_runtime.c. INGEST_KIND keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a content-type flag; the redundant "structured" value (an alias for "file" that hinted the now-deleted JSON branch) is removed. ingest_dir drops its file-extension filter for the same reason: transduce() takes anything now. Verification: local manifold construction confirmed correct against a real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte real prefix slice) — exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes. Compiles clean via the real elc + the modified el_runtime.c/engram_*.c (built and booted an actual sandbox engram off this exact source with `nsbx create --branch`). NOT verified this session, disclosed rather than papered over: end-to-end server-confirmed persistence (a real before/after /api/stats delta, and a fetched node by id) for the audio, prose, and JSON-fixture cases. Every local nsbx sandbox engram tried tonight (two stock pre-#109 binaries hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW binary built from current dev) took minutes-to indefinitely long on the final /api/load-merge write's embedding step and hit the client's 60s HTTP timeout before responding, even for a 5-node write. This is confirmed as real (if slow) forward progress, not a hang: the sandbox's WAL file was observed growing steadily across every attempt. The code's own pre-existing HONESTY GATE correctly refused to report success in every case, returning "load-merge failed: ..." with a "nothing below this manifold was confirmed persisted by the server" note instead — exactly as designed. This is an environment/infrastructure limitation, not a defect introduced by this change: the engram server binary itself is untouched by this commit.
This commit is contained in:
@@ -2188,6 +2188,22 @@ el_val_t fs_exists(el_val_t pathv) {
|
||||
return (el_val_t)(stat(path, &st) == 0 ? 1 : 0);
|
||||
}
|
||||
|
||||
/* fs_size — real on-disk byte count of a file, via stat() (not strlen).
|
||||
* Needed alongside fs_read_b64_chunk() below: fs_read()'s el_val_t string
|
||||
* result is NUL-terminated and length-unsafe for arbitrary binary content
|
||||
* (str_len/str_slice fall back to strlen when the buffer isn't a tagged
|
||||
* binary value — see el_input_len), so any caller that needs to walk a
|
||||
* binary file in fixed-size windows (e.g. transduce()'s raw/opaque chunking
|
||||
* of audio bytes) must learn the true length here instead of from the
|
||||
* decoded string. Returns -1 if the path doesn't exist or isn't stat-able. */
|
||||
el_val_t fs_size(el_val_t pathv) {
|
||||
const char* path = EL_CSTR(pathv);
|
||||
if (!path || !*path) return -1;
|
||||
struct stat st;
|
||||
if (stat(path, &st) != 0) return -1;
|
||||
return (el_val_t)st.st_size;
|
||||
}
|
||||
|
||||
/* fs_mkdir — create directory at path with mode 0755, mkdir -p semantics.
|
||||
* Returns 1 if path exists or was created (incl. all parents); 0 on failure.
|
||||
* Walks the path component-by-component so missing intermediate dirs are
|
||||
@@ -14260,6 +14276,41 @@ el_val_t el_base64_encode_n(const unsigned char* data, size_t len, int url_safe)
|
||||
return el_wrap_str(out);
|
||||
}
|
||||
|
||||
/* fs_read_b64_chunk — binary-safe windowed file read: read up to `length`
|
||||
* raw bytes starting at byte `offset` from `path` and return them base64-
|
||||
* encoded (RFC 4648, standard alphabet, padded). The raw bytes are read into
|
||||
* a local C buffer and base64-encoded directly here — they never pass
|
||||
* through an el_val_t string as raw bytes, so embedded NUL bytes (common in
|
||||
* real PCM audio) never hit a strlen()-based code path. This mirrors the
|
||||
* existing llm_vision() image-attachment path (read file -> base64 in C ->
|
||||
* hand back a plain-ASCII string) and the http_*_to_file() rationale above:
|
||||
* bypass the string wrapper entirely for the part that must stay binary.
|
||||
*
|
||||
* Returns "" if the path can't be opened, offset is negative or past EOF,
|
||||
* or length <= 0. A short final chunk (less than `length` bytes remaining)
|
||||
* is returned truncated to what's actually on disk — never padded/invented. */
|
||||
el_val_t fs_read_b64_chunk(el_val_t pathv, el_val_t offsetv, el_val_t lengthv) {
|
||||
const char* path = EL_CSTR(pathv);
|
||||
int64_t offset = (int64_t)offsetv;
|
||||
int64_t length = (int64_t)lengthv;
|
||||
if (!path || !*path || offset < 0 || length <= 0) return el_wrap_str(el_strdup(""));
|
||||
FILE* f = fopen(path, "rb");
|
||||
if (!f) return el_wrap_str(el_strdup(""));
|
||||
fseek(f, 0, SEEK_END);
|
||||
long sz = ftell(f);
|
||||
if (sz < 0 || offset >= sz) { fclose(f); return el_wrap_str(el_strdup("")); }
|
||||
fseek(f, (long)offset, SEEK_SET);
|
||||
long remain = sz - (long)offset;
|
||||
size_t want = (size_t)(((int64_t)remain < length) ? remain : length);
|
||||
unsigned char* buf = malloc(want > 0 ? want : 1);
|
||||
if (!buf) { fclose(f); return el_wrap_str(el_strdup("")); }
|
||||
size_t got = fread(buf, 1, want, f);
|
||||
fclose(f);
|
||||
el_val_t out = el_base64_encode_n(buf, got, /*url_safe=*/0);
|
||||
free(buf);
|
||||
return out;
|
||||
}
|
||||
|
||||
/* Decode either alphabet — accepts both '+/' and '-_' transparently, and
|
||||
* tolerates missing padding (which JWTs typically omit). Whitespace is
|
||||
* skipped for robustness. Invalid characters cause the decode to stop and
|
||||
|
||||
Reference in New Issue
Block a user