501 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
bigmerge	b5a0a729e6	codegen: Bool is int-like, so Bool comparisons stop lowering to str_eq ... fn check(label: String, cond: Bool, want: Bool) -> Void { if cond == want { ... } -> if (str_eq(cond, want)) SIGSEGV } Bool has always been an integer in the value model — type_to_c maps Bool to "int", and el_runtime.h states "Bool -> el_val_t (0 = false, nonzero = true)". But Bool names were registered NOWHERE: build_int_names_for_params tracked Int and Float params, and the `let` path tracked Int and Float bindings. Neither knew about Bool. So comparing two Bools fell through to str_eq, which dereferenced 0 or 1 as a char* and segfaulted immediately. This is the third instance of one family found tonight, after el #137 (a call on either side of == poisoned the operator) and el #136 (a missing import compiled clean). All three are the same shape: something the compiler could not type, silently handled as a string. Found while writing #137's own test harness — the first version of that harness crashed on exactly this, on both the old and new compiler, which is how it surfaced. A test harness that cannot compare two Bools is a good way to notice. VERIFIED: - the harness that segfaulted on every prior compiler (exit 139, no output) now runs clean: 14 passed, 0 failed - self-hosting fixpoint byte-identical - the compiler's own generated C differs by 8 lines — only the intended registration - neuron's full soul amalgam regenerates in 424ms, exit 0, BYTE-IDENTICAL - test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12 Adds tests/runtime/operator_typing_test.el, the 15-case suite from #137, so this family is covered going forward rather than rediscovered.	2026-08-15 21:54:10 -05:00
will.anderson	b26dd47aef	Merge pull request 'codegen: either side Int is enough for == and !=, not both' (#137) from fix/eq-operand-inference into dev	2026-08-16 02:52:02 +00:00
bigmerge	b55e6bfd53	codegen: either side Int is enough for == and !=, not both ... let a: Int = 5 getint(5) == a -> str_eq(getint(5), a) SIGSEGV getint(5) == 5 -> getint(5) == 5 fine A function call whose return type codegen cannot infer poisoned the operator, and a declared Int on the other side did not save it. str_eq then read an integer as a char* and segfaulted. Only an integer LITERAL on one side forced the numeric form, which is why the bug stayed invisible: the common case happened to be safe. The check required BOTH operands to be provably Int: if is_int_expr(left) { if is_int_expr(right) { numeric } } Loosening to OR is strictly safer, not a trade: - when one side is a known Int, str_eq is ALWAYS wrong — it dereferences that integer — while numeric comparison is at worst a wrong answer on a program that was already ill-typed; - when neither side is Int nothing changes at all, so string comparison is untouched. Found by the test-framework agent while building the benchmark harness; it correctly declined to fix it mid-phase since it is a codegen semantics change. VERIFIED, because a semantics change earns more than an assertion: - 15/15 on a dedicated operator suite covering string literals, string vars, string-returning calls, mixed var/call, and != in every combination. The pre-change compiler scores 0/15 on the same file: it segfaults before printing anything. - self-hosting fixpoint byte-identical - the ONLY difference in the compiler's own generated C is the intended one: a nested if becoming two sequential ifs, in EqEq and NotEq. Nothing else moved. - neuron's full soul amalgam regenerates in 400ms, exit 0, output BYTE-IDENTICAL at 1,270,212 bytes - test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12 — 62 tests, 190 assertions, zero failures NOT fixed here, same family, flagged for a decision: Bool PARAMETERS are not tracked as int-like, so `cond == want` between two Bool params still lowers to str_eq and segfaults. Found while writing this commit's own test harness — the first version of it crashed on exactly that, on both the old and new compiler. It needs the same treatment, and it wants its own change.	2026-08-15 21:51:35 -05:00
will.anderson	dbb06f6ee4	Merge pull request 'compiler: a missing import is an error, not an empty string' (#136) from fix/missing-import-is-an-error into dev	2026-08-16 02:48:01 +00:00
bigmerge	906c664a65	compiler: a missing import is an error, not an empty string ... import "../../NOPE/does_not_exist.el" compiled CLEANLY — exit 0, empty stderr, and a program silently missing everything it imported. resolve_imports did `fs_read(src_path)` and used the result without checking. fs_read returns "" both for "file is empty" and "file does not exist", so a typo, a moved file, or a relative path resolved from the wrong working directory all produced a successful build of nothing. It caused a real wrong conclusion during test-framework work: a bisection run from a subdirectory where ../../runtime/ did not resolve produced ELEVEN consecutive "successful" compiles that had included no runtime at all, and the results were believed before anyone noticed. Missing dependency, confident success — the same shape as a test suite reporting pass for tests that never ran, and as a benchmark reporting 0us because the optimiser deleted the loop. fs_exists separates the two cases, so a legitimately empty file still resolves to "" and is fine. A path that does not exist now prints the resolved path and exits 1, which is what build scripts check. Verified: - bad import: exit 1 (was 0), message names the resolved path - elc-cli.el still compiles, self-hosting fixpoint byte-identical - neuron's full soul amalgam regeneration: exit 0, 405ms, output byte-identical at 1,270,212 bytes	2026-08-15 21:47:32 -05:00
will.anderson	9e96d74f6a	Merge pull request 'runtime: count container allocations too, not just strings' (#135) from feat/alloc-accounting-containers into dev	2026-08-16 02:37:14 +00:00
bigmerge	a8908908df	runtime: count container allocations too, not just strings ... el #131 instrumented the four string allocators, which meant list- and map-heavy code reported ZERO allocations — a benchmark over lists would have been fitted against a flat line and passed anything. Caught during framework work: a "linear" specimen read 0 allocs until it was rewritten to allocate strings. A gate is only as good as its blind spots are small, and a signal that silently reads zero is worse than no signal: it produces a confident pass. Now counted at every container allocation — ElList and ElMap bodies, their backing arrays, the copy-on-write clones, and the realloc growth path. Verified on an append loop (n = 100..800): allocs 7, 8, 9, 10 +1 per doubling = O(log n) reallocations bytes 2048, 4096, 8192, 16384 exactly 2x per doubling = O(n) Both curves are what correct amortized growth should look like, and both read zero before this change. Known remaining scope, stated rather than left implicit: these counters cover the runtime's own allocations. They do not see malloc inside engram_*.c or libcurl, which is correct — the gate is for El-level complexity, not for third-party memory behaviour.	2026-08-15 21:36:52 -05:00
will.anderson	a69a4a5894	Merge pull request 'runtime: math_log is base-10, not natural log' (#134) from fix/math-log-base10 into dev	2026-08-16 02:34:09 +00:00
bigmerge	edafd8cce8	runtime: math_log is base-10, not natural log ... el_val_t math_log(el_val_t f) { return el_from_float(log(el_to_float(f))); } el_val_t math_ln(el_val_t f) { return el_from_float(log(el_to_float(f))); } Both were natural log, so math_log and math_ln were the same function. log10(100) returned 4.605 instead of 2. Three sources already agreed it should be base-10 and were being contradicted by this one line: - runtime/math.el:55 "// math_log — base-10 logarithm." - el_seed.c:1278 __log_f -> log10() (the path math.el actually calls) - tests/native/test_math.el:133 asserts log10(100) == 2 FOUND BY THE NEW TEST FRAMEWORK ON ITS FIRST RUN (el #133). The assertion had been sitting in the suite the whole time; nothing could report it. The old harness printed "N passed, M failed" with no per-test detail, and half the suites were not compiling at all — so a failing assertion in a suite nobody could run was indistinguishable from no failure. That is the entire argument for the framework, demonstrated on day one: this is not a bug the framework introduced, it is a bug the framework made VISIBLE. Verified: tests/native/test_math.el goes 12/13 -> 13/13, math-log passing.	2026-08-15 21:33:20 -05:00
will.anderson	5e3e69d326	Merge pull request 'test framework phase 1: compile-time registry + El-side runner with per-test timing' (#133) from wt/soul-runtime-reconcile into dev	2026-08-16 02:31:08 +00:00
will.anderson	0288024396	Merge pull request 'compiler: fix the quadratic — strlen() on every character access' (#132) from fix/compiler-quadratic-strlen into dev	2026-08-16 02:29:02 +00:00
Neuron	3e7ab07e82	test framework phase 1: forward decls, void-return fix, suite migration ... Completes the Phase 1 runner and migrates the 11 test files onto it. - forward-declare the registry accessors in the test preamble; they are defined at the end of the unit but the El runner is compiled in between - eltest.el: explicit trailing return in the void emit_* helpers, which otherwise lower to 'return println(...)' and fail to compile - test files import runtime/eltest.el explicitly, using the language's own textual import mechanism rather than compiler-side auto-injection - DESIGN.md 6.5: gate on allocation COUNT AND BYTES, not count alone Verified: self-hosting fixpoint byte-identical (gen2 == gen3). 6 of 11 suites run and report per-test timing. The other 5 fail to COMPILE, and fail identically under the committed compiler -- pre-existing breakage this framework makes visible for the first time.	2026-08-15 21:28:30 -05:00
bigmerge	d231b7e5e7	compiler: fix the quadratic — strlen() on every character access ... THE BUG. str_char_code() and str_slice() each called strlen() on every invocation. The lexer walks source one character at a time, so every character access rescanned the whole remaining input: O(n) per character over n characters = O(n^2). el_val_t str_char_code(el_val_t s, el_val_t i) { ... int64_t n = (int64_t)strlen(str); // <- O(n), every call if (idx < 0 || idx >= n) return 0; return str[idx]; } HOW IT WAS FOUND. Not by reading code — by sampling the running process, which is the same method that resolved tonight's engram outage after four wrong theories. A geometric sweep of synthetic sources showed wall-clock rising 3.0x, 3.0x, 4.0x, 4.14x per doubling (converging on 4x = quadratic), and a stack sample put 779 of 779 samples inside lex(), every one bottoming out in _platform_strlen via str_char_code and str_slice. THE FIX. Remember the length instead of recomputing it. The subtlety is INVALIDATION: El strings are arena-allocated, so a freed pointer can be reused for a different string at the same address, and a naive pointer-keyed cache would hand back a stale length and read past the end of the new string — trading a performance bug for a memory-safety one. So entries carry a generation, a hit requires pointer AND generation to match, and every path that frees or mutates a runtime string bumps the generation: el_arena_pop, seed_request_end, __str_set_char. Stale entries cannot be believed; they miss and recompute. MEASURED, same host, same inputs: n(fns) before after 512 0.10s 0.01s 1024 0.37s 0.02s 2048 1.51s 0.03s 50x the compiler's own 422 KB source concatenated (DESIGN.md's 3.58s case): 3.55s -> 0.03s 118x The speedup GROWS with input size, which is the signature of removing a complexity class rather than a constant factor. After the fix each doubling adds ~0.01s: linear. CORRECTNESS, verified rather than assumed: - byte-identical output on every sweep input (n = 128..2048) - byte-identical output on the 422 KB compiler concatenation - byte-identical output on tests/runtime/string_test.el - self-hosting fixpoint byte-identical - new tests/runtime/str_cache_test.el: 17 assertions covering bounds, empty strings, negative indices, slice clamping, distinct strings not sharing a cached length, 1000 interleaved strings forcing cache-slot collisions, and a grown string not reporting its old length. All pass. This is the defect that made dist/soul.c a committed artifact: elc could not run in CI because it needed 24 GB+ and minutes. It needs neither now.	2026-08-15 21:28:23 -05:00
bigmerge	a668062e38	Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile	2026-08-15 21:24:03 -05:00
Neuron	24fac765a6	test framework phase 1: compile-time registry + El-side runner ... Replace the hardcoded test harness main() with a generated static registry and index-based accessors, and move all reporting into runtime/eltest.el. The old harness inlined direct calls into main() and counted assertions in two globals. That shape cannot report which test failed, how long any test took, or whether a test ran at all -- a misspelled registration reported success for a test that never executed. - assertions record into per-test state instead of global counters - registry table emitted at compile time; discovery strictly precedes execution, which is what later enables --list, filtering and sharding - per-test wall timing on CLOCK_MONOTONIC, taken in C around the call - runner in El: structured NDJSON events as source of truth, human output rendered from the same fields	2026-08-15 21:24:03 -05:00
will.anderson	cb1f2a74af	Merge pull request 'runtime: allocation accounting — deterministic signal for complexity gating' (#131) from feat/alloc-accounting into dev	2026-08-16 02:22:13 +00:00
bigmerge	37bcf7eb74	runtime: allocation accounting — the deterministic signal for complexity gating ... Implements the three primitives the test-framework design (DESIGN.md §6.5) requires for gating on growth curves: el_alloc_count, el_alloc_bytes, el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per the project's C-builtin recipe. WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build failure, which means the signal cannot flake. Wall-clock needs warmup, statistics, and a quiet machine; on shared CI it is unusable as a gate. Allocation counts are perfectly deterministic — same input, same number, every machine, every run. Fit them against n and a complexity regression becomes a build failure with zero noise. All four runtime string allocators (el_strdup, el_strbuf, and their _persist variants) funnel every allocation the language performs, so instrumenting there counts everything. WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate. Measured with two El programs, one allocating once per item, one rebuilding its accumulator each iteration: n linear allocs / bytes quadratic allocs / bytes 100 100 / 290 100 / 5,150 200 200 / 690 200 / 20,300 400 400 / 1,490 400 / 80,600 800 800 / 3,090 800 / 321,200 The quadratic program's allocation COUNT is exactly linear — identical to the healthy one. Counting allocations alone would have missed it completely. Bytes catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 -> converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0. That shape — count linear, per-allocation size growing — is the classic accidental quadratic, and it is exactly elc's defect: quadratic allocation VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across both; RSS and wall-clock were just the two ways it surfaced. el_peak_rss is exported for context and is explicitly NOT a gating signal — it is perturbed by allocator internals, the page cache, and the OS. Gate on the deterministic numbers; report the physical one. Counters are unsynchronised by design: this is measurement, and a lock would change the thing being measured. Exact on the single-threaded compile path, approximate under threads.	2026-08-15 21:21:42 -05:00
will.anderson	2240d26c32	Merge pull request 'store: judge memory pressure by swap RATE, not level' (#130) from fix/elc-rebuildable-compiler-builtins into dev	2026-08-16 02:12:22 +00:00
bigmerge	19cc99e57d	store: judge memory pressure by swap RATE, not swap level ... The guard I added minutes ago checked swap availability as a level (avail < total/8 -> report zero available). That is the wrong signal, and the same host proved it twice within minutes: 47.65 / 48.00 GiB swap used, 2047 swapouts/s -> genuinely thrashing 26.67 / 28.00 GiB swap used, 0 swapouts/s -> healthy, 15.6 GiB free Both are ~97% "used". macOS grows swap files on demand and trims them lazily, so the level says almost nothing about now — it is a high-water mark. The level check calls the second state an emergency and starves the pool for no reason, which is its own failure mode: a guard that fires on healthy machines gets disabled, and then guards nothing. What separates the two is whether pages are moving. So sample the swapout counter across calls and judge the delta: - > 200 pages/s (~3 MiB/s) sustained outward paging => report zero available; callers refuse to grow and pc_relieve_pressure hands frames back. - The first call primes the baseline and reports no pressure. One sample cannot have a rate, and inferring one from a single reading is exactly the mistake this commit removes. Measured thresholds, not guessed: idle sat at 0/s, recovery burst hit 24,845/s while the compressor drained (transient, correctly not a growth decision since growth is only evaluated on eviction passes), and real thrash held ~2000/s. 200/s sits clearly above noise and far below either. The compressor-footprint subtraction stays: that RAM is genuinely spoken for regardless of paging rate.	2026-08-15 21:12:00 -05:00
will.anderson	f39ae40047	Merge pull request 'store: bound the pool by available memory and let it shrink' (#129) from fix/elc-rebuildable-compiler-builtins into dev	2026-08-16 02:02:24 +00:00
bigmerge	e52415f0e0	store: bound the pool by AVAILABLE memory and let it shrink ... The adaptive budget I added an hour ago could only grow, and grew toward a share of TOTAL ram (80%, ~38 GiB on a 48 GB host). That is a memory leak with extra steps: total never shrinks when other processes need memory, so the pool had no way to notice it was starving the machine it runs on. Deployed briefly; caught as memory pressure on the host. A control loop with only one direction is not a control loop. - pc_available_ram(): free + inactive + purgeable via host_statistics64 on Darwin, MemAvailable on Linux. Availability is the quantity that moves when the machine is under pressure; total is not. Returns 0 when it cannot be read, and callers then refuse to grow — a cache is never worth swapping the host, so unknown means no. - Growth is bounded by availability minus a free-memory floor (2 GiB default, ENGRAM_POOL_FREE_FLOOR_MB), not by total. The share-of-total ceiling stays as a second bound and drops 80% -> 50%. - pc_relieve_pressure(): the missing direction. On every eviction pass, if available memory is under the floor, hand back ~25% of held frames; the resident set follows on the next pass so the memory is actually returned rather than merely re-labelled. Counted as adapt_shrinks alongside adapt_grows so both directions are visible in the same report. - pc_default_cap() also clamps the STARTING budget to what is spare right now, so a cold boot on a loaded machine does not open at a size the host cannot afford. Verified on a 48 GB host: engram boots in ~30s, RSS settles at 2.22 GiB (the store's actual size, resident, not creeping), 0.0% CPU, 13,439 nodes / 37,670 edges, embeddings complete. Guard reports 9.71 GiB available against a 2.00 GiB floor — 7.71 GiB of headroom it is permitted to use and no more.	2026-08-15 21:02:05 -05:00
will.anderson	7a479111ac	Merge pull request 'store: extend the write barrier to edges — kills the full-store walk' (#128) from fix/elc-rebuildable-compiler-builtins into dev	2026-08-16 01:44:33 +00:00
bigmerge	e917b3d439	store: make the buffer pool sense its own state and correct from it ... Follow-on to the edge write barrier. That fix removed the full-store walk; this one makes the pool able to notice if anything like it happens again. WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill twice on 2026-08-15 and said nothing. From outside it was indistinguishable from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each costing a deploy or a rollback. The whole time, hits/misses/evictions were already being counted in PgCache, and the struct comment read: /* stats (introspection only — never affect semantics) */ That comment was the bug. Self-measurement treated as decoration is why the pool could not correct itself and why no one outside could see what it was doing. A system that cannot read its own state cannot correct, and neither can anyone watching it. - pc_adapt_budget(): the loop, closed. Over a sliding window, evictions running at a large fraction of accesses WHILE reuse is real means the working set exceeds the budget — so grow it, geometrically, bounded by a LIVE re-read of physical memory. Evictions alone are not pressure (a scan evicts and never returns); evictions with reuse are. An explicit ENGRAM_POOL_FRAMES still wins — an operator override must not be silently overruled. - Budget derived, not declared. A constant cannot be right: 16 GiB of frames is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM at startup" is a guess about the future — it cannot know the store grew or the machine changed. Hence the live re-read. - pc_report(): ONE structured emission carrying the entire sensed state, through emit_log — El's existing telemetry, already exporting to OTLP. Deliberately not a function per stat, and deliberately not a bespoke /api/pool endpoint: both make observability something hand-written per noun instead of the uniform mechanism every component already has. - engram_pool_stats_json(): the same state readable live, wired through the normal builtin path (codegen arity + el_seed wrapper), so the pool can be observed in real time rather than reconstructed afterward from a stack sample. Verified: with the exact configuration that took production down (ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete — and NO pressure event fires, because the barrier removed the walk that caused it. The controller is defense in depth; the barrier is the fix.	2026-08-15 20:44:23 -05:00
bigmerge	777ccc02f0	store: extend the durable-hash write barrier to edges (kills the full-store walk) ... Checkpointing pushes the ENTIRE resident graph through store_put_node and store_put_edge (see engram_store_checkpoint). Nodes were cheap: a durable-hash compare skipped unchanged records with zero page I/O. Edges had no barrier at all — struct comment at PgCache.barrier_on even says "node durable-hash barrier" — so every edge was rewritten on every checkpoint, and each rewrite runs the idempotency probe max_page_lsn_for_id -> btree lookup -> page_read. Edges outnumber nodes ~3:1 here (37,663 vs 13,436), so routine checkpointing degenerated into a FULL-STORE WALK in id order: random page access across the whole 2 GiB store, repeated, overwhelmingly to rediscover nothing had changed. LRU is worst-case under exactly that pattern — it evicts the page it is about to want — so once the page cache was smaller than the store, the walk collapsed into thrashing: 100% CPU, flat RSS, no forward progress, port never bound. That took the live engram down twice on 2026-08-15. The walk is the defect. Sizing the cache to survive it treats the symptom. Changes: - dh_edge_hash(): edge counterpart of dh_node_hash, with a kind discriminator byte so an edge can never collide with a node of the same id in the shared map. created_at/updated_at/last_fired are excluded deliberately: last_fired is touched by activation without changing what the edge IS, and folding it in would defeat the barrier on precisely the hot edges that most need it. - store_put_edge(): barrier check + dh_set on success, mirroring store_put_node exactly. - store_scan_edges(): seed the barrier map from on-disk truth at load, so the FIRST post-boot checkpoint already skips unchanged edges. store_scan_nodes already did this and its comment says why; edges were simply never done. Verified: with the exact configuration that killed production (ENGRAM_POOL_FRAMES=65536 -> 1 GiB cache against a 2 GiB store), the engram now boots clean and serves — LISTENING, 13,436 nodes / 37,663 edges, embeddings complete, 0.0% CPU, RSS 1.14 GiB (cache resting at its budget rather than thrashing against it). Same small cache, same store, no walk.	2026-08-15 20:38:11 -05:00
will.anderson	c21074b547	Merge pull request 'runtime: engram_edges_json — kill the whole-graph file round trip' (#127) from fix/elc-rebuildable-compiler-builtins into dev	2026-08-16 01:13:44 +00:00
bigmerge	4e24d7d3f1	runtime: engram_edges_json — read edges without a whole-graph file round trip ... /api/graph/edges answered a read query by calling engram_save() to serialize the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in one line, and both bit production on 2026-08-15: 1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram server's CANONICAL store. A READ route overwriting the persistence owner's canonical file. This defect had been fixed once (export moved to a scratch path); it came back when the hand-written dispatch block was replaced by @route dispatch and the unfixed copy is the one that survived the merge. 2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole graph, per request, to return a bounded slice. Calling it tonight overwrote the canonical snapshot and immediately preceded an engram crash loop. engram_edges_json(limit, offset) is the builtin that route's own TODO asked for ("Future: add an engram_edges_json() builtin and drop the file round trip entirely"). It walks g->edges directly and emits every persisted field. limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell over, and an unbounded default would preserve the failure mode under a new name. Callers page explicitly. Registered in codegen.el's builtin_arity (both plain and __ spellings) and wrapped in el_seed.c per the project's C-builtin recipe.	2026-08-15 20:10:48 -05:00
will.anderson	7557ea6e19	Merge pull request 'runtime: restore engram_recall_json + cgi_* accessors (unblocks the soul build)' (#126) from fix/elc-rebuildable-compiler-builtins into dev	2026-08-16 00:57:11 +00:00
bigmerge	7351fb0a8d	runtime: restore engram_recall_json + cgi_* accessors ... neuron's soul calls engram_recall_json (neuron-api.el:618, memory.el:80) and cgi_principal (studio.el:72). Both existed in the runtime neuron vendored (v1.0.0-20260501) and were absent here, so the soul could not link against current el at all. The dangerous part is what the obvious "fix" would have done. These look like redundant wrappers over one impl: engram_search_json(q, limit) -> eg_search_json_impl(q, limit, 0) LEXICAL engram_recall_json(q, limit) -> eg_search_json_impl(q, limit, 1) SEMANTIC They are not interchangeable, and the split is documented at neuron-api.el:613: search stays LEXICAL because ~40 internal call sites pass a KEY and seven of them DELETE every record returned. Point those at a semantic matcher and they delete fuzzy matches. Conversely, pointing recall at search silently downgrades the mind's entire retrieval surface from semantic to lexical — no error, just permanently worse recall. Implemented over engram_activate(), which in this runtime already IS the semantic path the old with_legs=1 branch built by hand (embeds the query via eg_embed_fetch, scores by cosine, then spreads activation one hop). Output shape matches engram_search_json — a flat array via engram_emit_node_json — because callers parse search's shape, not activate's envelope. Verified: neuron's soul now compiles and links against current el, boots, and serves /health with layers initialized. NOTE for follow-up: current el also ships engram_retrieve_geometric_json, a structure-first retrieval that appears to be the intended successor to recall. Repointing the two recall call sites at it may well be the right end state and would remove the two-wrapper shape entirely — but that is a behavioral change that must be measured against neuron/tools/retrieval-eval/'s gold set, not assumed. This commit preserves existing behavior exactly; it does not decide that question.	2026-08-15 19:56:43 -05:00
will.anderson	d545b69614	Merge pull request 'runtime: restore the three builtins that made elc unrebuildable' (#125) from fix/elc-rebuildable-compiler-builtins into dev	2026-08-16 00:50:43 +00:00
bigmerge	598915cc61	runtime: restore the three builtins that made elc unrebuildable ... The committed elc binary could not be refreshed from its own source. Rebuilding failed with three implicit-declaration errors: el_mem_check, stdout_to_file, stdout_restore. The compiler's own source calls all three (compiler.el:472,479,574 and codegen.el:4248) and two are registered in codegen.el's builtin_arity table — but none were defined in this runtime. They were found intact in ui/examples/native-hello-ios/NativeHello/el_runtime.c, a divergent private copy of this runtime that still carried them. Ported verbatim. Consequence of them being missing: the canonical elc binary was frozen. Source gained @route dispatch codegen (emit_route_dispatch, codegen.el:3948) and the @manager boundary-beat seam, but no rebuilt binary could carry them, so neuron's soul — whose routes.el now calls the compiler-synthesized el_route_dispatch — could not be built at all. Verified after the fix: - elc rebuilds from current source, clean. - Self-hosting fixpoint byte-identical (stage3 == stage2). - The rebuilt elc emits el_route_dispatch (2 occurrences in the soul amalgam, previously 0) and injects engram_boundary_beat at @manager boundaries, i.e. the decorator seam is live rather than inert. el_mem_check is itself the compiler's memory guard (ELC_MAX_MEM_MB, default 512MB, self-terminates before the OS OOM-killer fires) — so the runtime was missing the very guard that would have surfaced the compiler's memory blowup as a clean error instead of a 27GB host-killer.	2026-08-15 19:50:17 -05:00
will.anderson	dab14f9100	Merge pull request 'engram: fix silently-wrong query params + make el_seed.o/el_runtime.o link' (#124) from fix/engram-query-param-and-seed-link into dev	2026-08-16 00:34:33 +00:00
bigmerge	40eb48e92f	engram: fix silently-wrong query params, and make el_seed.o + el_runtime.o link ... Three real bugs, all found by actually running the thing rather than reading it. 1. query_param never URL-decoded. A GET of /api/search?q=neural%20network searched for the literal string "neural%20network" and returned []. Every multi-word search against the live engram has been silently returning empty results — not an error, an empty result, which is why it went unnoticed. Affects every GET route that reads query params, not just search. 2. query_param matched key names unanchored. str_index_of(qs, "q=") matches inside "faq=", so "?faq=X&q=Y" returned X for key "q". Verified live before the fix. Now searches for "&key=" against "&"+querystring so a match can only land on a real parameter boundary. 3. el_request_start/el_request_end were defined in BOTH el_seed.c and el_runtime.c, so linking the two objects together — which is exactly what the product build does — failed with duplicate symbols. el_seed.c's own comment already says these moved there ("formerly defined in el_runtime.c. Now self-contained in el_seed.c"); the el_runtime.c copies were left behind during that move. Removed them, kept declarations since http_worker calls them. Also added the three missing prototypes (engram_op_assert_json, engram_node_full_in, engram_connect_in) that el_seed.c wraps but never declared, which made it fail to compile standalone under C99+. Verified: engram builds and links clean from canonical source; before/after comparison on a copy of the real store shows "neural network" returning a real match where the live build returns [], and "?faq=WRONG&q=MetaColloc" now resolving to MetaColloc. Live engram on :8742 was never touched.	2026-08-15 19:34:04 -05:00
will.anderson	c9f75e2592	Merge pull request 'engine: land op_assert + purview mutation wrappers (clean re-merge)' (#123) from merge-pr103-v2 into dev	2026-08-15 23:27:17 +00:00
bigmerge	09dade0613	Merge remote-tracking branch 'origin/pr/103' into HEAD ... # Conflicts: # lang/AGENTS.md # lang/runtime/el_runtime.h	2026-08-15 18:26:57 -05:00
will.anderson	38a8e32d6c	Merge pull request 'engram: batch-cosine Adapter/Strategy/Factory over ggml (supersedes #114)' (#116) from feat/engram-ggml-cosine-batch into dev	2026-08-15 23:22:56 +00:00
will.anderson	15b66c8b1a	Merge pull request 'swarm: land wt/swarm-ccr onto dev (clean re-merge)' (#122) from merge-swarm-ccr-v2 into dev	2026-08-15 23:17:56 +00:00
bigmerge	9883aa7564	Merge remote-tracking branch 'origin/wt/swarm-ccr' into merge-swarm-ccr-v2 ... # Conflicts: # lang/runtime/el_runtime.c # lang/runtime/el_runtime.h	2026-08-15 18:17:33 -05:00
will.anderson	d45a0882f3	Merge pull request 'nsbx: fail loud on daemon-not-ready + el_seed.c standalone compile' (#118) from fix/nsbx-tooling-hardening into dev	2026-08-15 23:06:01 +00:00
will.anderson	1c9de03fdb	Merge pull request 'engram: make the ggml batch-cosine strategy actually compute in fp32 (recall 0.9933 -> 0.9987)' (#121) from improve/ggml-cosine-fp32-and-init into feat/engram-ggml-cosine-batch	2026-08-15 23:01:29 +00:00
bigmerge	c008b7228a	engram: make the ggml batch-cosine strategy actually compute in fp32 ... #116 shipped the ggml strategy at 0.9933 id-recall against the CPU oracle while the hand-rolled Metal kernel it replaced scored 0.9997 — a ~150x worse error margin. That was not an inherent property of ggml. It was a usage bug in this file, and this commit fixes it. ggml-metal has two F32xF32 matmul kernels and picks between them purely on ne11, the number of B rows, which for us is the query-batch size: ne11 <= 8 -> kernel_mul_mv_ext_f32_f32_* / kernel_mul_mv_f32_f32_*, templated <float, float> — genuine F32. ne11 > 8 -> kernel_mul_mm_f32_f32, templated <half, half4x4, simdgroup_half8x8, half, half2x4, ...> — BOTH operands narrowed to F16, despite F32 tensors on both sides. The old code issued one ggml_mul_mat with ne11 = nq (300 in the benchmark), landing squarely on the F16 path. The file's own header comment asserted the opposite ("computes in F32 on the Metal backend"); that claim was wrong and is replaced with the measurement. Fix: emit ceil(nq/8) mul_mats over ne11<=8 ggml_view_2d slices of one query tensor, all expanded into ONE graph and one ggml_backend_graph_compute, so the node matrix is still uploaded and shared exactly once. EL_GGML_MULMAT_CHUNK overrides the 8; setting it >= nq reproduces the old behaviour exactly, which is also how the before/after below was measured in a single binary. Measured, real store snapshot, 13415 live embedded nodes, dim=768, 300 real queries, vs the CPU double-accumulated oracle (vindex_bench, offline copy of the store — no live service touched): id-recall same-rank |Δdist| max mean old (ne11=300) 0.9933 6.80e-05 1.43e-05 new (ne11<=8) 0.9987 4.77e-07 9.30e-08 hand-rolled 0.9997 3.58e-07 7.55e-08 ~145x better max error, ~154x better mean — now the same order of magnitude as the hand-rolled kernel rather than 150x off it. The cost is real and is documented rather than buried. Median of 15 reps of the whole batch_multi() call, three runs: 13.2-14.4ms unchunked, 19.9-20.2ms chunked, 17.7-18.0ms hand-rolled. Correctness costs ~+6.7ms per 300-query batch and leaves ggml ~12% behind the hand-rolled kernel instead of ~35% ahead. It cannot be recovered inside ggml: an fp32 matmul on Metal must re-stream the node matrix once per <=8 queries, and ggml's Metal backend ships no fp32 TILED matmul, so "fast" and "fp32" are genuinely exclusive there. Two things that did NOT work, recorded so nobody retries them: - ggml_mul_mat_set_prec(t, GGML_PREC_F32) does nothing here. Error was bit-identical with and without it (1.038e-05 either way) — ggml-metal has no F32-accumulating mul_mm kernel to switch to. ne11 is the only lever. - The ACCEL/BLAS device looked excellent in an isolated compute-only probe (3.4-4.0ms, mean |Δdot| 1.5e-08) but is dominated on BOTH axes end-to-end (0.191 ms/query at 0.9973 recall vs 0.125-0.142 at 0.9987), because the probe was not competing for the same CPU cores the real call path is. It stays reachable via EL_GGML_DEVICE as a no-Metal fallback, labelled as measured-and-rejected, not as a recommendation. Also corrected: the ~7.8s "cold start" blamed on this file is not this file re-initialising per call — init was already cached. It is Apple's shader cache missing on ggml's embedded metallib (~650 kernels), keyed on the library and shared across processes: the first load on a machine reports "loaded in 7.670 sec", the next run of a *different* binary reports 0.009 sec. Once per machine per ggml version, not once per process, and not ours to fix. Warm ggml init is 44-53ms vs 36-117ms for the hand-rolled strategy. Loading only libggml-metal.so instead of every plugin in the directory is kept for tidiness, and explicitly documented as NOT a speedup: 44.7-52.4ms against 46.9-58.9ms, the same number inside noise. The -2.0 sentinel contract is unchanged and re-verified at batch sizes that straddle the chunk boundary (1,7,8,9,16,17,33), plus NULL rows, dim mismatches, zero-norm rows, and an all-invalid population. Notably the old ne11=300 path fails that same check at a 2e-6 cosine tolerance with 2299 mismatches, which is an independent confirmation of the defect.	2026-08-15 17:57:09 -05:00
bigmerge	3718bf0380	runtime: port missing __channel_* primitives into el_seed.c ... runtime/channel.el has always called __channel_new/__channel_send/ __channel_recv/__channel_try_recv/__channel_close, but these were only ever implemented in the pre-restructure lang/el-compiler/runtime/el_runtime.c. When the canonical runtime was consolidated onto the release copy (lang/runtime/el_runtime.c) and el_seed.c became the sole C dependency, the channel implementation was never carried forward — __mutex_new made the move, __channel_* did not. Any El program using Go-style channels currently fails to link on dev. Ported the working buffered-MPMC-channel implementation (mutex+condvar+ circular buffer, bounded and unbounded modes) from the old el_runtime.c verbatim, adapted only to el_seed.c's arena API (seed_arena_track in place of el_arena_track). Declared in el_seed.h alongside the existing mutex primitives.	2026-08-15 17:50:04 -05:00
bigmerge	9d40f87926	ingest: unify transduce_prose/transduce_structured into one transduce() ... transduce() is now THE single mechanism: one function, no content-type branch inside it. It never asks whether `source` is prose, JSON, or raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally: split on "\n\n" as a universal boundary-marker check, and if that finds no boundary, fall back to fixed 4096-char windows. Same node/edge wiring (root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets a heading/section_of link) regardless of what's inside a chunk. Dedup is the existing find_existing_by_content path via merge_manifold, applied uniformly. The old transduce_structured JSON dataset/records/feature-node interpretation is deleted outright, not just unused — a JSON file now gets chunked and deduped like anything else, with no pre-computed structure. All five ingest_* entry points still exist unchanged in name and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one transduce() (ingest_stream builds its own turn-nodes directly and never called either old function, so it's untouched). This unlocks raw/opaque content (audio, or anything else with no natural text/JSON shape) without any DSP, LLM call, or external API: transduce() chunks it exactly like it chunks anything else. There is zero semantic understanding of audio (or any payload) claimed or built here — any meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time. Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk) because El strings are NUL-unsafe under strlen-based ops and fs_read()'s result silently truncates at the first embedded NUL, which is routine in real binary/audio bytes. ingest_file compares fs_read()'s string length against a real fs_size() stat() count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim, no invention), joined with the same "\n\n" marker transduce()'s boundary scan already looks for. This is a mechanical fidelity fix, not interpretation of content — transduce() never learns a fallback happened. Registered both builtins' arity in codegen.el; did not rebuild the elc compiler binary itself (unrelated, pre-existing gap: self-hosting elc via el_seed.c fails on this worktree independent of this change, reproduced with codegen.el reverted) — the existing elc binary compiles calls to unregistered builtins via its already-existing arity=-1 passthrough, confirmed by an actual clean `elc ingest.el` + `cc` build against the modified el_runtime.c. INGEST_KIND keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a content-type flag; the redundant "structured" value (an alias for "file" that hinted the now-deleted JSON branch) is removed. ingest_dir drops its file-extension filter for the same reason: transduce() takes anything now. Verification: local manifold construction confirmed correct against a real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte real prefix slice) — exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes. Compiles clean via the real elc + the modified el_runtime.c/engram_*.c (built and booted an actual sandbox engram off this exact source with `nsbx create --branch`). NOT verified this session, disclosed rather than papered over: end-to-end server-confirmed persistence (a real before/after /api/stats delta, and a fetched node by id) for the audio, prose, and JSON-fixture cases. Every local nsbx sandbox engram tried tonight (two stock pre-#109 binaries hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW binary built from current dev) took minutes-to indefinitely long on the final /api/load-merge write's embedding step and hit the client's 60s HTTP timeout before responding, even for a 5-node write. This is confirmed as real (if slow) forward progress, not a hang: the sandbox's WAL file was observed growing steadily across every attempt. The code's own pre-existing HONESTY GATE correctly refused to report success in every case, returning "load-merge failed: ..." with a "nothing below this manifold was confirmed persisted by the server" note instead — exactly as designed. This is an environment/infrastructure limitation, not a defect introduced by this change: the engram server binary itself is untouched by this commit.	2026-08-15 17:50:04 -05:00
bigmerge	90d3f0bc76	engram: port PR #105's 3 genuine wins onto dev's existing cosq/e_eff semantic layer ... Reconciles PR #105 ("fix: engram search latency — pin embed model, cache query embeddings, bound activate BFS") with dev's ACTUAL current engram_activate, rather than the ancient pre-restructure snapshot #105 was built against. WHY THIS NEEDED RECONCILIATION, NOT A DIRECT PORT: #105's single commit (1dc49b1) modifies `lang/el-compiler/runtime/el_runtime.c` — a path that does not exist on dev (dev has `lang/runtime/el_runtime.c`; the restructure that renamed it happened after #105's branch point, which traces to a July 22 merge-base, weeks before the M8/M8.1/qgate/fan-effect/adjacency-index work this file has grown since). #105's own engram_activate is consequently the PRE-restructure version: no adjacency index (O(E) full edge scan per hop), no query-aware qgate, no ACT-R fan effect, no eg_edge_eff_weight, and no awareness of dev's cosq/e_eff embedding-blend semantic layer — it built a parallel `g_qcache`/`engram_embed_raw` mechanism from scratch against code that no longer exists at that path. A raw merge/cherry-pick was not possible and would have been wrong even if it were: taking #105's tree wholesale would have thrown away everything dev grew in the meantime (qgate, fan effect, adjacency index, and this session's own M8 HNSW vindex integration). RECONCILIATION: kept dev's cosq/e_eff mechanism as the semantic layer entirely intact (unchanged by this commit) and ported #105's three genuinely additive wins on TOP of it, at their equivalent sites in the CURRENT eg_embed_fetch/engram_activate: 1. keep_alive:-1 on the Ollama embed request body (eg_embed_fetch) — pins the embed model resident so a larger generation model loading under unified-memory pressure can't evict it and force a cold reload on the next search (#105 measured ~2.2s cold vs ~0.02-0.05s warm). 2. Query-embedding cache upgraded from dev's single-slot (`_eg_qcache_text`, only ever remembered the LAST query) to a direct-mapped, FNV-1a-keyed, 1024-slot cache (reusing the existing engram_id_hash) — so the curiosity loop's rotating phrases actually hit the cache instead of evicting each other every call. Same "pointer owned by the cache, not freed by caller" contract as before, just per-slot instead of global. 3. Beam cap on the layer-1 spreading-activation BFS (new engram_activate_beam(), tunable via ENGRAM_ACTIVATE_BEAM, default 128). The FIFO frontier is processed in hop-level batches (entries sharing .hops are provably contiguous — see the code comment); when a level exceeds the beam width, only the top-`beam` by activation actually EXPAND. Every node in an oversized level still gets reached[]/best_bg[] recorded (that happens at enqueue time, one level up) and appears in the reported/promoted set — the cap bounds associative SPREAD width only, never recall of what was already found. Kept as a genuine additional bound even though the adjacency index + qgate + fan effect already mitigate #105's original "hub-node explosion" failure mode for a different reason: those prune WHICH targets matter; this bounds worst-case width regardless. Everything else in dev's engram_activate — cosq/e_eff, the qgate rescale, the fan effect, eg_edge_eff_weight, the M8 HNSW vindex seed discovery from the #109 reconciliation earlier this session — is untouched. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): cc -std=c11 -O2 clean build; booted in an isolated sandbox against a real cloned production snapshot (13,424 nodes / 37,656 edges); ran 5 activate() calls across rotating queries at depth 3, including the same query issued twice non-consecutively (2nd hit landed at 476ms vs the 1st at 483ms — consistent with a cache hit once Ollama's own warm-model latency is accounted for; no crash, correct varied result counts (367-2610 nodes) each call; act-stats JSON read correctly throughout. Built on top of the M8/#109 reconciliation (bacaf3d, merged to dev as #109) — dev's current HEAD at the time of this commit.	2026-08-15 17:50:04 -05:00
will.anderson	ee39aa5f17	Merge pull request 'ingest: unify transduce_prose/transduce_structured into one transduce()' (#117) from feat/transduce-unify into dev	2026-08-15 22:36:12 +00:00
bigmerge	2d0aef4ef8	lang: declare the el_runtime.c symbols el_seed.c's wrappers call ... runtime/el_seed.c does not compile standalone via the exact command tools/install.sh uses (`cc -std=c11 -O2 -I runtime -c runtime/el_seed.c`): 51 of its __-prefixed wrapper functions (http serving, JSON access, key-val state, URL/HTML escaping, and the whole engram_* node/edge/layer/search surface) call unprefixed counterparts that are implemented in el_runtime.c, not in el_seed.c itself, and el_seed.c never declared them -- a toolchain that treats an implicit function declaration as a hard error under C11 fails the compile outright. install.sh already compiles el_seed.c and el_runtime.c as separate objects and archives both into libel.a, so the symbols are always present at link time; el_seed.c alone was just missing the prototypes. A plain `#include "el_runtime.h"` was tried first and rejected: it redefines el_to_float/el_from_float, which el_seed.h already provides -- a real compile error, not a style preference. Added narrow prototypes instead, copied verbatim from el_runtime.h, for exactly the 51 symbols el_seed.c's wrappers reference and nothing else. Verified clean: - `cc -std=c11 -O2 -I runtime -c runtime/el_seed.c` (install.sh's exact per-file compile) -- 0 errors, 0 warnings, even with -ferror-limit=0. - full `tools/install.sh` run -- compiles both objects and archives them into libel.a successfully. Separately (not fixed here, out of scope): AGENTS.md's documented compiler self-rebuild command links elc-new.c against el_seed.c, but elc-new.c's own generated `#include "el_runtime.h"` line and 3 undeclared symbols (el_mem_check, stdout_to_file, stdout_restore -- present in neither el_runtime.c nor el_seed.c) mean that command fails regardless of which runtime file it's linked against; and install.sh's libel.a only archives el_seed.o + el_runtime.o, so any program that calls into the engram_* surface fails to link against it (el_runtime.c's engram_* wrappers need engram_store.c/engram_geometry.c/engram_reason.c/engram_cognition.c/ engram_vindex.c, none of which install.sh compiles in). Both are real, pre-existing, and independent of this fix -- worth their own look.	2026-08-15 17:32:58 -05:00
bigmerge	979e820f68	nsbx: fail loud on daemon-not-ready instead of printing a false success banner ... Three confirmed-live bugs tonight: - `nsbx up` printed "daemon did not become ready" immediately followed by a green "your sandbox is ready" banner and exited 0, because the existing- sandbox restart path (`daemon_alive || start_daemon`) never checked start_daemon's return code. `cmd_build` had the identical unguarded pattern, plus `cmd_run`/`cmd_validate`'s own start-if-dead calls. All four now `|| die` with a message pointing at daemon.log. - `nsbx status`/`nsbx list` reported bare "state: running" for a process that's alive (passes kill -0) but not actually answering /api/stats -- pegged, hung, or mid-boot. Added daemon_health(), which does the real stats fetch and distinguishes stopped/running/unresponsive; both commands now say "running but NOT RESPONDING" with a next-step hint instead of silently going quiet on the stats field. Reproduced live against another agent's actively-running (CPU-pinned, non-responsive) sandbox tonight, and again via a deliberate SIGSTOP on a throwaway sandbox. - Sandboxes carried no visible signal that their binary predated a relevant fix. `status`/`list` now show the binary's sha + real build timestamp (mtime survives `cp -p`), plus a best-effort staleness note: for stock-prod clones, compare against the currently-configured live binary; for source/branch builds, compare the recorded source commit against local origin/dev via merge-base --is-ancestor. Also, found live while verifying the above: - A cold boot under concurrent sandbox/CPU load can legitimately take past the old hardcoded 15s readiness window. Made it configurable (NSBX_READY_TIMEOUT_SECS) rather than just widening the default blindly. - cmd_create's post-boot baseline capture could silently record sbx_baseline as 0/0 when the stats fetch came back empty right after the auto-remerge step -- which would make every future `nsbx validate` zero-loss/reboot- prove check trivially PASS regardless of real data loss. Added a bounded retry and a loud warning if it still comes back empty. - Sharpened a handful of "no such sandbox" / missing-binary errors to name the next command instead of just stating the failure.	2026-08-15 17:32:44 -05:00
bigmerge	e29fe4fd0b	ingest: unify transduce_prose/transduce_structured into one transduce() ... transduce() is now THE single mechanism: one function, no content-type branch inside it. It never asks whether `source` is prose, JSON, or raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally: split on "\n\n" as a universal boundary-marker check, and if that finds no boundary, fall back to fixed 4096-char windows. Same node/edge wiring (root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets a heading/section_of link) regardless of what's inside a chunk. Dedup is the existing find_existing_by_content path via merge_manifold, applied uniformly. The old transduce_structured JSON dataset/records/feature-node interpretation is deleted outright, not just unused — a JSON file now gets chunked and deduped like anything else, with no pre-computed structure. All five ingest_* entry points still exist unchanged in name and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one transduce() (ingest_stream builds its own turn-nodes directly and never called either old function, so it's untouched). This unlocks raw/opaque content (audio, or anything else with no natural text/JSON shape) without any DSP, LLM call, or external API: transduce() chunks it exactly like it chunks anything else. There is zero semantic understanding of audio (or any payload) claimed or built here — any meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time. Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk) because El strings are NUL-unsafe under strlen-based ops and fs_read()'s result silently truncates at the first embedded NUL, which is routine in real binary/audio bytes. ingest_file compares fs_read()'s string length against a real fs_size() stat() count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim, no invention), joined with the same "\n\n" marker transduce()'s boundary scan already looks for. This is a mechanical fidelity fix, not interpretation of content — transduce() never learns a fallback happened. Registered both builtins' arity in codegen.el; did not rebuild the elc compiler binary itself (unrelated, pre-existing gap: self-hosting elc via el_seed.c fails on this worktree independent of this change, reproduced with codegen.el reverted) — the existing elc binary compiles calls to unregistered builtins via its already-existing arity=-1 passthrough, confirmed by an actual clean `elc ingest.el` + `cc` build against the modified el_runtime.c. INGEST_KIND keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a content-type flag; the redundant "structured" value (an alias for "file" that hinted the now-deleted JSON branch) is removed. ingest_dir drops its file-extension filter for the same reason: transduce() takes anything now. Verification: local manifold construction confirmed correct against a real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte real prefix slice) — exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes. Compiles clean via the real elc + the modified el_runtime.c/engram_*.c (built and booted an actual sandbox engram off this exact source with `nsbx create --branch`). NOT verified this session, disclosed rather than papered over: end-to-end server-confirmed persistence (a real before/after /api/stats delta, and a fetched node by id) for the audio, prose, and JSON-fixture cases. Every local nsbx sandbox engram tried tonight (two stock pre-#109 binaries hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW binary built from current dev) took minutes-to indefinitely long on the final /api/load-merge write's embedding step and hit the client's 60s HTTP timeout before responding, even for a 5-node write. This is confirmed as real (if slow) forward progress, not a hang: the sandbox's WAL file was observed growing steadily across every attempt. The code's own pre-existing HONESTY GATE correctly refused to report success in every case, returning "load-merge failed: ..." with a "nothing below this manifold was confirmed persisted by the server" note instead — exactly as designed. This is an environment/infrastructure limitation, not a defect introduced by this change: the engram server binary itself is untouched by this commit.	2026-08-15 17:28:19 -05:00
bigmerge	b3f410fc91	engram: batch-cosine Adapter/Strategy/Factory over ggml, supersedes hand-rolled PR #114 ... Stop hand-rolling GPU kernels for batch cosine similarity — use ggml (the MIT-licensed compute library underneath llama.cpp, installed standalone via Homebrew) as the preferred backend, without ripping out PR #114's carefully-verified hand-rolled Metal shader. Structure: one stable public adapter (eg_cosine_batch.h, zero #ifdef at call sites) backed by three selectable concrete Strategies behind an internal vtable (eg_cosine_batch_strategy.h) chosen by a Factory (eg_cosine_batch.c): - eg_cosine_batch_strategy_ggml.c — NEW. ggml + dynamically-loaded Metal backend plugin (ggml_backend_load_all_from_path + ggml_mul_mat for the batched dot product), gather/scatter around the -2.0 sentinel contract. - eg_cosine_batch_strategy_metal_hand.m — PR #114's original hand-rolled Metal shader bridge, preserved almost verbatim, now one strategy among several rather than the only option. eg_cosine_batch.metal kept byte-identical to the original. - eg_cosine_batch_strategy_cpu.c — universal always-false fallback (direct descendant of PR #114's eg_metal_cosine_stub.c). Selection: EL_COSINE_BATCH_STRATEGY=ggml|metal|cpu|auto (default: ggml first, then hand-rolled Metal, then CPU — first available wins), plus back-compat EL_METAL_COSINE=0 to disable every GPU-backed strategy. build_vindex_bench.sh compiles all three strategies on Darwin, CPU-fallback-only elsewhere. vindex_bench.c now reports BRUTE-GGML and BRUTE-METAL side by side against the same CPU oracle, on the same dataset, in one run (real numbers vs. real store snapshot in the PR body).	2026-08-15 17:16:41 -05:00
will.anderson	0e924f7df9	Merge pull request 'engram: reconcile #105's embed-cache/beam-BFS latency fixes onto current dev' (#115) from fix/engram-search-latency-reconciled into dev	2026-08-15 22:07:45 +00:00
bigmerge	1bb1edc851	engram: port PR #105's 3 genuine wins onto dev's existing cosq/e_eff semantic layer ... Reconciles PR #105 ("fix: engram search latency — pin embed model, cache query embeddings, bound activate BFS") with dev's ACTUAL current engram_activate, rather than the ancient pre-restructure snapshot #105 was built against. WHY THIS NEEDED RECONCILIATION, NOT A DIRECT PORT: #105's single commit (1dc49b1) modifies `lang/el-compiler/runtime/el_runtime.c` — a path that does not exist on dev (dev has `lang/runtime/el_runtime.c`; the restructure that renamed it happened after #105's branch point, which traces to a July 22 merge-base, weeks before the M8/M8.1/qgate/fan-effect/adjacency-index work this file has grown since). #105's own engram_activate is consequently the PRE-restructure version: no adjacency index (O(E) full edge scan per hop), no query-aware qgate, no ACT-R fan effect, no eg_edge_eff_weight, and no awareness of dev's cosq/e_eff embedding-blend semantic layer — it built a parallel `g_qcache`/`engram_embed_raw` mechanism from scratch against code that no longer exists at that path. A raw merge/cherry-pick was not possible and would have been wrong even if it were: taking #105's tree wholesale would have thrown away everything dev grew in the meantime (qgate, fan effect, adjacency index, and this session's own M8 HNSW vindex integration). RECONCILIATION: kept dev's cosq/e_eff mechanism as the semantic layer entirely intact (unchanged by this commit) and ported #105's three genuinely additive wins on TOP of it, at their equivalent sites in the CURRENT eg_embed_fetch/engram_activate: 1. keep_alive:-1 on the Ollama embed request body (eg_embed_fetch) — pins the embed model resident so a larger generation model loading under unified-memory pressure can't evict it and force a cold reload on the next search (#105 measured ~2.2s cold vs ~0.02-0.05s warm). 2. Query-embedding cache upgraded from dev's single-slot (`_eg_qcache_text`, only ever remembered the LAST query) to a direct-mapped, FNV-1a-keyed, 1024-slot cache (reusing the existing engram_id_hash) — so the curiosity loop's rotating phrases actually hit the cache instead of evicting each other every call. Same "pointer owned by the cache, not freed by caller" contract as before, just per-slot instead of global. 3. Beam cap on the layer-1 spreading-activation BFS (new engram_activate_beam(), tunable via ENGRAM_ACTIVATE_BEAM, default 128). The FIFO frontier is processed in hop-level batches (entries sharing .hops are provably contiguous — see the code comment); when a level exceeds the beam width, only the top-`beam` by activation actually EXPAND. Every node in an oversized level still gets reached[]/best_bg[] recorded (that happens at enqueue time, one level up) and appears in the reported/promoted set — the cap bounds associative SPREAD width only, never recall of what was already found. Kept as a genuine additional bound even though the adjacency index + qgate + fan effect already mitigate #105's original "hub-node explosion" failure mode for a different reason: those prune WHICH targets matter; this bounds worst-case width regardless. Everything else in dev's engram_activate — cosq/e_eff, the qgate rescale, the fan effect, eg_edge_eff_weight, the M8 HNSW vindex seed discovery from the #109 reconciliation earlier this session — is untouched. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): cc -std=c11 -O2 clean build; booted in an isolated sandbox against a real cloned production snapshot (13,424 nodes / 37,656 edges); ran 5 activate() calls across rotating queries at depth 3, including the same query issued twice non-consecutively (2nd hit landed at 476ms vs the 1st at 483ms — consistent with a cache hit once Ollama's own warm-model latency is accounted for; no crash, correct varied result counts (367-2610 nodes) each call; act-stats JSON read correctly throughout. Built on top of the M8/#109 reconciliation (bacaf3d, merged to dev as #109) — dev's current HEAD at the time of this commit.	2026-08-15 16:55:32 -05:00