50 Commits

Author	SHA1	Message	Date
Neuron	8e9d88fc01	runtime: publish the vector index instead of guarding it ... The crash (SIGTRAP in engram_activate -> eg_vindex_sync -> vindex_insert -> _realloc) had three read paths mutating five process-global statics. engram_activate, eg_knn_for_node (whose own comment says "No writes.") and engram_geo_reify_run_json all called eg_vindex_sync, which frees the index, reallocs the seen-map and inserts — on a read. Three moves, in decreasing order of how much they dissolve: 1. Misfiled scratch is not shared state. visited/visit_epoch/visited_cap were never owned by the index; they are one traversal's local, hoisted into struct VIndex as an allocation optimisation. They want neither a lock nor a capability nor a pool — just to go back in the call frame. Two concurrent READS stomped each other purely because of this. 2. const IS the capability. Once the scratch leaves the struct, search reads and nothing else, so vindex_search takes a const VIndex*. That is exactly what a capability-pointer ABI would have bought — a read path physically cannot call vindex_insert, enforced by the compiler on every future caller — for one qualifier instead of an ABI swept across hundreds of builtins. 3. What survives is publication, not ownership. HNSW insert is NOT an append: it rewires the neighbour links of already-existing elements and reallocs elems[], so the store's append-only property does not transfer to the index derived from it. eg_vindex_sync therefore splits into eg_vindex_maintain (exclusive, sole mutator) and eg_vindex_view (shared, returns const VIndex*). A read path may demand that a current snapshot exist — a request to the owner, not a mutation by the reader. Write-side owner: eg_vindex_note_embedded hooks the embedding-ASSIGNMENT sites rather than the append sites, because a node with no embedding cannot be in a vector index — embedding assignment is the event that owns index membership. One O(log n) insert, no O(node_count) presence scan. This also retires the "STALENESS (honest tradeoff)" note where a lazily-embedded older node stayed invisible to route_nearest/autoconnect until a full rebuild (the embed-gap #20 shape). Evidence. The existing harness conflated two hazards, which is why fixing half of it read as failure. Split into four: single (3000 vec, ASan+UBSan) clean -> clean readers (4 readers, no writer, TSan) RACE -> clean unsynchronized (writer+reader, bare) race -> race, expected forever published (owner + 4 readers) n/a -> clean, 3000/3000 landed RESULT: PASS. recall@10 = 0.9365 at ef_search=128 (gate >= 0.90); determinism byte-identical across two independent builds. The unsynchronized half is now permanently expected to race, deliberately: it is the executable proof that the boundary must live above the data structure, not inside it. fb32d15's guard is KEPT, correcting this design's own section 5. Measured, it guards TWO structures and only one was converted here: g->nodes/g->edges are realloc'd in place (el_runtime.c:7618,7629) and engram_activate_inner's embed-backfill writes n->emb through exactly such a borrowed pointer. Deleting the guard reintroduces a measured 11171->9579 edge loss. Its comment is narrowed to the RAM graph and the deletion precondition named. That corrects the ordering claim too: the residual is not one ABI that dissolves everything at once, it is a PROPERTY applied per structure. Residues evaporate in the order the property is applied, and a residue whose structure has not been converted must be left standing.	2026-08-16 11:29:17 -05:00
bigmerge	bdc1f99fb9	runtime: guard engram activation against the unsynchronized awareness thread ... The soul daemon had two engram callers and only one of them locked. soul.el:729 starts the HTTP server via http_serve_async (spawning http_worker threads); soul.el:731 then runs awareness_run() on the MAIN thread. awareness.el's perceive() -> engram_activate_json() -> engram_activate() -> eg_vindex_sync() -> vindex_insert() mutates the same g->nodes/g->edges and the process-global _eg_vindex HNSW index that the workers touch. g_engram_req_lock existed to serialize exactly this, but it was only ever taken inside http_worker: engram_req_lock/engram_req_unlock appear in ZERO .el sources, so the awareness loop ran lock-free beside the workers on every tick (SOUL_TICK_MS=1000). Result was a crash-loop under launchd KeepAlive: five crashes in ~4 minutes on 2026-08-16 with varying faulting frames -- search_layer<-vindex_insert <-eg_vindex_sync, engram_activate, abort, and one inside xzm_realloc's own freelist. Varying sites plus a fault in allocator metadata means heap corruption. The SIGSEGV address 0x65646f4e6d617267 is little-endian ASCII "gramNode": string bytes dereferenced as an Elem vector pointer. Diagnosed by bisection rather than inspection: - Replaying all 13,820 real dim-768 vectors harvested from the live store through the index single-threaded under ASan is 100% clean, which rules out an HNSW logic/bounds bug. - Two threads on one index trip ThreadSanitizer immediately at engram_vindex.c:195 (visited_reset), reached from both vindex_search and vindex_insert. VIndex keeps a SHARED visited-epoch scratch buffer, so even two concurrent READS corrupt each other's traversal and walk bogus element indices. So this is purely a concurrency defect, not an HNSW logic error. (An inspection-derived hypothesis about an out-of-bounds reverse-link write at engram_vindex.c:340 was disproved by the single-threaded run.) Fix: a thread-local ownership depth (_eg_req_depth) lets engram entry points self-guard. engram_activate() becomes a wrapper over engram_activate_inner() that acquires g_engram_req_lock when called with depth 0 (the awareness thread) and passes through when depth > 0 (nested inside an http_worker that already holds it), so the non-recursive mutex cannot self-deadlock. The depth is a plain counter, never a recursive-mutex count, preserving engram_self_reify_beat_json's contract of genuinely releasing the lock mid-beat.	2026-08-16 11:29:17 -05:00
Neuron	ded6ca546f	runtime: anchor the think read, so Neuron can think at all ... engram_think_json passed NULL as the anchor. NULL is not "no opinion": engram_think re-origins at `anchor ? anchor : region->centroid`, so NULL means "read from the centroid" — and the centroid is the one point where the gradient is zero by construction. r = x - centroid = 0, so every axis projection is 0, grad is 0, and direction takes the "at rest" branch at engram_cognition.c:137. Measured consequence: EVERY faculty returned an identical null result, differing only in its label — {"direction":[0,0,0,0,0,0,0,0],"spread":0,"magnitude":1,"confidence":0.5} magnitude 1 is membership evaluated at the centroid, spread 0 is its distance to itself, confidence 0.5 is the stance fallback. The geometry was never at fault: /api/drift computes real values (centroid_sep 0.104, core_disp 0.045) over the very same 87 members. Neuron could not think because the read was always taken from the region's own centre. The seeds choose WHICH region; they must also supply the VANTAGE. Anchor at the first resolvable embedded seed — the same seed eg_geo_build_desc infers dim from, so the two can never disagree. One seed still yields a real gradient because the descriptor expands to that seed's neighbourhood, so the seed's position is distinct from the neighbourhood centroid. The vector is COPIED, never borrowed: g->nodes is realloc'd in place on append, so a borrowed EngramNode* dangles across any concurrent write. Verified against a clone of the production store (13,616 nodes / 37,865 edges): self anchor n_support 87 magnitude 0.00282 spread 18.79 values hub n_support 28 magnitude 0.00318 spread 17.72 with distinct unit direction vectors. Previously both returned the zero vector with magnitude 1 and spread 0. STILL OPEN, now isolated by this fix: all five faculties return identical numbers and confidence stays 0.5, because cog_stance_init is passed NULL for the stance and the faculty enters the computation only through the stance's axis_gain[] and bias_dir. The faculty label is inert until a stance is loaded — which is what learn()'s correspondence-beat calibrates. Same shape as this bug: a neutral parameter collapsing a capability to a constant.	2026-08-16 11:25:24 -05:00
Neuron	c79033b749	runtime: let signal enter as geometry, not as prose about signal ... No ingest path could carry a vector. engram_node/_full/_layered take text only, and a node acquired an embedding solely via engram_embed_backfill DERIVING one from n->content. That made text the mandatory entry medium: any non-text modality had to be described in prose first, so the geometry we then reasoned over was the geometry OF THE DESCRIPTION, not of the signal. Measured: POST /api/nodes accepted an "emb" field, returned 200 with a fresh id, and stored nothing — emb_dim=None, embedded=false. engram_node_set_emb attaches a vector to an existing node. Off-dimension vectors are stored but not indexed (the HNSW build loop already filters on emb_dim), so modality geometry is durable and addressable without perturbing the canonical index. Setting emb also makes the node ineligible for embed_backfill, so a realizer's vector is never overwritten by a text-derived one. Two reporting fixes ride along, because both are how the drop stayed invisible: the create response now reports emb_set instead of being success-shaped regardless, and the node document now always emits emb_dim and embedded — without which a genuine ingest drop and a mere reporting gap are indistinguishable. Verified live: voice node emb_dim=64 embedded=true; text control emb_dim=0 embedded=false; malformed hex, length mismatch and dim<=0 all reject. KNOWN PLACEMENT DEFECT: this is at the consumer. Ingest is a language concern, not an engram feature — every el program touching any modality needs it. The vector also marshals as a hex STRING because el has no first-class geometry value, which reintroduces text as the transport medium one layer below the problem being fixed. The durable shape is geometry as an el value plus declarable realizers, after which the engram stops having an ingest concept at all. Landing this as the verified probe that proves the path.	2026-08-16 11:12:48 -05:00
bigmerge	9c07970943	runtime: state_get leaked its value on every call ... char* result = el_strdup_persist(e ? e->value : ""); // never freed pthread_mutex_unlock(&_state_mu); char* copy = el_strdup(result); // arena-tracked return el_wrap_str(copy); Two copies were made. `result` existed only as the source for `copy` — never returned, never freed — and el_strdup_persist bypasses the arena BY DESIGN ("state_set, engram internals"), so arena-pop could never reclaim it. Every state_get leaked its full value string, permanently. MEASURED: 200,000 state_get calls against a 64-byte value. before 15 MB peak RSS growth (~75 bytes/call — the value plus overhead) after 0 MB IMPACT. The soul's awareness loop has 68 state_get call sites and ticks every 200ms. Live measurement before the fix: RSS climbing 112 MB per 20s, about 19 GB/hour, in awareness_run -> one_cycle -> perceive, while node_count stayed flat at ~13,479 — growth with no data behind it. It drove the host from 20 GB free to 4.3 GB in roughly an hour. WHY NOW, since the code is old: the soul used to restart constantly (no write-through, divergent graph, 2.11 GB). Stabilising it (neuron #162) let it stay up long enough to accumulate. The fix did not cause this leak; it removed the crashes that were hiding it. Same pattern as the test framework surfacing math_log — the defect was always there, something finally made it visible. Found by Ishikawa rather than by reading the nearest code: method (arena push/pop IS correctly paired per tick), material (node count flat, so not data growth), environment (19 GB/hr / 18,000 ticks = ~1.1 MB per tick, so per-tick not one-shot), machine (an allocator that bypasses the arena) — which is where the evidence pointed. el_strdup tracks into the thread-local arena, which touches no shared state, so taking the single copy under _state_mu is safe and removes the temporary entirely. Verified: self-hosting fixpoint byte-identical; state round-trip correct for hit, miss, and overwrite.	2026-08-16 08:09:32 -05:00
Neuron	e0b2c0ea54	bench: arm the Phase 4 gate -- proven to pass clean AND fire on a quadratic ... Adds tests/native/test_lexer_scaling.el, the regression gate for el #132. Both directions are proven on LIVE workloads, not synthetic series: healthy per-character scan 1821 3251 6007 10422 us -> O(n) PASS rescan-from-zero (the #132 shape) 922 3667 13524 44792 -> O(n^2) FAIL A gate only proven to pass is decoration. The quadratic specimen exists so the gate is proven to FIRE. Also fixes elb_spread_ok to judge the ASYMPTOTIC TAIL (last three ratios) rather than the whole sweep. Measured on a genuinely linear scan the ratios ran 3.37 2.92 1.76 1.65 -- the head looks quadratic because it is cold cache, the tail is the truth. Whole-sweep spread rejected correct data. A complexity bound is an asymptotic claim and must be judged asymptotically. That fix came from the classifier refusing to rubber-stamp my own bad measurement: it reported INDETERMINATE on an unwarmed sweep rather than passing it. Warmup is now taken and discarded at every sweep point. Reverts the == workarounds in test_elbench.el now that el #137 has landed; the natural form generates no str_eq and all 13 fitter tests stay green. The workaround remains -- the Plus arm is still open.	2026-08-15 21:58:46 -05:00
Neuron	6a6b589ba0	bench: real black_box barrier + three-signal growth-curve gate ... Adds el_black_box (inline asm, +r constraint, memory clobber) and runtime/elbench.el: a growth-curve classifier that gates time AND allocation-count AND allocation-bytes, failing if any exceeds its declared curve. Refusal is a first-class verdict. The classifier REFUSES rather than classifying when the largest measurement is below the floor, or when a series is hard-flat across an 8x input range -- the shape produced when the optimiser deletes the work. Reporting O(1) there would be a confident answer with nothing behind it. Disagreeing ratios report INDETERMINATE rather than a guess. Deviation from DESIGN.md 6.2, stated in the source: uses consecutive ratios on a mandated geometric sweep rather than least-squares over candidate curves. Ratios are directly interpretable on a doubling sweep and need no floating point; the cost is weaker O(n) vs O(n log n) separation, reported as an ambiguous band rather than guessed. Documents the counter scope limit: engram_*.c and libcurl malloc are NOT tracked, so a flat curve over engram/HTTP-dominated work is not evidence of anything. 13 tests prove the classifier against real measured series from fitprobe.el -- including that an accumulator's allocation COUNT is linear while its bytes are quadratic, and that el #132's pure-CPU shape reads FLAT on both allocation signals and is caught only by time.	2026-08-15 21:45:13 -05:00
bigmerge	a8908908df	runtime: count container allocations too, not just strings ... el #131 instrumented the four string allocators, which meant list- and map-heavy code reported ZERO allocations — a benchmark over lists would have been fitted against a flat line and passed anything. Caught during framework work: a "linear" specimen read 0 allocs until it was rewritten to allocate strings. A gate is only as good as its blind spots are small, and a signal that silently reads zero is worse than no signal: it produces a confident pass. Now counted at every container allocation — ElList and ElMap bodies, their backing arrays, the copy-on-write clones, and the realloc growth path. Verified on an append loop (n = 100..800): allocs 7, 8, 9, 10 +1 per doubling = O(log n) reallocations bytes 2048, 4096, 8192, 16384 exactly 2x per doubling = O(n) Both curves are what correct amortized growth should look like, and both read zero before this change. Known remaining scope, stated rather than left implicit: these counters cover the runtime's own allocations. They do not see malloc inside engram_*.c or libcurl, which is correct — the gate is for El-level complexity, not for third-party memory behaviour.	2026-08-15 21:36:52 -05:00
bigmerge	edafd8cce8	runtime: math_log is base-10, not natural log ... el_val_t math_log(el_val_t f) { return el_from_float(log(el_to_float(f))); } el_val_t math_ln(el_val_t f) { return el_from_float(log(el_to_float(f))); } Both were natural log, so math_log and math_ln were the same function. log10(100) returned 4.605 instead of 2. Three sources already agreed it should be base-10 and were being contradicted by this one line: - runtime/math.el:55 "// math_log — base-10 logarithm." - el_seed.c:1278 __log_f -> log10() (the path math.el actually calls) - tests/native/test_math.el:133 asserts log10(100) == 2 FOUND BY THE NEW TEST FRAMEWORK ON ITS FIRST RUN (el #133). The assertion had been sitting in the suite the whole time; nothing could report it. The old harness printed "N passed, M failed" with no per-test detail, and half the suites were not compiling at all — so a failing assertion in a suite nobody could run was indistinguishable from no failure. That is the entire argument for the framework, demonstrated on day one: this is not a bug the framework introduced, it is a bug the framework made VISIBLE. Verified: tests/native/test_math.el goes 12/13 -> 13/13, math-log passing.	2026-08-15 21:33:20 -05:00
will.anderson	5e3e69d326	Merge pull request 'test framework phase 1: compile-time registry + El-side runner with per-test timing' (#133) from wt/soul-runtime-reconcile into dev	2026-08-16 02:31:08 +00:00
Neuron	3e7ab07e82	test framework phase 1: forward decls, void-return fix, suite migration ... Completes the Phase 1 runner and migrates the 11 test files onto it. - forward-declare the registry accessors in the test preamble; they are defined at the end of the unit but the El runner is compiled in between - eltest.el: explicit trailing return in the void emit_* helpers, which otherwise lower to 'return println(...)' and fail to compile - test files import runtime/eltest.el explicitly, using the language's own textual import mechanism rather than compiler-side auto-injection - DESIGN.md 6.5: gate on allocation COUNT AND BYTES, not count alone Verified: self-hosting fixpoint byte-identical (gen2 == gen3). 6 of 11 suites run and report per-test timing. The other 5 fail to COMPILE, and fail identically under the committed compiler -- pre-existing breakage this framework makes visible for the first time.	2026-08-15 21:28:30 -05:00
bigmerge	d231b7e5e7	compiler: fix the quadratic — strlen() on every character access ... THE BUG. str_char_code() and str_slice() each called strlen() on every invocation. The lexer walks source one character at a time, so every character access rescanned the whole remaining input: O(n) per character over n characters = O(n^2). el_val_t str_char_code(el_val_t s, el_val_t i) { ... int64_t n = (int64_t)strlen(str); // <- O(n), every call if (idx < 0 || idx >= n) return 0; return str[idx]; } HOW IT WAS FOUND. Not by reading code — by sampling the running process, which is the same method that resolved tonight's engram outage after four wrong theories. A geometric sweep of synthetic sources showed wall-clock rising 3.0x, 3.0x, 4.0x, 4.14x per doubling (converging on 4x = quadratic), and a stack sample put 779 of 779 samples inside lex(), every one bottoming out in _platform_strlen via str_char_code and str_slice. THE FIX. Remember the length instead of recomputing it. The subtlety is INVALIDATION: El strings are arena-allocated, so a freed pointer can be reused for a different string at the same address, and a naive pointer-keyed cache would hand back a stale length and read past the end of the new string — trading a performance bug for a memory-safety one. So entries carry a generation, a hit requires pointer AND generation to match, and every path that frees or mutates a runtime string bumps the generation: el_arena_pop, seed_request_end, __str_set_char. Stale entries cannot be believed; they miss and recompute. MEASURED, same host, same inputs: n(fns) before after 512 0.10s 0.01s 1024 0.37s 0.02s 2048 1.51s 0.03s 50x the compiler's own 422 KB source concatenated (DESIGN.md's 3.58s case): 3.55s -> 0.03s 118x The speedup GROWS with input size, which is the signature of removing a complexity class rather than a constant factor. After the fix each doubling adds ~0.01s: linear. CORRECTNESS, verified rather than assumed: - byte-identical output on every sweep input (n = 128..2048) - byte-identical output on the 422 KB compiler concatenation - byte-identical output on tests/runtime/string_test.el - self-hosting fixpoint byte-identical - new tests/runtime/str_cache_test.el: 17 assertions covering bounds, empty strings, negative indices, slice clamping, distinct strings not sharing a cached length, 1000 interleaved strings forcing cache-slot collisions, and a grown string not reporting its old length. All pass. This is the defect that made dist/soul.c a committed artifact: elc could not run in CI because it needed 24 GB+ and minutes. It needs neither now.	2026-08-15 21:28:23 -05:00
bigmerge	a668062e38	Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile	2026-08-15 21:24:03 -05:00
Neuron	24fac765a6	test framework phase 1: compile-time registry + El-side runner ... Replace the hardcoded test harness main() with a generated static registry and index-based accessors, and move all reporting into runtime/eltest.el. The old harness inlined direct calls into main() and counted assertions in two globals. That shape cannot report which test failed, how long any test took, or whether a test ran at all -- a misspelled registration reported success for a test that never executed. - assertions record into per-test state instead of global counters - registry table emitted at compile time; discovery strictly precedes execution, which is what later enables --list, filtering and sharding - per-test wall timing on CLOCK_MONOTONIC, taken in C around the call - runner in El: structured NDJSON events as source of truth, human output rendered from the same fields	2026-08-15 21:24:03 -05:00
bigmerge	37bcf7eb74	runtime: allocation accounting — the deterministic signal for complexity gating ... Implements the three primitives the test-framework design (DESIGN.md §6.5) requires for gating on growth curves: el_alloc_count, el_alloc_bytes, el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per the project's C-builtin recipe. WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build failure, which means the signal cannot flake. Wall-clock needs warmup, statistics, and a quiet machine; on shared CI it is unusable as a gate. Allocation counts are perfectly deterministic — same input, same number, every machine, every run. Fit them against n and a complexity regression becomes a build failure with zero noise. All four runtime string allocators (el_strdup, el_strbuf, and their _persist variants) funnel every allocation the language performs, so instrumenting there counts everything. WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate. Measured with two El programs, one allocating once per item, one rebuilding its accumulator each iteration: n linear allocs / bytes quadratic allocs / bytes 100 100 / 290 100 / 5,150 200 200 / 690 200 / 20,300 400 400 / 1,490 400 / 80,600 800 800 / 3,090 800 / 321,200 The quadratic program's allocation COUNT is exactly linear — identical to the healthy one. Counting allocations alone would have missed it completely. Bytes catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 -> converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0. That shape — count linear, per-allocation size growing — is the classic accidental quadratic, and it is exactly elc's defect: quadratic allocation VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across both; RSS and wall-clock were just the two ways it surfaced. el_peak_rss is exported for context and is explicitly NOT a gating signal — it is perturbed by allocator internals, the page cache, and the OS. Gate on the deterministic numbers; report the physical one. Counters are unsynchronised by design: this is measurement, and a lock would change the thing being measured. Exact on the single-threaded compile path, approximate under threads.	2026-08-15 21:21:42 -05:00
bigmerge	19cc99e57d	store: judge memory pressure by swap RATE, not swap level ... The guard I added minutes ago checked swap availability as a level (avail < total/8 -> report zero available). That is the wrong signal, and the same host proved it twice within minutes: 47.65 / 48.00 GiB swap used, 2047 swapouts/s -> genuinely thrashing 26.67 / 28.00 GiB swap used, 0 swapouts/s -> healthy, 15.6 GiB free Both are ~97% "used". macOS grows swap files on demand and trims them lazily, so the level says almost nothing about now — it is a high-water mark. The level check calls the second state an emergency and starves the pool for no reason, which is its own failure mode: a guard that fires on healthy machines gets disabled, and then guards nothing. What separates the two is whether pages are moving. So sample the swapout counter across calls and judge the delta: - > 200 pages/s (~3 MiB/s) sustained outward paging => report zero available; callers refuse to grow and pc_relieve_pressure hands frames back. - The first call primes the baseline and reports no pressure. One sample cannot have a rate, and inferring one from a single reading is exactly the mistake this commit removes. Measured thresholds, not guessed: idle sat at 0/s, recovery burst hit 24,845/s while the compressor drained (transient, correctly not a growth decision since growth is only evaluated on eviction passes), and real thrash held ~2000/s. 200/s sits clearly above noise and far below either. The compressor-footprint subtraction stays: that RAM is genuinely spoken for regardless of paging rate.	2026-08-15 21:12:00 -05:00
bigmerge	e52415f0e0	store: bound the pool by AVAILABLE memory and let it shrink ... The adaptive budget I added an hour ago could only grow, and grew toward a share of TOTAL ram (80%, ~38 GiB on a 48 GB host). That is a memory leak with extra steps: total never shrinks when other processes need memory, so the pool had no way to notice it was starving the machine it runs on. Deployed briefly; caught as memory pressure on the host. A control loop with only one direction is not a control loop. - pc_available_ram(): free + inactive + purgeable via host_statistics64 on Darwin, MemAvailable on Linux. Availability is the quantity that moves when the machine is under pressure; total is not. Returns 0 when it cannot be read, and callers then refuse to grow — a cache is never worth swapping the host, so unknown means no. - Growth is bounded by availability minus a free-memory floor (2 GiB default, ENGRAM_POOL_FREE_FLOOR_MB), not by total. The share-of-total ceiling stays as a second bound and drops 80% -> 50%. - pc_relieve_pressure(): the missing direction. On every eviction pass, if available memory is under the floor, hand back ~25% of held frames; the resident set follows on the next pass so the memory is actually returned rather than merely re-labelled. Counted as adapt_shrinks alongside adapt_grows so both directions are visible in the same report. - pc_default_cap() also clamps the STARTING budget to what is spare right now, so a cold boot on a loaded machine does not open at a size the host cannot afford. Verified on a 48 GB host: engram boots in ~30s, RSS settles at 2.22 GiB (the store's actual size, resident, not creeping), 0.0% CPU, 13,439 nodes / 37,670 edges, embeddings complete. Guard reports 9.71 GiB available against a 2.00 GiB floor — 7.71 GiB of headroom it is permitted to use and no more.	2026-08-15 21:02:05 -05:00
bigmerge	e917b3d439	store: make the buffer pool sense its own state and correct from it ... Follow-on to the edge write barrier. That fix removed the full-store walk; this one makes the pool able to notice if anything like it happens again. WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill twice on 2026-08-15 and said nothing. From outside it was indistinguishable from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each costing a deploy or a rollback. The whole time, hits/misses/evictions were already being counted in PgCache, and the struct comment read: /* stats (introspection only — never affect semantics) */ That comment was the bug. Self-measurement treated as decoration is why the pool could not correct itself and why no one outside could see what it was doing. A system that cannot read its own state cannot correct, and neither can anyone watching it. - pc_adapt_budget(): the loop, closed. Over a sliding window, evictions running at a large fraction of accesses WHILE reuse is real means the working set exceeds the budget — so grow it, geometrically, bounded by a LIVE re-read of physical memory. Evictions alone are not pressure (a scan evicts and never returns); evictions with reuse are. An explicit ENGRAM_POOL_FRAMES still wins — an operator override must not be silently overruled. - Budget derived, not declared. A constant cannot be right: 16 GiB of frames is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM at startup" is a guess about the future — it cannot know the store grew or the machine changed. Hence the live re-read. - pc_report(): ONE structured emission carrying the entire sensed state, through emit_log — El's existing telemetry, already exporting to OTLP. Deliberately not a function per stat, and deliberately not a bespoke /api/pool endpoint: both make observability something hand-written per noun instead of the uniform mechanism every component already has. - engram_pool_stats_json(): the same state readable live, wired through the normal builtin path (codegen arity + el_seed wrapper), so the pool can be observed in real time rather than reconstructed afterward from a stack sample. Verified: with the exact configuration that took production down (ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete — and NO pressure event fires, because the barrier removed the walk that caused it. The controller is defense in depth; the barrier is the fix.	2026-08-15 20:44:23 -05:00
bigmerge	777ccc02f0	store: extend the durable-hash write barrier to edges (kills the full-store walk) ... Checkpointing pushes the ENTIRE resident graph through store_put_node and store_put_edge (see engram_store_checkpoint). Nodes were cheap: a durable-hash compare skipped unchanged records with zero page I/O. Edges had no barrier at all — struct comment at PgCache.barrier_on even says "node durable-hash barrier" — so every edge was rewritten on every checkpoint, and each rewrite runs the idempotency probe max_page_lsn_for_id -> btree lookup -> page_read. Edges outnumber nodes ~3:1 here (37,663 vs 13,436), so routine checkpointing degenerated into a FULL-STORE WALK in id order: random page access across the whole 2 GiB store, repeated, overwhelmingly to rediscover nothing had changed. LRU is worst-case under exactly that pattern — it evicts the page it is about to want — so once the page cache was smaller than the store, the walk collapsed into thrashing: 100% CPU, flat RSS, no forward progress, port never bound. That took the live engram down twice on 2026-08-15. The walk is the defect. Sizing the cache to survive it treats the symptom. Changes: - dh_edge_hash(): edge counterpart of dh_node_hash, with a kind discriminator byte so an edge can never collide with a node of the same id in the shared map. created_at/updated_at/last_fired are excluded deliberately: last_fired is touched by activation without changing what the edge IS, and folding it in would defeat the barrier on precisely the hot edges that most need it. - store_put_edge(): barrier check + dh_set on success, mirroring store_put_node exactly. - store_scan_edges(): seed the barrier map from on-disk truth at load, so the FIRST post-boot checkpoint already skips unchanged edges. store_scan_nodes already did this and its comment says why; edges were simply never done. Verified: with the exact configuration that killed production (ENGRAM_POOL_FRAMES=65536 -> 1 GiB cache against a 2 GiB store), the engram now boots clean and serves — LISTENING, 13,436 nodes / 37,663 edges, embeddings complete, 0.0% CPU, RSS 1.14 GiB (cache resting at its budget rather than thrashing against it). Same small cache, same store, no walk.	2026-08-15 20:38:11 -05:00
bigmerge	4e24d7d3f1	runtime: engram_edges_json — read edges without a whole-graph file round trip ... /api/graph/edges answered a read query by calling engram_save() to serialize the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in one line, and both bit production on 2026-08-15: 1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram server's CANONICAL store. A READ route overwriting the persistence owner's canonical file. This defect had been fixed once (export moved to a scratch path); it came back when the hand-written dispatch block was replaced by @route dispatch and the unfixed copy is the one that survived the merge. 2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole graph, per request, to return a bounded slice. Calling it tonight overwrote the canonical snapshot and immediately preceded an engram crash loop. engram_edges_json(limit, offset) is the builtin that route's own TODO asked for ("Future: add an engram_edges_json() builtin and drop the file round trip entirely"). It walks g->edges directly and emits every persisted field. limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell over, and an unbounded default would preserve the failure mode under a new name. Callers page explicitly. Registered in codegen.el's builtin_arity (both plain and __ spellings) and wrapped in el_seed.c per the project's C-builtin recipe.	2026-08-15 20:10:48 -05:00
bigmerge	7351fb0a8d	runtime: restore engram_recall_json + cgi_* accessors ... neuron's soul calls engram_recall_json (neuron-api.el:618, memory.el:80) and cgi_principal (studio.el:72). Both existed in the runtime neuron vendored (v1.0.0-20260501) and were absent here, so the soul could not link against current el at all. The dangerous part is what the obvious "fix" would have done. These look like redundant wrappers over one impl: engram_search_json(q, limit) -> eg_search_json_impl(q, limit, 0) LEXICAL engram_recall_json(q, limit) -> eg_search_json_impl(q, limit, 1) SEMANTIC They are not interchangeable, and the split is documented at neuron-api.el:613: search stays LEXICAL because ~40 internal call sites pass a KEY and seven of them DELETE every record returned. Point those at a semantic matcher and they delete fuzzy matches. Conversely, pointing recall at search silently downgrades the mind's entire retrieval surface from semantic to lexical — no error, just permanently worse recall. Implemented over engram_activate(), which in this runtime already IS the semantic path the old with_legs=1 branch built by hand (embeds the query via eg_embed_fetch, scores by cosine, then spreads activation one hop). Output shape matches engram_search_json — a flat array via engram_emit_node_json — because callers parse search's shape, not activate's envelope. Verified: neuron's soul now compiles and links against current el, boots, and serves /health with layers initialized. NOTE for follow-up: current el also ships engram_retrieve_geometric_json, a structure-first retrieval that appears to be the intended successor to recall. Repointing the two recall call sites at it may well be the right end state and would remove the two-wrapper shape entirely — but that is a behavioral change that must be measured against neuron/tools/retrieval-eval/'s gold set, not assumed. This commit preserves existing behavior exactly; it does not decide that question.	2026-08-15 19:56:43 -05:00
bigmerge	598915cc61	runtime: restore the three builtins that made elc unrebuildable ... The committed elc binary could not be refreshed from its own source. Rebuilding failed with three implicit-declaration errors: el_mem_check, stdout_to_file, stdout_restore. The compiler's own source calls all three (compiler.el:472,479,574 and codegen.el:4248) and two are registered in codegen.el's builtin_arity table — but none were defined in this runtime. They were found intact in ui/examples/native-hello-ios/NativeHello/el_runtime.c, a divergent private copy of this runtime that still carried them. Ported verbatim. Consequence of them being missing: the canonical elc binary was frozen. Source gained @route dispatch codegen (emit_route_dispatch, codegen.el:3948) and the @manager boundary-beat seam, but no rebuilt binary could carry them, so neuron's soul — whose routes.el now calls the compiler-synthesized el_route_dispatch — could not be built at all. Verified after the fix: - elc rebuilds from current source, clean. - Self-hosting fixpoint byte-identical (stage3 == stage2). - The rebuilt elc emits el_route_dispatch (2 occurrences in the soul amalgam, previously 0) and injects engram_boundary_beat at @manager boundaries, i.e. the decorator seam is live rather than inert. el_mem_check is itself the compiler's memory guard (ELC_MAX_MEM_MB, default 512MB, self-terminates before the OS OOM-killer fires) — so the runtime was missing the very guard that would have surfaced the compiler's memory blowup as a clean error instead of a 27GB host-killer.	2026-08-15 19:50:17 -05:00
bigmerge	40eb48e92f	engram: fix silently-wrong query params, and make el_seed.o + el_runtime.o link ... Three real bugs, all found by actually running the thing rather than reading it. 1. query_param never URL-decoded. A GET of /api/search?q=neural%20network searched for the literal string "neural%20network" and returned []. Every multi-word search against the live engram has been silently returning empty results — not an error, an empty result, which is why it went unnoticed. Affects every GET route that reads query params, not just search. 2. query_param matched key names unanchored. str_index_of(qs, "q=") matches inside "faq=", so "?faq=X&q=Y" returned X for key "q". Verified live before the fix. Now searches for "&key=" against "&"+querystring so a match can only land on a real parameter boundary. 3. el_request_start/el_request_end were defined in BOTH el_seed.c and el_runtime.c, so linking the two objects together — which is exactly what the product build does — failed with duplicate symbols. el_seed.c's own comment already says these moved there ("formerly defined in el_runtime.c. Now self-contained in el_seed.c"); the el_runtime.c copies were left behind during that move. Removed them, kept declarations since http_worker calls them. Also added the three missing prototypes (engram_op_assert_json, engram_node_full_in, engram_connect_in) that el_seed.c wraps but never declared, which made it fail to compile standalone under C99+. Verified: engram builds and links clean from canonical source; before/after comparison on a copy of the real store shows "neural network" returning a real match where the live build returns [], and "?faq=WRONG&q=MetaColloc" now resolving to MetaColloc. Live engram on :8742 was never touched.	2026-08-15 19:34:04 -05:00
bigmerge	09dade0613	Merge remote-tracking branch 'origin/pr/103' into HEAD ... # Conflicts: # lang/AGENTS.md # lang/runtime/el_runtime.h	2026-08-15 18:26:57 -05:00
will.anderson	38a8e32d6c	Merge pull request 'engram: batch-cosine Adapter/Strategy/Factory over ggml (supersedes #114)' (#116) from feat/engram-ggml-cosine-batch into dev	2026-08-15 23:22:56 +00:00
bigmerge	9883aa7564	Merge remote-tracking branch 'origin/wt/swarm-ccr' into merge-swarm-ccr-v2 ... # Conflicts: # lang/runtime/el_runtime.c # lang/runtime/el_runtime.h	2026-08-15 18:17:33 -05:00
bigmerge	c008b7228a	engram: make the ggml batch-cosine strategy actually compute in fp32 ... #116 shipped the ggml strategy at 0.9933 id-recall against the CPU oracle while the hand-rolled Metal kernel it replaced scored 0.9997 — a ~150x worse error margin. That was not an inherent property of ggml. It was a usage bug in this file, and this commit fixes it. ggml-metal has two F32xF32 matmul kernels and picks between them purely on ne11, the number of B rows, which for us is the query-batch size: ne11 <= 8 -> kernel_mul_mv_ext_f32_f32_* / kernel_mul_mv_f32_f32_*, templated <float, float> — genuine F32. ne11 > 8 -> kernel_mul_mm_f32_f32, templated <half, half4x4, simdgroup_half8x8, half, half2x4, ...> — BOTH operands narrowed to F16, despite F32 tensors on both sides. The old code issued one ggml_mul_mat with ne11 = nq (300 in the benchmark), landing squarely on the F16 path. The file's own header comment asserted the opposite ("computes in F32 on the Metal backend"); that claim was wrong and is replaced with the measurement. Fix: emit ceil(nq/8) mul_mats over ne11<=8 ggml_view_2d slices of one query tensor, all expanded into ONE graph and one ggml_backend_graph_compute, so the node matrix is still uploaded and shared exactly once. EL_GGML_MULMAT_CHUNK overrides the 8; setting it >= nq reproduces the old behaviour exactly, which is also how the before/after below was measured in a single binary. Measured, real store snapshot, 13415 live embedded nodes, dim=768, 300 real queries, vs the CPU double-accumulated oracle (vindex_bench, offline copy of the store — no live service touched): id-recall same-rank |Δdist| max mean old (ne11=300) 0.9933 6.80e-05 1.43e-05 new (ne11<=8) 0.9987 4.77e-07 9.30e-08 hand-rolled 0.9997 3.58e-07 7.55e-08 ~145x better max error, ~154x better mean — now the same order of magnitude as the hand-rolled kernel rather than 150x off it. The cost is real and is documented rather than buried. Median of 15 reps of the whole batch_multi() call, three runs: 13.2-14.4ms unchunked, 19.9-20.2ms chunked, 17.7-18.0ms hand-rolled. Correctness costs ~+6.7ms per 300-query batch and leaves ggml ~12% behind the hand-rolled kernel instead of ~35% ahead. It cannot be recovered inside ggml: an fp32 matmul on Metal must re-stream the node matrix once per <=8 queries, and ggml's Metal backend ships no fp32 TILED matmul, so "fast" and "fp32" are genuinely exclusive there. Two things that did NOT work, recorded so nobody retries them: - ggml_mul_mat_set_prec(t, GGML_PREC_F32) does nothing here. Error was bit-identical with and without it (1.038e-05 either way) — ggml-metal has no F32-accumulating mul_mm kernel to switch to. ne11 is the only lever. - The ACCEL/BLAS device looked excellent in an isolated compute-only probe (3.4-4.0ms, mean |Δdot| 1.5e-08) but is dominated on BOTH axes end-to-end (0.191 ms/query at 0.9973 recall vs 0.125-0.142 at 0.9987), because the probe was not competing for the same CPU cores the real call path is. It stays reachable via EL_GGML_DEVICE as a no-Metal fallback, labelled as measured-and-rejected, not as a recommendation. Also corrected: the ~7.8s "cold start" blamed on this file is not this file re-initialising per call — init was already cached. It is Apple's shader cache missing on ggml's embedded metallib (~650 kernels), keyed on the library and shared across processes: the first load on a machine reports "loaded in 7.670 sec", the next run of a *different* binary reports 0.009 sec. Once per machine per ggml version, not once per process, and not ours to fix. Warm ggml init is 44-53ms vs 36-117ms for the hand-rolled strategy. Loading only libggml-metal.so instead of every plugin in the directory is kept for tidiness, and explicitly documented as NOT a speedup: 44.7-52.4ms against 46.9-58.9ms, the same number inside noise. The -2.0 sentinel contract is unchanged and re-verified at batch sizes that straddle the chunk boundary (1,7,8,9,16,17,33), plus NULL rows, dim mismatches, zero-norm rows, and an all-invalid population. Notably the old ne11=300 path fails that same check at a 2e-6 cosine tolerance with 2299 mismatches, which is an independent confirmation of the defect.	2026-08-15 17:57:09 -05:00
bigmerge	3718bf0380	runtime: port missing __channel_* primitives into el_seed.c ... runtime/channel.el has always called __channel_new/__channel_send/ __channel_recv/__channel_try_recv/__channel_close, but these were only ever implemented in the pre-restructure lang/el-compiler/runtime/el_runtime.c. When the canonical runtime was consolidated onto the release copy (lang/runtime/el_runtime.c) and el_seed.c became the sole C dependency, the channel implementation was never carried forward — __mutex_new made the move, __channel_* did not. Any El program using Go-style channels currently fails to link on dev. Ported the working buffered-MPMC-channel implementation (mutex+condvar+ circular buffer, bounded and unbounded modes) from the old el_runtime.c verbatim, adapted only to el_seed.c's arena API (seed_arena_track in place of el_arena_track). Declared in el_seed.h alongside the existing mutex primitives.	2026-08-15 17:50:04 -05:00
bigmerge	9d40f87926	ingest: unify transduce_prose/transduce_structured into one transduce() ... transduce() is now THE single mechanism: one function, no content-type branch inside it. It never asks whether `source` is prose, JSON, or raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally: split on "\n\n" as a universal boundary-marker check, and if that finds no boundary, fall back to fixed 4096-char windows. Same node/edge wiring (root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets a heading/section_of link) regardless of what's inside a chunk. Dedup is the existing find_existing_by_content path via merge_manifold, applied uniformly. The old transduce_structured JSON dataset/records/feature-node interpretation is deleted outright, not just unused — a JSON file now gets chunked and deduped like anything else, with no pre-computed structure. All five ingest_* entry points still exist unchanged in name and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one transduce() (ingest_stream builds its own turn-nodes directly and never called either old function, so it's untouched). This unlocks raw/opaque content (audio, or anything else with no natural text/JSON shape) without any DSP, LLM call, or external API: transduce() chunks it exactly like it chunks anything else. There is zero semantic understanding of audio (or any payload) claimed or built here — any meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time. Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk) because El strings are NUL-unsafe under strlen-based ops and fs_read()'s result silently truncates at the first embedded NUL, which is routine in real binary/audio bytes. ingest_file compares fs_read()'s string length against a real fs_size() stat() count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim, no invention), joined with the same "\n\n" marker transduce()'s boundary scan already looks for. This is a mechanical fidelity fix, not interpretation of content — transduce() never learns a fallback happened. Registered both builtins' arity in codegen.el; did not rebuild the elc compiler binary itself (unrelated, pre-existing gap: self-hosting elc via el_seed.c fails on this worktree independent of this change, reproduced with codegen.el reverted) — the existing elc binary compiles calls to unregistered builtins via its already-existing arity=-1 passthrough, confirmed by an actual clean `elc ingest.el` + `cc` build against the modified el_runtime.c. INGEST_KIND keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a content-type flag; the redundant "structured" value (an alias for "file" that hinted the now-deleted JSON branch) is removed. ingest_dir drops its file-extension filter for the same reason: transduce() takes anything now. Verification: local manifold construction confirmed correct against a real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte real prefix slice) — exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes. Compiles clean via the real elc + the modified el_runtime.c/engram_*.c (built and booted an actual sandbox engram off this exact source with `nsbx create --branch`). NOT verified this session, disclosed rather than papered over: end-to-end server-confirmed persistence (a real before/after /api/stats delta, and a fetched node by id) for the audio, prose, and JSON-fixture cases. Every local nsbx sandbox engram tried tonight (two stock pre-#109 binaries hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW binary built from current dev) took minutes-to indefinitely long on the final /api/load-merge write's embedding step and hit the client's 60s HTTP timeout before responding, even for a 5-node write. This is confirmed as real (if slow) forward progress, not a hang: the sandbox's WAL file was observed growing steadily across every attempt. The code's own pre-existing HONESTY GATE correctly refused to report success in every case, returning "load-merge failed: ..." with a "nothing below this manifold was confirmed persisted by the server" note instead — exactly as designed. This is an environment/infrastructure limitation, not a defect introduced by this change: the engram server binary itself is untouched by this commit.	2026-08-15 17:50:04 -05:00
bigmerge	90d3f0bc76	engram: port PR #105's 3 genuine wins onto dev's existing cosq/e_eff semantic layer ... Reconciles PR #105 ("fix: engram search latency — pin embed model, cache query embeddings, bound activate BFS") with dev's ACTUAL current engram_activate, rather than the ancient pre-restructure snapshot #105 was built against. WHY THIS NEEDED RECONCILIATION, NOT A DIRECT PORT: #105's single commit (1dc49b1) modifies `lang/el-compiler/runtime/el_runtime.c` — a path that does not exist on dev (dev has `lang/runtime/el_runtime.c`; the restructure that renamed it happened after #105's branch point, which traces to a July 22 merge-base, weeks before the M8/M8.1/qgate/fan-effect/adjacency-index work this file has grown since). #105's own engram_activate is consequently the PRE-restructure version: no adjacency index (O(E) full edge scan per hop), no query-aware qgate, no ACT-R fan effect, no eg_edge_eff_weight, and no awareness of dev's cosq/e_eff embedding-blend semantic layer — it built a parallel `g_qcache`/`engram_embed_raw` mechanism from scratch against code that no longer exists at that path. A raw merge/cherry-pick was not possible and would have been wrong even if it were: taking #105's tree wholesale would have thrown away everything dev grew in the meantime (qgate, fan effect, adjacency index, and this session's own M8 HNSW vindex integration). RECONCILIATION: kept dev's cosq/e_eff mechanism as the semantic layer entirely intact (unchanged by this commit) and ported #105's three genuinely additive wins on TOP of it, at their equivalent sites in the CURRENT eg_embed_fetch/engram_activate: 1. keep_alive:-1 on the Ollama embed request body (eg_embed_fetch) — pins the embed model resident so a larger generation model loading under unified-memory pressure can't evict it and force a cold reload on the next search (#105 measured ~2.2s cold vs ~0.02-0.05s warm). 2. Query-embedding cache upgraded from dev's single-slot (`_eg_qcache_text`, only ever remembered the LAST query) to a direct-mapped, FNV-1a-keyed, 1024-slot cache (reusing the existing engram_id_hash) — so the curiosity loop's rotating phrases actually hit the cache instead of evicting each other every call. Same "pointer owned by the cache, not freed by caller" contract as before, just per-slot instead of global. 3. Beam cap on the layer-1 spreading-activation BFS (new engram_activate_beam(), tunable via ENGRAM_ACTIVATE_BEAM, default 128). The FIFO frontier is processed in hop-level batches (entries sharing .hops are provably contiguous — see the code comment); when a level exceeds the beam width, only the top-`beam` by activation actually EXPAND. Every node in an oversized level still gets reached[]/best_bg[] recorded (that happens at enqueue time, one level up) and appears in the reported/promoted set — the cap bounds associative SPREAD width only, never recall of what was already found. Kept as a genuine additional bound even though the adjacency index + qgate + fan effect already mitigate #105's original "hub-node explosion" failure mode for a different reason: those prune WHICH targets matter; this bounds worst-case width regardless. Everything else in dev's engram_activate — cosq/e_eff, the qgate rescale, the fan effect, eg_edge_eff_weight, the M8 HNSW vindex seed discovery from the #109 reconciliation earlier this session — is untouched. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): cc -std=c11 -O2 clean build; booted in an isolated sandbox against a real cloned production snapshot (13,424 nodes / 37,656 edges); ran 5 activate() calls across rotating queries at depth 3, including the same query issued twice non-consecutively (2nd hit landed at 476ms vs the 1st at 483ms — consistent with a cache hit once Ollama's own warm-model latency is accounted for; no crash, correct varied result counts (367-2610 nodes) each call; act-stats JSON read correctly throughout. Built on top of the M8/#109 reconciliation (bacaf3d, merged to dev as #109) — dev's current HEAD at the time of this commit.	2026-08-15 17:50:04 -05:00
bigmerge	2d0aef4ef8	lang: declare the el_runtime.c symbols el_seed.c's wrappers call ... runtime/el_seed.c does not compile standalone via the exact command tools/install.sh uses (`cc -std=c11 -O2 -I runtime -c runtime/el_seed.c`): 51 of its __-prefixed wrapper functions (http serving, JSON access, key-val state, URL/HTML escaping, and the whole engram_* node/edge/layer/search surface) call unprefixed counterparts that are implemented in el_runtime.c, not in el_seed.c itself, and el_seed.c never declared them -- a toolchain that treats an implicit function declaration as a hard error under C11 fails the compile outright. install.sh already compiles el_seed.c and el_runtime.c as separate objects and archives both into libel.a, so the symbols are always present at link time; el_seed.c alone was just missing the prototypes. A plain `#include "el_runtime.h"` was tried first and rejected: it redefines el_to_float/el_from_float, which el_seed.h already provides -- a real compile error, not a style preference. Added narrow prototypes instead, copied verbatim from el_runtime.h, for exactly the 51 symbols el_seed.c's wrappers reference and nothing else. Verified clean: - `cc -std=c11 -O2 -I runtime -c runtime/el_seed.c` (install.sh's exact per-file compile) -- 0 errors, 0 warnings, even with -ferror-limit=0. - full `tools/install.sh` run -- compiles both objects and archives them into libel.a successfully. Separately (not fixed here, out of scope): AGENTS.md's documented compiler self-rebuild command links elc-new.c against el_seed.c, but elc-new.c's own generated `#include "el_runtime.h"` line and 3 undeclared symbols (el_mem_check, stdout_to_file, stdout_restore -- present in neither el_runtime.c nor el_seed.c) mean that command fails regardless of which runtime file it's linked against; and install.sh's libel.a only archives el_seed.o + el_runtime.o, so any program that calls into the engram_* surface fails to link against it (el_runtime.c's engram_* wrappers need engram_store.c/engram_geometry.c/engram_reason.c/engram_cognition.c/ engram_vindex.c, none of which install.sh compiles in). Both are real, pre-existing, and independent of this fix -- worth their own look.	2026-08-15 17:32:58 -05:00
bigmerge	b3f410fc91	engram: batch-cosine Adapter/Strategy/Factory over ggml, supersedes hand-rolled PR #114 ... Stop hand-rolling GPU kernels for batch cosine similarity — use ggml (the MIT-licensed compute library underneath llama.cpp, installed standalone via Homebrew) as the preferred backend, without ripping out PR #114's carefully-verified hand-rolled Metal shader. Structure: one stable public adapter (eg_cosine_batch.h, zero #ifdef at call sites) backed by three selectable concrete Strategies behind an internal vtable (eg_cosine_batch_strategy.h) chosen by a Factory (eg_cosine_batch.c): - eg_cosine_batch_strategy_ggml.c — NEW. ggml + dynamically-loaded Metal backend plugin (ggml_backend_load_all_from_path + ggml_mul_mat for the batched dot product), gather/scatter around the -2.0 sentinel contract. - eg_cosine_batch_strategy_metal_hand.m — PR #114's original hand-rolled Metal shader bridge, preserved almost verbatim, now one strategy among several rather than the only option. eg_cosine_batch.metal kept byte-identical to the original. - eg_cosine_batch_strategy_cpu.c — universal always-false fallback (direct descendant of PR #114's eg_metal_cosine_stub.c). Selection: EL_COSINE_BATCH_STRATEGY=ggml|metal|cpu|auto (default: ggml first, then hand-rolled Metal, then CPU — first available wins), plus back-compat EL_METAL_COSINE=0 to disable every GPU-backed strategy. build_vindex_bench.sh compiles all three strategies on Darwin, CPU-fallback-only elsewhere. vindex_bench.c now reports BRUTE-GGML and BRUTE-METAL side by side against the same CPU oracle, on the same dataset, in one run (real numbers vs. real store snapshot in the PR body).	2026-08-15 17:16:41 -05:00
bigmerge	bacaf3d39c	engram: reconcile M8 HNSW vindex (#109) onto current dev, restore 3 fixes the branch predated ... Lands feat/reframe-region-setop (PR #109: native set-based reframe_region, decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's actual current HEAD, plus engram-tiered-storage's still-unique test suite. RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/ verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/ select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two branches; #109's copy is a strict superset (adds vindex_harvest_from_store, used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and engram_verify.c are also byte-identical. #109's own branch point already carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing left to merge into #109 for those files. The one thing engram-tiered-storage had that #109's tree dropped: its full test suite (test_vindex.c, test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh harnesses) — ported over here unchanged. WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch forked from dev on 2026-08-14 15:40 (before restructure-adjacent history diverged the file's merge-base for `git merge` — it presented as an add/add conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but it silently reverted THREE dev fixes landed on 2026-08-14/15, after the branch point, that the PR's diff had no way to know about: 1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of the query-aware propagation gate dropped the shift-and-floor rescale about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate 0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi] is computed) is orthogonal to WHAT it gates on and both are kept. 2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global). PR's tree predates this and dropped all three counters + their JSON stats fields. Restored declarations, all 4 direct increment sites, the eg_wm_carry_over bll increment, and the act-stats JSON fields -- alongside (not instead of) the PR's own P4 afferent / API-reshape counters already in that same struct/JSON. 3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev selects the STRONGEST qualifying candidate for consolidation each call; PR's tree predates this and reverted to hash-slot order (arbitrary wrt association strength) for edge formation -- the one path that writes PERMANENT structure. Restored the strongest-candidate while-loop, keeping the PR's own genuine improvement at that site (engram_adj_on_edge_added incremental-index append instead of a bare adj_dirty=1 full-rebuild flag). engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env flags, /api/nodes connected-count in responses) were pure additive: dev's side was empty, PR's side added the feature. Took PR's side whole. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): - cc -std=c11 -O2, clean link against the real engram/src/server.el via elc, zero errors. - vindex_bench (built standalone, read-only harvest) against the real production store clone (13,671 embedded nodes, 768-dim nomic-embed-text): recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs 2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s for the full 13,671-node set -- see the flagged risk below. - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned snapshot of the live store, 13,424 nodes / 37,656 edges) and called /api/activate for real: first call after boot 41.5s (pays the one-time HNSW build inline -- matches the standalone bench), second/third calls 356ms/605ms, no crash, correct results, act-stats JSON (including the restored evict_floor/cap/bll fields) reads correctly. KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync builds the HNSW index synchronously, inline, on the first engram_activate() call after every process start (or index invalidation). On the real node count that is a ~46s blocking stall on a single-threaded server -- the first request after every restart (or its concurrent siblings) waits the full build. Recommend a background/incremental build (or a bounded per-call build budget) before this ever reaches the live daemon. See PR description / final report for the fuller writeup.	2026-08-15 16:46:44 -05:00
will.anderson	e34ebd4b3d	Merge pull request 'nsbx + cognitive architecture design + engram self-review series' (#113) from feat/neuron-sandbox into dev	2026-08-15 19:59:07 +00:00
will.anderson	d4a04bb944	Merge pull request 'swarm: native interruptibility for dispatched agent workers' (#107) from worktree-agent-a6177cda24c71d1df into dev	2026-08-15 19:57:10 +00:00
will.anderson	09ae14a970	Merge pull request 'fix: float arithmetic codegen (segfault/garbage) and math_log aliasing' (#104) from worktree-agent-a456e0cf8cd2ee361 into dev	2026-08-15 19:54:46 +00:00
will.anderson	fdf0d6cb64	Merge pull request 'nsbx: one-command dev onboarding (branch + worktree + isolated engram)' (#99) from feat/nsbx-dev-env into dev	2026-08-15 19:52:35 +00:00
bigmerge	08cbcef5d9	engram: fix lazy-embed index gap (#20) and make activate's cosine scan lazy; extract vindex harvest primitive with a bench/oracle harness ... Adds an O(1) "seen" bitmap so lazily-embedded older nodes get picked up incrementally instead of only on a full rebuild (embed-gap #20). Replaces engram_activate's O(N*D) cosine prescan with a lazy-memoized cosine cache (eg_cosq_at), proven bit-identical to the old path. Extracts a clean vindex_harvest_from_store primitive (read-only vector harvest, careful malloc/ownership/error-path handling) reused by both index-build and the new vindex_bench.c — a read-only proof harness comparing brute-force vs HNSW recall/latency on both the real store and synthetic data. .nsbx-env intentionally excluded — local sandbox config (ports, paths, dev-only placeholder key), not checked in.	2026-08-15 14:26:16 -05:00
bigmerge	708722b7ff	Add native interruptibility for dispatched agent workers ... Cancellation-token control channel checked at every step boundary lets a coordinator PAUSE/RESUME/REDIRECT/KILL a running worker mid-task instead of waiting for the whole (possibly wrong) plan to finish. Bounded purviews mean no half-committed state to unwind on interrupt. Includes a proof harness (proof.el, run.sh) comparing a broken non-interruptible worker against the new one under identical kill/redirect/pause timing. Distinct from the already-preserved swarm-ccr orchestrator (fan-out/ converge dispatch): this is single-worker interruptibility, a complementary mechanism, not a duplicate.	2026-08-15 14:25:55 -05:00
bigmerge	01826421c4	seam: implement decorated-fn boundary auto-emit; prove on clone ... Will waived diff review -> build it for real. Add engram_boundary_beat() to the runtime (afferent counter++ + engram_chrono_tick + engram_strengthen(self-anchor) + dharma_emit) and two act-stats counters (aff_boundary_ops, dharma_emits). codegen cg_fn injects ONE engram_boundary_beat(op) at the entry of every @manager/@accessor fn (fn_has_decorator, so it fires under @route @manager too) — a decorated op self-reports with ZERO hand-written instrumentation. Rebuilt elc self-host + the cognition engram in the worktree; ran it as the clone daemon on :8900. Proof (/api/boundary-proof, @manager, empty body, 5x): aff_boundary_ops 0->5, dharma_emits 0->5, self activation_count 1510->1513, chrono stamp advanced. Brought in feat/cognitive-architecture engram runtime+server for the build. strengthen = activation bump (not content/edge write) -> identity protection intact. Live :8742 untouched; no push, no cutover.	2026-08-14 21:20:18 -05:00
will.anderson	7aa847e32a	Merge origin/dev into engram-tiered-storage ... Resolve 3 conflicts: - lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork; single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh). - lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated artifact folder, not a source path; a release is a git tag, not a folder). - lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL. Nothing in dev's build consumes the deprecated fork or releases/ folder.	2026-08-12 15:23:02 -05:00
will.anderson	bb64a236ed	engram tiered storage: engram-service wiring + elc fold-hang fix + prune-store mirror ... - Wire paged store into the ENGRAM SERVICE (server.el, the authoritative durable owner): boot->engram_store_boot, persist_canonical->engram_store_checkpoint, gated by ENGRAM_STORE. - elc (lang/elc.c + src/parser.el + codegen.el + elc-combined.el): OOB guard in tok_kind/tok_value + parse_block progress backstop — fixes the pre-existing unbounded-memory fold hang on sessions.el. - engram_prune_telemetry mirrors ISE prune to the store (store_forget) so store live-count tracks resident and stale telemetry stays bounded. - Deployed live 2026-08-12: engram :8742 on neuron.egm+WAL, count reconciled 11552.	2026-08-12 14:14:20 -05:00
will.anderson	9a0266cbf9	engram tiered storage M3.5: persist activation field updates (pre-flip gate) ... Flag-on checkpoint now full-walks the resident graph: store_put_node (WM weight, activation_count, last_activated, wm_anchor) + store_put_edge (hebb, last_fired) for every node/edge, then engram_checkpoint. Uses store_put_edge (idempotent upsert) not store_hebb_batch, because activation FORMS new hebbian-associate edges that bypass the create hook and delta-only hebb_batch can't create them. Store-on boot now applies the same WM-halving + floor + cap transforms as engram_load. This is the hebb-survives-restart fix. Gate: reboot from neuron.egm with snapshot.json deleted -> edge hebb + activation_count survive unchanged, WM weight survives with identical boot transform; negative control proves persist is load-bearing (hebb->0 without it). M1 33/33 + M2 36/36 + M3 parity PASS, ASan/UBSan clean, flag-off untouched. Engine unchanged (boundary held).	2026-08-11 23:37:52 -05:00
will.anderson	a72145b44e	engram tiered storage M3: wire store behind ENGRAM_STORE (default off) + .egm rename ... Caller-side shim in el_runtime.c maps EngramNode/Edge <-> StoreNode/Edge; engine keeps zero soul deps (libengram boundary, design §10). Flag off = today's JSON path byte-for-byte (proven: no neuron.egm created, graph identical). Flag on = engram_open (import snapshot.json once into neuron.egm, else WAL-replay) + resident load; node/edge create + forget dual-write via guarded hooks. Files renamed engram.store->neuron.egm, engram.wal->neuron.wal. Gate: M3 parity PASS (graph on==off byte-exact modulo ordering; snapshot round-trip; reboot-from-egm with snapshot.json deleted; activation set+sequence identical; ASan/UBSan clean). M1 33/33 + M2 36/36 green post-rename. Known gap (pre-flip): in-place hebb/WM/activation_count updates during activation are not yet persisted to the store (create/connect/forget are). Must close before live flip so learned edges survive restart.	2026-08-11 23:21:21 -05:00
will.anderson	8affb1d6e0	engram tiered storage M2: WAL + checkpoint + crash recovery + legacy import ... Write-back no-steal buffer pool makes the fsync'd WAL load-bearing (M1 was write-through). Logical WAL with record-granularity page-LSN redo idempotency. Checkpoint = flush dirty pages, fsync store, advance last_checkpoint_lsn, reclaim WAL prefix. One-time snapshot.json import only when store absent; JSON never read as the ongoing store thereafter. Gates: 33/33 M1 (no regression) + 36/36 M2 — replay parity, torn-tail fuzz (every byte offset), checkpoint-crash at all 5 phases, torn-page+WAL redo, legacy-import parity, hebb-survives-crash.	2026-08-11 23:00:20 -05:00
will.anderson	fa47b98d18	engram tiered storage M1: on-disk paged store format + round-trip tests ... Self-contained paged store (lang/runtime/engram_store.{c,h}): 16KiB slotted pages, u32 TLV self-describing records (forward-compatible), overflow chains, B+-tree id-index + from/to adjacency, page free-list, tombstones, double superblock + crc recovery. Not yet wired to activation (M3). 33/33 tests pass (ASan/UBSan clean); 5k nodes/20k edges round-trip bit-exact incl 768xf32 emb + hebb; store 25MB vs 64MB JSON. Format is final — see design §2.4.	2026-08-11 22:26:05 -05:00
will.anderson	0a72fced28	engram: WAL persistence + integrity hardening + single canonical runtime ... Establish lang/runtime/ as the ONE canonical el runtime (from the active runtime that carries hebb/emb persistence + the new WAL); repoint the el CI publish, engram build, elb default, and in-repo build scripts to it; delete the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh drift guard. Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian edge weights — learned co-activation was wiped on every restart. Publishing from canonical ships the stranded 'learning that cannot outlive the process' fix. WAL storage engine + integrity fixes (DELETE->tombstone + store-layer protection, safe data-dir default) ride in behind ENGRAM_WAL (default off = byte-identical to today). Verified: engram elb per-module build clean, WAL gate 66/66, native smoke ok, drift-guard green.	2026-08-11 21:31:37 -05:00
Tim Lingo	c2afcbddf5	fix(engram): allow SessionSummary node_type in validation allowlist ... handle_api_consolidate writes a "SessionSummary" node, but engram_valid_node_type omitted it — so once this validation ships, every consolidate() would be silently REJECTED at the engram boundary. Add SessionSummary to the allowlist. Found in Will's PR review of neuron #1 / el #52. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-10 06:26:25 -05:00
Tim Lingo	dfe4e83ed1	Fix engram_node_full wrapper field corruption + add node_type/tier validation ... The wrapper signature was stale and didn't match the C primitive __engram_node_full(content, node_type, label, salience, importance, confidence, tier, tags). Because el_val_t is an untyped machine word, the compiler coerced caller args to the wrong declared param types and forwarded them BY POSITION — so tier received an int, importance/confidence received strings, label received a float, etc. (~100 corrupt nodes). - Correct the wrapper to match the C contract 1:1 (no coercion, no reorder). - Add engram_valid_node_type / engram_valid_tier allowlists; engram_node and engram_node_full now reject invalid values with __println + return "" (fail loud, no silent malformed write). See neuron repo: HANDOFF-engram-write-corruption.md for the full write-up + deploy runbook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-08 16:13:43 -05:00
Will Anderson	1ae68962cf	restructure: move el compiler content into lang/	2026-05-05 01:38:51 -05:00