10 Commits

Author	SHA1	Message	Date
bigmerge	d231b7e5e7	compiler: fix the quadratic — strlen() on every character access ... THE BUG. str_char_code() and str_slice() each called strlen() on every invocation. The lexer walks source one character at a time, so every character access rescanned the whole remaining input: O(n) per character over n characters = O(n^2). el_val_t str_char_code(el_val_t s, el_val_t i) { ... int64_t n = (int64_t)strlen(str); // <- O(n), every call if (idx < 0 || idx >= n) return 0; return str[idx]; } HOW IT WAS FOUND. Not by reading code — by sampling the running process, which is the same method that resolved tonight's engram outage after four wrong theories. A geometric sweep of synthetic sources showed wall-clock rising 3.0x, 3.0x, 4.0x, 4.14x per doubling (converging on 4x = quadratic), and a stack sample put 779 of 779 samples inside lex(), every one bottoming out in _platform_strlen via str_char_code and str_slice. THE FIX. Remember the length instead of recomputing it. The subtlety is INVALIDATION: El strings are arena-allocated, so a freed pointer can be reused for a different string at the same address, and a naive pointer-keyed cache would hand back a stale length and read past the end of the new string — trading a performance bug for a memory-safety one. So entries carry a generation, a hit requires pointer AND generation to match, and every path that frees or mutates a runtime string bumps the generation: el_arena_pop, seed_request_end, __str_set_char. Stale entries cannot be believed; they miss and recompute. MEASURED, same host, same inputs: n(fns) before after 512 0.10s 0.01s 1024 0.37s 0.02s 2048 1.51s 0.03s 50x the compiler's own 422 KB source concatenated (DESIGN.md's 3.58s case): 3.55s -> 0.03s 118x The speedup GROWS with input size, which is the signature of removing a complexity class rather than a constant factor. After the fix each doubling adds ~0.01s: linear. CORRECTNESS, verified rather than assumed: - byte-identical output on every sweep input (n = 128..2048) - byte-identical output on the 422 KB compiler concatenation - byte-identical output on tests/runtime/string_test.el - self-hosting fixpoint byte-identical - new tests/runtime/str_cache_test.el: 17 assertions covering bounds, empty strings, negative indices, slice clamping, distinct strings not sharing a cached length, 1000 interleaved strings forcing cache-slot collisions, and a grown string not reporting its old length. All pass. This is the defect that made dist/soul.c a committed artifact: elc could not run in CI because it needed 24 GB+ and minutes. It needs neither now.	2026-08-15 21:28:23 -05:00
bigmerge	37bcf7eb74	runtime: allocation accounting — the deterministic signal for complexity gating ... Implements the three primitives the test-framework design (DESIGN.md §6.5) requires for gating on growth curves: el_alloc_count, el_alloc_bytes, el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per the project's C-builtin recipe. WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build failure, which means the signal cannot flake. Wall-clock needs warmup, statistics, and a quiet machine; on shared CI it is unusable as a gate. Allocation counts are perfectly deterministic — same input, same number, every machine, every run. Fit them against n and a complexity regression becomes a build failure with zero noise. All four runtime string allocators (el_strdup, el_strbuf, and their _persist variants) funnel every allocation the language performs, so instrumenting there counts everything. WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate. Measured with two El programs, one allocating once per item, one rebuilding its accumulator each iteration: n linear allocs / bytes quadratic allocs / bytes 100 100 / 290 100 / 5,150 200 200 / 690 200 / 20,300 400 400 / 1,490 400 / 80,600 800 800 / 3,090 800 / 321,200 The quadratic program's allocation COUNT is exactly linear — identical to the healthy one. Counting allocations alone would have missed it completely. Bytes catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 -> converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0. That shape — count linear, per-allocation size growing — is the classic accidental quadratic, and it is exactly elc's defect: quadratic allocation VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across both; RSS and wall-clock were just the two ways it surfaced. el_peak_rss is exported for context and is explicitly NOT a gating signal — it is perturbed by allocator internals, the page cache, and the OS. Gate on the deterministic numbers; report the physical one. Counters are unsynchronised by design: this is measurement, and a lock would change the thing being measured. Exact on the single-threaded compile path, approximate under threads.	2026-08-15 21:21:42 -05:00
bigmerge	e917b3d439	store: make the buffer pool sense its own state and correct from it ... Follow-on to the edge write barrier. That fix removed the full-store walk; this one makes the pool able to notice if anything like it happens again. WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill twice on 2026-08-15 and said nothing. From outside it was indistinguishable from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each costing a deploy or a rollback. The whole time, hits/misses/evictions were already being counted in PgCache, and the struct comment read: /* stats (introspection only — never affect semantics) */ That comment was the bug. Self-measurement treated as decoration is why the pool could not correct itself and why no one outside could see what it was doing. A system that cannot read its own state cannot correct, and neither can anyone watching it. - pc_adapt_budget(): the loop, closed. Over a sliding window, evictions running at a large fraction of accesses WHILE reuse is real means the working set exceeds the budget — so grow it, geometrically, bounded by a LIVE re-read of physical memory. Evictions alone are not pressure (a scan evicts and never returns); evictions with reuse are. An explicit ENGRAM_POOL_FRAMES still wins — an operator override must not be silently overruled. - Budget derived, not declared. A constant cannot be right: 16 GiB of frames is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM at startup" is a guess about the future — it cannot know the store grew or the machine changed. Hence the live re-read. - pc_report(): ONE structured emission carrying the entire sensed state, through emit_log — El's existing telemetry, already exporting to OTLP. Deliberately not a function per stat, and deliberately not a bespoke /api/pool endpoint: both make observability something hand-written per noun instead of the uniform mechanism every component already has. - engram_pool_stats_json(): the same state readable live, wired through the normal builtin path (codegen arity + el_seed wrapper), so the pool can be observed in real time rather than reconstructed afterward from a stack sample. Verified: with the exact configuration that took production down (ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete — and NO pressure event fires, because the barrier removed the walk that caused it. The controller is defense in depth; the barrier is the fix.	2026-08-15 20:44:23 -05:00
bigmerge	4e24d7d3f1	runtime: engram_edges_json — read edges without a whole-graph file round trip ... /api/graph/edges answered a read query by calling engram_save() to serialize the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in one line, and both bit production on 2026-08-15: 1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram server's CANONICAL store. A READ route overwriting the persistence owner's canonical file. This defect had been fixed once (export moved to a scratch path); it came back when the hand-written dispatch block was replaced by @route dispatch and the unfixed copy is the one that survived the merge. 2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole graph, per request, to return a bounded slice. Calling it tonight overwrote the canonical snapshot and immediately preceded an engram crash loop. engram_edges_json(limit, offset) is the builtin that route's own TODO asked for ("Future: add an engram_edges_json() builtin and drop the file round trip entirely"). It walks g->edges directly and emits every persisted field. limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell over, and an unbounded default would preserve the failure mode under a new name. Callers page explicitly. Registered in codegen.el's builtin_arity (both plain and __ spellings) and wrapped in el_seed.c per the project's C-builtin recipe.	2026-08-15 20:10:48 -05:00
bigmerge	40eb48e92f	engram: fix silently-wrong query params, and make el_seed.o + el_runtime.o link ... Three real bugs, all found by actually running the thing rather than reading it. 1. query_param never URL-decoded. A GET of /api/search?q=neural%20network searched for the literal string "neural%20network" and returned []. Every multi-word search against the live engram has been silently returning empty results — not an error, an empty result, which is why it went unnoticed. Affects every GET route that reads query params, not just search. 2. query_param matched key names unanchored. str_index_of(qs, "q=") matches inside "faq=", so "?faq=X&q=Y" returned X for key "q". Verified live before the fix. Now searches for "&key=" against "&"+querystring so a match can only land on a real parameter boundary. 3. el_request_start/el_request_end were defined in BOTH el_seed.c and el_runtime.c, so linking the two objects together — which is exactly what the product build does — failed with duplicate symbols. el_seed.c's own comment already says these moved there ("formerly defined in el_runtime.c. Now self-contained in el_seed.c"); the el_runtime.c copies were left behind during that move. Removed them, kept declarations since http_worker calls them. Also added the three missing prototypes (engram_op_assert_json, engram_node_full_in, engram_connect_in) that el_seed.c wraps but never declared, which made it fail to compile standalone under C99+. Verified: engram builds and links clean from canonical source; before/after comparison on a copy of the real store shows "neural network" returning a real match where the live build returns [], and "?faq=WRONG&q=MetaColloc" now resolving to MetaColloc. Live engram on :8742 was never touched.	2026-08-15 19:34:04 -05:00
bigmerge	09dade0613	Merge remote-tracking branch 'origin/pr/103' into HEAD ... # Conflicts: # lang/AGENTS.md # lang/runtime/el_runtime.h	2026-08-15 18:26:57 -05:00
bigmerge	3718bf0380	runtime: port missing __channel_* primitives into el_seed.c ... runtime/channel.el has always called __channel_new/__channel_send/ __channel_recv/__channel_try_recv/__channel_close, but these were only ever implemented in the pre-restructure lang/el-compiler/runtime/el_runtime.c. When the canonical runtime was consolidated onto the release copy (lang/runtime/el_runtime.c) and el_seed.c became the sole C dependency, the channel implementation was never carried forward — __mutex_new made the move, __channel_* did not. Any El program using Go-style channels currently fails to link on dev. Ported the working buffered-MPMC-channel implementation (mutex+condvar+ circular buffer, bounded and unbounded modes) from the old el_runtime.c verbatim, adapted only to el_seed.c's arena API (seed_arena_track in place of el_arena_track). Declared in el_seed.h alongside the existing mutex primitives.	2026-08-15 17:50:04 -05:00
bigmerge	2d0aef4ef8	lang: declare the el_runtime.c symbols el_seed.c's wrappers call ... runtime/el_seed.c does not compile standalone via the exact command tools/install.sh uses (`cc -std=c11 -O2 -I runtime -c runtime/el_seed.c`): 51 of its __-prefixed wrapper functions (http serving, JSON access, key-val state, URL/HTML escaping, and the whole engram_* node/edge/layer/search surface) call unprefixed counterparts that are implemented in el_runtime.c, not in el_seed.c itself, and el_seed.c never declared them -- a toolchain that treats an implicit function declaration as a hard error under C11 fails the compile outright. install.sh already compiles el_seed.c and el_runtime.c as separate objects and archives both into libel.a, so the symbols are always present at link time; el_seed.c alone was just missing the prototypes. A plain `#include "el_runtime.h"` was tried first and rejected: it redefines el_to_float/el_from_float, which el_seed.h already provides -- a real compile error, not a style preference. Added narrow prototypes instead, copied verbatim from el_runtime.h, for exactly the 51 symbols el_seed.c's wrappers reference and nothing else. Verified clean: - `cc -std=c11 -O2 -I runtime -c runtime/el_seed.c` (install.sh's exact per-file compile) -- 0 errors, 0 warnings, even with -ferror-limit=0. - full `tools/install.sh` run -- compiles both objects and archives them into libel.a successfully. Separately (not fixed here, out of scope): AGENTS.md's documented compiler self-rebuild command links elc-new.c against el_seed.c, but elc-new.c's own generated `#include "el_runtime.h"` line and 3 undeclared symbols (el_mem_check, stdout_to_file, stdout_restore -- present in neither el_runtime.c nor el_seed.c) mean that command fails regardless of which runtime file it's linked against; and install.sh's libel.a only archives el_seed.o + el_runtime.o, so any program that calls into the engram_* surface fails to link against it (el_runtime.c's engram_* wrappers need engram_store.c/engram_geometry.c/engram_reason.c/engram_cognition.c/ engram_vindex.c, none of which install.sh compiles in). Both are real, pre-existing, and independent of this fix -- worth their own look.	2026-08-15 17:32:58 -05:00
will.anderson	7aa847e32a	Merge origin/dev into engram-tiered-storage ... Resolve 3 conflicts: - lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork; single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh). - lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated artifact folder, not a source path; a release is a git tag, not a folder). - lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL. Nothing in dev's build consumes the deprecated fork or releases/ folder.	2026-08-12 15:23:02 -05:00
will.anderson	0a72fced28	engram: WAL persistence + integrity hardening + single canonical runtime ... Establish lang/runtime/ as the ONE canonical el runtime (from the active runtime that carries hebb/emb persistence + the new WAL); repoint the el CI publish, engram build, elb default, and in-repo build scripts to it; delete the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh drift guard. Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian edge weights — learned co-activation was wiped on every restart. Publishing from canonical ships the stranded 'learning that cannot outlive the process' fix. WAL storage engine + integrity fixes (DELETE->tombstone + store-layer protection, safe data-dir default) ride in behind ENGRAM_WAL (default off = byte-identical to today). Verified: engram elb per-module build clean, WAL gate 66/66, native smoke ok, drift-guard green.	2026-08-11 21:31:37 -05:00