11 Commits

Author	SHA1	Message	Date
Neuron	5503e1d9a4	organ: el gets a speaker, and fetches the voice from the engram ... El could turn meaning into samples and could not make a sound. Every path from those samples to the air ran outside the language, through a 939-line Swift program that shelled out to afplay, so the voice was not a capability of El or of Neuron but a separate binary standing next to them. Two things land here. The speaker. el_audio_darwin.m is a CoreAudio AudioQueue realizer in its own translation unit, declared in el_runtime.h, deliberately not a patch to el_runtime.c — acquiring a device must not mean editing the middle of the language, the same rule the realizer registry follows for modalities. It takes samples straight out of memory, so nothing is written to disk and no process is spawned between the intent to speak and the sound. The async half (play/stop/playing/played_frames) exists because barge-in means stopping on the spot, and a blocking play cannot be interrupted. el_peripheral_null.c is the same entry points everywhere else, so El that speaks links anywhere and truthfully reports having no speaker. The voice. organ_voice_fetch asks the engram for a voice region by query and reads the geometry off the node that comes back. A voice is not a JSON file next to the code; it is a memory, and the organ retrieves it the way anything retrieves a memory. An absent region returns empty rather than a plausible default, because a caller must be able to tell 'this is how they sound' from 'I never heard them'. Underneath both: __str_set_char bounds-checked writes against strlen(), which is 0 for the zero-filled buffer __str_alloc hands back, so every write was rejected and every El-authored WAV in this repo was 55,244 bytes of silence that reported ok=true. Byte buffers now carry their capacity in a side table; text keeps the exact strlen behaviour it had. This is why nobody noticed El was mute. Measured: voice fetched from the engram reads f0=137 f0_end=116 kf=1269 f1=500 f2=2093 f3=3531, matching the 30s LPC measurement; render is 20160 samples at 16 kHz; both the rendered utterance and an own-core tone played aloud through CoreAudio with no Swift and no afplay in the chain.	2026-08-16 16:27:30 -05:00
bigmerge	d231b7e5e7	compiler: fix the quadratic — strlen() on every character access ... THE BUG. str_char_code() and str_slice() each called strlen() on every invocation. The lexer walks source one character at a time, so every character access rescanned the whole remaining input: O(n) per character over n characters = O(n^2). el_val_t str_char_code(el_val_t s, el_val_t i) { ... int64_t n = (int64_t)strlen(str); // <- O(n), every call if (idx < 0 || idx >= n) return 0; return str[idx]; } HOW IT WAS FOUND. Not by reading code — by sampling the running process, which is the same method that resolved tonight's engram outage after four wrong theories. A geometric sweep of synthetic sources showed wall-clock rising 3.0x, 3.0x, 4.0x, 4.14x per doubling (converging on 4x = quadratic), and a stack sample put 779 of 779 samples inside lex(), every one bottoming out in _platform_strlen via str_char_code and str_slice. THE FIX. Remember the length instead of recomputing it. The subtlety is INVALIDATION: El strings are arena-allocated, so a freed pointer can be reused for a different string at the same address, and a naive pointer-keyed cache would hand back a stale length and read past the end of the new string — trading a performance bug for a memory-safety one. So entries carry a generation, a hit requires pointer AND generation to match, and every path that frees or mutates a runtime string bumps the generation: el_arena_pop, seed_request_end, __str_set_char. Stale entries cannot be believed; they miss and recompute. MEASURED, same host, same inputs: n(fns) before after 512 0.10s 0.01s 1024 0.37s 0.02s 2048 1.51s 0.03s 50x the compiler's own 422 KB source concatenated (DESIGN.md's 3.58s case): 3.55s -> 0.03s 118x The speedup GROWS with input size, which is the signature of removing a complexity class rather than a constant factor. After the fix each doubling adds ~0.01s: linear. CORRECTNESS, verified rather than assumed: - byte-identical output on every sweep input (n = 128..2048) - byte-identical output on the 422 KB compiler concatenation - byte-identical output on tests/runtime/string_test.el - self-hosting fixpoint byte-identical - new tests/runtime/str_cache_test.el: 17 assertions covering bounds, empty strings, negative indices, slice clamping, distinct strings not sharing a cached length, 1000 interleaved strings forcing cache-slot collisions, and a grown string not reporting its old length. All pass. This is the defect that made dist/soul.c a committed artifact: elc could not run in CI because it needed 24 GB+ and minutes. It needs neither now.	2026-08-15 21:28:23 -05:00
bigmerge	37bcf7eb74	runtime: allocation accounting — the deterministic signal for complexity gating ... Implements the three primitives the test-framework design (DESIGN.md §6.5) requires for gating on growth curves: el_alloc_count, el_alloc_bytes, el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per the project's C-builtin recipe. WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build failure, which means the signal cannot flake. Wall-clock needs warmup, statistics, and a quiet machine; on shared CI it is unusable as a gate. Allocation counts are perfectly deterministic — same input, same number, every machine, every run. Fit them against n and a complexity regression becomes a build failure with zero noise. All four runtime string allocators (el_strdup, el_strbuf, and their _persist variants) funnel every allocation the language performs, so instrumenting there counts everything. WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate. Measured with two El programs, one allocating once per item, one rebuilding its accumulator each iteration: n linear allocs / bytes quadratic allocs / bytes 100 100 / 290 100 / 5,150 200 200 / 690 200 / 20,300 400 400 / 1,490 400 / 80,600 800 800 / 3,090 800 / 321,200 The quadratic program's allocation COUNT is exactly linear — identical to the healthy one. Counting allocations alone would have missed it completely. Bytes catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 -> converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0. That shape — count linear, per-allocation size growing — is the classic accidental quadratic, and it is exactly elc's defect: quadratic allocation VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across both; RSS and wall-clock were just the two ways it surfaced. el_peak_rss is exported for context and is explicitly NOT a gating signal — it is perturbed by allocator internals, the page cache, and the OS. Gate on the deterministic numbers; report the physical one. Counters are unsynchronised by design: this is measurement, and a lock would change the thing being measured. Exact on the single-threaded compile path, approximate under threads.	2026-08-15 21:21:42 -05:00
bigmerge	e917b3d439	store: make the buffer pool sense its own state and correct from it ... Follow-on to the edge write barrier. That fix removed the full-store walk; this one makes the pool able to notice if anything like it happens again. WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill twice on 2026-08-15 and said nothing. From outside it was indistinguishable from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each costing a deploy or a rollback. The whole time, hits/misses/evictions were already being counted in PgCache, and the struct comment read: /* stats (introspection only — never affect semantics) */ That comment was the bug. Self-measurement treated as decoration is why the pool could not correct itself and why no one outside could see what it was doing. A system that cannot read its own state cannot correct, and neither can anyone watching it. - pc_adapt_budget(): the loop, closed. Over a sliding window, evictions running at a large fraction of accesses WHILE reuse is real means the working set exceeds the budget — so grow it, geometrically, bounded by a LIVE re-read of physical memory. Evictions alone are not pressure (a scan evicts and never returns); evictions with reuse are. An explicit ENGRAM_POOL_FRAMES still wins — an operator override must not be silently overruled. - Budget derived, not declared. A constant cannot be right: 16 GiB of frames is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM at startup" is a guess about the future — it cannot know the store grew or the machine changed. Hence the live re-read. - pc_report(): ONE structured emission carrying the entire sensed state, through emit_log — El's existing telemetry, already exporting to OTLP. Deliberately not a function per stat, and deliberately not a bespoke /api/pool endpoint: both make observability something hand-written per noun instead of the uniform mechanism every component already has. - engram_pool_stats_json(): the same state readable live, wired through the normal builtin path (codegen arity + el_seed wrapper), so the pool can be observed in real time rather than reconstructed afterward from a stack sample. Verified: with the exact configuration that took production down (ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete — and NO pressure event fires, because the barrier removed the walk that caused it. The controller is defense in depth; the barrier is the fix.	2026-08-15 20:44:23 -05:00
bigmerge	4e24d7d3f1	runtime: engram_edges_json — read edges without a whole-graph file round trip ... /api/graph/edges answered a read query by calling engram_save() to serialize the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in one line, and both bit production on 2026-08-15: 1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram server's CANONICAL store. A READ route overwriting the persistence owner's canonical file. This defect had been fixed once (export moved to a scratch path); it came back when the hand-written dispatch block was replaced by @route dispatch and the unfixed copy is the one that survived the merge. 2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole graph, per request, to return a bounded slice. Calling it tonight overwrote the canonical snapshot and immediately preceded an engram crash loop. engram_edges_json(limit, offset) is the builtin that route's own TODO asked for ("Future: add an engram_edges_json() builtin and drop the file round trip entirely"). It walks g->edges directly and emits every persisted field. limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell over, and an unbounded default would preserve the failure mode under a new name. Callers page explicitly. Registered in codegen.el's builtin_arity (both plain and __ spellings) and wrapped in el_seed.c per the project's C-builtin recipe.	2026-08-15 20:10:48 -05:00
bigmerge	40eb48e92f	engram: fix silently-wrong query params, and make el_seed.o + el_runtime.o link ... Three real bugs, all found by actually running the thing rather than reading it. 1. query_param never URL-decoded. A GET of /api/search?q=neural%20network searched for the literal string "neural%20network" and returned []. Every multi-word search against the live engram has been silently returning empty results — not an error, an empty result, which is why it went unnoticed. Affects every GET route that reads query params, not just search. 2. query_param matched key names unanchored. str_index_of(qs, "q=") matches inside "faq=", so "?faq=X&q=Y" returned X for key "q". Verified live before the fix. Now searches for "&key=" against "&"+querystring so a match can only land on a real parameter boundary. 3. el_request_start/el_request_end were defined in BOTH el_seed.c and el_runtime.c, so linking the two objects together — which is exactly what the product build does — failed with duplicate symbols. el_seed.c's own comment already says these moved there ("formerly defined in el_runtime.c. Now self-contained in el_seed.c"); the el_runtime.c copies were left behind during that move. Removed them, kept declarations since http_worker calls them. Also added the three missing prototypes (engram_op_assert_json, engram_node_full_in, engram_connect_in) that el_seed.c wraps but never declared, which made it fail to compile standalone under C99+. Verified: engram builds and links clean from canonical source; before/after comparison on a copy of the real store shows "neural network" returning a real match where the live build returns [], and "?faq=WRONG&q=MetaColloc" now resolving to MetaColloc. Live engram on :8742 was never touched.	2026-08-15 19:34:04 -05:00
bigmerge	09dade0613	Merge remote-tracking branch 'origin/pr/103' into HEAD ... # Conflicts: # lang/AGENTS.md # lang/runtime/el_runtime.h	2026-08-15 18:26:57 -05:00
bigmerge	3718bf0380	runtime: port missing __channel_* primitives into el_seed.c ... runtime/channel.el has always called __channel_new/__channel_send/ __channel_recv/__channel_try_recv/__channel_close, but these were only ever implemented in the pre-restructure lang/el-compiler/runtime/el_runtime.c. When the canonical runtime was consolidated onto the release copy (lang/runtime/el_runtime.c) and el_seed.c became the sole C dependency, the channel implementation was never carried forward — __mutex_new made the move, __channel_* did not. Any El program using Go-style channels currently fails to link on dev. Ported the working buffered-MPMC-channel implementation (mutex+condvar+ circular buffer, bounded and unbounded modes) from the old el_runtime.c verbatim, adapted only to el_seed.c's arena API (seed_arena_track in place of el_arena_track). Declared in el_seed.h alongside the existing mutex primitives.	2026-08-15 17:50:04 -05:00
bigmerge	2d0aef4ef8	lang: declare the el_runtime.c symbols el_seed.c's wrappers call ... runtime/el_seed.c does not compile standalone via the exact command tools/install.sh uses (`cc -std=c11 -O2 -I runtime -c runtime/el_seed.c`): 51 of its __-prefixed wrapper functions (http serving, JSON access, key-val state, URL/HTML escaping, and the whole engram_* node/edge/layer/search surface) call unprefixed counterparts that are implemented in el_runtime.c, not in el_seed.c itself, and el_seed.c never declared them -- a toolchain that treats an implicit function declaration as a hard error under C11 fails the compile outright. install.sh already compiles el_seed.c and el_runtime.c as separate objects and archives both into libel.a, so the symbols are always present at link time; el_seed.c alone was just missing the prototypes. A plain `#include "el_runtime.h"` was tried first and rejected: it redefines el_to_float/el_from_float, which el_seed.h already provides -- a real compile error, not a style preference. Added narrow prototypes instead, copied verbatim from el_runtime.h, for exactly the 51 symbols el_seed.c's wrappers reference and nothing else. Verified clean: - `cc -std=c11 -O2 -I runtime -c runtime/el_seed.c` (install.sh's exact per-file compile) -- 0 errors, 0 warnings, even with -ferror-limit=0. - full `tools/install.sh` run -- compiles both objects and archives them into libel.a successfully. Separately (not fixed here, out of scope): AGENTS.md's documented compiler self-rebuild command links elc-new.c against el_seed.c, but elc-new.c's own generated `#include "el_runtime.h"` line and 3 undeclared symbols (el_mem_check, stdout_to_file, stdout_restore -- present in neither el_runtime.c nor el_seed.c) mean that command fails regardless of which runtime file it's linked against; and install.sh's libel.a only archives el_seed.o + el_runtime.o, so any program that calls into the engram_* surface fails to link against it (el_runtime.c's engram_* wrappers need engram_store.c/engram_geometry.c/engram_reason.c/engram_cognition.c/ engram_vindex.c, none of which install.sh compiles in). Both are real, pre-existing, and independent of this fix -- worth their own look.	2026-08-15 17:32:58 -05:00
will.anderson	7aa847e32a	Merge origin/dev into engram-tiered-storage ... Resolve 3 conflicts: - lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork; single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh). - lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated artifact folder, not a source path; a release is a git tag, not a folder). - lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL. Nothing in dev's build consumes the deprecated fork or releases/ folder.	2026-08-12 15:23:02 -05:00
will.anderson	0a72fced28	engram: WAL persistence + integrity hardening + single canonical runtime ... Establish lang/runtime/ as the ONE canonical el runtime (from the active runtime that carries hebb/emb persistence + the new WAL); repoint the el CI publish, engram build, elb default, and in-repo build scripts to it; delete the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh drift guard. Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian edge weights — learned co-activation was wiped on every restart. Publishing from canonical ships the stranded 'learning that cannot outlive the process' fix. WAL storage engine + integrity fixes (DELETE->tombstone + store-layer protection, safe data-dir default) ride in behind ENGRAM_WAL (default off = byte-identical to today). Verified: engram elb per-module build clean, WAL gate 66/66, native smoke ok, drift-guard green.	2026-08-11 21:31:37 -05:00