2 Commits

Author	SHA1	Message	Date
bigmerge	addd51209f	runtime: extract engram_text.c, and repair 10 harnesses that could not link ... First concern moved out of el_runtime.c under the ratchet, and the move is deliberately small: it exists to prove the mechanism end to end before anything large depends on it. engram_text.{c,h} — query tokenization, candidate-token hygiene, word-boundary matching, and the text-damage signature. Four functions, moved verbatim; only `static` was dropped and each doc comment travelled with the code. They touch no EL value type and no engram store type: plain C over <ctype.h>/<string.h> over char buffers. They were never el_runtime.c's business. el_runtime.c 20,527 -> 20,427 lines (BUDGET max_lines ratcheted down) engram fns 279 -> 275 (BUDGET max_engram_fns ratcheted down) The Stage 1 extension point worked as designed: adding the file to lang/runtime/SOURCES was one line, and every build path picked it up. The Stage 2 drift guard then caught that I had NOT added it to install.sh's standalone list — the exact class of drift it was written for, on its first real change, before the commit rather than after a broken SDK shipped. WHY ONLY 100 LINES, AND WHAT ACTUALLY BLOCKS THE REST Measured, not estimated: of 273 engram-domain functions in el_runtime.c (~9,700 lines), only 75 (~1,058 lines) can move today, and they are scattered rather than clustered. The blocker is a single fact: EngramNode, EngramEdge, EngramStore, EngramLayer, EngramWal and EngramIdSlot are typedef'd INSIDE el_runtime.c. No sibling can see them. engram_store.h defines a SEPARATE serializable "node view" struct and maps between the two. So every engram function that takes an EngramNode* — which is most of them, 109 of 273 by direct type reference — cannot compile in engram_store.c until those types move to a shared header. That extraction is the real Stage 3 enabler and it deserves its own change: it touches the most load-bearing struct in the system, and doing it in the same commit as a code move would make a regression impossible to bisect. REPAIRED: 10 engram harnesses that had silently stopped linking Not new breakage from this move — verified against unmodified dev, where el_runtime.c + engram_store.c alone already failed with undefined symbols. They had been dead for as long as el_runtime.c has been calling into the siblings, and nothing noticed because nothing ran them. run_m3_parity, run_m7_traversal, run_m35_hebb_persist, run_interoception_p0..p5 — now build from $(scripts/el-runtime-sources.sh) run_wal_tests — its two TUs #include "el_runtime.c" directly, so it links the SIBLINGS ONLY; adding el_runtime.c to that link line would define every symbol twice (That #include'd .c is worth recording: the runtime does have one, in engram/test/test_wal.c and the generated test_failloud.c.) Verified locally — every one of these was run, not assumed: * m3_parity ............ PASS, incl. ASan+UBSan clean across seed/on/reboot * m7_traversal ......... PASS * m35_hebb_persist ..... PASS (the gate over the original prod hebb bug) * interoception p0..p5 . PASS (all six) * wal_tests ............ 66 passed, 0 failed, + fail-loud exit check * self-host fixpoint ... byte-identical, AND the emitted C is byte-identical to the pre-move compiler output — the move changes nothing the compiler produces * engram/src/server.el . compiles and links * native suites ........ 8 of 13, unchanged from before the move; the same 5 pre-existing failures, no regression * both runtime guards .. green at the new, lower budget Also fixes a block comment left unterminated by the extraction (the deleted range carried its closing */), restoring the compile to its single pre-existing -Wcomment warning.	2026-08-16 19:59:43 -05:00
bigmerge	bacaf3d39c	engram: reconcile M8 HNSW vindex (#109) onto current dev, restore 3 fixes the branch predated ... Lands feat/reframe-region-setop (PR #109: native set-based reframe_region, decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's actual current HEAD, plus engram-tiered-storage's still-unique test suite. RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/ verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/ select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two branches; #109's copy is a strict superset (adds vindex_harvest_from_store, used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and engram_verify.c are also byte-identical. #109's own branch point already carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing left to merge into #109 for those files. The one thing engram-tiered-storage had that #109's tree dropped: its full test suite (test_vindex.c, test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh harnesses) — ported over here unchanged. WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch forked from dev on 2026-08-14 15:40 (before restructure-adjacent history diverged the file's merge-base for `git merge` — it presented as an add/add conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but it silently reverted THREE dev fixes landed on 2026-08-14/15, after the branch point, that the PR's diff had no way to know about: 1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of the query-aware propagation gate dropped the shift-and-floor rescale about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate 0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi] is computed) is orthogonal to WHAT it gates on and both are kept. 2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global). PR's tree predates this and dropped all three counters + their JSON stats fields. Restored declarations, all 4 direct increment sites, the eg_wm_carry_over bll increment, and the act-stats JSON fields -- alongside (not instead of) the PR's own P4 afferent / API-reshape counters already in that same struct/JSON. 3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev selects the STRONGEST qualifying candidate for consolidation each call; PR's tree predates this and reverted to hash-slot order (arbitrary wrt association strength) for edge formation -- the one path that writes PERMANENT structure. Restored the strongest-candidate while-loop, keeping the PR's own genuine improvement at that site (engram_adj_on_edge_added incremental-index append instead of a bare adj_dirty=1 full-rebuild flag). engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env flags, /api/nodes connected-count in responses) were pure additive: dev's side was empty, PR's side added the feature. Took PR's side whole. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): - cc -std=c11 -O2, clean link against the real engram/src/server.el via elc, zero errors. - vindex_bench (built standalone, read-only harvest) against the real production store clone (13,671 embedded nodes, 768-dim nomic-embed-text): recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs 2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s for the full 13,671-node set -- see the flagged risk below. - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned snapshot of the live store, 13,424 nodes / 37,656 edges) and called /api/activate for real: first call after boot 41.5s (pays the one-time HNSW build inline -- matches the standalone bench), second/third calls 356ms/605ms, no crash, correct results, act-stats JSON (including the restored evict_floor/cap/bll fields) reads correctly. KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync builds the HNSW index synchronously, inline, on the first engram_activate() call after every process start (or index invalidation). On the real node count that is a ~46s blocking stall on a single-threaded server -- the first request after every restart (or its concurrent siblings) waits the full build. Recommend a background/incremental build (or a bounded per-call build budget) before this ever reaches the live daemon. See PR description / final report for the fuller writeup.	2026-08-15 16:46:44 -05:00