12 Commits

Author	SHA1	Message	Date
bigmerge	9a13547fe2	runtime: put el_runtime.c on a ratchet, and actually run the guards ... scripts/check-single-runtime.sh guards against el_runtime.c being COPIED — it was written after a lagging fork shipped to prod and dropped learned hebb edges. Nothing guarded against it GROWING. So it grew: 10,607 -> 20,527 lines, 94% in 3.5 months, the whole time under an explicit commit-message promise that it was a temporary shim about to be deleted. Worse, the copy guard was never wired in. Its own footer described the CI wire-in as a TODO, and the TODO had never been done — the script existed but ran nowhere, in no workflow and in no hook, so it had caught nothing for as long as it has been in the tree. A guard that does not run is a comment. This adds the missing guard and runs both. * lang/runtime/BUDGET — a RATCHET, not a limit. max_lines is set at the current 20,527 with NO headroom: the file cannot grow by one line. A second cap, max_engram_fns (279), counts top-level engram_/eg_/cog_ definitions in it — ~47.5% of the file is engram code and engram already owns six sibling .c files, so this is the scoreboard for moving it out. Both may only go DOWN. * scripts/check-runtime-growth.sh — enforces the ratchet, and three invariants that keep the multi-file runtime honest: every .c in lang/runtime/ is either in SOURCES or explicitly platform-optional (an unaccounted .c is compiled by nothing and is silently dead); install.sh's hardcoded download list matches SOURCES (it cannot call the helper — it runs where there is no checkout — so that copy is checked, not trusted); and an advisory nudge to lower the budget when you have earned it. * Both guards now run as early steps in ci-dev.yaml, ci-stage.yaml and sdk-release.yaml, and in .githooks/pre-commit. The failure message is the point. The guard that existed said what was wrong but not where the code should go, which makes it easy to "fix" by arguing with the guard. This one names the destination: the concern-owning .c, or a new .c plus one line in SOURCES, or c_source in a program's manifest.el — and it prints the `nm` command that proves placement is link-time and that the shipped compiler already links from ten translation units. Every runtime file except el_runtime.c is deliberately uncapped, because that is where code is supposed to go. Proven with negative controls, per lang/AGENTS.md step 5 — each shown FAILING: * +1 line to el_runtime.c -> FAIL (20528/20527) * +1 engram fn, net-zero lines -> FAIL (280/279) * a new unaccounted lang/runtime/*.c -> FAIL * engram_store.c removed from install.sh -> FAIL, names the missing file * el_runtime.c truncated to 20,000 lines -> PASS + "lower max_lines to 20000" * baseline, tree unmodified -> OK, and both guards green el_runtime.c is byte-identical after the controls; this commit changes zero lines of it.	2026-08-16 19:59:43 -05:00
will.anderson	e1bc6fe944	Merge pull request 'singleton: guard the state, not the program's name' (#157) from fix/singleton-guards-the-state into dev	2026-08-17 00:57:49 +00:00
bigmerge	8c2406ff6b	runtime: the link set is multi-file — name it once, ship all of it ... el_runtime.c was created 2026-05-03 as an explicitly temporary build shim. It was deleted that afternoon ("runtime is 100% native El") and restored 25 minutes later "UNTIL the compiler is updated to emit #include el_seed.h". The `until` never came. 3.5 months on it is 20,527 lines, and nothing was ever set up to notice — a file scheduled for deletion gets no owner, no budget, no boundary. What kept it growing is not inertia, it is an instruction. lang/AGENTS.md said el_runtime.c "is the authoritative single-file link target ... THIS IS WHERE A NEW C BUILTIN'S IMPLEMENTATION MUST CURRENTLY LIVE TO BE LINKABLE", and made it step 1 of the add-a-builtin recipe. That is false. Placement is a link-time concern: builtin_arity maps NAME -> ARITY INT only, the El name is emitted as the exact C symbol, and `ld` resolves it — the compiler cannot tell which .c a symbol came from. `nm lang/dist/platform/elc` on the shipped compiler already shows T _engram_geo_reify_index_new, T _vindex_insert, T _engram_think, T _engram_reason_abduce: it is linked from ten translation units today. In a repo where agents write most of the code, a false instruction in the instruction file is the forcing function. The file grew because the recipe said to grow it. The multi-file runtime is therefore already real, and the docs and the distribution never caught up — which left a live, shipped bug: * Linking el_runtime.c alone FAILS at `ld` (undefined engram_ground_json, engram_activate_inner, eg_find_relation, cog_assert_two_axis, ...) because el_runtime.c #includes six engram headers and calls into all six siblings. * sdk-release.yaml shipped el_runtime.c/.h + engram_store.c/.h and none of the other five required .c files, so downstream consumers of the el-runtime-c Artifact Registry package and of install.sh got a lib/ that cannot link. * .githooks/pre-commit linked el_runtime.c alone with stderr to /dev/null, so it reported all 13 native suites as FAILED with the real ld error invisible. * AGENTS.md's self-host recipe compiled el-compiler/runtime/el_runtime.c — a path the same file's "DO NOT EDIT" list names as a lagging fork. The root fix is to stop writing the list down eight times: * lang/runtime/SOURCES — the canonical link set, in one place, in link order. * scripts/el-runtime-sources.sh — prints it, optionally prefixed; --check fails loudly on a missing file, --headers for the shipped headers. * Every link line in AGENTS.md, lang/AGENTS.md, DESIGN.md, lang/spec/language.md, the three workflows and the pre-commit hook now reads that one list. * Adding a concern's .c is one line in SOURCES, so a new builtin no longer has to be appended to el_runtime.c just because appending was the cheaper edit. Distribution: ship the siblings rather than amalgamate. Amalgamation needs a new tool and contradicts DESIGN.md's compile-once-link-many; the siblings are already independently authored and independently tested (engram/test/*.sh link subsets directly), and engram_store.c was already shipped, so this completes a mechanism that existed rather than inventing one. Source is also a superset: a consumer that wants one file can concatenate, one that wants separate TUs cannot undo an amalgamation. el-runtime-c/-h stay for backward compatibility; el-runtime-src is added carrying the complete set plus SOURCES. lang/AGENTS.md now points new C builtins at the concern-owning .c and states plainly that the compiler cannot tell which .c a symbol came from, with the nm evidence. AGENTS.md's "reconcile which is canonical (verify)" note is resolved: neither file supersedes the other, the canonical unit is the set. Verified locally (the bar; not CI): * engram/src/server.el compiles and links against the SOURCES set. * Compile-once-link-many into libel.a links the same program. * elb builds from the corrected recipe. * Self-host fixpoint byte-identical (11,110 lines, stage2 == stage3) built with the SOURCES-driven link line. * pre-commit hook: 0 of 13 native suites passing -> 8 of 13. The 5 still-failing suites are PRE-EXISTING and untouched here: test_fs (fs_list_json undeclared), test_state (state_has, state_get_or undeclared), test_json (json_build_array/json_build_object/json_escape_string undefined), test_time (now_ns undefined), test_env (1 assertion). Builtins registered in builtin_arity with no implementation or no declaration anywhere — the same recipe defect, now visible because the linker error is no longer suppressed. Not attempted: making elc emit #include el_seed.h and dropping elb's hardcoded runtime path. That is the correct long-term fix and finishes the 2026-05-03 migration, but it touches codegen and self-hosting and belongs in its own change.	2026-08-16 16:44:26 -05:00
bigmerge	45325f7391	singleton: guard the state, not the program's name ... The singleton lock protected a filename, not a store. It was keyed on $EL_SINGLETON_DIR|$TMPDIR|/tmp + /el-singleton-<program>.lock — the program's NAME and a temp directory — and never consulted the state it claimed to protect, while its own refusal message read "Refusing to start a second instance against the same state." Measured, it failed in both directions. A second engram against a DIFFERENT data dir was refused, naming the first's pid. And TMPDIR=/tmp/other let a second engram start against the SAME data dir with no complaint — the two-writer data-loss condition the guard exists to prevent, defeated by one environment variable. Both are one error: the identity of the resource had been replaced by a label for it. The lock now lives inside the state it guards — <state>/.el-singleton-<id>.lock — and the program block says what that state is. Same directory is the same file is the same inode, so it contends and there is no TMPDIR left in the key to change. Different directories are different files, so they don't. Different spellings of one directory (trailing slash, x/../x, symlink) collapse in the kernel's own path walk, so they contend without this code comparing strings; canonicalisation is for the message, never the decision. `guards:` is an expression so a program can point at the resolver that already owns its path — guards: engram_resolve_data_dir() — instead of restating that resolver's default, which is the two-owners defect spec 18.4 exists to prevent. A `singleton:` without `guards:` is now a compile error; emitting a name-keyed lock instead would be emitting the defect. Kept: the flock (the kernel drops it on crash and SIGKILL, so there is still no "delete the lock file to get unstuck" ritual — a stale file inside a copied data dir is inert), and the holder's pid in the message. Changed: the message is true. It says "the same state" because the lock it failed to take is in that state, and it names the state it checked. An unguardable state (missing, read-only) now refuses rather than starting unguarded. Also corrects lang/AGENTS.md's compiler rebuild line, which had gone stale: linking el_runtime.c alone no longer resolves.	2026-08-16 16:08:40 -05:00
bigmerge	caa1206af5	docs: the nine-op surface shipped, and two of its primitives are the wrong shape ... lang/AGENTS.md said the collapse was 'not yet compiled into the MCP server'. Verified against the live tool surface: it is exactly the nine ops. Noted that think's faculty parameter and ground's minted edge are both documented as the wrong shape.	2026-08-16 15:49:44 -05:00
Neuron	fe820928b0	docs: the builtin recipe never required a test ... lang/AGENTS.md:71-77 gives four steps for adding a C builtin and ends at 'confirm the self-host fixpoint is byte-identical'. No step asks for a test. The only 'verify' in the file is that fixpoint, which proves the COMPILER REPRODUCES ITSELF and says nothing about whether the builtin works — so the recipe reads as complete while having checked nothing about the thing just added. Measured on 2026-08-16: engram_node_set_emb, engram_curiosity_json and dream_set_handler were all added in a single session with zero tests, by an agent following this recipe. Separately a UTF-8 fix was written and tested and THE TEST PASSED ON THE UNPATCHED BUILD — the real defect was elsewhere, and only building the pre-fix binary exposed it. Without a negative control that fix would have merged as verified. Adds step 5 with the two failure shapes actually encountered: a test that never exercises the change (a route default bypassed the code under test), and an induction that loses a race (curl --max-time left BOTH builds alive; only SO_LINGER 0, a real RST, reproduced it). Plus the port-binding check, because a stale instance answering has silently produced false results here more than once and pkill -f does not reliably match argv './engram'. Documentation only. Does not touch the (a) split-the-C / (b) close-the- compiler-gap question, which is a separate decision.	2026-08-16 13:53:08 -05:00
bigmerge	09dade0613	Merge remote-tracking branch 'origin/pr/103' into HEAD ... # Conflicts: # lang/AGENTS.md # lang/runtime/el_runtime.h	2026-08-15 18:26:57 -05:00
bigmerge	bacaf3d39c	engram: reconcile M8 HNSW vindex (#109) onto current dev, restore 3 fixes the branch predated ... Lands feat/reframe-region-setop (PR #109: native set-based reframe_region, decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's actual current HEAD, plus engram-tiered-storage's still-unique test suite. RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/ verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/ select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two branches; #109's copy is a strict superset (adds vindex_harvest_from_store, used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and engram_verify.c are also byte-identical. #109's own branch point already carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing left to merge into #109 for those files. The one thing engram-tiered-storage had that #109's tree dropped: its full test suite (test_vindex.c, test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh harnesses) — ported over here unchanged. WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch forked from dev on 2026-08-14 15:40 (before restructure-adjacent history diverged the file's merge-base for `git merge` — it presented as an add/add conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but it silently reverted THREE dev fixes landed on 2026-08-14/15, after the branch point, that the PR's diff had no way to know about: 1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of the query-aware propagation gate dropped the shift-and-floor rescale about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate 0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi] is computed) is orthogonal to WHAT it gates on and both are kept. 2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global). PR's tree predates this and dropped all three counters + their JSON stats fields. Restored declarations, all 4 direct increment sites, the eg_wm_carry_over bll increment, and the act-stats JSON fields -- alongside (not instead of) the PR's own P4 afferent / API-reshape counters already in that same struct/JSON. 3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev selects the STRONGEST qualifying candidate for consolidation each call; PR's tree predates this and reverted to hash-slot order (arbitrary wrt association strength) for edge formation -- the one path that writes PERMANENT structure. Restored the strongest-candidate while-loop, keeping the PR's own genuine improvement at that site (engram_adj_on_edge_added incremental-index append instead of a bare adj_dirty=1 full-rebuild flag). engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env flags, /api/nodes connected-count in responses) were pure additive: dev's side was empty, PR's side added the feature. Took PR's side whole. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): - cc -std=c11 -O2, clean link against the real engram/src/server.el via elc, zero errors. - vindex_bench (built standalone, read-only harvest) against the real production store clone (13,671 embedded nodes, 768-dim nomic-embed-text): recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs 2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s for the full 13,671-node set -- see the flagged risk below. - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned snapshot of the live store, 13,424 nodes / 37,656 edges) and called /api/activate for real: first call after boot 41.5s (pays the one-time HNSW build inline -- matches the standalone bench), second/third calls 356ms/605ms, no crash, correct results, act-stats JSON (including the restored evict_floor/cap/bll fields) reads correctly. KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync builds the HNSW index synchronously, inline, on the first engram_activate() call after every process start (or index invalidation). On the real node count that is a ~46s blocking stall on a single-threaded server -- the first request after every restart (or its concurrent siblings) waits the full build. Recommend a background/incremental build (or a bounded per-call build budget) before this ever reaches the live daemon. See PR description / final report for the fuller writeup.	2026-08-15 16:46:44 -05:00
bigmerge	08cbcef5d9	engram: fix lazy-embed index gap (#20) and make activate's cosine scan lazy; extract vindex harvest primitive with a bench/oracle harness ... Adds an O(1) "seen" bitmap so lazily-embedded older nodes get picked up incrementally instead of only on a full rebuild (embed-gap #20). Replaces engram_activate's O(N*D) cosine prescan with a lazy-memoized cosine cache (eg_cosq_at), proven bit-identical to the old path. Extracts a clean vindex_harvest_from_store primitive (read-only vector harvest, careful malloc/ownership/error-path handling) reused by both index-build and the new vindex_bench.c — a read-only proof harness comparing brute-force vs HNSW recall/latency on both the real store and synthetic data. .nsbx-env intentionally excluded — local sandbox config (ports, paths, dev-only placeholder key), not checked in.	2026-08-15 14:26:16 -05:00
bigmerge	1010185978	Add op_assert grounded-envelope primitive and purview-bounded mutation wrappers ... Adds engram_assert_json — a grounded "assertion envelope" primitive for a realizer/op_assert seam (per backlog bl-53/#57) — plus purview-scoped mutation wrappers engram_node_full_in/engram_connect_in, which refuse non-default purviews rather than silently mutating the live store. Threads through el_seed.c/h wrappers and the codegen.el arity table per the project's existing C-builtin recipe. Also rewrites lang/AGENTS.md build docs with verified (2026-08-15) findings that el_seed.c does not compile standalone.	2026-08-15 14:24:11 -05:00
will.anderson	0a72fced28	engram: WAL persistence + integrity hardening + single canonical runtime ... Establish lang/runtime/ as the ONE canonical el runtime (from the active runtime that carries hebb/emb persistence + the new WAL); repoint the el CI publish, engram build, elb default, and in-repo build scripts to it; delete the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh drift guard. Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian edge weights — learned co-activation was wiped on every restart. Publishing from canonical ships the stranded 'learning that cannot outlive the process' fix. WAL storage engine + integrity fixes (DELETE->tombstone + store-layer protection, safe data-dir default) ride in behind ENGRAM_WAL (default off = byte-identical to today). Verified: engram elb per-module build clean, WAL gate 66/66, native smoke ok, drift-guard green.	2026-08-11 21:31:37 -05:00
Will Anderson	1ae68962cf	restructure: move el compiler content into lang/	2026-05-05 01:38:51 -05:00