ci: make official engram build store-enabled (publish + link engram_store.{c,h}) #95
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: neuron-technologies/el#95
Reference in New Issue
Block a user
Delete Branch "engram-tiered-storage"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Make the official engram build store-enabled (revert-proofing)
Background — the live store transition
The live engram (
:8742) now runs the paged store (neuron.egm+ WAL) insteadof the in-memory graph. The engine lives in
lang/runtime/engram_store.{c,h}and iswired into
lang/runtime/el_runtime.c(#include "engram_store.h",ENGRAM_STORE=1gated boot + structural-mutation hooks). Source is already committed on this branch
(
engram-tiered-storage, base commitbb64a23).The problem this PR fixes
The official CI could silently revert the store transition:
sdk-release.yamlpublishes onlyelc,el_runtime.c,el_runtime.has SDKrelease assets. It never publishes
engram_store.{c,h}.engram/.gitea/workflows/engram-release.yamldownloads onlyel_runtime.{c,h}from
RELEASE_BASEand links onlyel_runtime.cin the finalcc. It neverfetches or links
engram_store.c.So the next official engram release would compile
el_runtime.cwithoutengram_store.c, producing an in-memory binary — a silent regression away from thelive paged store.
Changes (CI recipe only — minimal, mirrors existing style)
.gitea/workflows/sdk-release.yaml(El SDK publish pipeline)engram_store.{c,h}into the SDK tarball (dist/sdk/runtime/).engram_store.candengram_store.has per-file release assets alongsideel_runtime.{c,h}— this is what downstream CI pulls fromRELEASE_BASE.engram/.gitea/workflows/engram-release.yaml(engram build)engram_store.c+engram_store.hinto/usr/local/lib/el/(mirrors theel_runtime.{c,h}downloads; the existing-I /usr/local/lib/elflag makes the#include "engram_store.h"inel_runtime.cresolve)./usr/local/lib/el/engram_store.cto thecclink line.No changes to any prod branch, no deployment touched, no builds run.
Related follow-up (not blocking for engram) — elc fold-hang fix
This branch also carries the elc fold-hang fix (
lang/elc.c,el-compiler/src/{parser,codegen}.el,elc-combined.el). That fix must reach theSDK's published
elcfor the SOUL build (soul importssessions.el, whichtriggers the fold hang). The ENGRAM build's
server.elfolds fine without it(no
sessions.elimport), so it does not block this engram revert-proofing —flagging it as a related item to land via the SDK re-release.
Remaining GATED steps (require Will's approval to complete official revert-proofing)
This PR only lands the CI recipe. To actually make official builds store-enabled,
two gated releases must run in order after this merges through
dev → stage → main:sdk-release.yamlonmain) solatestpublishesengram_store.{c,h}(+ the store-enabledel_runtime.{c,h}, and ideally thefold-hang-fixed
elc).engram-release.yaml) so it downloads + linksengram_store.cand produces the store-enabled binary.Until both run, the recipe is in place but the published artifacts are unchanged.
Guardrails observed
Feature branch + PR only. Targets
dev(integration branch;stage ← dev,main ← stage). No merge/push to prod branches, no force-push. Live deploymentand
~/.neuronuntouched. Noelcfold run locally.Working memory was thrashing behind a healthy-looking gauge. wm_active sat at 22-24 while breakthroughs ran 661-903 and evictions 485-717 PER 60s tick - roughly 825-1125 nodes cycling in 5-call lockstep. Root cause: the breakthrough path was an anti-starvation mechanism that reset its own counter on firing, with no budget and no refractory. A node failing its type threshold 5 times was force-promoted at exactly 0.10 and had its suppression_count reset to 0, so it immediately restarted the identical climb. Since BREAKTHROUGH_WEIGHT (0.10) > WM_FLOOR (0.05), every one of them cleared the admission floor and entered the rank contest tied at 0.10, where the tie-break degenerated to node-array index order. Cap-evicted nodes are skipped by retrieval reinforcement, so they never got an access_ts record and the STI inhibition-of-return damper never applied to them. That closed the loop: re-suppressed, completely unmarked, forever. An anti-starvation rule that resets its own counter without a bound is not a fairness valve, it is an oscillator. Fixes in engram_activate Pass 2: - ENGRAM_BREAKTHROUGH_BUDGET (WM_CAP/4 = 6) caps intrusive thoughts per call. - ENGRAM_BREAKTHROUGH_COOLDOWN (55) via NEGATIVE suppression_count. The field already serializes as %d and parses through eg_get_int_field, so negatives round-trip through snapshots with no struct or format change. - Blocked breakthroughs no longer reset the counter; it saturates so a starved node surfaces on a later call instead of restarting from zero. - Graded breakthrough weight by nearness to own threshold, so the rank tie-break is cognitive rather than insertion order. Invariant preserved: WM_FLOOR < weight < min(type_threshold). Also: moved the additive cosine term AFTER the STI multiplier. It was applied before, so an incumbent re-reached 30s later took t_n/(t_n+120) = 0.2x, which cut the semantic term's ceiling from 0.20 to 0.04 - below every per-type threshold. Meaning-match was being punished for having been recently useful. Inhibition-of-return should rotate the structural score, not the semantic one. Also: _eg_act_wm_evicted counted 3 of 5 eviction paths. The two carry-over paths were silent, so the reported rate was an undercount of unknown magnitude - while being used to diagnose an eviction pathology. All five now increment. Also: route_sync returned {"nodes":[],"edges":[]} when the snapshot export failed. The soul's sync_ok check only tests for "" and "{}", so that placeholder passed as a healthy sync: last_sync_ok_ts stamped, sync_age_ms green, sync_empty never fired, added:0 forever. A broken sync was indistinguishable from a quiet healthy one - the exact class this route was added to fix. Returns a real error now. Verified live (boot 20 vs boot 19): breakthroughs 661-903 -> 36/tick, evictions 485-717 -> 12-46/tick against a counter that now covers more paths, wm_active unchanged at 22-24, wm_avg_weight 0.138-0.273 -> 0.186-0.446. Working memory is holding strong nodes instead of breakthrough-floor filler.The soul's curiosity auto-term extractor takes the first word of a top-WM node label. It has no term-quality scoring, so three prior self-reviews each bolted on another hand-curated blocklist (genre words 07-23, quoted titles 07-25, stopwords 07-30). Every one was written reactively, after a flood was already observed. A list can only contain floods that already happened. Two were in flight and unfixed when this review ran: "<!--" label df 220 -> 252 nodes activated "SELF" label df 175 -> 541 nodes activated (list has "Self" Title-case; str_eq is case-sensitive, so the uppercase token sailed through) engram_label_df(term) counts nodes whose label contains term. Low-specificity tokens are corpus-frequent by definition, so this catches the flood class prospectively and tracks the corpus as the world-ingestor changes it. This is Sparck Jones (1972), which introduced IDF under the name 'term specificity'; automatic stopword compilation from it is the textbook application. NOT a replacement for the stopword list -- verified against all 86 listed terms, not assumed. Catches 13 (Will:306, Self:175, Over:116, Knowledge:112), misses 73 (Whose:0, Would:0, Could:0, This:9). Labels are terse titles, so English function words are genuinely rare in them. The gates cover disjoint failure modes; both are required. Policy lives in awareness.el, not here: the runtime measures, the soul decides.route_load was a stub response over the most destructive operation in the server: engram_load resets the store before parsing, so a readable-but- malformed snapshot left a hollow graph and the route answered {"ok":true}. With 37GB of stale dated snapshots in the data dir as restore targets, that is a live risk. Now returns the real return value plus node/edge counts and an explicit hollow flag. route_save discarded engram_save's return the same way; persist_canonical returned a hardcoded 1, making 'let saved: Int = persist_canonical()' a dead variable at six durable write paths.Self-contained paged store (lang/runtime/engram_store.{c,h}): 16KiB slotted pages, u32 TLV self-describing records (forward-compatible), overflow chains, B+-tree id-index + from/to adjacency, page free-list, tombstones, double superblock + crc recovery. Not yet wired to activation (M3). 33/33 tests pass (ASan/UBSan clean); 5k nodes/20k edges round-trip bit-exact incl 768xf32 emb + hebb; store 25MB vs 64MB JSON. Format is final — see design §2.4.The live engram now runs the paged store (neuron.egm+WAL), but the SDK release publishes only el_runtime.{c,h} and the engram build links only el_runtime.c — so a future official release would silently revert to the in-memory store. Publish engram_store.{c,h} as SDK release assets and add them to the engram build's download + cc link so the store transition cannot regress.