Archived
promote dev -> stage: engram tiered-storage + WAL + self-review fixes (2026-08-15) #96
No Assignees
No due date set.
Dependencies
No dependencies set.
Reference: neuron-technologies/el-retired#96
Reference in New Issue
Block a user
Delete Branch "dev"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Advance stage to dev. Linear: stage is an ancestor of dev (31 commits: engram tiered storage M1-M3.5, WAL persistence, self-review batch). Fast-forward, nothing lost.
Working memory was thrashing behind a healthy-looking gauge. wm_active sat at 22-24 while breakthroughs ran 661-903 and evictions 485-717 PER 60s tick - roughly 825-1125 nodes cycling in 5-call lockstep. Root cause: the breakthrough path was an anti-starvation mechanism that reset its own counter on firing, with no budget and no refractory. A node failing its type threshold 5 times was force-promoted at exactly 0.10 and had its suppression_count reset to 0, so it immediately restarted the identical climb. Since BREAKTHROUGH_WEIGHT (0.10) > WM_FLOOR (0.05), every one of them cleared the admission floor and entered the rank contest tied at 0.10, where the tie-break degenerated to node-array index order. Cap-evicted nodes are skipped by retrieval reinforcement, so they never got an access_ts record and the STI inhibition-of-return damper never applied to them. That closed the loop: re-suppressed, completely unmarked, forever. An anti-starvation rule that resets its own counter without a bound is not a fairness valve, it is an oscillator. Fixes in engram_activate Pass 2: - ENGRAM_BREAKTHROUGH_BUDGET (WM_CAP/4 = 6) caps intrusive thoughts per call. - ENGRAM_BREAKTHROUGH_COOLDOWN (55) via NEGATIVE suppression_count. The field already serializes as %d and parses through eg_get_int_field, so negatives round-trip through snapshots with no struct or format change. - Blocked breakthroughs no longer reset the counter; it saturates so a starved node surfaces on a later call instead of restarting from zero. - Graded breakthrough weight by nearness to own threshold, so the rank tie-break is cognitive rather than insertion order. Invariant preserved: WM_FLOOR < weight < min(type_threshold). Also: moved the additive cosine term AFTER the STI multiplier. It was applied before, so an incumbent re-reached 30s later took t_n/(t_n+120) = 0.2x, which cut the semantic term's ceiling from 0.20 to 0.04 - below every per-type threshold. Meaning-match was being punished for having been recently useful. Inhibition-of-return should rotate the structural score, not the semantic one. Also: _eg_act_wm_evicted counted 3 of 5 eviction paths. The two carry-over paths were silent, so the reported rate was an undercount of unknown magnitude - while being used to diagnose an eviction pathology. All five now increment. Also: route_sync returned {"nodes":[],"edges":[]} when the snapshot export failed. The soul's sync_ok check only tests for "" and "{}", so that placeholder passed as a healthy sync: last_sync_ok_ts stamped, sync_age_ms green, sync_empty never fired, added:0 forever. A broken sync was indistinguishable from a quiet healthy one - the exact class this route was added to fix. Returns a real error now. Verified live (boot 20 vs boot 19): breakthroughs 661-903 -> 36/tick, evictions 485-717 -> 12-46/tick against a counter that now covers more paths, wm_active unchanged at 22-24, wm_avg_weight 0.138-0.273 -> 0.186-0.446. Working memory is holding strong nodes instead of breakthrough-floor filler.The soul's curiosity auto-term extractor takes the first word of a top-WM node label. It has no term-quality scoring, so three prior self-reviews each bolted on another hand-curated blocklist (genre words 07-23, quoted titles 07-25, stopwords 07-30). Every one was written reactively, after a flood was already observed. A list can only contain floods that already happened. Two were in flight and unfixed when this review ran: "<!--" label df 220 -> 252 nodes activated "SELF" label df 175 -> 541 nodes activated (list has "Self" Title-case; str_eq is case-sensitive, so the uppercase token sailed through) engram_label_df(term) counts nodes whose label contains term. Low-specificity tokens are corpus-frequent by definition, so this catches the flood class prospectively and tracks the corpus as the world-ingestor changes it. This is Sparck Jones (1972), which introduced IDF under the name 'term specificity'; automatic stopword compilation from it is the textbook application. NOT a replacement for the stopword list -- verified against all 86 listed terms, not assumed. Catches 13 (Will:306, Self:175, Over:116, Knowledge:112), misses 73 (Whose:0, Would:0, Could:0, This:9). Labels are terse titles, so English function words are genuinely rare in them. The gates cover disjoint failure modes; both are required. Policy lives in awareness.el, not here: the runtime measures, the soul decides.route_load was a stub response over the most destructive operation in the server: engram_load resets the store before parsing, so a readable-but- malformed snapshot left a hollow graph and the route answered {"ok":true}. With 37GB of stale dated snapshots in the data dir as restore targets, that is a live risk. Now returns the real return value plus node/edge counts and an explicit hollow flag. route_save discarded engram_save's return the same way; persist_canonical returned a hardcoded 1, making 'let saved: Int = persist_canonical()' a dead variable at six durable write paths.Self-contained paged store (lang/runtime/engram_store.{c,h}): 16KiB slotted pages, u32 TLV self-describing records (forward-compatible), overflow chains, B+-tree id-index + from/to adjacency, page free-list, tombstones, double superblock + crc recovery. Not yet wired to activation (M3). 33/33 tests pass (ASan/UBSan clean); 5k nodes/20k edges round-trip bit-exact incl 768xf32 emb + hebb; store 25MB vs 64MB JSON. Format is final — see design §2.4.The live engram now runs the paged store (neuron.egm+WAL), but the SDK release publishes only el_runtime.{c,h} and the engram build links only el_runtime.c — so a future official release would silently revert to the in-memory store. Publish engram_store.{c,h} as SDK release assets and add them to the engram build's download + cc link so the store transition cannot regress.