store: extend the write barrier to edges — kills the full-store walk #128
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: neuron-technologies/el#128
Reference in New Issue
Block a user
Delete Branch "fix/elc-rebuildable-compiler-builtins"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This took the live engram down twice tonight. Root cause is not the cache size.
Checkpointing pushes the entire resident graph through
store_put_node/store_put_edge. Nodes were cheap — a durable-hash compare skipped unchanged records with zero page I/O. Edges had no barrier at all (the struct comment literally readsENGRAM_WRITE_BARRIER: node durable-hash barrier), so every edge was rewritten on every checkpoint, each runningmax_page_lsn_for_id→ btree lookup →page_read.Edges outnumber nodes ~3:1 (37,663 vs 13,436), so a routine checkpoint became a full-store walk in id order — random page access across 2 GiB, mostly to rediscover nothing had changed. LRU is worst-case under exactly that pattern: it evicts the page it is about to want. Once the page cache was smaller than the store, the walk collapsed into thrashing — 100% CPU, flat RSS, no progress, port never bound.
The walk is the defect. Sizing the cache to survive it treats the symptom.
dh_edge_hash()— edge counterpart ofdh_node_hash, with a kind-discriminator byte so an edge can never collide with a node of the same id.created_at/updated_at/last_firedexcluded deliberately:last_firedis touched by activation without changing what the edge is, and folding it in would defeat the barrier on precisely the hot edges that need it most.store_put_edge()— barrier check +dh_seton success, mirroringstore_put_node.store_scan_edges()— seed the barrier map from disk at load, so the first post-boot checkpoint already skips.store_scan_nodesalready did this and its comment says why; edges were never done.Verified with the exact config that killed production (
ENGRAM_POOL_FRAMES=65536→ 1 GiB cache vs 2 GiB store): boots clean and serves — LISTENING, 13,436 nodes / 37,663 edges, embeddings complete, 0.0% CPU, RSS 1.14 GiB (cache resting at budget instead of thrashing against it). Same small cache, same store, no walk.Also in this branch: frame budget derived from physical RAM instead of a hardcoded constant (a constant is wrong in both directions — 16 GiB of frames is arbitrary on a 48 GB host and suicidal on a 16 GB one), and a thrash detector that prints eviction/hit counts and the remedy, because thrashing was indistinguishable from "busy loading" and that ambiguity cost hours.
Checkpointing pushes the ENTIRE resident graph through store_put_node and store_put_edge (see engram_store_checkpoint). Nodes were cheap: a durable-hash compare skipped unchanged records with zero page I/O. Edges had no barrier at all — struct comment at PgCache.barrier_on even says "node durable-hash barrier" — so every edge was rewritten on every checkpoint, and each rewrite runs the idempotency probe max_page_lsn_for_id -> btree lookup -> page_read. Edges outnumber nodes ~3:1 here (37,663 vs 13,436), so routine checkpointing degenerated into a FULL-STORE WALK in id order: random page access across the whole 2 GiB store, repeated, overwhelmingly to rediscover nothing had changed. LRU is worst-case under exactly that pattern — it evicts the page it is about to want — so once the page cache was smaller than the store, the walk collapsed into thrashing: 100% CPU, flat RSS, no forward progress, port never bound. That took the live engram down twice on 2026-08-15. The walk is the defect. Sizing the cache to survive it treats the symptom. Changes: - dh_edge_hash(): edge counterpart of dh_node_hash, with a kind discriminator byte so an edge can never collide with a node of the same id in the shared map. created_at/updated_at/last_fired are excluded deliberately: last_fired is touched by activation without changing what the edge IS, and folding it in would defeat the barrier on precisely the hot edges that most need it. - store_put_edge(): barrier check + dh_set on success, mirroring store_put_node exactly. - store_scan_edges(): seed the barrier map from on-disk truth at load, so the FIRST post-boot checkpoint already skips unchanged edges. store_scan_nodes already did this and its comment says why; edges were simply never done. Verified: with the exact configuration that killed production (ENGRAM_POOL_FRAMES=65536 -> 1 GiB cache against a 2 GiB store), the engram now boots clean and serves — LISTENING, 13,436 nodes / 37,663 edges, embeddings complete, 0.0% CPU, RSS 1.14 GiB (cache resting at its budget rather than thrashing against it). Same small cache, same store, no walk.Follow-on to the edge write barrier. That fix removed the full-store walk; this one makes the pool able to notice if anything like it happens again. WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill twice on 2026-08-15 and said nothing. From outside it was indistinguishable from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each costing a deploy or a rollback. The whole time, hits/misses/evictions were already being counted in PgCache, and the struct comment read: /* stats (introspection only — never affect semantics) */ That comment was the bug. Self-measurement treated as decoration is why the pool could not correct itself and why no one outside could see what it was doing. A system that cannot read its own state cannot correct, and neither can anyone watching it. - pc_adapt_budget(): the loop, closed. Over a sliding window, evictions running at a large fraction of accesses WHILE reuse is real means the working set exceeds the budget — so grow it, geometrically, bounded by a LIVE re-read of physical memory. Evictions alone are not pressure (a scan evicts and never returns); evictions with reuse are. An explicit ENGRAM_POOL_FRAMES still wins — an operator override must not be silently overruled. - Budget derived, not declared. A constant cannot be right: 16 GiB of frames is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM at startup" is a guess about the future — it cannot know the store grew or the machine changed. Hence the live re-read. - pc_report(): ONE structured emission carrying the entire sensed state, through emit_log — El's existing telemetry, already exporting to OTLP. Deliberately not a function per stat, and deliberately not a bespoke /api/pool endpoint: both make observability something hand-written per noun instead of the uniform mechanism every component already has. - engram_pool_stats_json(): the same state readable live, wired through the normal builtin path (codegen arity + el_seed wrapper), so the pool can be observed in real time rather than reconstructed afterward from a stack sample. Verified: with the exact configuration that took production down (ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete — and NO pressure event fires, because the barrier removed the walk that caused it. The controller is defense in depth; the barrier is the fix.