store_scan_nodes/store_scan_edges deduplicated emitted records by their
64-bit id_hash (FNV-1a-64) rather than the full id string. Two distinct ids
that collide under id_hash emitted only the first; the second was durably on
a live page and findable by store_get_node (which disambiguates by strcmp),
yet silently dropped from the resident boot-load. After any store reopen that
node was unretrievable by id, absent from lexical search, and missing from the
recent list — the reported memory-integrity gap.
Replace the hash-keyed U64Set with a StrSet: bucket by id_hash for O(1) probing
but compare full ids by strcmp, mirroring the primary B+-tree readers. Same
change for edges. Adds test_scan_collision.c (real FNV-1a-64 colliding ids).
Reclaims space held by dead records (tombstoned prune/forget nodes, superseded
ids, stale re-put/hebb versions, and their orphaned overflow chains). On-disk
format UNCHANGED — pure behavior.
COMPACTION (store_compact): copy-live + atomic-swap.
A. checkpoint/sync to quiesce (WAL reduced to CHECKPOINT{C}); crash here => pre.
B. build <path>.compact with only the live records, re-placed bit-exact into
fresh densely-packed pages + fresh id/adjacency B+-trees, every page stamped
LSN=C, new SB last_checkpoint_lsn=C; fsync. crash here => pre-compaction.
C. rename(<path>.compact -> <path>) — POSIX-atomic commit; crash after => post.
D. reopen in place: swap fd, INVALIDATE every pool frame (M4 remap of relocated
pages), reload SB, re-autopin.
Crash at any instant recovers to pre- OR post-compaction, never a corrupt mix.
Single-threaded => "online" = safe between mutations; takes a checkpoint quiesce
at entry. M4 cooperation: temp build has its own pool honoring ENGRAM_POOL_FRAMES
(evict/re-fault + no-steal + pins); live pool fully invalidated on reopen.
BACKGROUND CHECKPOINTER: ckpt_maybe now fires on ANY armed trigger — ops (default
100000), dirty pool frames, WAL bytes-since-reclaim (default 64 MiB), or a
wall-clock interval (checked on the write path; no extra thread). Same M2
checkpoint semantics (calls engram_checkpoint). Env: ENGRAM_CKPT_OPS/_DIRTY/
_WAL_BYTES/_INTERVAL_MS; runtime setter store_set_checkpoint_policy().
Tests: engram/test/test_compaction.c (+runner). Plain gcc, ASan/UBSan clean.
36 passed, 0 failed (O2 and ASan+UBSan builds).
reclaim: page_count 655 -> 153, file 10731520 -> 2506752 bytes (76.6% reclaimed),
every live record + adjacency bit-exact at new locations.
crash-during-compaction phases 0/1/2: crc clean, live set intact, writable.
background checkpointer: ops / WAL-bytes / dirty triggers each auto-fire; WAL
prefix reclaimed (30 B after 600 puts); recovery correct.
pool cooperation: compact under 24-frame pool correct, no stale frames.
No regression: M1 33, M2 36, M3 parity, M3.5, M4 bufpool 37 — all green.
On-disk format unchanged (additive).
Turn M2's write-back/no-steal cache into a bounded, demand-paged buffer pool so
the paged store can exceed RAM while keeping only hot pages resident. On-disk
format UNCHANGED (additive residency only; no migration). Default budget is large
enough that today's store stays fully resident, so default behaviour == Phase 1.
- Frame table capped at `cap` frames (env ENGRAM_POOL_FRAMES; 0 = unlimited;
default 1<<20). Not-resident access faults in from neuron.egm.
- LRU eviction of CLEAN, unpinned frames only. Dirty frames are never stolen
(M2 no-steal / WAL durability preserved) — turned evictable by a checkpoint's
pc_flush, which then trims the pool back to budget.
- Pinning: superblocks (0,1) + index root/interior pages auto-pinned; explicit
store_pin_page/unpin and store_pin_layer/unpin (hot WM/core layers).
- Bounded sequential read-ahead on scans (env ENGRAM_PREFETCH, default 8).
- Correctness rests on callers copying page bytes into local buffers and never
retaining a frame pointer across another access, so evict+re-fault is safe.
Gates (plain gcc, ASan/UBSan clean):
M4 run_bufpool_tests.sh ...... 37 passed, 0 failed (+ ASan/UBSan: 37/0)
small-pool round-trip (cap=32 vs 1599 pages, 2708 evictions): 5000 nodes +
4000 sampled edges bit-exact, crc clean, pool bounded to cap.
eviction: hot set 0 re-faults, cold evicted, hit-rate 0.989; no-steal burst
(cap=8) holds 309 dirty frames > cap, reads correct from dirty pages.
pinning: superblocks/roots/explicit page/hot-layer(19 pages) stay resident;
unpin makes them evictable.
prefetch: sequential scan 511 demand-faults OFF -> 4 ON.
crash-under-paging (ENGRAM_POOL_FRAMES=16): WAL replay + checkpoint-crash
phases 0-4 all recover bit-exact.
default pool: 0 evictions, whole store resident (== Phase 1).
No regression: M1 33/0, M2 36/0, M3 parity PASS, M3.5 PASS.
Flag-on checkpoint now full-walks the resident graph: store_put_node (WM weight,
activation_count, last_activated, wm_anchor) + store_put_edge (hebb, last_fired)
for every node/edge, then engram_checkpoint. Uses store_put_edge (idempotent
upsert) not store_hebb_batch, because activation FORMS new hebbian-associate edges
that bypass the create hook and delta-only hebb_batch can't create them. Store-on
boot now applies the same WM-halving + floor + cap transforms as engram_load.
This is the hebb-survives-restart fix. Gate: reboot from neuron.egm with
snapshot.json deleted -> edge hebb + activation_count survive unchanged, WM weight
survives with identical boot transform; negative control proves persist is
load-bearing (hebb->0 without it). M1 33/33 + M2 36/36 + M3 parity PASS, ASan/UBSan
clean, flag-off untouched. Engine unchanged (boundary held).
Caller-side shim in el_runtime.c maps EngramNode/Edge <-> StoreNode/Edge; engine
keeps zero soul deps (libengram boundary, design §10). Flag off = today's JSON
path byte-for-byte (proven: no neuron.egm created, graph identical). Flag on =
engram_open (import snapshot.json once into neuron.egm, else WAL-replay) +
resident load; node/edge create + forget dual-write via guarded hooks. Files
renamed engram.store->neuron.egm, engram.wal->neuron.wal.
Gate: M3 parity PASS (graph on==off byte-exact modulo ordering; snapshot round-trip;
reboot-from-egm with snapshot.json deleted; activation set+sequence identical;
ASan/UBSan clean). M1 33/33 + M2 36/36 green post-rename.
Known gap (pre-flip): in-place hebb/WM/activation_count updates during activation
are not yet persisted to the store (create/connect/forget are). Must close before
live flip so learned edges survive restart.
Write-back no-steal buffer pool makes the fsync'd WAL load-bearing (M1 was
write-through). Logical WAL with record-granularity page-LSN redo idempotency.
Checkpoint = flush dirty pages, fsync store, advance last_checkpoint_lsn,
reclaim WAL prefix. One-time snapshot.json import only when store absent;
JSON never read as the ongoing store thereafter.
Gates: 33/33 M1 (no regression) + 36/36 M2 — replay parity, torn-tail fuzz
(every byte offset), checkpoint-crash at all 5 phases, torn-page+WAL redo,
legacy-import parity, hebb-survives-crash.
Establish lang/runtime/ as the ONE canonical el runtime (from the active
runtime that carries hebb/emb persistence + the new WAL); repoint the el CI
publish, engram build, elb default, and in-repo build scripts to it; delete
the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh
drift guard.
Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING
el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian
edge weights — learned co-activation was wiped on every restart. Publishing
from canonical ships the stranded 'learning that cannot outlive the process'
fix.
WAL storage engine + integrity fixes (DELETE->tombstone + store-layer
protection, safe data-dir default) ride in behind ENGRAM_WAL (default off =
byte-identical to today). Verified: engram elb per-module build clean, WAL
gate 66/66, native smoke ok, drift-guard green.
will.anderson