The nomic-embed-text space over the corpus is strongly anisotropic (mean
pairwise cosine ~0.55), which compresses cosine-based domain separation almost
to nothing so the design-doc s5 operators (distance/overlap/Wasserstein) cannot
discriminate. Subtracting the global mean of the normalized embeddings restores
isotropy (mean pairwise cosine ~0) and sharpens the operators.
- add GeoMeanCache (engram_geo_mean_build / _maybe_refresh / _vec / _free): a
store-derived centering offset over the embed-eligible set, cached and
refreshed on significant drift; lives in geometry.c, not the store.
- engram_geometry_descriptor gains an optional global_mean: when supplied the
centroid, per-member cosine distance, and co-registration run in centered
space (GM=zeros reproduces the legacy raw path exactly).
- co-registration choice (b): the ANN query stays in raw unit space (index
unchanged) since centering is a rigid translation that ~preserves neighborhood
membership; only the descriptor statistics move to the centered frame.
Covariance/axes/radius are translation-invariant and therefore unchanged.
- test: synthetic ground-truth suite stays green (PERF + ASan/UBSan), plus new
centered/raw/mean-cache assertions.
- add bench_discrimination.c (env-gated, read-only, skips in CI): on a copy of
the real store the two-domain overlap operator drops 1.13 -> 0.008 and
cross-centroid cosine 0.899 -> 0.003 after centering, Euclid distance
unchanged (translation-invariant control).
No change to activation/retrieval behavior; wiring geometry into retrieval is a
separate, behavior-changing cutover.
The stone the operator/drift/occupation work stands on: express a relational
neighborhood as the compact joint geometry Will specified (design §3/§5, node
e94371bd) — semantic side (centroid, principal-axis ellipsoid via dual-PCA,
radius) braided with the relational side (k-core skeleton, hub->periphery
centrality gradient), plus soft membership and a co-registration diagnostic
(corr of hebb strength vs semantic proximity — >0 reifies, <0 flags dreams).
Built ONLY on the two standalone modules — engram_vindex (ANN, the cloud) and
engram_store (embeddings + hebb adjacency, the skeleton). Pure C11 + libm; does
not link or touch el_runtime.c. Strictly READ-ONLY: never mutates nodes, edges,
activation, the index, or any retrieval path. Not yet wired into retrieval —
foundation only.
Self-contained test (test_geometry.c) synthesizes two known embedding clusters
with intra-cluster hebb edges and verifies the descriptor recovers the shape:
centroid on the seeded cluster, hub = relational center, skeleton = the strong
intra-cluster wiring, positive co-registration, sorted axis extents. PERF +
ASan/UBSan passes both green; needs no live data.
int_max_keys() computed an internal B+tree node's key capacity as
IDX_BODY/8 - 1 (2041 at a 16 KB page), dividing by 8 and ignoring that
each key also carries an 8-byte child pointer. The true capacity is
(IDX_BODY-8)/16 = 1020. An internal node was therefore allowed to grow to
~2x what a page holds; once it crossed 1020 keys, btree_insert's write-back
overran its STORE_PAGE_SIZE stack page buffer and smashed the stack canary
(__stack_chk_fail / SIGABRT). A clean/small store never grows an internal
node that large, so it never tripped; the ~8x-bloated live store (a day of
tombstone churn) plus a 44 MB un-checkpointed WAL replayed on open pushed a
node over the boundary during redo -> deterministic crash loop
(btree_insert <- apply_edge_put <- engram_open <- engram_store_boot).
Fixes:
- int_max_keys: use (IDX_BODY-8)/16 so internal nodes split at the real
page capacity.
- btree_insert: reject any page whose on-disk nkeys exceeds physical
capacity (fail loud, never smash the stack) -- overflow is now impossible
regardless of on-disk content.
- read_body: bound the slot (off,len) and record length to the page before
dereferencing; a stale/torn index entry could otherwise make store_get_node
read off the stack (observed EXC_BAD_ACCESS on the bloated store). Fail safe.
Verified on a COPY of the live store: unfixed binary SIGABRTs in btree_insert
on open; fixed binary boots clean, recovers the store, checkpoints the WAL,
and M5 compaction shrinks 458 MB -> 57.7 MB with node/edge counts preserved.
Persistent process-lifetime HNSW index (engram_vindex) over resident node
embeddings, node_id == resident g->nodes[] index. Built lazily on first
activation, grown incrementally as appended nodes get embedded, rebuilt on
emb-dim change or resident-array shrink.
Seed selection queries the ANN for the nearest embedded nodes to the effective
query vector, replacing the O(K*N) exact argmax scan as the DISCOVERY step.
Each ANN candidate is admitted through the identical gate the exact scan uses
(exact cosine >= SEED_MIN via cosq, reached/dup skips, content dedup, same
decay/dampen shaping), so scoring/dynamics are unchanged. The exact argmax
scan is preserved verbatim and tops up any unfilled seed slot, and runs in
full when the index is empty/too-small/unavailable -> pre-M8 behaviour exactly.
cosq (O(N*dim) cosine fill) is intentionally retained: it still feeds the
propagation qgate and the Pass-2 WM term (activation dynamics, out of M8
scope). ANN accelerates SELECTION only.
store_scan_nodes/store_scan_edges deduplicated emitted records by their
64-bit id_hash (FNV-1a-64) rather than the full id string. Two distinct ids
that collide under id_hash emitted only the first; the second was durably on
a live page and findable by store_get_node (which disambiguates by strcmp),
yet silently dropped from the resident boot-load. After any store reopen that
node was unretrievable by id, absent from lexical search, and missing from the
recent list — the reported memory-integrity gap.
Replace the hash-keyed U64Set with a StrSet: bucket by id_hash for O(1) probing
but compare full ids by strcmp, mirroring the primary B+-tree readers. Same
change for edges. Adds test_scan_collision.c (real FNV-1a-64 colliding ids).
Spreading activation rebuilt the entire per-node adjacency index
(engram_adj_rebuild, O(E)) lazily before every BFS whenever any edge/node
was added — so a curiosity-loop query that touched a small frontier still
paid to rebuild the whole edge set. This makes the index incrementally
maintained behind ENGRAM_STORE: single node/edge creates APPEND to the live
adjacency in amortized O(1) instead of marking it dirty, so a query only
pays for the frontier it touches (one initial O(E) build, then O(1)/edge).
Approach (b), not (a): the store's from/to adjacency B-tree was rejected
because with the store on the whole graph is already resident and activation
reads in-RAM edges, whose hebb/weight only sync to the store at checkpoint
cadence — reading StoreEdge copies would use stale weights and break
byte-identical parity. The incremental in-RAM index reads the exact same
g->edges[ei] the scan path does, so activation is identical by construction.
Correctness: edges are only ever appended, so incremental append reproduces
the rebuild's ascending-edge-index ordering exactly (same skip rule for null
endpoints). Any index-invalidating mutation (forget/prune/clear) still frees
the index + sets adj_dirty=1, falling back to a full rebuild. Flag-off is
untouched: the mutation hooks just set adj_dirty=1 as before — proven
byte-identical.
engram_store.c is NOT modified (avoids the M5 compaction collision).
Tests (plain gcc, ASan/UBSan clean): engram/test/{test_m7_traversal.c,
run_m7_traversal.sh}. Parity gate proves flag-on incremental == flag-on
forced-full-rebuild == flag-off scan, byte-identical on a mutating query
sequence (activated set, weights, ordering, hops, WM promotion). Perf on a
13k-node / 43k-edge graph over 120 (add-edge + activate) iterations:
adjacency edge-touches 5,167,260 -> 43,001 (120x fewer), full rebuilds
120 -> 1, adjacency-maintenance wall-time 0.74s -> 0.006s (~121x). Prior
gates green: M1 store (33), M2 (36), M3 parity, M3.5, M4 bufpool (37).
Reclaims space held by dead records (tombstoned prune/forget nodes, superseded
ids, stale re-put/hebb versions, and their orphaned overflow chains). On-disk
format UNCHANGED — pure behavior.
COMPACTION (store_compact): copy-live + atomic-swap.
A. checkpoint/sync to quiesce (WAL reduced to CHECKPOINT{C}); crash here => pre.
B. build <path>.compact with only the live records, re-placed bit-exact into
fresh densely-packed pages + fresh id/adjacency B+-trees, every page stamped
LSN=C, new SB last_checkpoint_lsn=C; fsync. crash here => pre-compaction.
C. rename(<path>.compact -> <path>) — POSIX-atomic commit; crash after => post.
D. reopen in place: swap fd, INVALIDATE every pool frame (M4 remap of relocated
pages), reload SB, re-autopin.
Crash at any instant recovers to pre- OR post-compaction, never a corrupt mix.
Single-threaded => "online" = safe between mutations; takes a checkpoint quiesce
at entry. M4 cooperation: temp build has its own pool honoring ENGRAM_POOL_FRAMES
(evict/re-fault + no-steal + pins); live pool fully invalidated on reopen.
BACKGROUND CHECKPOINTER: ckpt_maybe now fires on ANY armed trigger — ops (default
100000), dirty pool frames, WAL bytes-since-reclaim (default 64 MiB), or a
wall-clock interval (checked on the write path; no extra thread). Same M2
checkpoint semantics (calls engram_checkpoint). Env: ENGRAM_CKPT_OPS/_DIRTY/
_WAL_BYTES/_INTERVAL_MS; runtime setter store_set_checkpoint_policy().
Tests: engram/test/test_compaction.c (+runner). Plain gcc, ASan/UBSan clean.
36 passed, 0 failed (O2 and ASan+UBSan builds).
reclaim: page_count 655 -> 153, file 10731520 -> 2506752 bytes (76.6% reclaimed),
every live record + adjacency bit-exact at new locations.
crash-during-compaction phases 0/1/2: crc clean, live set intact, writable.
background checkpointer: ops / WAL-bytes / dirty triggers each auto-fire; WAL
prefix reclaimed (30 B after 600 puts); recovery correct.
pool cooperation: compact under 24-frame pool correct, no stale frames.
No regression: M1 33, M2 36, M3 parity, M3.5, M4 bufpool 37 — all green.
On-disk format unchanged (additive).
Turn M2's write-back/no-steal cache into a bounded, demand-paged buffer pool so
the paged store can exceed RAM while keeping only hot pages resident. On-disk
format UNCHANGED (additive residency only; no migration). Default budget is large
enough that today's store stays fully resident, so default behaviour == Phase 1.
- Frame table capped at `cap` frames (env ENGRAM_POOL_FRAMES; 0 = unlimited;
default 1<<20). Not-resident access faults in from neuron.egm.
- LRU eviction of CLEAN, unpinned frames only. Dirty frames are never stolen
(M2 no-steal / WAL durability preserved) — turned evictable by a checkpoint's
pc_flush, which then trims the pool back to budget.
- Pinning: superblocks (0,1) + index root/interior pages auto-pinned; explicit
store_pin_page/unpin and store_pin_layer/unpin (hot WM/core layers).
- Bounded sequential read-ahead on scans (env ENGRAM_PREFETCH, default 8).
- Correctness rests on callers copying page bytes into local buffers and never
retaining a frame pointer across another access, so evict+re-fault is safe.
Gates (plain gcc, ASan/UBSan clean):
M4 run_bufpool_tests.sh ...... 37 passed, 0 failed (+ ASan/UBSan: 37/0)
small-pool round-trip (cap=32 vs 1599 pages, 2708 evictions): 5000 nodes +
4000 sampled edges bit-exact, crc clean, pool bounded to cap.
eviction: hot set 0 re-faults, cold evicted, hit-rate 0.989; no-steal burst
(cap=8) holds 309 dirty frames > cap, reads correct from dirty pages.
pinning: superblocks/roots/explicit page/hot-layer(19 pages) stay resident;
unpin makes them evictable.
prefetch: sequential scan 511 demand-faults OFF -> 4 ON.
crash-under-paging (ENGRAM_POOL_FRAMES=16): WAL replay + checkpoint-crash
phases 0-4 all recover bit-exact.
default pool: 0 evictions, whole store resident (== Phase 1).
No regression: M1 33/0, M2 36/0, M3 parity PASS, M3.5 PASS.
Resolve 3 conflicts:
- lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork;
single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh).
- lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated
artifact folder, not a source path; a release is a git tag, not a folder).
- lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval
wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL.
Nothing in dev's build consumes the deprecated fork or releases/ folder.
- Wire paged store into the ENGRAM SERVICE (server.el, the authoritative durable
owner): boot->engram_store_boot, persist_canonical->engram_store_checkpoint,
gated by ENGRAM_STORE.
- elc (lang/elc.c + src/parser.el + codegen.el + elc-combined.el): OOB guard in
tok_kind/tok_value + parse_block progress backstop — fixes the pre-existing
unbounded-memory fold hang on sessions.el.
- engram_prune_telemetry mirrors ISE prune to the store (store_forget) so store
live-count tracks resident and stale telemetry stays bounded.
- Deployed live 2026-08-12: engram :8742 on neuron.egm+WAL, count reconciled 11552.
Flag-on checkpoint now full-walks the resident graph: store_put_node (WM weight,
activation_count, last_activated, wm_anchor) + store_put_edge (hebb, last_fired)
for every node/edge, then engram_checkpoint. Uses store_put_edge (idempotent
upsert) not store_hebb_batch, because activation FORMS new hebbian-associate edges
that bypass the create hook and delta-only hebb_batch can't create them. Store-on
boot now applies the same WM-halving + floor + cap transforms as engram_load.
This is the hebb-survives-restart fix. Gate: reboot from neuron.egm with
snapshot.json deleted -> edge hebb + activation_count survive unchanged, WM weight
survives with identical boot transform; negative control proves persist is
load-bearing (hebb->0 without it). M1 33/33 + M2 36/36 + M3 parity PASS, ASan/UBSan
clean, flag-off untouched. Engine unchanged (boundary held).
Caller-side shim in el_runtime.c maps EngramNode/Edge <-> StoreNode/Edge; engine
keeps zero soul deps (libengram boundary, design §10). Flag off = today's JSON
path byte-for-byte (proven: no neuron.egm created, graph identical). Flag on =
engram_open (import snapshot.json once into neuron.egm, else WAL-replay) +
resident load; node/edge create + forget dual-write via guarded hooks. Files
renamed engram.store->neuron.egm, engram.wal->neuron.wal.
Gate: M3 parity PASS (graph on==off byte-exact modulo ordering; snapshot round-trip;
reboot-from-egm with snapshot.json deleted; activation set+sequence identical;
ASan/UBSan clean). M1 33/33 + M2 36/36 green post-rename.
Known gap (pre-flip): in-place hebb/WM/activation_count updates during activation
are not yet persisted to the store (create/connect/forget are). Must close before
live flip so learned edges survive restart.
Write-back no-steal buffer pool makes the fsync'd WAL load-bearing (M1 was
write-through). Logical WAL with record-granularity page-LSN redo idempotency.
Checkpoint = flush dirty pages, fsync store, advance last_checkpoint_lsn,
reclaim WAL prefix. One-time snapshot.json import only when store absent;
JSON never read as the ongoing store thereafter.
Gates: 33/33 M1 (no regression) + 36/36 M2 — replay parity, torn-tail fuzz
(every byte offset), checkpoint-crash at all 5 phases, torn-page+WAL redo,
legacy-import parity, hebb-survives-crash.
Establish lang/runtime/ as the ONE canonical el runtime (from the active
runtime that carries hebb/emb persistence + the new WAL); repoint the el CI
publish, engram build, elb default, and in-repo build scripts to it; delete
the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh
drift guard.
Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING
el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian
edge weights — learned co-activation was wiped on every restart. Publishing
from canonical ships the stranded 'learning that cannot outlive the process'
fix.
WAL storage engine + integrity fixes (DELETE->tombstone + store-layer
protection, safe data-dir default) ride in behind ENGRAM_WAL (default off =
byte-identical to today). Verified: engram elb per-module build clean, WAL
gate 66/66, native smoke ok, drift-guard green.
handle_api_consolidate writes a "SessionSummary" node, but engram_valid_node_type
omitted it — so once this validation ships, every consolidate() would be silently
REJECTED at the engram boundary. Add SessionSummary to the allowlist.
Found in Will's PR review of neuron #1 / el #52.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The wrapper signature was stale and didn't match the C primitive
__engram_node_full(content, node_type, label, salience, importance, confidence, tier, tags).
Because el_val_t is an untyped machine word, the compiler coerced caller args to the
wrong declared param types and forwarded them BY POSITION — so tier received an int,
importance/confidence received strings, label received a float, etc. (~100 corrupt nodes).
- Correct the wrapper to match the C contract 1:1 (no coercion, no reorder).
- Add engram_valid_node_type / engram_valid_tier allowlists; engram_node and
engram_node_full now reject invalid values with __println + return "" (fail loud,
no silent malformed write).
See neuron repo: HANDOFF-engram-write-corruption.md for the full write-up + deploy runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
will.anderson
Tim Lingo