neuron-technologies/el
3
0
Fork 0
Code Issues 4 Pull Requests 4 Actions Packages Projects Releases 5 Wiki Activity

engram tiered storage M5: online compaction + background checkpointer (Phase 3, additive)

Browse Source 
Reclaims space held by dead records (tombstoned prune/forget nodes, superseded
ids, stale re-put/hebb versions, and their orphaned overflow chains). On-disk
format UNCHANGED — pure behavior.

COMPACTION (store_compact): copy-live + atomic-swap.
  A. checkpoint/sync to quiesce (WAL reduced to CHECKPOINT{C}); crash here => pre.
  B. build <path>.compact with only the live records, re-placed bit-exact into
     fresh densely-packed pages + fresh id/adjacency B+-trees, every page stamped
     LSN=C, new SB last_checkpoint_lsn=C; fsync. crash here => pre-compaction.
  C. rename(<path>.compact -> <path>) — POSIX-atomic commit; crash after => post.
  D. reopen in place: swap fd, INVALIDATE every pool frame (M4 remap of relocated
     pages), reload SB, re-autopin.
Crash at any instant recovers to pre- OR post-compaction, never a corrupt mix.
Single-threaded => "online" = safe between mutations; takes a checkpoint quiesce
at entry. M4 cooperation: temp build has its own pool honoring ENGRAM_POOL_FRAMES
(evict/re-fault + no-steal + pins); live pool fully invalidated on reopen.

BACKGROUND CHECKPOINTER: ckpt_maybe now fires on ANY armed trigger — ops (default
100000), dirty pool frames, WAL bytes-since-reclaim (default 64 MiB), or a
wall-clock interval (checked on the write path; no extra thread). Same M2
checkpoint semantics (calls engram_checkpoint). Env: ENGRAM_CKPT_OPS/_DIRTY/
_WAL_BYTES/_INTERVAL_MS; runtime setter store_set_checkpoint_policy().

Tests: engram/test/test_compaction.c (+runner). Plain gcc, ASan/UBSan clean.
  36 passed, 0 failed (O2 and ASan+UBSan builds).
  reclaim: page_count 655 -> 153, file 10731520 -> 2506752 bytes (76.6% reclaimed),
    every live record + adjacency bit-exact at new locations.
  crash-during-compaction phases 0/1/2: crc clean, live set intact, writable.
  background checkpointer: ops / WAL-bytes / dirty triggers each auto-fire; WAL
    prefix reclaimed (30 B after 600 puts); recovery correct.
  pool cooperation: compact under 24-frame pool correct, no stale frames.
No regression: M1 33, M2 36, M3 parity, M3.5, M4 bufpool 37 — all green.
On-disk format unchanged (additive).
This commit is contained in:
Will Anderson
2026-08-12 16:03:14 -05:00
parent 02dc12d785
commit ce0d33ba93
4 changed files with 703 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lang/runtime/engram_store.c
+221 -6
View File
@@ -141,6 +141,11 @@ struct EngramPagedStore {
int recovering; /* set during WAL replay */
uint64_t ops_since_ckpt; /* checkpoint threshold counter */
uint64_t ckpt_threshold; /* auto-checkpoint after this many ops (0 = never) */
/* ── M5 background-checkpointer triggers (0 = that trigger disabled) ────── */
size_t ckpt_dirty_threshold; /* auto-checkpoint at this many dirty frames */
uint64_t ckpt_wal_threshold; /* auto-checkpoint at this many WAL bytes */
long long ckpt_interval_ms; /* auto-checkpoint after this many ms elapse */
long long last_ckpt_ms; /* wall-clock time of the last checkpoint */
};
/* M2/M4 buffer-pool hooks (defined in the pool section at the bottom of this file).
@@ -176,6 +181,7 @@ struct PgCache {
PgEnt* mru; PgEnt* lru; /* MRU (front) → LRU (back) recency list */
unsigned prefetch; /* read-ahead window (pages); 0 = off */
LayerPin* lp; size_t lp_n, lp_cap; /* hot-layer pin bookkeeping */
size_t dirty_count; /* # dirty frames, maintained incrementally (M5) */
/* stats (introspection only — never affect semantics) */
uint64_t hits, misses, evictions, prefetch_reads;
};
@@ -1424,7 +1430,7 @@ static int pc_put(EngramPagedStore* s, uint64_t id, const uint8_t* buf, int dirt
}
memcpy(e->buf, buf, STORE_PAGE_SIZE);
e->lsn = get_u64(buf + 16);
if (dirty) e->dirty = 1;
if (dirty){ if (!e->dirty) c->dirty_count++; e->dirty = 1; } /* clean→dirty transition */
pc_evict_to_budget(c);
return 0;
}
@@ -1434,6 +1440,7 @@ static int pc_flush(EngramPagedStore* s){
for (size_t i=0;i<c->nbuckets;i++)
for (PgEnt* e=c->buckets[i]; e; e=e->next)
if (e->dirty){ if (page_write_raw(s, e->id, e->buf)!=0) return -1; e->dirty=0; }
c->dirty_count = 0; /* all frames clean after flush */
/* Post-checkpoint the just-cleaned frames are now evictable; trim the pool
* back to budget so a dirty-heavy burst that transiently overshot cap does
* not leave the pool oversized. No-op at the default (unlimited-ish) cap. */
@@ -1601,6 +1608,7 @@ struct EngramWal {
uint64_t last_fsync_lsn;
long long last_fsync_ms;
uint64_t appended_since_fsync;
uint64_t bytes_since_reclaim; /* WAL bytes appended since last reclaim (M5) */
};
static long long now_ms(void){
@@ -1658,12 +1666,14 @@ static int wal_append(EngramPagedStore* s, uint8_t op, const uint8_t* payload,
free(fr);
if (wr != (ssize_t)fl) return -1;
w->appended_since_fsync++;
w->bytes_since_reclaim += fl;
wal_maybe_fsync(s, lsn);
return 0;
}
static int wal_reclaim(EngramPagedStore* s, uint64_t ckpt_lsn){
EngramWal* w = s->wal; if (!w) return 0;
if (ftruncate(w->fd, 0) != 0) return -1; /* prefix <= ckpt reclaimed */
w->bytes_since_reclaim = 0; /* WAL just shrank to the marker */
uint8_t p[8]; put_u64(p, ckpt_lsn);
if (wal_append(s, OP_CHECKPOINT, p, 8, ckpt_lsn) != 0) return -1;
fsync(w->fd); w->last_fsync_ms = now_ms();
@@ -2008,12 +2018,24 @@ static int wal_recover(EngramPagedStore* s){
return rc;
}
/* ── checkpoint threshold trigger ────────────────────────────────────────────── */
/* ── background checkpointer: fire on ops / dirty-frames / WAL-bytes / timer ─────
* Single-threaded model: the triggers are evaluated on the write path (no
* background thread), so a checkpoint fires on the first mutation after any armed
* threshold trips. This reclaims the WAL prefix automatically instead of only at
* an explicit engram_checkpoint. Same checkpoint semantics as M2 (it calls the
* very same engram_checkpoint). */
static void ckpt_maybe(EngramPagedStore* s){
if (s->recovering) return;
s->ops_since_ckpt++;
if (s->ckpt_threshold && s->ops_since_ckpt >= s->ckpt_threshold)
engram_checkpoint(s);
int fire = 0;
if (s->ckpt_threshold && s->ops_since_ckpt >= s->ckpt_threshold) fire = 1;
if (!fire && s->ckpt_dirty_threshold && s->cache &&
s->cache->dirty_count >= s->ckpt_dirty_threshold) fire = 1;
if (!fire && s->ckpt_wal_threshold && s->wal &&
s->wal->bytes_since_reclaim >= s->ckpt_wal_threshold) fire = 1;
if (!fire && s->ckpt_interval_ms &&
(now_ms() - s->last_ckpt_ms) >= s->ckpt_interval_ms) fire = 1;
if (fire) engram_checkpoint(s);
}
/* ── public mutation entry points (log-then-apply when a WAL is attached) ─────── */
@@ -2148,6 +2170,7 @@ int store__checkpoint_crashat(EngramPagedStore* s, int phase){
if (phase == 3){ store__crash(s); return 0; }
if (wal_reclaim(s, C) != 0) return -1; /* 4: reclaim WAL prefix */
s->ops_since_ckpt = 0;
s->last_ckpt_ms = now_ms(); /* arm the interval trigger */
if (phase == 4){ store__crash(s); return 0; }
return 0;
}
@@ -2375,6 +2398,26 @@ static int import_snapshot(EngramPagedStore* s, const char* path){
return 0;
}
/* Arm the background checkpointer with sensible defaults, overridable by env:
* ENGRAM_CKPT_OPS mutations since last checkpoint (default 100000)
* ENGRAM_CKPT_DIRTY dirty pool frames (default 0 = off)
* ENGRAM_CKPT_WAL_BYTES WAL bytes since reclaim (default 64 MiB)
* ENGRAM_CKPT_INTERVAL_MS wall-clock ms (default 0 = off)
* Any of these tripping on the write path triggers a checkpoint (→ WAL reclaimed).
* The M4 tests set tiny pools but never hit these bounds, so behaviour is unchanged. */
static void engram__default_ckpt_policy(EngramPagedStore* s){
s->ckpt_threshold = 100000;
s->ckpt_dirty_threshold = 0;
s->ckpt_wal_threshold = 64u*1024u*1024u;
s->ckpt_interval_ms = 0;
s->last_ckpt_ms = now_ms();
const char* e;
if ((e=getenv("ENGRAM_CKPT_OPS")) && *e) s->ckpt_threshold = strtoull(e,NULL,10);
if ((e=getenv("ENGRAM_CKPT_DIRTY")) && *e) s->ckpt_dirty_threshold = (size_t)strtoull(e,NULL,10);
if ((e=getenv("ENGRAM_CKPT_WAL_BYTES")) && *e) s->ckpt_wal_threshold = strtoull(e,NULL,10);
if ((e=getenv("ENGRAM_CKPT_INTERVAL_MS")) && *e) s->ckpt_interval_ms = strtoll(e,NULL,10);
}
/* ── durable-engram boot / close ─────────────────────────────────────────────── */
EngramPagedStore* engram_open(const char* data_dir){
if (!data_dir) return NULL;
@@ -2398,7 +2441,7 @@ EngramPagedStore* engram_open(const char* data_dir){
if (!s) return NULL;
s->wal = wal_open(wal_path, sync);
if (!s->wal){ store_close(s); return NULL; }
s->ckpt_threshold = 100000;
engram__default_ckpt_policy(s);
wal_recover(s); /* replay post-checkpoint tail */
return s;
}
@@ -2407,7 +2450,7 @@ EngramPagedStore* engram_open(const char* data_dir){
if (!s) return NULL;
s->wal = wal_open(wal_path, sync);
if (!s->wal){ store_close(s); return NULL; }
s->ckpt_threshold = 100000;
engram__default_ckpt_policy(s);
if (stat(snap_path, &st) == 0) import_snapshot(s, snap_path);
engram_checkpoint(s); /* store is now authoritative */
return s;
@@ -2417,3 +2460,175 @@ int engram_close(EngramPagedStore* s){
engram_checkpoint(s);
return store_close(s);
}
/* ══════════════════════════════════════════════════════════════════════════════
* M5 — ONLINE COMPACTION + background-checkpointer policy setter
*
* Dead space accrues in three shapes, all reclaimed here:
* 1. DEAD slots on NODE/EDGE pages — tombstones (prune/forget), superseded ids,
* and the stale prior versions a re-put / hebb-batch leaves (apply_*_put
* appends a new record + index entry; the old slot is marked DEAD).
* 2. Duplicate primary/adjacency index entries pointing at those DEAD records.
* 3. OVERFLOW chains orphaned when a large record died (tombstone only flips the
* slot; it never frees the record's overflow pages).
*
* Strategy — copy-live + atomic swap (the safest crash-safe relocation):
* A. Quiesce: checkpoint (or sync) so the on-disk .egm fully reflects state and
* the WAL is reduced to its CHECKPOINT{C} marker (C = current LSN watermark).
* B. Build a brand-new store file `<path>.compact` holding ONLY the live records
* — walked canonically (latest-live per id) and re-placed bit-exact into
* fresh, densely packed pages with fresh id + adjacency B+-trees. Every page
* is stamped with LSN = C and the new superblock records last_checkpoint_lsn
* = C, so it is LSN-consistent with the (unchanged) WAL. fsync it.
* C. Commit by rename(<path>.compact → <path>) — POSIX-atomic: recovery sees
* either the whole old file or the whole new file, never a torn mix.
* D. Reopen in place: swap the fd, INVALIDATE every pool frame (old page ids now
* hold different data — this is the M4 "remap relocated pages" step), reload
* the superblock, re-autopin.
*
* Crash safety (proven by the test at phases 0/1/2):
* • crash in A/B (before rename): old .egm is byte-for-byte intact and the WAL
* still matches it → recovery = PRE-compaction (all live records present).
* The half-built `.compact` temp is ignored by engram_open and unlinked at the
* next compaction.
* • crash after rename (C/D): the new .egm is fully fsync'd with last_checkpoint
* = C and the WAL (CHECKPOINT{C}, nothing newer) matches it → recovery =
* POST-compaction. No undo ever needed because relocation is copy-then-swap,
* never in-place mutation of a still-referenced page.
*
* Online vs quiesce: the store is single-threaded, so "online" means it is safe
* to interleave between mutations (each mutation is a synchronous call) — NOT that
* it runs concurrently with one. It takes a checkpoint quiesce point at entry.
*
* M4 cooperation: the build writes into a SEPARATE store `d` whose own pool obeys
* ENGRAM_POOL_FRAMES (so a small pool evicts/re-faults throughout the build,
* no-steal + pins honoured there); the live store's pool is fully invalidated on
* reopen, guaranteeing no stale frame maps a relocated page.
* ════════════════════════════════════════════════════════════════════════════ */
/* Drop every resident frame (relocated pages are no longer valid) but keep the
* pool object with its cap / prefetch / stats. Layer-pin bookkeeping is cleared
* (those page ids belong to the old image). */
static void pc_invalidate_all(PgCache* c){
if (!c) return;
for (size_t i=0;i<c->nbuckets;i++){
PgEnt* e = c->buckets[i];
while (e){ PgEnt* n=e->next; free(e->buf); free(e); e=n; }
c->buckets[i] = NULL;
}
for (size_t i=0;i<c->lp_n;i++) free(c->lp[i].pages);
c->lp_n = 0;
c->count = 0; c->dirty_count = 0; c->mru = c->lru = NULL;
}
/* Re-open the store file in place after an atomic swap: swap fd, invalidate the
* pool, reload the superblock, resume the LSN watermark, re-autopin. Keeps the
* attached WAL (it references the same checkpoint LSN the new file carries). */
static int store__reopen_swapped(EngramPagedStore* s){
if (s->fd >= 0) close(s->fd);
s->fd = open(s->path, O_RDWR);
if (s->fd < 0) return -1;
pc_invalidate_all(s->cache); /* M4: no stale frame for a relocated page */
uint8_t b0[STORE_PAGE_SIZE], b1[STORE_PAGE_SIZE];
uint64_t s0=0, s1=0;
int ok0 = sb_load_one(s, 0, b0, &s0) == 0;
int ok1 = sb_load_one(s, 1, b1, &s1) == 0;
if (!ok0 && !ok1) return -1;
const uint8_t* pick = (ok0&&ok1) ? ((s0>=s1)?b0:b1) : (ok0?b0:b1);
sb_apply(s, pick);
s->next_lsn = (s->last_checkpoint_lsn > s->sb_seq) ? s->last_checkpoint_lsn : s->sb_seq;
s->cur_node_page = 0; s->cur_edge_page = 0;
s->ops_since_ckpt = 0; s->last_ckpt_ms = now_ms();
store__autopin(s);
return 0;
}
/* Callback context for copying live records into the compacted store `d`. */
typedef struct { EngramPagedStore* d; int err; } CompactCtx;
static void compact_node_cb(const StoreNode* n, void* ctx){
CompactCtx* c = (CompactCtx*)ctx;
if (c->err) return;
if (node_place(c->d, n) != 0) c->err = 1; /* bit-exact re-place; stamped d->stamp_lsn */
}
static void compact_edge_cb(const StoreEdge* e, void* ctx){
CompactCtx* c = (CompactCtx*)ctx;
if (c->err) return;
if (edge_place(c->d, e) != 0) c->err = 1;
}
/* Build the compacted image (only live records, fresh indexes) into a new file. */
static int compact_build(EngramPagedStore* s, const char* tmp_path){
unlink(tmp_path); /* drop any temp from a crashed run */
EngramPagedStore* d = store_create(tmp_path);
if (!d) return -1;
uint64_t W = s->next_lsn; /* LSN watermark == checkpoint LSN */
memcpy(d->uuid, s->uuid, 16); /* preserve store identity */
d->last_checkpoint_lsn = W;
d->next_lsn = W;
d->stamp_lsn = W; /* every compacted page → LSN W */
int rc = 0;
/* live layers */
StoreLayer* layers = NULL; size_t nlay = 0;
if (store_list_layers(s, &layers, &nlay) == 0){
for (size_t i=0;i<nlay && rc==0;i++)
if (apply_layer_put(d, &layers[i], W) != 0) rc = -1;
store_layers_free(layers, nlay);
}
/* live nodes + edges (canonical latest-live, dedup by id — see store_scan_*) */
CompactCtx ctx = { d, 0 };
if (rc == 0 && store_scan_nodes(s, compact_node_cb, &ctx) < 0) rc = -1;
if (rc == 0 && store_scan_edges(s, compact_edge_cb, &ctx) < 0) rc = -1;
if (ctx.err) rc = -1;
d->stamp_lsn = 0;
if (rc == 0 && store_sync(d) != 0) rc = -1; /* flush + fsync + both superblocks */
store_close(d);
return rc;
}
int store__compact_crashat(EngramPagedStore* s, int phase){
if (!s) return -1;
/* A. quiesce → on-disk store consistent, WAL reduced to its checkpoint marker */
if (s->wal){ if (engram_checkpoint(s) != 0) return -1; }
else { if (store_sync(s) != 0) return -1; }
if (phase == 0){ store__crash(s); return 0; } /* → recovers pre-compaction */
char tmp[1200];
snprintf(tmp, sizeof tmp, "%s.compact", s->path);
if (compact_build(s, tmp) != 0){ unlink(tmp); return -1; }
if (phase == 1){ store__crash(s); return 0; } /* built, not renamed → pre-compaction */
/* C. atomic commit */
if (rename(tmp, s->path) != 0){ unlink(tmp); return -1; }
if (phase == 2){ store__crash(s); return 0; } /* renamed, not reopened → post-compaction */
/* D. reopen RAM state against the compacted file */
return store__reopen_swapped(s);
}
int store_compact(EngramPagedStore* s){ return store__compact_crashat(s, -1); }
void store_set_checkpoint_policy(EngramPagedStore* s, uint64_t ops,
size_t dirty_pages, uint64_t wal_bytes,
long long interval_ms){
if (!s) return;
s->ckpt_threshold = ops;
s->ckpt_dirty_threshold = dirty_pages;
s->ckpt_wal_threshold = wal_bytes;
s->ckpt_interval_ms = interval_ms;
s->last_ckpt_ms = now_ms();
}
uint64_t store_free_page_count(const EngramPagedStore* s){
if (!s) return 0;
uint64_t n = 0, id = s->free_list_head;
uint8_t buf[STORE_PAGE_SIZE];
while (id){
if (page_read((EngramPagedStore*)s, id, buf) != 0) break;
if (buf[8] != STORE_PT_FREE) break;
n++;
id = get_u64(buf + OVF_NEXT_OFF);
}
return n;
}
lang/runtime/engram_store.h
+39
View File
@@ -253,4 +253,43 @@ void store__crash(EngramPagedStore* s);
int store__flush_pages(EngramPagedStore* s);
int store__checkpoint_crashat(EngramPagedStore* s, int phase);
/* ── M5: online compaction + background checkpointer (additive; format UNCHANGED) ──
*
* COMPACTION reclaims the space held by DEAD records — tombstoned nodes/edges
* (telemetry prune, forget), superseded ids, and the stale prior versions a
* re-put/hebb-batch leaves behind — plus the overflow pages they orphaned. It
* rewrites only the LIVE records (bit-exact) into a fresh, densely packed image
* with fresh id + adjacency indexes, then commits the swap atomically, so the
* .egm file physically SHRINKS and the freed pages are reclaimed. Crash-safe:
* a crash at any instant recovers to either the pre- or the post-compaction
* store, never a corrupt mix (atomic rename is the commit point). It cooperates
* with the M4 pool (no-steal, pins) by building into a separate store whose own
* pool honours ENGRAM_POOL_FRAMES, then INVALIDATING every frame of the live
* pool so no stale frame survives for a relocated page.
*
* Requires a quiesce point: store_compact performs a checkpoint (or sync) at
* entry, so it is called between mutations, not concurrently with one. */
int store_compact(EngramPagedStore* s);
/* Test hook: run compaction but stop (then power-loss) after `phase`:
* 0 = after the entry checkpoint, before building (→ recovers pre-compaction)
* 1 = after building+fsync the new image, before rename (→ pre-compaction)
* 2 = after the atomic rename, before reopening RAM state (→ post-compaction)
* phase<0 = full compaction. Frees `s` on a crash phase (like the checkpoint hook). */
int store__compact_crashat(EngramPagedStore* s, int phase);
/* BACKGROUND CHECKPOINTER policy. A checkpoint fires automatically on the write
* path when ANY armed trigger trips, reclaiming the WAL prefix without an explicit
* engram_checkpoint. 0 disables that trigger. Same checkpoint semantics as M2.
* ops — mutations since last checkpoint (default 100000)
* dirty_pages — dirty (un-checkpointed) pool frames
* wal_bytes — bytes appended to the WAL since it was last reclaimed
* interval_ms — wall-clock ms since the last checkpoint (checked on writes) */
void store_set_checkpoint_policy(EngramPagedStore* s, uint64_t ops,
size_t dirty_pages, uint64_t wal_bytes,
long long interval_ms);
/* Introspection: number of pages currently on the free-list. */
uint64_t store_free_page_count(const EngramPagedStore* s);
#endif /* ENGRAM_STORE_H */