engram tiered storage M5: online compaction + background checkpointer (Phase 3, additive)
Reclaims space held by dead records (tombstoned prune/forget nodes, superseded
ids, stale re-put/hebb versions, and their orphaned overflow chains). On-disk
format UNCHANGED — pure behavior.
COMPACTION (store_compact): copy-live + atomic-swap.
A. checkpoint/sync to quiesce (WAL reduced to CHECKPOINT{C}); crash here => pre.
B. build <path>.compact with only the live records, re-placed bit-exact into
fresh densely-packed pages + fresh id/adjacency B+-trees, every page stamped
LSN=C, new SB last_checkpoint_lsn=C; fsync. crash here => pre-compaction.
C. rename(<path>.compact -> <path>) — POSIX-atomic commit; crash after => post.
D. reopen in place: swap fd, INVALIDATE every pool frame (M4 remap of relocated
pages), reload SB, re-autopin.
Crash at any instant recovers to pre- OR post-compaction, never a corrupt mix.
Single-threaded => "online" = safe between mutations; takes a checkpoint quiesce
at entry. M4 cooperation: temp build has its own pool honoring ENGRAM_POOL_FRAMES
(evict/re-fault + no-steal + pins); live pool fully invalidated on reopen.
BACKGROUND CHECKPOINTER: ckpt_maybe now fires on ANY armed trigger — ops (default
100000), dirty pool frames, WAL bytes-since-reclaim (default 64 MiB), or a
wall-clock interval (checked on the write path; no extra thread). Same M2
checkpoint semantics (calls engram_checkpoint). Env: ENGRAM_CKPT_OPS/_DIRTY/
_WAL_BYTES/_INTERVAL_MS; runtime setter store_set_checkpoint_policy().
Tests: engram/test/test_compaction.c (+runner). Plain gcc, ASan/UBSan clean.
36 passed, 0 failed (O2 and ASan+UBSan builds).
reclaim: page_count 655 -> 153, file 10731520 -> 2506752 bytes (76.6% reclaimed),
every live record + adjacency bit-exact at new locations.
crash-during-compaction phases 0/1/2: crc clean, live set intact, writable.
background checkpointer: ops / WAL-bytes / dirty triggers each auto-fire; WAL
prefix reclaimed (30 B after 600 puts); recovery correct.
pool cooperation: compact under 24-frame pool correct, no stale frames.
No regression: M1 33, M2 36, M3 parity, M3.5, M4 bufpool 37 — all green.
On-disk format unchanged (additive).
This commit is contained in:
@@ -253,4 +253,43 @@ void store__crash(EngramPagedStore* s);
|
||||
int store__flush_pages(EngramPagedStore* s);
|
||||
int store__checkpoint_crashat(EngramPagedStore* s, int phase);
|
||||
|
||||
/* ── M5: online compaction + background checkpointer (additive; format UNCHANGED) ──
|
||||
*
|
||||
* COMPACTION reclaims the space held by DEAD records — tombstoned nodes/edges
|
||||
* (telemetry prune, forget), superseded ids, and the stale prior versions a
|
||||
* re-put/hebb-batch leaves behind — plus the overflow pages they orphaned. It
|
||||
* rewrites only the LIVE records (bit-exact) into a fresh, densely packed image
|
||||
* with fresh id + adjacency indexes, then commits the swap atomically, so the
|
||||
* .egm file physically SHRINKS and the freed pages are reclaimed. Crash-safe:
|
||||
* a crash at any instant recovers to either the pre- or the post-compaction
|
||||
* store, never a corrupt mix (atomic rename is the commit point). It cooperates
|
||||
* with the M4 pool (no-steal, pins) by building into a separate store whose own
|
||||
* pool honours ENGRAM_POOL_FRAMES, then INVALIDATING every frame of the live
|
||||
* pool so no stale frame survives for a relocated page.
|
||||
*
|
||||
* Requires a quiesce point: store_compact performs a checkpoint (or sync) at
|
||||
* entry, so it is called between mutations, not concurrently with one. */
|
||||
int store_compact(EngramPagedStore* s);
|
||||
|
||||
/* Test hook: run compaction but stop (then power-loss) after `phase`:
|
||||
* 0 = after the entry checkpoint, before building (→ recovers pre-compaction)
|
||||
* 1 = after building+fsync the new image, before rename (→ pre-compaction)
|
||||
* 2 = after the atomic rename, before reopening RAM state (→ post-compaction)
|
||||
* phase<0 = full compaction. Frees `s` on a crash phase (like the checkpoint hook). */
|
||||
int store__compact_crashat(EngramPagedStore* s, int phase);
|
||||
|
||||
/* BACKGROUND CHECKPOINTER policy. A checkpoint fires automatically on the write
|
||||
* path when ANY armed trigger trips, reclaiming the WAL prefix without an explicit
|
||||
* engram_checkpoint. 0 disables that trigger. Same checkpoint semantics as M2.
|
||||
* ops — mutations since last checkpoint (default 100000)
|
||||
* dirty_pages — dirty (un-checkpointed) pool frames
|
||||
* wal_bytes — bytes appended to the WAL since it was last reclaimed
|
||||
* interval_ms — wall-clock ms since the last checkpoint (checked on writes) */
|
||||
void store_set_checkpoint_policy(EngramPagedStore* s, uint64_t ops,
|
||||
size_t dirty_pages, uint64_t wal_bytes,
|
||||
long long interval_ms);
|
||||
|
||||
/* Introspection: number of pages currently on the free-list. */
|
||||
uint64_t store_free_page_count(const EngramPagedStore* s);
|
||||
|
||||
#endif /* ENGRAM_STORE_H */
|
||||
|
||||
Reference in New Issue
Block a user