will.anderson
ce0d33ba93
Reclaims space held by dead records (tombstoned prune/forget nodes, superseded
ids, stale re-put/hebb versions, and their orphaned overflow chains). On-disk
format UNCHANGED — pure behavior.
COMPACTION (store_compact): copy-live + atomic-swap.
A. checkpoint/sync to quiesce (WAL reduced to CHECKPOINT{C}); crash here => pre.
B. build <path>.compact with only the live records, re-placed bit-exact into
fresh densely-packed pages + fresh id/adjacency B+-trees, every page stamped
LSN=C, new SB last_checkpoint_lsn=C; fsync. crash here => pre-compaction.
C. rename(<path>.compact -> <path>) — POSIX-atomic commit; crash after => post.
D. reopen in place: swap fd, INVALIDATE every pool frame (M4 remap of relocated
pages), reload SB, re-autopin.
Crash at any instant recovers to pre- OR post-compaction, never a corrupt mix.
Single-threaded => "online" = safe between mutations; takes a checkpoint quiesce
at entry. M4 cooperation: temp build has its own pool honoring ENGRAM_POOL_FRAMES
(evict/re-fault + no-steal + pins); live pool fully invalidated on reopen.
BACKGROUND CHECKPOINTER: ckpt_maybe now fires on ANY armed trigger — ops (default
100000), dirty pool frames, WAL bytes-since-reclaim (default 64 MiB), or a
wall-clock interval (checked on the write path; no extra thread). Same M2
checkpoint semantics (calls engram_checkpoint). Env: ENGRAM_CKPT_OPS/_DIRTY/
_WAL_BYTES/_INTERVAL_MS; runtime setter store_set_checkpoint_policy().
Tests: engram/test/test_compaction.c (+runner). Plain gcc, ASan/UBSan clean.
36 passed, 0 failed (O2 and ASan+UBSan builds).
reclaim: page_count 655 -> 153, file 10731520 -> 2506752 bytes (76.6% reclaimed),
every live record + adjacency bit-exact at new locations.
crash-during-compaction phases 0/1/2: crc clean, live set intact, writable.
background checkpointer: ops / WAL-bytes / dirty triggers each auto-fire; WAL
prefix reclaimed (30 B after 600 puts); recovery correct.
pool cooperation: compact under 24-frame pool correct, no stale frames.
No regression: M1 33, M2 36, M3 parity, M3.5, M4 bufpool 37 — all green.
On-disk format unchanged (additive).
23 lines
896 B
Bash
Executable File
23 lines
896 B
Bash
Executable File
#!/usr/bin/env bash
|
|
# M5 online-compaction + background-checkpointer gate. Pure C (NOT elb/elc).
|
|
# Writes only under /tmp. Runs an -O2 correctness build then an ASan+UBSan build.
|
|
set -e
|
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
SRC="$HERE/../../lang/runtime/engram_store.c"
|
|
TST="$HERE/test_compaction.c"
|
|
|
|
echo "== compiling (gcc -O2): test_compaction.c engram_store.c =="
|
|
BIN="/tmp/test_compaction.$$"
|
|
gcc -O2 -Wall -Wextra -std=c11 "$TST" "$SRC" -o "$BIN"
|
|
"$BIN"; rc=$?
|
|
rm -f "$BIN"; rm -rf /tmp/engram-compact-test-*
|
|
[ $rc -ne 0 ] && exit $rc
|
|
|
|
echo
|
|
echo "== ASan+UBSan build (memory-error + UB checks; LSan unavailable on macOS) =="
|
|
ABIN="/tmp/test_compaction_asan.$$"
|
|
gcc -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -std=c11 "$TST" "$SRC" -o "$ABIN"
|
|
ASAN_OPTIONS=detect_leaks=0 UBSAN_OPTIONS=halt_on_error=1 "$ABIN"; rc=$?
|
|
rm -f "$ABIN"; rm -rf /tmp/engram-compact-test-*
|
|
exit $rc
|