will.anderson
02dc12d785
Turn M2's write-back/no-steal cache into a bounded, demand-paged buffer pool so
the paged store can exceed RAM while keeping only hot pages resident. On-disk
format UNCHANGED (additive residency only; no migration). Default budget is large
enough that today's store stays fully resident, so default behaviour == Phase 1.
- Frame table capped at `cap` frames (env ENGRAM_POOL_FRAMES; 0 = unlimited;
default 1<<20). Not-resident access faults in from neuron.egm.
- LRU eviction of CLEAN, unpinned frames only. Dirty frames are never stolen
(M2 no-steal / WAL durability preserved) — turned evictable by a checkpoint's
pc_flush, which then trims the pool back to budget.
- Pinning: superblocks (0,1) + index root/interior pages auto-pinned; explicit
store_pin_page/unpin and store_pin_layer/unpin (hot WM/core layers).
- Bounded sequential read-ahead on scans (env ENGRAM_PREFETCH, default 8).
- Correctness rests on callers copying page bytes into local buffers and never
retaining a frame pointer across another access, so evict+re-fault is safe.
Gates (plain gcc, ASan/UBSan clean):
M4 run_bufpool_tests.sh ...... 37 passed, 0 failed (+ ASan/UBSan: 37/0)
small-pool round-trip (cap=32 vs 1599 pages, 2708 evictions): 5000 nodes +
4000 sampled edges bit-exact, crc clean, pool bounded to cap.
eviction: hot set 0 re-faults, cold evicted, hit-rate 0.989; no-steal burst
(cap=8) holds 309 dirty frames > cap, reads correct from dirty pages.
pinning: superblocks/roots/explicit page/hot-layer(19 pages) stay resident;
unpin makes them evictable.
prefetch: sequential scan 511 demand-faults OFF -> 4 ON.
crash-under-paging (ENGRAM_POOL_FRAMES=16): WAL replay + checkpoint-crash
phases 0-4 all recover bit-exact.
default pool: 0 evictions, whole store resident (== Phase 1).
No regression: M1 33/0, M2 36/0, M3 parity PASS, M3.5 PASS.
23 lines
882 B
Bash
Executable File
23 lines
882 B
Bash
Executable File
#!/usr/bin/env bash
|
|
# M4 demand-paging buffer-pool gate. Pure C (NOT elb/elc). Writes only under /tmp.
|
|
# Runs the suite twice: an -O2 correctness build and an ASan+UBSan build.
|
|
set -e
|
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
SRC="$HERE/../../lang/runtime/engram_store.c"
|
|
TST="$HERE/test_bufpool.c"
|
|
|
|
echo "== compiling (gcc -O2): test_bufpool.c engram_store.c =="
|
|
BIN="/tmp/test_bufpool.$$"
|
|
gcc -O2 -Wall -Wextra -std=c11 "$TST" "$SRC" -o "$BIN"
|
|
"$BIN"; rc=$?
|
|
rm -f "$BIN"; rm -rf /tmp/engram-bufpool-test-*
|
|
[ $rc -ne 0 ] && exit $rc
|
|
|
|
echo
|
|
echo "== ASan+UBSan build (memory-error + UB checks; LSan unavailable on macOS) =="
|
|
ABIN="/tmp/test_bufpool_asan.$$"
|
|
gcc -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -std=c11 "$TST" "$SRC" -o "$ABIN"
|
|
ASAN_OPTIONS=detect_leaks=0 UBSAN_OPTIONS=halt_on_error=1 "$ABIN"; rc=$?
|
|
rm -f "$ABIN"; rm -rf /tmp/engram-bufpool-test-*
|
|
exit $rc
|