store: make the buffer pool sense its own state and correct from it
El SDK CI - dev / build-and-test (pull_request) Failing after 14m35s

Follow-on to the edge write barrier. That fix removed the full-store walk;
this one makes the pool able to notice if anything like it happens again.

WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill
twice on 2026-08-15 and said nothing. From outside it was indistinguishable
from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories
got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each
costing a deploy or a rollback. The whole time, hits/misses/evictions were
already being counted in PgCache, and the struct comment read:

    /* stats (introspection only — never affect semantics) */

That comment was the bug. Self-measurement treated as decoration is why the
pool could not correct itself and why no one outside could see what it was
doing. A system that cannot read its own state cannot correct, and neither can
anyone watching it.

  - pc_adapt_budget(): the loop, closed. Over a sliding window, evictions
    running at a large fraction of accesses WHILE reuse is real means the
    working set exceeds the budget — so grow it, geometrically, bounded by a
    LIVE re-read of physical memory. Evictions alone are not pressure (a scan
    evicts and never returns); evictions with reuse are. An explicit
    ENGRAM_POOL_FRAMES still wins — an operator override must not be silently
    overruled.

  - Budget derived, not declared. A constant cannot be right: 16 GiB of frames
    is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM
    at startup" is a guess about the future — it cannot know the store grew or
    the machine changed. Hence the live re-read.

  - pc_report(): ONE structured emission carrying the entire sensed state,
    through emit_log — El's existing telemetry, already exporting to OTLP.
    Deliberately not a function per stat, and deliberately not a bespoke
    /api/pool endpoint: both make observability something hand-written per noun
    instead of the uniform mechanism every component already has.

  - engram_pool_stats_json(): the same state readable live, wired through the
    normal builtin path (codegen arity + el_seed wrapper), so the pool can be
    observed in real time rather than reconstructed afterward from a stack
    sample.

Verified: with the exact configuration that took production down
(ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots
clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete —
and NO pressure event fires, because the barrier removed the walk that caused
it. The controller is defense in depth; the barrier is the fix.
This commit is contained in:
bigmerge
2026-08-15 20:44:23 -05:00
parent 777ccc02f0
commit e917b3d439
5 changed files with 157 additions and 27 deletions
+38
View File
@@ -18373,3 +18373,41 @@ el_val_t engram_edges_json(el_val_t limit, el_val_t offset) {
jb_putc(&b, ']');
return el_wrap_str(b.buf);
}
/* engram_pool_stats_json() — the buffer pool's interoception, exposed.
*
* StorePoolStats and store_pool_stats() already existed and were surfaced
* NOWHERE. On 2026-08-15 the engram thrashed itself to a standstill twice while
* these exact counters sat in memory, unread, and four wrong theories were tried
* from the outside instead. Sensing state is only corrective if the state can be
* read by the process itself (pc_adapt_budget) and by anything watching it.
*
* Serves the live numbers plus the derived signals that actually diagnose:
* hit_rate sustained low hit rate with high evictions is the thrash shape
* evict_ratio evictions per access; ~1 means every fetch displaces a live page
* pressure 1 when evicting into genuine reuse (working set > budget)
* cap_gib/resident_gib budget vs what is actually held
*/
el_val_t engram_pool_stats_json(void) {
if (!g_engram_store) return el_wrap_str(el_strdup("{\"store\":false}"));
StorePoolStats st;
store_pool_stats(g_engram_store, &st);
uint64_t acc = st.hits + st.misses;
double hit_rate = acc ? (double)st.hits / (double)acc : 0.0;
double evict_ratio = acc ? (double)st.evictions / (double)acc : 0.0;
int pressure = (acc > 100000 && evict_ratio > 0.33 && hit_rate > 0.25) ? 1 : 0;
char b[768];
snprintf(b, sizeof b,
"{\"store\":true,\"cap_frames\":%zu,\"resident_frames\":%zu,\"pinned\":%zu,"
"\"dirty\":%zu,\"prefetch\":%u,\"hits\":%llu,\"misses\":%llu,\"evictions\":%llu,"
"\"prefetch_reads\":%llu,\"hit_rate\":%.4f,\"evict_ratio\":%.4f,\"pressure\":%d,"
"\"cap_gib\":%.3f,\"resident_gib\":%.3f,\"page_size\":%u}",
st.cap, st.resident, st.pinned, st.dirty, st.prefetch,
(unsigned long long)st.hits, (unsigned long long)st.misses,
(unsigned long long)st.evictions, (unsigned long long)st.prefetch_reads,
hit_rate, evict_ratio, pressure,
(double)st.cap * (double)STORE_PAGE_SIZE / (1024.0*1024.0*1024.0),
(double)st.resident * (double)STORE_PAGE_SIZE / (1024.0*1024.0*1024.0),
(unsigned)STORE_PAGE_SIZE);
return el_wrap_str(el_strdup(b));
}