store: make the buffer pool sense its own state and correct from it
El SDK CI - dev / build-and-test (pull_request) Failing after 14m35s
El SDK CI - dev / build-and-test (pull_request) Failing after 14m35s
Follow-on to the edge write barrier. That fix removed the full-store walk;
this one makes the pool able to notice if anything like it happens again.
WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill
twice on 2026-08-15 and said nothing. From outside it was indistinguishable
from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories
got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each
costing a deploy or a rollback. The whole time, hits/misses/evictions were
already being counted in PgCache, and the struct comment read:
/* stats (introspection only — never affect semantics) */
That comment was the bug. Self-measurement treated as decoration is why the
pool could not correct itself and why no one outside could see what it was
doing. A system that cannot read its own state cannot correct, and neither can
anyone watching it.
- pc_adapt_budget(): the loop, closed. Over a sliding window, evictions
running at a large fraction of accesses WHILE reuse is real means the
working set exceeds the budget — so grow it, geometrically, bounded by a
LIVE re-read of physical memory. Evictions alone are not pressure (a scan
evicts and never returns); evictions with reuse are. An explicit
ENGRAM_POOL_FRAMES still wins — an operator override must not be silently
overruled.
- Budget derived, not declared. A constant cannot be right: 16 GiB of frames
is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM
at startup" is a guess about the future — it cannot know the store grew or
the machine changed. Hence the live re-read.
- pc_report(): ONE structured emission carrying the entire sensed state,
through emit_log — El's existing telemetry, already exporting to OTLP.
Deliberately not a function per stat, and deliberately not a bespoke
/api/pool endpoint: both make observability something hand-written per noun
instead of the uniform mechanism every component already has.
- engram_pool_stats_json(): the same state readable live, wired through the
normal builtin path (codegen arity + el_seed wrapper), so the pool can be
observed in real time rather than reconstructed afterward from a stack
sample.
Verified: with the exact configuration that took production down
(ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots
clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete —
and NO pressure event fires, because the barrier removed the walk that caused
it. The controller is defense in depth; the barrier is the fix.
This commit is contained in:
@@ -18373,3 +18373,41 @@ el_val_t engram_edges_json(el_val_t limit, el_val_t offset) {
|
||||
jb_putc(&b, ']');
|
||||
return el_wrap_str(b.buf);
|
||||
}
|
||||
|
||||
/* engram_pool_stats_json() — the buffer pool's interoception, exposed.
|
||||
*
|
||||
* StorePoolStats and store_pool_stats() already existed and were surfaced
|
||||
* NOWHERE. On 2026-08-15 the engram thrashed itself to a standstill twice while
|
||||
* these exact counters sat in memory, unread, and four wrong theories were tried
|
||||
* from the outside instead. Sensing state is only corrective if the state can be
|
||||
* read — by the process itself (pc_adapt_budget) and by anything watching it.
|
||||
*
|
||||
* Serves the live numbers plus the derived signals that actually diagnose:
|
||||
* hit_rate — sustained low hit rate with high evictions is the thrash shape
|
||||
* evict_ratio — evictions per access; ~1 means every fetch displaces a live page
|
||||
* pressure — 1 when evicting into genuine reuse (working set > budget)
|
||||
* cap_gib/resident_gib — budget vs what is actually held
|
||||
*/
|
||||
el_val_t engram_pool_stats_json(void) {
|
||||
if (!g_engram_store) return el_wrap_str(el_strdup("{\"store\":false}"));
|
||||
StorePoolStats st;
|
||||
store_pool_stats(g_engram_store, &st);
|
||||
uint64_t acc = st.hits + st.misses;
|
||||
double hit_rate = acc ? (double)st.hits / (double)acc : 0.0;
|
||||
double evict_ratio = acc ? (double)st.evictions / (double)acc : 0.0;
|
||||
int pressure = (acc > 100000 && evict_ratio > 0.33 && hit_rate > 0.25) ? 1 : 0;
|
||||
char b[768];
|
||||
snprintf(b, sizeof b,
|
||||
"{\"store\":true,\"cap_frames\":%zu,\"resident_frames\":%zu,\"pinned\":%zu,"
|
||||
"\"dirty\":%zu,\"prefetch\":%u,\"hits\":%llu,\"misses\":%llu,\"evictions\":%llu,"
|
||||
"\"prefetch_reads\":%llu,\"hit_rate\":%.4f,\"evict_ratio\":%.4f,\"pressure\":%d,"
|
||||
"\"cap_gib\":%.3f,\"resident_gib\":%.3f,\"page_size\":%u}",
|
||||
st.cap, st.resident, st.pinned, st.dirty, st.prefetch,
|
||||
(unsigned long long)st.hits, (unsigned long long)st.misses,
|
||||
(unsigned long long)st.evictions, (unsigned long long)st.prefetch_reads,
|
||||
hit_rate, evict_ratio, pressure,
|
||||
(double)st.cap * (double)STORE_PAGE_SIZE / (1024.0*1024.0*1024.0),
|
||||
(double)st.resident * (double)STORE_PAGE_SIZE / (1024.0*1024.0*1024.0),
|
||||
(unsigned)STORE_PAGE_SIZE);
|
||||
return el_wrap_str(el_strdup(b));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user