Compare commits

..

15 Commits

Author SHA1 Message Date
bigmerge 37bcf7eb74 runtime: allocation accounting — the deterministic signal for complexity gating
El SDK CI - dev / build-and-test (pull_request) Failing after 12m7s
Implements the three primitives the test-framework design (DESIGN.md §6.5)
requires for gating on growth curves: el_alloc_count, el_alloc_bytes,
el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per
the project's C-builtin recipe.

WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build
failure, which means the signal cannot flake. Wall-clock needs warmup,
statistics, and a quiet machine; on shared CI it is unusable as a gate.
Allocation counts are perfectly deterministic — same input, same number, every
machine, every run. Fit them against n and a complexity regression becomes a
build failure with zero noise.

All four runtime string allocators (el_strdup, el_strbuf, and their _persist
variants) funnel every allocation the language performs, so instrumenting there
counts everything.

WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate.
Measured with two El programs, one allocating once per item, one rebuilding its
accumulator each iteration:

    n     linear allocs / bytes      quadratic allocs / bytes
    100        100 /    290               100 /   5,150
    200        200 /    690               200 /  20,300
    400        400 /  1,490               400 /  80,600
    800        800 /  3,090               800 / 321,200

The quadratic program's allocation COUNT is exactly linear — identical to the
healthy one. Counting allocations alone would have missed it completely. Bytes
catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 ->
converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0.

That shape — count linear, per-allocation size growing — is the classic
accidental quadratic, and it is exactly elc's defect: quadratic allocation
VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt
one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across
both; RSS and wall-clock were just the two ways it surfaced.

el_peak_rss is exported for context and is explicitly NOT a gating signal — it
is perturbed by allocator internals, the page cache, and the OS. Gate on the
deterministic numbers; report the physical one.

Counters are unsynchronised by design: this is measurement, and a lock would
change the thing being measured. Exact on the single-threaded compile path,
approximate under threads.
2026-08-15 21:21:42 -05:00
will.anderson 2240d26c32 Merge pull request 'store: judge memory pressure by swap RATE, not level' (#130) from fix/elc-rebuildable-compiler-builtins into dev
El SDK CI - dev / build-and-test (push) Failing after 11m44s
2026-08-16 02:12:22 +00:00
bigmerge 19cc99e57d store: judge memory pressure by swap RATE, not swap level
El SDK CI - dev / build-and-test (pull_request) Failing after 11m59s
The guard I added minutes ago checked swap availability as a level
(avail < total/8 -> report zero available). That is the wrong signal, and the
same host proved it twice within minutes:

    47.65 / 48.00 GiB swap used, 2047 swapouts/s  -> genuinely thrashing
    26.67 / 28.00 GiB swap used,    0 swapouts/s  -> healthy, 15.6 GiB free

Both are ~97% "used". macOS grows swap files on demand and trims them lazily,
so the level says almost nothing about now — it is a high-water mark. The level
check calls the second state an emergency and starves the pool for no reason,
which is its own failure mode: a guard that fires on healthy machines gets
disabled, and then guards nothing.

What separates the two is whether pages are moving. So sample the swapout
counter across calls and judge the delta:

  - > 200 pages/s (~3 MiB/s) sustained outward paging => report zero available;
    callers refuse to grow and pc_relieve_pressure hands frames back.
  - The first call primes the baseline and reports no pressure. One sample
    cannot have a rate, and inferring one from a single reading is exactly the
    mistake this commit removes.

Measured thresholds, not guessed: idle sat at 0/s, recovery burst hit 24,845/s
while the compressor drained (transient, correctly not a growth decision since
growth is only evaluated on eviction passes), and real thrash held ~2000/s.
200/s sits clearly above noise and far below either.

The compressor-footprint subtraction stays: that RAM is genuinely spoken for
regardless of paging rate.
2026-08-15 21:12:00 -05:00
will.anderson f39ae40047 Merge pull request 'store: bound the pool by available memory and let it shrink' (#129) from fix/elc-rebuildable-compiler-builtins into dev
El SDK CI - dev / build-and-test (push) Failing after 4m43s
2026-08-16 02:02:24 +00:00
bigmerge e52415f0e0 store: bound the pool by AVAILABLE memory and let it shrink
El SDK CI - dev / build-and-test (pull_request) Failing after 11m47s
The adaptive budget I added an hour ago could only grow, and grew toward a
share of TOTAL ram (80%, ~38 GiB on a 48 GB host). That is a memory leak with
extra steps: total never shrinks when other processes need memory, so the pool
had no way to notice it was starving the machine it runs on. Deployed briefly;
caught as memory pressure on the host.

A control loop with only one direction is not a control loop.

  - pc_available_ram(): free + inactive + purgeable via host_statistics64 on
    Darwin, MemAvailable on Linux. Availability is the quantity that moves when
    the machine is under pressure; total is not. Returns 0 when it cannot be
    read, and callers then refuse to grow — a cache is never worth swapping the
    host, so unknown means no.

  - Growth is bounded by availability minus a free-memory floor (2 GiB default,
    ENGRAM_POOL_FREE_FLOOR_MB), not by total. The share-of-total ceiling stays
    as a second bound and drops 80% -> 50%.

  - pc_relieve_pressure(): the missing direction. On every eviction pass, if
    available memory is under the floor, hand back ~25% of held frames; the
    resident set follows on the next pass so the memory is actually returned
    rather than merely re-labelled. Counted as adapt_shrinks alongside
    adapt_grows so both directions are visible in the same report.

  - pc_default_cap() also clamps the STARTING budget to what is spare right
    now, so a cold boot on a loaded machine does not open at a size the host
    cannot afford.

Verified on a 48 GB host: engram boots in ~30s, RSS settles at 2.22 GiB (the
store's actual size, resident, not creeping), 0.0% CPU, 13,439 nodes / 37,670
edges, embeddings complete. Guard reports 9.71 GiB available against a 2.00 GiB
floor — 7.71 GiB of headroom it is permitted to use and no more.
2026-08-15 21:02:05 -05:00
will.anderson 7a479111ac Merge pull request 'store: extend the write barrier to edges — kills the full-store walk' (#128) from fix/elc-rebuildable-compiler-builtins into dev
El SDK CI - dev / build-and-test (push) Failing after 14m27s
2026-08-16 01:44:33 +00:00
bigmerge e917b3d439 store: make the buffer pool sense its own state and correct from it
El SDK CI - dev / build-and-test (pull_request) Failing after 14m35s
Follow-on to the edge write barrier. That fix removed the full-store walk;
this one makes the pool able to notice if anything like it happens again.

WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill
twice on 2026-08-15 and said nothing. From outside it was indistinguishable
from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories
got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each
costing a deploy or a rollback. The whole time, hits/misses/evictions were
already being counted in PgCache, and the struct comment read:

    /* stats (introspection only — never affect semantics) */

That comment was the bug. Self-measurement treated as decoration is why the
pool could not correct itself and why no one outside could see what it was
doing. A system that cannot read its own state cannot correct, and neither can
anyone watching it.

  - pc_adapt_budget(): the loop, closed. Over a sliding window, evictions
    running at a large fraction of accesses WHILE reuse is real means the
    working set exceeds the budget — so grow it, geometrically, bounded by a
    LIVE re-read of physical memory. Evictions alone are not pressure (a scan
    evicts and never returns); evictions with reuse are. An explicit
    ENGRAM_POOL_FRAMES still wins — an operator override must not be silently
    overruled.

  - Budget derived, not declared. A constant cannot be right: 16 GiB of frames
    is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM
    at startup" is a guess about the future — it cannot know the store grew or
    the machine changed. Hence the live re-read.

  - pc_report(): ONE structured emission carrying the entire sensed state,
    through emit_log — El's existing telemetry, already exporting to OTLP.
    Deliberately not a function per stat, and deliberately not a bespoke
    /api/pool endpoint: both make observability something hand-written per noun
    instead of the uniform mechanism every component already has.

  - engram_pool_stats_json(): the same state readable live, wired through the
    normal builtin path (codegen arity + el_seed wrapper), so the pool can be
    observed in real time rather than reconstructed afterward from a stack
    sample.

Verified: with the exact configuration that took production down
(ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots
clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete —
and NO pressure event fires, because the barrier removed the walk that caused
it. The controller is defense in depth; the barrier is the fix.
2026-08-15 20:44:23 -05:00
bigmerge 777ccc02f0 store: extend the durable-hash write barrier to edges (kills the full-store walk)
El SDK CI - dev / build-and-test (pull_request) Failing after 10m45s
Checkpointing pushes the ENTIRE resident graph through store_put_node and
store_put_edge (see engram_store_checkpoint). Nodes were cheap: a durable-hash
compare skipped unchanged records with zero page I/O. Edges had no barrier at
all — struct comment at PgCache.barrier_on even says "node durable-hash
barrier" — so every edge was rewritten on every checkpoint, and each rewrite
runs the idempotency probe max_page_lsn_for_id -> btree lookup -> page_read.

Edges outnumber nodes ~3:1 here (37,663 vs 13,436), so routine checkpointing
degenerated into a FULL-STORE WALK in id order: random page access across the
whole 2 GiB store, repeated, overwhelmingly to rediscover nothing had changed.
LRU is worst-case under exactly that pattern — it evicts the page it is about
to want — so once the page cache was smaller than the store, the walk collapsed
into thrashing: 100% CPU, flat RSS, no forward progress, port never bound.
That took the live engram down twice on 2026-08-15.

The walk is the defect. Sizing the cache to survive it treats the symptom.

Changes:
  - dh_edge_hash(): edge counterpart of dh_node_hash, with a kind discriminator
    byte so an edge can never collide with a node of the same id in the shared
    map. created_at/updated_at/last_fired are excluded deliberately: last_fired
    is touched by activation without changing what the edge IS, and folding it
    in would defeat the barrier on precisely the hot edges that most need it.
  - store_put_edge(): barrier check + dh_set on success, mirroring
    store_put_node exactly.
  - store_scan_edges(): seed the barrier map from on-disk truth at load, so the
    FIRST post-boot checkpoint already skips unchanged edges. store_scan_nodes
    already did this and its comment says why; edges were simply never done.

Verified: with the exact configuration that killed production
(ENGRAM_POOL_FRAMES=65536 -> 1 GiB cache against a 2 GiB store), the engram now
boots clean and serves — LISTENING, 13,436 nodes / 37,663 edges, embeddings
complete, 0.0% CPU, RSS 1.14 GiB (cache resting at its budget rather than
thrashing against it). Same small cache, same store, no walk.
2026-08-15 20:38:11 -05:00
will.anderson c21074b547 Merge pull request 'runtime: engram_edges_json — kill the whole-graph file round trip' (#127) from fix/elc-rebuildable-compiler-builtins into dev
El SDK CI - dev / build-and-test (push) Failing after 10m19s
2026-08-16 01:13:44 +00:00
bigmerge 4e24d7d3f1 runtime: engram_edges_json — read edges without a whole-graph file round trip
El SDK CI - dev / build-and-test (pull_request) Failing after 13m4s
/api/graph/edges answered a read query by calling engram_save() to serialize
the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in
one line, and both bit production on 2026-08-15:

  1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram
     server's CANONICAL store. A READ route overwriting the persistence
     owner's canonical file. This defect had been fixed once (export moved to
     a scratch path); it came back when the hand-written dispatch block was
     replaced by @route dispatch and the unfixed copy is the one that
     survived the merge.
  2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole
     graph, per request, to return a bounded slice.

Calling it tonight overwrote the canonical snapshot and immediately preceded
an engram crash loop.

engram_edges_json(limit, offset) is the builtin that route's own TODO asked
for ("Future: add an engram_edges_json() builtin and drop the file round trip
entirely"). It walks g->edges directly and emits every persisted field.

limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell
over, and an unbounded default would preserve the failure mode under a new
name. Callers page explicitly.

Registered in codegen.el's builtin_arity (both plain and __ spellings) and
wrapped in el_seed.c per the project's C-builtin recipe.
2026-08-15 20:10:48 -05:00
will.anderson 7557ea6e19 Merge pull request 'runtime: restore engram_recall_json + cgi_* accessors (unblocks the soul build)' (#126) from fix/elc-rebuildable-compiler-builtins into dev
El SDK CI - dev / build-and-test (push) Failing after 10m15s
2026-08-16 00:57:11 +00:00
bigmerge 7351fb0a8d runtime: restore engram_recall_json + cgi_* accessors
El SDK CI - dev / build-and-test (pull_request) Failing after 10m24s
neuron's soul calls engram_recall_json (neuron-api.el:618, memory.el:80) and
cgi_principal (studio.el:72). Both existed in the runtime neuron vendored
(v1.0.0-20260501) and were absent here, so the soul could not link against
current el at all.

The dangerous part is what the obvious "fix" would have done. These look like
redundant wrappers over one impl:

    engram_search_json(q, limit)  -> eg_search_json_impl(q, limit, 0)  LEXICAL
    engram_recall_json(q, limit)  -> eg_search_json_impl(q, limit, 1)  SEMANTIC

They are not interchangeable, and the split is documented at neuron-api.el:613:
search stays LEXICAL because ~40 internal call sites pass a KEY and seven of
them DELETE every record returned. Point those at a semantic matcher and they
delete fuzzy matches. Conversely, pointing recall at search silently downgrades
the mind's entire retrieval surface from semantic to lexical — no error, just
permanently worse recall.

Implemented over engram_activate(), which in this runtime already IS the
semantic path the old with_legs=1 branch built by hand (embeds the query via
eg_embed_fetch, scores by cosine, then spreads activation one hop). Output
shape matches engram_search_json — a flat array via engram_emit_node_json —
because callers parse search's shape, not activate's envelope.

Verified: neuron's soul now compiles and links against current el, boots, and
serves /health with layers initialized.

NOTE for follow-up: current el also ships engram_retrieve_geometric_json, a
structure-first retrieval that appears to be the intended successor to recall.
Repointing the two recall call sites at it may well be the right end state and
would remove the two-wrapper shape entirely — but that is a behavioral change
that must be measured against neuron/tools/retrieval-eval/'s gold set, not
assumed. This commit preserves existing behavior exactly; it does not decide
that question.
2026-08-15 19:56:43 -05:00
will.anderson d545b69614 Merge pull request 'runtime: restore the three builtins that made elc unrebuildable' (#125) from fix/elc-rebuildable-compiler-builtins into dev
El SDK CI - dev / build-and-test (push) Failing after 11m4s
2026-08-16 00:50:43 +00:00
bigmerge 598915cc61 runtime: restore the three builtins that made elc unrebuildable
El SDK CI - dev / build-and-test (pull_request) Failing after 3m52s
The committed elc binary could not be refreshed from its own source. Rebuilding
failed with three implicit-declaration errors: el_mem_check, stdout_to_file,
stdout_restore. The compiler's own source calls all three (compiler.el:472,479,574
and codegen.el:4248) and two are registered in codegen.el's builtin_arity table —
but none were defined in this runtime.

They were found intact in ui/examples/native-hello-ios/NativeHello/el_runtime.c,
a divergent private copy of this runtime that still carried them. Ported verbatim.

Consequence of them being missing: the canonical elc binary was frozen. Source
gained @route dispatch codegen (emit_route_dispatch, codegen.el:3948) and the
@manager boundary-beat seam, but no rebuilt binary could carry them, so
neuron's soul — whose routes.el now calls the compiler-synthesized
el_route_dispatch — could not be built at all.

Verified after the fix:
  - elc rebuilds from current source, clean.
  - Self-hosting fixpoint byte-identical (stage3 == stage2).
  - The rebuilt elc emits el_route_dispatch (2 occurrences in the soul amalgam,
    previously 0) and injects engram_boundary_beat at @manager boundaries,
    i.e. the decorator seam is live rather than inert.

el_mem_check is itself the compiler's memory guard (ELC_MAX_MEM_MB, default
512MB, self-terminates before the OS OOM-killer fires) — so the runtime was
missing the very guard that would have surfaced the compiler's memory blowup
as a clean error instead of a 27GB host-killer.
2026-08-15 19:50:17 -05:00
will.anderson dab14f9100 Merge pull request 'engram: fix silently-wrong query params + make el_seed.o/el_runtime.o link' (#124) from fix/engram-query-param-and-seed-link into dev
El SDK CI - dev / build-and-test (push) Failing after 14m30s
2026-08-16 00:34:33 +00:00
5 changed files with 706 additions and 8 deletions
+10
View File
@@ -2764,6 +2764,11 @@ fn builtin_arity(name: String) -> Int {
if str_eq(name, "__engram_node_full_in") { return 9 }
if str_eq(name, "__engram_connect_in") { return 5 }
if str_eq(name, "__engram_scan_nodes_json") { return 2 }
if str_eq(name, "__engram_edges_json") { return 2 }
if str_eq(name, "__engram_pool_stats_json") { return 0 }
if str_eq(name, "__el_alloc_count") { return 0 }
if str_eq(name, "__el_alloc_bytes") { return 0 }
if str_eq(name, "__el_peak_rss") { return 0 }
if str_eq(name, "__generate") { return 1 }
// Filesystem
if str_eq(name, "fs_read") { return 1 }
@@ -2862,6 +2867,11 @@ fn builtin_arity(name: String) -> Int {
if str_eq(name, "engram_get_node_by_label") { return 1 }
if str_eq(name, "engram_search_json") { return 2 }
if str_eq(name, "engram_scan_nodes_json") { return 2 }
if str_eq(name, "engram_edges_json") { return 2 }
if str_eq(name, "engram_pool_stats_json") { return 0 }
if str_eq(name, "el_alloc_count") { return 0 }
if str_eq(name, "el_alloc_bytes") { return 0 }
if str_eq(name, "el_peak_rss") { return 0 }
if str_eq(name, "engram_neighbors_json") { return 3 }
if str_eq(name, "engram_activate_json") { return 2 }
if str_eq(name, "engram_stats_json") { return 0 }
+275 -2
View File
@@ -155,21 +155,55 @@ el_val_t el_arena_pop(el_val_t mark) {
return 0;
}
/* ── Allocation accounting ───────────────────────────────────────────────────
*
* Every string allocation in the runtime funnels through the four functions
* below, so counting here counts everything the language does.
*
* WHY THIS EXISTS: a growth-curve gate needs a signal that is DETERMINISTIC.
* Wall-clock needs statistics, warmup, and a quiet machine; it is noisy on
* shared CI and unusable as a hard build gate. Allocation COUNT has none of
* those problems the same input allocates the same number of times on every
* machine, every run. Fit allocations against input size and a complexity
* regression becomes a build failure with zero flake.
*
* This is not hypothetical. elc's known defect is quadratic ALLOCATION VOLUME.
* The old shipped binary paid it in RSS (27 GB, OOM); the rebuilt one pays the
* same quadratic in malloc/free churn (42s on a 1.4 MB input). The allocation
* count was the invariant across both RSS and wall-clock were just the two
* ways it surfaced. An `expect allocs O(n)` assertion on the compile path
* would have failed the build the day it was introduced.
*
* Peak RSS is exported too but is explicitly NOT the gating signal: it is
* perturbed by allocator behaviour, page cache, and the OS. Gate on counts,
* report RSS as context.
*
* Counters are plain unsigned longs, incremented on the allocating thread with
* no synchronisation: this is measurement, and a lock here would change the
* thing being measured. Under threads the count is approximate; for the
* single-threaded compile path it is exact.
* */
static unsigned long _el_alloc_count = 0;
static unsigned long _el_alloc_bytes = 0;
/* Persistent allocation — bypasses the arena (state_set, engram internals). */
static char* el_strdup_persist(const char* s) {
if (!s) return strdup("");
if (!s) { _el_alloc_count++; _el_alloc_bytes += 1; return strdup(""); }
_el_alloc_count++; _el_alloc_bytes += strlen(s) + 1;
return strdup(s);
}
static char* el_strbuf_persist(size_t n) {
char* p = malloc(n + 1);
if (!p) { fputs("el_runtime: out of memory\n", stderr); exit(1); }
p[0] = '\0';
_el_alloc_count++; _el_alloc_bytes += n + 1;
return p;
}
static char* el_strdup(const char* s) {
if (!s) { char* p = strdup(""); el_arena_track(p); return p; }
if (!s) { char* p = strdup(""); _el_alloc_count++; _el_alloc_bytes += 1; el_arena_track(p); return p; }
char* p = strdup(s);
_el_alloc_count++; _el_alloc_bytes += strlen(s) + 1;
el_arena_track(p);
return p;
}
@@ -178,6 +212,7 @@ static char* el_strbuf(size_t n) {
char* p = malloc(n + 1);
if (!p) { fputs("el_runtime: out of memory\n", stderr); exit(1); }
p[0] = '\0';
_el_alloc_count++; _el_alloc_bytes += n + 1;
el_arena_track(p);
return p;
}
@@ -18204,3 +18239,241 @@ el_val_t __http_do(el_val_t m, el_val_t u, el_val_t b, el_val_t h, el_val_t t) {
el_val_t __http_do_map(el_val_t m, el_val_t u, el_val_t b, el_val_t h, el_val_t t) { (void)m; (void)u; (void)b; (void)h; (void)t; return _no_curl_err(); }
el_val_t __http_do_map_to_file(el_val_t m, el_val_t u, el_val_t b, el_val_t h, el_val_t p) { (void)m; (void)u; (void)b; (void)h; (void)p; return _no_curl_err(); }
#endif /* !HAVE_CURL */
/* ── Compiler-support builtins ───────────────────────────────────────────────
* stdout_to_file / stdout_restore / el_mem_check are called by the El compiler's
* own source (compiler.el:472,479,574 and codegen.el:4248) and are registered in
* codegen.el's builtin_arity table, but were missing from this runtime so
* rebuilding elc from source failed with three implicit-declaration errors and
* the committed elc binary could never be refreshed. The definitions below are
* ported verbatim from ui/examples/native-hello-ios/NativeHello/el_runtime.c,
* a divergent private copy of this runtime that still carried them.
* */
#include <sys/resource.h>
static int _el_saved_stdout_fd = -1;
/* Redirect process stdout to a file; used by the compiler's JS post-processing
* pipeline to capture codegen output before piping it onward. */
el_val_t stdout_to_file(el_val_t pathv) {
const char* path = EL_CSTR(pathv);
if (!path) return (el_val_t)(int64_t)-1;
fflush(stdout);
_el_saved_stdout_fd = dup(STDOUT_FILENO);
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
if (fd < 0) return (el_val_t)(int64_t)-1;
dup2(fd, STDOUT_FILENO);
close(fd);
return (el_val_t)(int64_t)0;
}
el_val_t stdout_restore(void) {
if (_el_saved_stdout_fd >= 0) {
fflush(stdout);
dup2(_el_saved_stdout_fd, STDOUT_FILENO);
close(_el_saved_stdout_fd);
_el_saved_stdout_fd = -1;
}
return (el_val_t)(int64_t)0;
}
/* el_mem_check — self-terminating memory guard for long-running compiler runs.
* Called periodically by the compiler to catch runaway growth before the OS
* OOM-killer fires. Limit comes from ELC_MAX_MEM_MB (default 512 MB).
* macOS reports ru_maxrss in bytes, Linux in kilobytes; normalised to MB. */
el_val_t el_mem_check(void) {
long limit_mb = 512;
const char* env_val = getenv("ELC_MAX_MEM_MB");
if (env_val && *env_val) {
long v = atol(env_val);
if (v > 0) limit_mb = v;
}
struct rusage ru;
if (getrusage(RUSAGE_SELF, &ru) != 0) return 0; /* can't read — skip check */
long rss_mb;
#if defined(__APPLE__) || defined(__MACH__)
rss_mb = (long)(ru.ru_maxrss / (1024L * 1024L));
#else
rss_mb = (long)(ru.ru_maxrss / 1024L);
#endif
if (rss_mb >= limit_mb) {
fprintf(stderr, "elc: memory limit exceeded (%ldMB), aborting\n", limit_mb);
exit(1);
}
return 0;
}
/* ── engram_recall_json / cgi_* accessors — restored 2026-08-15 ──────────────
*
* These existed in the runtime neuron vendored (v1.0.0-20260501) and were lost
* when this runtime moved on, so a soul built against current el would fail to
* link and, worse, the naive "fix" of pointing recall at engram_search_json
* would have SILENTLY DOWNGRADED the mind's whole retrieval surface from
* semantic to lexical, with no error at any layer.
*
* The lexical/semantic split is a real safety boundary, not redundant naming
* (neuron-api.el:613 documents it): engram_search_json stays LEXICAL because
* ~40 internal call sites pass a KEY and seven of them DELETE every record
* returned making those semantic would delete fuzzy matches. recall is the
* SEMANTIC surface, used by the retrieval routes.
*
* The old implementation was eg_search_json_impl(q, limit, with_legs=1): embed
* the query, cosine over the corpus, then a graph leg from semantic seeds.
* In this runtime that is exactly what engram_activate() already does (it
* embeds via eg_embed_fetch, scores by cosine, then spreads activation), so
* recall delegates to it rather than re-deriving a second semantic path.
* Output shape matches engram_search_json a flat array of node objects via
* engram_emit_node_json because existing callers (memory.el:80,
* neuron-api.el:618) parse it as search's shape, not activate's envelope.
* */
el_val_t engram_recall_json(el_val_t query, el_val_t limit) {
int64_t lim = (int64_t)limit;
if (lim <= 0) lim = 100;
/* depth 1: the associative leg, one hop out from the semantic seeds. */
el_val_t lst = engram_activate(query, (el_val_t)(int64_t)1);
ElList* arr = (ElList*)(uintptr_t)lst;
JsonBuf b; jb_init(&b);
jb_putc(&b, '[');
int64_t emitted = 0;
if (arr) {
for (int64_t i = 0; i < arr->length && emitted < lim; i++) {
if (!arr->elems[i]) continue;
el_val_t node_map = el_map_get(arr->elems[i], EL_STR("node"));
el_val_t id_v = el_map_get(node_map, EL_STR("id"));
const char* id_s = EL_CSTR(id_v);
EngramNode* n = id_s ? engram_find_node(id_s) : NULL;
if (!n) continue;
if (emitted > 0) jb_putc(&b, ',');
engram_emit_node_json(&b, n, 0);
emitted++;
}
}
jb_putc(&b, ']');
return el_wrap_str(b.buf);
}
/* cgi_* — read-only identity accessors over the process-wide CGI registration
* set by cgi_register(). Read-only by design: there is no setter (studio.el:66). */
el_val_t cgi_principal(void) { return EL_STR(_el_cgi_principal ? _el_cgi_principal : ""); }
el_val_t cgi_network(void) { return EL_STR(_el_cgi_network ? _el_cgi_network : ""); }
el_val_t cgi_engram(void) { return EL_STR(_el_cgi_engram ? _el_cgi_engram : ""); }
/* engram_edges_json(limit, offset) — emit edges straight from the store.
*
* Replaces a serialize-and-reread round trip that took production down on
* 2026-08-15: /api/graph/edges called engram_save() to write the ENTIRE graph
* to disk (128 MB) and then fs_read it back, just to answer a read query for
* edges. One debug request cost a full snapshot write, a 128 MB read, and the
* peak memory to hold it on top of being O(whole graph) for a bounded slice.
* The route's own comment had already named the fix: "Future: add an
* engram_edges_json() builtin and drop the file round trip entirely."
*
* limit <= 0 defaults to 1000 rather than unbounded: this is the endpoint that
* fell over, and an unbounded default would preserve the failure mode under a
* different name. Pass an explicit limit to page.
*/
el_val_t engram_edges_json(el_val_t limit, el_val_t offset) {
EngramStore* g = engram_get();
int64_t lim = (int64_t)limit; if (lim <= 0) lim = 1000;
int64_t off = (int64_t)offset; if (off < 0) off = 0;
JsonBuf b; jb_init(&b);
jb_putc(&b, '[');
int64_t emitted = 0;
char t[192];
for (int64_t i = off; i < g->edge_count && emitted < lim; i++) {
EngramEdge* e = &g->edges[i];
if (emitted > 0) jb_putc(&b, ',');
jb_puts(&b, "{\"id\":"); jb_emit_escaped(&b, e->id ? e->id : "");
jb_puts(&b, ",\"from_id\":"); jb_emit_escaped(&b, e->from_id ? e->from_id : "");
jb_puts(&b, ",\"to_id\":"); jb_emit_escaped(&b, e->to_id ? e->to_id : "");
jb_puts(&b, ",\"relation\":"); jb_emit_escaped(&b, e->relation ? e->relation : "");
snprintf(t, sizeof t,
",\"weight\":%.6g,\"hebb\":%.6g,\"confidence\":%.6g,"
"\"created_at\":%lld,\"updated_at\":%lld,\"last_fired\":%lld,"
"\"inhibitory\":%d,\"layer_id\":%u}",
e->weight, e->hebb, e->confidence,
(long long)e->created_at, (long long)e->updated_at,
(long long)e->last_fired, e->inhibitory, (unsigned)e->layer_id);
jb_puts(&b, t);
emitted++;
}
jb_putc(&b, ']');
return el_wrap_str(b.buf);
}
/* engram_pool_stats_json() — the buffer pool's interoception, exposed.
*
* StorePoolStats and store_pool_stats() already existed and were surfaced
* NOWHERE. On 2026-08-15 the engram thrashed itself to a standstill twice while
* these exact counters sat in memory, unread, and four wrong theories were tried
* from the outside instead. Sensing state is only corrective if the state can be
* read by the process itself (pc_adapt_budget) and by anything watching it.
*
* Serves the live numbers plus the derived signals that actually diagnose:
* hit_rate sustained low hit rate with high evictions is the thrash shape
* evict_ratio evictions per access; ~1 means every fetch displaces a live page
* pressure 1 when evicting into genuine reuse (working set > budget)
* cap_gib/resident_gib budget vs what is actually held
*/
el_val_t engram_pool_stats_json(void) {
if (!g_engram_store) return el_wrap_str(el_strdup("{\"store\":false}"));
StorePoolStats st;
store_pool_stats(g_engram_store, &st);
uint64_t acc = st.hits + st.misses;
double hit_rate = acc ? (double)st.hits / (double)acc : 0.0;
double evict_ratio = acc ? (double)st.evictions / (double)acc : 0.0;
int pressure = (acc > 100000 && evict_ratio > 0.33 && hit_rate > 0.25) ? 1 : 0;
char b[768];
snprintf(b, sizeof b,
"{\"store\":true,\"cap_frames\":%zu,\"resident_frames\":%zu,\"pinned\":%zu,"
"\"dirty\":%zu,\"prefetch\":%u,\"hits\":%llu,\"misses\":%llu,\"evictions\":%llu,"
"\"prefetch_reads\":%llu,\"hit_rate\":%.4f,\"evict_ratio\":%.4f,\"pressure\":%d,"
"\"cap_gib\":%.3f,\"resident_gib\":%.3f,\"page_size\":%u}",
st.cap, st.resident, st.pinned, st.dirty, st.prefetch,
(unsigned long long)st.hits, (unsigned long long)st.misses,
(unsigned long long)st.evictions, (unsigned long long)st.prefetch_reads,
hit_rate, evict_ratio, pressure,
(double)st.cap * (double)STORE_PAGE_SIZE / (1024.0*1024.0*1024.0),
(double)st.resident * (double)STORE_PAGE_SIZE / (1024.0*1024.0*1024.0),
(unsigned)STORE_PAGE_SIZE);
return el_wrap_str(el_strdup(b));
}
/* ── Allocation/RSS introspection (test-framework complexity gate, §6.5) ─────
*
* el_alloc_count() total runtime string allocations since process start.
* THE gating signal. Deterministic: same input => same count, every machine,
* every run. A benchmark harness samples it before and after an operation at
* several input sizes and fits the deltas against n; a curve worse than the
* declared one fails the build. No warmup, no statistics, no baseline file,
* no flake none of which is true of wall-clock.
*
* el_alloc_bytes() total bytes requested. Same determinism; catches the case
* where allocation COUNT stays linear but per-allocation SIZE grows, which is
* the classic accidental-quadratic shape (rebuilding a whole buffer per
* append). Count alone would miss it.
*
* el_peak_rss() peak resident set in bytes. Context, NOT a gate: perturbed by
* allocator internals, the page cache, and the OS. Reported so a human can
* see the physical consequence; never fitted.
*/
el_val_t el_alloc_count(void) { return (el_val_t)(int64_t)_el_alloc_count; }
el_val_t el_alloc_bytes(void) { return (el_val_t)(int64_t)_el_alloc_bytes; }
el_val_t el_peak_rss(void) {
struct rusage ru;
if (getrusage(RUSAGE_SELF, &ru) != 0) return (el_val_t)0;
#if defined(__APPLE__) || defined(__MACH__)
return (el_val_t)(int64_t)ru.ru_maxrss; /* macOS: bytes */
#else
return (el_val_t)(int64_t)(ru.ru_maxrss * 1024L); /* Linux: KB -> bytes */
#endif
}
+28
View File
@@ -1011,6 +1011,34 @@ el_val_t __uuid_v4(void);
/* Args */
el_val_t __args_json(void);
/* Compiler-support builtins — called by the El compiler's own source
* (compiler.el, codegen.el) and registered in codegen.el's builtin_arity. */
el_val_t stdout_to_file(el_val_t path);
el_val_t stdout_restore(void);
el_val_t el_mem_check(void);
/* Allocation accounting — the deterministic signal behind complexity gating.
* Gate on counts/bytes; peak RSS is context only. */
el_val_t el_alloc_count(void);
el_val_t el_alloc_bytes(void);
el_val_t el_peak_rss(void);
/* Semantic retrieval surface. NOT interchangeable with engram_search_json,
* which is lexical by design see the note at the definition. */
el_val_t engram_recall_json(el_val_t query, el_val_t limit);
/* Edges straight from the store — replaces the engram_save()+fs_read()
* whole-graph round trip that /api/graph/edges used to do. */
el_val_t engram_edges_json(el_val_t limit, el_val_t offset);
/* Buffer-pool interoception as JSON — live pool health for observation. */
el_val_t engram_pool_stats_json(void);
/* CGI identity accessors (read-only). */
el_val_t cgi_principal(void);
el_val_t cgi_network(void);
el_val_t cgi_engram(void);
#ifdef __cplusplus
}
#endif
+15
View File
@@ -1371,6 +1371,21 @@ el_val_t __engram_scan_nodes_json(el_val_t limit, el_val_t offset) {
return engram_scan_nodes_json(limit, offset);
}
el_val_t engram_edges_json(el_val_t limit, el_val_t offset);
el_val_t __engram_edges_json(el_val_t limit, el_val_t offset) {
return engram_edges_json(limit, offset);
}
el_val_t engram_pool_stats_json(void);
el_val_t __engram_pool_stats_json(void) { return engram_pool_stats_json(); }
el_val_t el_alloc_count(void);
el_val_t el_alloc_bytes(void);
el_val_t el_peak_rss(void);
el_val_t __el_alloc_count(void) { return el_alloc_count(); }
el_val_t __el_alloc_bytes(void) { return el_alloc_bytes(); }
el_val_t __el_peak_rss(void) { return el_peak_rss(); }
el_val_t __engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset) {
return engram_scan_nodes_by_type_json(node_type, limit, offset);
}
+378 -6
View File
@@ -44,6 +44,11 @@
#include <string.h>
#include <stdint.h>
#include <unistd.h>
#if defined(__APPLE__) || defined(__MACH__)
#include <sys/sysctl.h>
#include <mach/mach.h>
#include <mach/mach_host.h>
#endif
#include <fcntl.h>
#include <errno.h>
#include <time.h>
@@ -236,8 +241,16 @@ struct PgCache {
unsigned prefetch; /* read-ahead window (pages); 0 = off */
LayerPin* lp; size_t lp_n, lp_cap; /* hot-layer pin bookkeeping */
size_t dirty_count; /* # dirty frames, maintained incrementally (M5) */
/* stats (introspection only — never affect semantics) */
/* Interoception. These were "introspection only — never affect semantics",
* and that was the bug: the pool could not feel itself thrash, so it could
* not correct, and neither could anyone watching from outside. The sensed
* state IS the corrective mechanism (see pc_adapt_budget) the same way the
* engram's own boundary-beat/chronoception let it feel its own activity. */
uint64_t hits, misses, evictions, prefetch_reads;
/* sliding-window marks so pressure reflects NOW, not lifetime totals */
uint64_t adapt_last_acc, adapt_last_evic, adapt_last_hits;
uint64_t adapt_grows; /* budget corrections upward */
uint64_t adapt_shrinks; /* budget corrections downward (memory pressure) */
};
/* ── little-endian scalar codecs ──────────────────────────────────────────── */
@@ -334,6 +347,51 @@ static uint64_t dh_node_hash(const StoreNode* n){
return h;
}
/* dh_edge_hash — the edge counterpart of dh_node_hash.
*
* WHY THIS EXISTS (2026-08-15): the write barrier was node-only. Checkpointing
* pushes the WHOLE resident graph through store_put_node/store_put_edge (see
* engram_store_checkpoint), and nodes were cheaply skipped when unchanged
* a hash compare, no page I/O. Edges had no such check, so every edge was
* rewritten on every checkpoint, and each rewrite runs the idempotency probe
* max_page_lsn_for_id btree lookup page_read per stored copy.
*
* Edges outnumber nodes roughly 3:1 here (37,663 vs 13,430), so this turned
* routine checkpointing into a FULL-STORE WALK in id order random page access
* across the entire 2 GiB store, repeated, mostly to rediscover that nothing
* had changed. That walk is the failure mode: with a page cache smaller than
* the store it degenerates into thrashing and the engram never makes progress.
* Sizing the cache around that walk treats the symptom; the walk itself should
* not happen.
*
* The discriminator byte keeps the edge keyspace from ever colliding with a
* node of the same id in the shared dh map: distinct kinds cannot produce the
* same hash, so a stale skip is not reachable by collision. */
static uint64_t dh_edge_hash(const StoreEdge* e){
uint64_t h = 1469598103934665603ULL;
const uint8_t kind = 0xE0; /* edge discriminator */
dh_fold_bytes(&h, &kind, 1);
dh_fold_str(&h, e->id);
dh_fold_str(&h, e->from_id);
dh_fold_str(&h, e->to_id);
dh_fold_str(&h, e->relation);
dh_fold_str(&h, e->metadata);
uint8_t t8[8];
put_f64(t8, e->weight); dh_fold_bytes(&h, t8, 8);
put_f64(t8, e->hebb); dh_fold_bytes(&h, t8, 8);
put_f64(t8, e->confidence); dh_fold_bytes(&h, t8, 8);
uint8_t t4[4];
put_u32(t4, (uint32_t)e->inhibitory); dh_fold_bytes(&h, t4, 4);
put_u32(t4, e->layer_id); dh_fold_bytes(&h, t4, 4);
/* created_at/updated_at/last_fired are deliberately EXCLUDED: last_fired is
* touched by activation without changing what the edge IS, and including it
* would defeat the barrier on exactly the hot edges it most needs to skip.
* The fields that define the edge's durable content are all folded above. */
if (e->unknown && e->unknown_len) dh_fold_bytes(&h, e->unknown, e->unknown_len);
if (h == 0) h = 1; /* reserve 0 as "absent" in the map */
return h;
}
/* Open-addressing id(string)→durable-hash map. Keyed for O(1) bucketing on the
* id's FNV hash, compared by strcmp for correctness (full-id discipline, matching
* store_scan_*'s StrSet). Values are the 64-bit durable hash. */
@@ -1531,6 +1589,11 @@ int store_scan_edges(EngramPagedStore* s, StoreEdgeScanCb cb, void* ctx){
if (cand.id && *cand.id && strset_add(&seen, cand.id)){
StoreEdge canon;
if (store_get_edge(s, cand.id, &canon) == 1){
/* seed the write-barrier map from on-disk truth so the FIRST
* post-boot checkpoint full-walk already skips unchanged edges
* (mirrors store_scan_nodes; without it the barrier is empty at
* boot and the first checkpoint re-probes every edge) */
if (s->barrier_on) dh_set(s->dh, canon.id, dh_edge_hash(&canon));
cb(&canon, ctx); count++; /* canonical latest-live */
store_edge_free(&canon);
}
@@ -1594,19 +1657,76 @@ int store_scan_edges(EngramPagedStore* s, StoreEdgeScanCb cb, void* ctx){
* matches disk, so a re-fault reproduces identical bytes.
* */
/* default frame budget: large enough that today's whole store stays resident
* (== Phase 1). Override with env ENGRAM_POOL_FRAMES (0 = unlimited). */
#ifndef ENGRAM_POOL_FRAMES_DEFAULT
#define ENGRAM_POOL_FRAMES_DEFAULT (1u<<20) /* ~1M frames × 16KiB = 16 GiB */
/* ── Frame budget ────────────────────────────────────────────────────────────
*
* A FIXED frame count cannot be correct. It has no relationship to either
* quantity that decides whether a cache works: the size of the working set, or
* the memory actually available on the host. It is the same number on a 16 GB
* laptop and a 256 GB server, and it stays put while the store grows.
*
* That is not hypothetical. On 2026-08-15 the deployment pinned
* ENGRAM_POOL_FRAMES=65536 (1 GiB) while neuron.egm grew to 2.1 GiB. The
* working set was twice the budget, so boot-time WAL replay which walks
* pages in an order uncorrelated with reuse evicted each page shortly before
* it was needed again. The engram spun at 100% CPU inside pc_evict_to_budget
* and never bound its port. Not slow: making no progress. Denning's thrashing,
* exactly, and no eviction policy can fix it when the working set does not
* fit, only more frames or admission control help.
*
* So the budget is DERIVED, from the host's physical memory, and it scales
* with the machine instead of pretending memory is a constant.
*
* ENGRAM_POOL_FRAMES explicit frame count; 0 = unlimited. Overrides all.
* Prefer leaving it unset a hand-set number is how
* this failure happened.
* ENGRAM_POOL_MEM_PCT percent of physical RAM to budget (default 60).
*
* Fallback when RAM cannot be read is 16 GiB worth of frames the old
* default, retained only as a floor for that case.
* */
#ifndef ENGRAM_POOL_FRAMES_FALLBACK
#define ENGRAM_POOL_FRAMES_FALLBACK (1u<<20) /* ~1M frames × 16KiB = 16 GiB */
#endif
static uint64_t pc_available_ram(void); /* fwd — defined with the controller */
/* Physical RAM in bytes, 0 when it cannot be determined. */
static uint64_t pc_physical_ram(void){
#if defined(__APPLE__) || defined(__MACH__)
uint64_t v = 0; size_t len = sizeof v;
int mib[2] = { CTL_HW, HW_MEMSIZE };
if (sysctl(mib, 2, &v, &len, NULL, 0) == 0) return v;
return 0;
#else
long pages = sysconf(_SC_PHYS_PAGES);
long psz = sysconf(_SC_PAGESIZE);
if (pages > 0 && psz > 0) return (uint64_t)pages * (uint64_t)psz;
return 0;
#endif
}
static size_t pc_default_cap(void){
unsigned pct = 60;
const char* p = getenv("ENGRAM_POOL_MEM_PCT");
if (p && *p){ unsigned long v = strtoul(p, NULL, 10); if (v > 0 && v <= 95) pct = (unsigned)v; }
uint64_t ram = pc_physical_ram();
if (!ram) return ENGRAM_POOL_FRAMES_FALLBACK;
uint64_t budget_bytes = (ram / 100u) * pct;
/* Never start above what the machine can actually spare right now. */
uint64_t avail = pc_available_ram();
if (avail > (1ull<<30) && budget_bytes > avail - (1ull<<30)) budget_bytes = avail - (1ull<<30);
uint64_t frames = budget_bytes / (uint64_t)STORE_PAGE_SIZE;
if (frames < 4096) frames = 4096; /* never absurdly small */
return (size_t)frames;
}
static PgCache* pc_new(void){
PgCache* c = (PgCache*)calloc(1, sizeof *c);
if (!c) return NULL;
c->nbuckets = 1024;
c->buckets = (PgEnt**)calloc(c->nbuckets, sizeof(PgEnt*));
if (!c->buckets){ free(c); return NULL; }
c->cap = ENGRAM_POOL_FRAMES_DEFAULT;
c->cap = pc_default_cap();
c->prefetch = 8;
const char* pf = getenv("ENGRAM_POOL_FRAMES");
if (pf && *pf){ char* end=NULL; unsigned long long v = strtoull(pf,&end,10); c->cap = (size_t)v; }
@@ -1676,6 +1796,241 @@ static void pc_remove(PgCache* c, PgEnt* e){
/* Reclaim clean unpinned frames from the LRU end until under budget, or until no
* evictable frame remains (a dirty/pinned-heavy pool may transiently exceed cap
* that is the no-steal guarantee, not a bug: the next checkpoint frees them). */
/* ── Adaptive budget: close the loop ─────────────────────────────────────────
*
* THE LESSON THIS ENCODES (2026-08-15). The engram spent hours down while four
* separate theories were tried bad binary, corrupt snapshot, WAL replay,
* feature flags because nothing in the system said what was happening. It
* looked identical to "busy loading": 100% CPU, flat RSS, no output. Meanwhile
* hits/misses/evictions were ALREADY being counted, right here, and surfaced
* nowhere. One eviction-rate number would have ended it in seconds.
*
* So the counters are not decoration. They are the control signal.
*
* A budget chosen once a literal like 65536, or 60% of RAM read at startup
* is a guess about the future. It cannot know the store grew, the working set
* shifted, or another process took the memory. The cache already MEASURES the
* only thing that matters (am I evicting pages I am about to want again), so it
* should act on that measurement instead of on a number someone typed.
*
* The controller: over a sliding window, if evictions are running at a rate
* comparable to accesses AND there is genuine reuse (hits are material), the
* working set exceeds the budget grow it. Growth is geometric, bounded by a
* live re-read of physical memory rather than a value cached at boot, so it
* tracks the machine instead of a snapshot of it. It never shrinks on its own:
* cap is a ceiling, not an allocation, and frames are only ever held because a
* real access put them there.
*
* Two things this deliberately does NOT do: it does not attempt a cleverer
* eviction policy (when the working set does not fit, no policy helps that is
* Denning, and it is why "tune the LRU" was never the fix), and it does not stay
* silent (pool_report exposes the same numbers outward, so a human or a metric
* pipeline sees the pressure the controller is reacting to). */
/* El's native telemetry, already in the runtime and already exporting to OTLP.
* Declared weak so engram_store.c still links standalone; when the runtime is
* present (every real build) the pool's interoception flows into the SAME
* pipeline as every other metric.
*
* ONE emission carrying the whole sensed state not a function per stat, and
* not a bespoke per-subsystem endpoint. Both of those are the degenerate case:
* they make observability something you hand-write per noun instead of a
* uniform mechanism every component already has. el_val_t is int64_t; strings
* ride as pointers cast through it (see el_runtime.h's value model). */
__attribute__((weak)) int64_t emit_log(int64_t level, int64_t msg, int64_t fields_json);
static void pc_report(const PgCache* c, const char* cause){
if (!emit_log) return; /* runtime not linked: no-op */
uint64_t acc = c->hits + c->misses;
char f[512];
snprintf(f, sizeof f,
"{\"component\":\"engram.pool\",\"cause\":\"%s\",\"hits\":%llu,\"misses\":%llu,"
"\"evictions\":%llu,\"prefetch_reads\":%llu,\"cap_frames\":%zu,\"resident\":%zu,"
"\"dirty\":%zu,\"grows\":%llu,\"hit_rate\":%.4f,\"evict_ratio\":%.4f,"
"\"cap_gib\":%.3f,\"resident_gib\":%.3f}",
cause,
(unsigned long long)c->hits, (unsigned long long)c->misses,
(unsigned long long)c->evictions, (unsigned long long)c->prefetch_reads,
c->cap, c->count, c->dirty_count, (unsigned long long)c->adapt_grows,
acc ? (double)c->hits / (double)acc : 0.0,
acc ? (double)c->evictions / (double)acc : 0.0,
(double)c->cap * (double)STORE_PAGE_SIZE / (1024.0*1024.0*1024.0),
(double)c->count * (double)STORE_PAGE_SIZE / (1024.0*1024.0*1024.0));
emit_log((int64_t)(uintptr_t)"warn", (int64_t)(uintptr_t)"engram.pool pressure",
(int64_t)(uintptr_t)f);
}
static uint64_t pc_ram_bytes_live(void){ return pc_physical_ram(); }
/* AVAILABLE memory right now — free + reclaimable, not total.
*
* Sizing a cache against TOTAL ram is what turns a cache into a memory leak:
* total does not shrink when other processes need memory, so a pool that only
* grows never notices it is starving the machine it runs on. Availability does.
* Returns 0 when undeterminable callers then refuse to grow, the safe way. */
static uint64_t pc_available_ram(void){
#if defined(__APPLE__) || defined(__MACH__)
/* SWAP AND COMPRESSOR FIRST. free+inactive+purgeable is a LIE under memory
* pressure: a machine deep in swap still reports gigabytes "available",
* because inactive pages are only reclaimable by evicting them to swap.
* Observed 2026-08-15: this returned 9.43 GiB available while vm.swapusage
* showed 51.58 of 53.25 GiB used (97% full) and the compressor occupied
* 23.7 GiB the host was thrashing to disk and the pool would have been
* cleared to grow into it. Growing a cache in that state is how a guard
* becomes the crash.
*
* So: if swap is nearly spent, report ZERO available. Callers refuse to
* grow on 0 and pc_relieve_pressure hands frames back. Only when the
* machine is genuinely not swapping do free+inactive+purgeable mean
* anything, and even then the compressor's footprint is subtracted because
* that RAM is already spoken for. */
/* RATE, NOT LEVEL. Swap *level* is a terrible signal: macOS grows swap files
* on demand and reclaims them lazily, so "47 of 48 GiB used" can mean the
* machine is dying OR that it recovered ten minutes ago and the file has not
* been trimmed yet. Measured both states on one host within minutes:
* 47.65/48.00 GiB used, 2047 swapouts/s -> genuinely thrashing
* 26.67/28.00 GiB used, 0 swapouts/s -> perfectly healthy, 15.6 GiB free
* A level check calls the second one an emergency and starves the pool for
* no reason. What distinguishes them is whether pages are moving NOW.
*
* So sample the swapout counter across calls and judge the delta. First call
* establishes the baseline and reports no pressure one sample cannot have
* a rate, and guessing from a single reading is the whole mistake. */
{
static uint64_t prev_swapouts = 0;
static time_t prev_t = 0;
static int primed = 0;
mach_port_t h0 = mach_host_self();
vm_statistics64_data_t v0; mach_msg_type_number_t c0 = HOST_VM_INFO64_COUNT;
if (host_statistics64(h0, HOST_VM_INFO64, (host_info64_t)&v0, &c0) == KERN_SUCCESS){
uint64_t now_out = (uint64_t)v0.swapouts;
time_t now_t = time(NULL);
if (!primed){ prev_swapouts = now_out; prev_t = now_t; primed = 1; }
else if (now_t > prev_t){
double per_s = (double)(now_out - prev_swapouts) / (double)(now_t - prev_t);
prev_swapouts = now_out; prev_t = now_t;
/* Sustained outward paging with nothing coming back is the
* signature of a host being pushed into swap. ~200 pages/s is
* ~3 MiB/s well above idle noise, well below the 2000+/s seen
* while actually thrashing. */
if (per_s > 200.0) return 0;
}
}
}
mach_port_t host = mach_host_self();
vm_size_t page = 0;
if (host_page_size(host, &page) != KERN_SUCCESS) return 0;
vm_statistics64_data_t vm; mach_msg_type_number_t cnt = HOST_VM_INFO64_COUNT;
if (host_statistics64(host, HOST_VM_INFO64, (host_info64_t)&vm, &cnt) != KERN_SUCCESS) return 0;
uint64_t avail = (uint64_t)vm.free_count + (uint64_t)vm.inactive_count
+ (uint64_t)vm.purgeable_count;
/* the compressor is holding real RAM that nobody can hand us */
uint64_t compressed = (uint64_t)vm.compressor_page_count;
if (compressed >= avail) return 0;
avail -= compressed;
return avail * (uint64_t)page;
#else
FILE* f = fopen("/proc/meminfo", "r");
if (!f) return 0;
char line[256]; unsigned long long kb = 0;
while (fgets(line, sizeof line, f))
if (sscanf(line, "MemAvailable: %llu kB", &kb) == 1) break;
fclose(f);
return (uint64_t)kb * 1024ull;
#endif
}
/* Shrink the budget when the machine is short on memory.
*
* A pool that can only grow is a leak with extra steps. This is the other half
* of the control loop: if free memory drops below a floor, hand frames back.
* The resident set follows on the next eviction pass, so the memory is actually
* returned rather than merely re-labelled. */
#ifndef ENGRAM_POOL_FREE_FLOOR_BYTES
#define ENGRAM_POOL_FREE_FLOOR_BYTES (2ull*1024ull*1024ull*1024ull) /* 2 GiB */
#endif
static int pc_relieve_pressure(PgCache* c){
uint64_t avail = pc_available_ram();
if (!avail) return 0;
uint64_t floor_b = ENGRAM_POOL_FREE_FLOOR_BYTES;
const char* fe = getenv("ENGRAM_POOL_FREE_FLOOR_MB");
if (fe && *fe){ unsigned long v = strtoul(fe, NULL, 10); if (v) floor_b = (uint64_t)v * 1024ull * 1024ull; }
if (avail >= floor_b) return 0; /* machine has room */
if (!c->cap || c->count == 0) return 0;
size_t was = c->cap;
size_t want = c->count - (c->count / 4); /* give back ~25% of what we hold */
if (want < 4096) want = 4096;
if (want >= c->cap) return 0;
c->cap = want;
c->adapt_shrinks++;
fprintf(stderr,
"[engram] memory pressure: %.2f GiB available (floor %.2f GiB) — shrinking pool "
"budget %zu -> %zu frames (%.2f -> %.2f GiB) and releasing frames.\n",
(double)avail/(1024.0*1024.0*1024.0), (double)floor_b/(1024.0*1024.0*1024.0),
was, c->cap,
(double)was * (double)STORE_PAGE_SIZE/(1024.0*1024.0*1024.0),
(double)c->cap* (double)STORE_PAGE_SIZE/(1024.0*1024.0*1024.0));
fflush(stderr);
return 1;
}
static void pc_adapt_budget(PgCache* c){
if (!c->cap) return; /* unlimited: nothing to adapt */
if (getenv("ENGRAM_POOL_FRAMES")) return; /* explicit operator override wins */
/* Sliding window so the signal reflects NOW, not lifetime totals. */
uint64_t acc = c->hits + c->misses;
if (acc - c->adapt_last_acc < 100000) return;
uint64_t d_acc = acc - c->adapt_last_acc;
uint64_t d_evic = c->evictions - c->adapt_last_evic;
uint64_t d_hits = c->hits - c->adapt_last_hits;
c->adapt_last_acc = acc; c->adapt_last_evic = c->evictions; c->adapt_last_hits = c->hits;
/* Pressure = evicting on a large fraction of accesses while still getting
* real reuse. Evictions alone are normal (a scan evicts and never returns);
* evictions WITH reuse means the working set genuinely does not fit. */
if (d_evic * 3 < d_acc) return; /* < 1/3 of accesses evict: healthy */
if (d_hits * 4 < d_acc) return; /* little reuse: a scan, not pressure */
/* Growth is bounded by what is AVAILABLE, never by total RAM. Sizing against
* total is how a cache starves its own host: total never shrinks when other
* processes need memory. Refuse to grow at all if availability is unknown or
* already under the floor a cache is never worth swapping the machine. */
uint64_t avail = pc_available_ram();
uint64_t floor_b = ENGRAM_POOL_FREE_FLOOR_BYTES;
const char* fe = getenv("ENGRAM_POOL_FREE_FLOOR_MB");
if (fe && *fe){ unsigned long v = strtoul(fe, NULL, 10); if (v) floor_b = (uint64_t)v * 1024ull * 1024ull; }
if (!avail || avail <= floor_b) return;
uint64_t ram = pc_ram_bytes_live();
if (!ram) return;
unsigned pct = 50; /* ceiling as a share of TOTAL, belt-and-braces */
const char* mp = getenv("ENGRAM_POOL_MAX_PCT");
if (mp && *mp){ unsigned long v = strtoul(mp, NULL, 10); if (v > 0 && v <= 95) pct = (unsigned)v; }
size_t ceiling = (size_t)(((ram / 100u) * pct) / (uint64_t)STORE_PAGE_SIZE);
/* and never grow into the free-memory floor */
uint64_t headroom = avail - floor_b;
size_t ceil_avail = (size_t)((c->count * (uint64_t)STORE_PAGE_SIZE + headroom)
/ (uint64_t)STORE_PAGE_SIZE);
if (ceil_avail < ceiling) ceiling = ceil_avail;
if (c->cap >= ceiling) return; /* already at the machine's limit */
size_t want = c->cap + (c->cap / 2) + 1; /* ×1.5, geometric */
if (want > ceiling) want = ceiling;
size_t was = c->cap;
c->cap = want;
c->adapt_grows++;
/* Emit the sensed state, not just the reaction. These are the numbers that
* would have diagnosed 2026-08-15 in seconds instead of hours. */
pc_report(c, "budget-grow");
fprintf(stderr,
"[engram] pool pressure: %llu evictions / %llu accesses (%llu hits) at %zu frames "
"(%.2f GiB) — working set exceeds budget; growing to %zu frames (%.2f GiB).\n",
(unsigned long long)d_evic, (unsigned long long)d_acc, (unsigned long long)d_hits,
was, (double)was * (double)STORE_PAGE_SIZE / (1024.0*1024.0*1024.0),
c->cap,(double)c->cap * (double)STORE_PAGE_SIZE / (1024.0*1024.0*1024.0));
fflush(stderr);
}
static void pc_evict_to_budget(PgCache* c){
if (!c->cap) return; /* unlimited */
while (c->count > c->cap){
@@ -1687,6 +2042,7 @@ static void pc_evict_to_budget(PgCache* c){
}
if (!freed) break; /* nothing evictable — allowed to exceed cap */
}
if (!pc_relieve_pressure(c)) pc_adapt_budget(c);
}
static PgEnt* pc_get(EngramPagedStore* s, uint64_t id){
@@ -2377,6 +2733,18 @@ int store_put_node(EngramPagedStore* s, const StoreNode* n){
int store_put_edge(EngramPagedStore* s, const StoreEdge* e){
if (!s || !e || !e->id || !e->from_id || !e->to_id) return -1;
STORE_GUARD(s);
/* Durable-hash write barrier — mirrors store_put_node. An unchanged edge
* costs one hash compare and zero page I/O; without this, checkpointing
* re-probed every edge against the paged store (max_page_lsn_for_id
* page_read), turning a routine checkpoint into a full-store walk. */
uint64_t dh_h = 0;
if (s->barrier_on){
dh_h = dh_edge_hash(e);
if (dh_get(s->dh, e->id) == dh_h){
s->stat_barrier_skips++;
return 0;
}
}
uint64_t L = ++s->next_lsn;
if (s->wal){
size_t blen; uint8_t* body = edge_serialize(e, &blen);
@@ -2386,6 +2754,10 @@ int store_put_edge(EngramPagedStore* s, const StoreEdge* e){
if (wr != 0) return -1;
}
int r = apply_edge_put(s, e, L);
if (r == 0 && s->barrier_on){
if (!dh_h) dh_h = dh_edge_hash(e);
dh_set(s->dh, e->id, dh_h); /* remember the now-persisted durable hash */
}
ckpt_maybe(s);
return r;
}