runtime: allocation accounting — the deterministic signal for complexity gating
El SDK CI - dev / build-and-test (pull_request) Failing after 12m7s

Implements the three primitives the test-framework design (DESIGN.md §6.5)
requires for gating on growth curves: el_alloc_count, el_alloc_bytes,
el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per
the project's C-builtin recipe.

WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build
failure, which means the signal cannot flake. Wall-clock needs warmup,
statistics, and a quiet machine; on shared CI it is unusable as a gate.
Allocation counts are perfectly deterministic — same input, same number, every
machine, every run. Fit them against n and a complexity regression becomes a
build failure with zero noise.

All four runtime string allocators (el_strdup, el_strbuf, and their _persist
variants) funnel every allocation the language performs, so instrumenting there
counts everything.

WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate.
Measured with two El programs, one allocating once per item, one rebuilding its
accumulator each iteration:

    n     linear allocs / bytes      quadratic allocs / bytes
    100        100 /    290               100 /   5,150
    200        200 /    690               200 /  20,300
    400        400 /  1,490               400 /  80,600
    800        800 /  3,090               800 / 321,200

The quadratic program's allocation COUNT is exactly linear — identical to the
healthy one. Counting allocations alone would have missed it completely. Bytes
catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 ->
converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0.

That shape — count linear, per-allocation size growing — is the classic
accidental quadratic, and it is exactly elc's defect: quadratic allocation
VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt
one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across
both; RSS and wall-clock were just the two ways it surfaced.

el_peak_rss is exported for context and is explicitly NOT a gating signal — it
is perturbed by allocator internals, the page cache, and the OS. Gate on the
deterministic numbers; report the physical one.

Counters are unsynchronised by design: this is measurement, and a lock would
change the thing being measured. Exact on the single-threaded compile path,
approximate under threads.
This commit is contained in:
bigmerge
2026-08-15 21:21:42 -05:00
parent 2240d26c32
commit 37bcf7eb74
4 changed files with 87 additions and 2 deletions
+68 -2
View File
@@ -155,21 +155,55 @@ el_val_t el_arena_pop(el_val_t mark) {
return 0;
}
/* ── Allocation accounting ───────────────────────────────────────────────────
*
* Every string allocation in the runtime funnels through the four functions
* below, so counting here counts everything the language does.
*
* WHY THIS EXISTS: a growth-curve gate needs a signal that is DETERMINISTIC.
* Wall-clock needs statistics, warmup, and a quiet machine; it is noisy on
* shared CI and unusable as a hard build gate. Allocation COUNT has none of
* those problems the same input allocates the same number of times on every
* machine, every run. Fit allocations against input size and a complexity
* regression becomes a build failure with zero flake.
*
* This is not hypothetical. elc's known defect is quadratic ALLOCATION VOLUME.
* The old shipped binary paid it in RSS (27 GB, OOM); the rebuilt one pays the
* same quadratic in malloc/free churn (42s on a 1.4 MB input). The allocation
* count was the invariant across both RSS and wall-clock were just the two
* ways it surfaced. An `expect allocs O(n)` assertion on the compile path
* would have failed the build the day it was introduced.
*
* Peak RSS is exported too but is explicitly NOT the gating signal: it is
* perturbed by allocator behaviour, page cache, and the OS. Gate on counts,
* report RSS as context.
*
* Counters are plain unsigned longs, incremented on the allocating thread with
* no synchronisation: this is measurement, and a lock here would change the
* thing being measured. Under threads the count is approximate; for the
* single-threaded compile path it is exact.
* */
static unsigned long _el_alloc_count = 0;
static unsigned long _el_alloc_bytes = 0;
/* Persistent allocation — bypasses the arena (state_set, engram internals). */
static char* el_strdup_persist(const char* s) {
if (!s) return strdup("");
if (!s) { _el_alloc_count++; _el_alloc_bytes += 1; return strdup(""); }
_el_alloc_count++; _el_alloc_bytes += strlen(s) + 1;
return strdup(s);
}
static char* el_strbuf_persist(size_t n) {
char* p = malloc(n + 1);
if (!p) { fputs("el_runtime: out of memory\n", stderr); exit(1); }
p[0] = '\0';
_el_alloc_count++; _el_alloc_bytes += n + 1;
return p;
}
static char* el_strdup(const char* s) {
if (!s) { char* p = strdup(""); el_arena_track(p); return p; }
if (!s) { char* p = strdup(""); _el_alloc_count++; _el_alloc_bytes += 1; el_arena_track(p); return p; }
char* p = strdup(s);
_el_alloc_count++; _el_alloc_bytes += strlen(s) + 1;
el_arena_track(p);
return p;
}
@@ -178,6 +212,7 @@ static char* el_strbuf(size_t n) {
char* p = malloc(n + 1);
if (!p) { fputs("el_runtime: out of memory\n", stderr); exit(1); }
p[0] = '\0';
_el_alloc_count++; _el_alloc_bytes += n + 1;
el_arena_track(p);
return p;
}
@@ -18411,3 +18446,34 @@ el_val_t engram_pool_stats_json(void) {
(unsigned)STORE_PAGE_SIZE);
return el_wrap_str(el_strdup(b));
}
/* ── Allocation/RSS introspection (test-framework complexity gate, §6.5) ─────
*
* el_alloc_count() total runtime string allocations since process start.
* THE gating signal. Deterministic: same input => same count, every machine,
* every run. A benchmark harness samples it before and after an operation at
* several input sizes and fits the deltas against n; a curve worse than the
* declared one fails the build. No warmup, no statistics, no baseline file,
* no flake none of which is true of wall-clock.
*
* el_alloc_bytes() total bytes requested. Same determinism; catches the case
* where allocation COUNT stays linear but per-allocation SIZE grows, which is
* the classic accidental-quadratic shape (rebuilding a whole buffer per
* append). Count alone would miss it.
*
* el_peak_rss() peak resident set in bytes. Context, NOT a gate: perturbed by
* allocator internals, the page cache, and the OS. Reported so a human can
* see the physical consequence; never fitted.
*/
el_val_t el_alloc_count(void) { return (el_val_t)(int64_t)_el_alloc_count; }
el_val_t el_alloc_bytes(void) { return (el_val_t)(int64_t)_el_alloc_bytes; }
el_val_t el_peak_rss(void) {
struct rusage ru;
if (getrusage(RUSAGE_SELF, &ru) != 0) return (el_val_t)0;
#if defined(__APPLE__) || defined(__MACH__)
return (el_val_t)(int64_t)ru.ru_maxrss; /* macOS: bytes */
#else
return (el_val_t)(int64_t)(ru.ru_maxrss * 1024L); /* Linux: KB -> bytes */
#endif
}