design: gate on THREE signals -- the alloc gate would have missed el #132

el #132's quadratic (strlen per character in str_char_code/str_slice) is
pure CPU and allocates NOTHING. Measured on three controlled specimens:

  specimen  allocs        bytes         time
  linear    2.00 -> O(n)  2.16 -> O(n)  2.05 -> O(n)
  accum     2.00 -> O(n)  3.99 -> O(n2) noisy
  compute   FLAT          FLAT          3.96 -> O(n2)

'compute' is #132's shape. A gate fitting only allocation count and bytes
classifies it FLAT and passes -- it would not have caught the defect it
was created for. The gate now fits time AND count AND bytes, failing if
any exceeds its declared curve.

Also: black_box is mandatory and consuming the result is NOT sufficient.
The first 'compute' reported 0us at every n while returning a correct n2 --
clang closed the loop to a multiply. Only an opaque call restored the curve.

Adds lang/tests/bench/fitprobe.el as the fitter's known-good/known-bad set,
so the classifier is provable without depending on a real bug existing.
Marks DESIGN.md 1.3 stale: test_compiler 3.58s -> 0.03s (119x).
This commit is contained in:
Neuron
2026-08-15 21:34:39 -05:00
parent 4c3414072b
commit 6291a35bb9
2 changed files with 140 additions and 0 deletions
+49
View File
@@ -61,6 +61,13 @@ Per test file, current build model:
| `cc` test .c → .o | 0.02s |
| link | 0.02s |
> **STALE as of el #132 — re-measured 2026-08-16.** The `test_compiler` figure below was
> *entirely* the `strlen`-per-character quadratic, now fixed. Re-measured on the same host:
> **3.58s → 0.03s (119x)**, and the 422 KB compiler concatenation likewise compiles in 0.03s.
> The table is retained only as the historical record that motivated the gate. The remaining
> per-file cost is the redundant `el_runtime.c` rebuild, which §9's compile-once architecture
> addresses.
Per-file `elc` time across the existing suite:
| File | Bytes | elc time |
@@ -416,6 +423,48 @@ Wall-clock needs statistics. **Allocation counts do not.** They are perfectly de
> a level. That is why the gate fits a curve across a sweep instead of comparing one number to a
> threshold.
> **Second correction, same day — THE ALLOCATION GATE ALONE WOULD HAVE MISSED THE REAL BUG.**
>
> el #132 found the actual elc quadratic: `strlen()` called inside `str_char_code()` and
> `str_slice()`, so the lexer rescanned the remaining input on every character. Pure CPU.
> **Zero allocation.** `str_char_code` is a bounds check and an index — it allocates nothing.
>
> Measured on three controlled specimens (`lang/.work/fitprobe.el`), growth ratio per doubling of
> n across n = 200/400/800/1600:
>
> | specimen | allocs | bytes | time | what it proves |
> |---|---|---|---|---|
> | `linear` — one alloc per item | 2.00 2.00 2.00 → **O(n)** | 2.16 2.07 2.23 → **O(n)** | 0.83 2.00 2.05 → **O(n)** | clean baseline |
> | `accum` — rebuilds accumulator | 2.00 2.00 2.00 → **O(n)** | 3.97 3.99 3.99 → **O(n²)** | noisy | count misses, **bytes catches** |
> | `compute` — n scans over n chars | 0 → **FLAT** | 0 → **FLAT** | 3.93 4.01 3.96 → **O(n²)** | **both alloc signals blind; only time catches** |
>
> `compute` is el #132's shape exactly. A gate fitting only allocation count and bytes classifies
> it as FLAT and passes it. **The gate as originally specified would not have caught the defect it
> was created for.**
>
> Therefore the gate fits **THREE** signals and fails if ANY exceeds its declared curve:
>
> ```
> bench "elc_compile" over n in [...] expect time O(n) allocs O(n) bytes O(n) { ... }
> ```
>
> - **allocs (count)** — deterministic, zero-noise. Catches per-item allocation growth.
> - **allocs (bytes)** — deterministic, zero-noise. Catches accumulator-rebuild quadratics that
> count cannot see.
> - **time** — noisy, needs the sweep and statistics. The ONLY signal that sees pure-compute
> complexity regressions. Gate on the fitted *exponent*, never on absolute duration, so CI
> hardware variance scales the coefficient and leaves the classification intact.
>
> The deterministic signals remain preferable where they apply — they need no statistics and are
> correct on the first run. They are simply not sufficient.
>
> **`black_box` is mandatory, and consuming the result is NOT enough.** The first version of
> `compute` accumulated `total + 1` in a nested loop and reported **0 µs at every n** while
> returning a numerically correct n². Clang recognised the idiom and closed the loop to a
> multiply. Feeding the result into output did not prevent it. Only making the inner operation an
> opaque external call restored the real curve. A benchmark harness that trusts the user to defeat
> the optimiser will silently measure nothing — and report success while doing it.
Instrument the runtime with allocation counters and fit *those* against n instead of time:
```el