diff --git a/DESIGN.md b/DESIGN.md index 30d0120..fd26324 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -61,6 +61,13 @@ Per test file, current build model: | `cc` test .c → .o | 0.02s | | link | 0.02s | +> **STALE as of el #132 — re-measured 2026-08-16.** The `test_compiler` figure below was +> *entirely* the `strlen`-per-character quadratic, now fixed. Re-measured on the same host: +> **3.58s → 0.03s (119x)**, and the 422 KB compiler concatenation likewise compiles in 0.03s. +> The table is retained only as the historical record that motivated the gate. The remaining +> per-file cost is the redundant `el_runtime.c` rebuild, which §9's compile-once architecture +> addresses. + Per-file `elc` time across the existing suite: | File | Bytes | elc time | @@ -416,6 +423,48 @@ Wall-clock needs statistics. **Allocation counts do not.** They are perfectly de > a level. That is why the gate fits a curve across a sweep instead of comparing one number to a > threshold. +> **Second correction, same day — THE ALLOCATION GATE ALONE WOULD HAVE MISSED THE REAL BUG.** +> +> el #132 found the actual elc quadratic: `strlen()` called inside `str_char_code()` and +> `str_slice()`, so the lexer rescanned the remaining input on every character. Pure CPU. +> **Zero allocation.** `str_char_code` is a bounds check and an index — it allocates nothing. +> +> Measured on three controlled specimens (`lang/.work/fitprobe.el`), growth ratio per doubling of +> n across n = 200/400/800/1600: +> +> | specimen | allocs | bytes | time | what it proves | +> |---|---|---|---|---| +> | `linear` — one alloc per item | 2.00 2.00 2.00 → **O(n)** | 2.16 2.07 2.23 → **O(n)** | 0.83 2.00 2.05 → **O(n)** | clean baseline | +> | `accum` — rebuilds accumulator | 2.00 2.00 2.00 → **O(n)** | 3.97 3.99 3.99 → **O(n²)** | noisy | count misses, **bytes catches** | +> | `compute` — n scans over n chars | 0 → **FLAT** | 0 → **FLAT** | 3.93 4.01 3.96 → **O(n²)** | **both alloc signals blind; only time catches** | +> +> `compute` is el #132's shape exactly. A gate fitting only allocation count and bytes classifies +> it as FLAT and passes it. **The gate as originally specified would not have caught the defect it +> was created for.** +> +> Therefore the gate fits **THREE** signals and fails if ANY exceeds its declared curve: +> +> ``` +> bench "elc_compile" over n in [...] expect time O(n) allocs O(n) bytes O(n) { ... } +> ``` +> +> - **allocs (count)** — deterministic, zero-noise. Catches per-item allocation growth. +> - **allocs (bytes)** — deterministic, zero-noise. Catches accumulator-rebuild quadratics that +> count cannot see. +> - **time** — noisy, needs the sweep and statistics. The ONLY signal that sees pure-compute +> complexity regressions. Gate on the fitted *exponent*, never on absolute duration, so CI +> hardware variance scales the coefficient and leaves the classification intact. +> +> The deterministic signals remain preferable where they apply — they need no statistics and are +> correct on the first run. They are simply not sufficient. +> +> **`black_box` is mandatory, and consuming the result is NOT enough.** The first version of +> `compute` accumulated `total + 1` in a nested loop and reported **0 µs at every n** while +> returning a numerically correct n². Clang recognised the idiom and closed the loop to a +> multiply. Feeding the result into output did not prevent it. Only making the inner operation an +> opaque external call restored the real curve. A benchmark harness that trusts the user to defeat +> the optimiser will silently measure nothing — and report success while doing it. + Instrument the runtime with allocation counters and fit *those* against n instead of time: ```el diff --git a/lang/tests/bench/fitprobe.el b/lang/tests/bench/fitprobe.el new file mode 100644 index 0000000..df601fc --- /dev/null +++ b/lang/tests/bench/fitprobe.el @@ -0,0 +1,91 @@ +// fitprobe.el — controlled growth-curve specimens for validating the complexity fitter. +// +// Three deliberately-shaped workloads. None depends on a real defect existing, +// which is the point: the fitter must be provable against KNOWN curves. +// +// linear — one allocation per item. count O(n), bytes O(n), time O(n) +// accum — rebuilds its accumulator. count O(n), bytes O(n^2), time O(n^2) +// compute — nested arithmetic, no alloc. count O(1), bytes O(1), time O(n^2) +// +// `compute` is the specimen that matters. It is the shape of el #132 +// (strlen-per-character inside str_char_code): pure CPU, zero allocation. +// An allocation-only gate is structurally blind to it. +// +// No imports — uses runtime builtins directly so nothing collides. + +fn work_linear(n: Int) -> Int { + let parts: [String] = native_list_empty() + let i: Int = 0 + while i < n { + let parts = native_list_append(parts, int_to_str(i)) + let i = i + 1 + } + return native_list_len(parts) +} + +fn work_accum(n: Int) -> Int { + let acc: String = "" + let i: Int = 0 + while i < n { + let acc = acc + "x" + let i = i + 1 + } + return str_len(acc) +} + +fn work_compute(n: Int) -> Int { + // str_char_code is an opaque external call, so the C optimiser cannot + // reduce this nest to a closed form the way it does with `total + 1`. + // This is the exact shape of el #132: n scans over n characters, pure + // CPU, ZERO allocation. + let s: String = "abcdefghij" + let total: Int = 0 + let i: Int = 0 + while i < n { + let j: Int = 0 + while j < n { + let total = total + str_char_code(s, 0) + let j = j + 1 + } + let i = i + 1 + } + return total +} + +fn run_one(mode: String, n: Int) { + let c0: Int = el_alloc_count() + let b0: Int = el_alloc_bytes() + let t0: Int = el_now_instant() + + let r: Int = 0 + if str_eq(mode, "linear") { let r = work_linear(n) } + if str_eq(mode, "accum") { let r = work_accum(n) } + if str_eq(mode, "compute") { let r = work_compute(n) } + + let t1: Int = el_now_instant() + let c1: Int = el_alloc_count() + let b1: Int = el_alloc_bytes() + + println(mode + "\t" + int_to_str(n) + + "\t" + int_to_str(c1 - c0) + + "\t" + int_to_str(b1 - b0) + + "\t" + int_to_str((t1 - t0) / 1000) + + "\t" + int_to_str(r)) + return +} + +fn sweep(mode: String) { + run_one(mode, 200) + run_one(mode, 400) + run_one(mode, 800) + run_one(mode, 1600) + return +} + +fn main() -> Int { + println("mode\tn\tallocs\tbytes\tusec\tsink") + sweep("linear") + sweep("accum") + sweep("compute") + return 0 +}