design: gate on THREE signals -- the alloc gate would have missed el #132
el #132's quadratic (strlen per character in str_char_code/str_slice) is pure CPU and allocates NOTHING. Measured on three controlled specimens: specimen allocs bytes time linear 2.00 -> O(n) 2.16 -> O(n) 2.05 -> O(n) accum 2.00 -> O(n) 3.99 -> O(n2) noisy compute FLAT FLAT 3.96 -> O(n2) 'compute' is #132's shape. A gate fitting only allocation count and bytes classifies it FLAT and passes -- it would not have caught the defect it was created for. The gate now fits time AND count AND bytes, failing if any exceeds its declared curve. Also: black_box is mandatory and consuming the result is NOT sufficient. The first 'compute' reported 0us at every n while returning a correct n2 -- clang closed the loop to a multiply. Only an opaque call restored the curve. Adds lang/tests/bench/fitprobe.el as the fitter's known-good/known-bad set, so the classifier is provable without depending on a real bug existing. Marks DESIGN.md 1.3 stale: test_compiler 3.58s -> 0.03s (119x).
This commit is contained in:
@@ -61,6 +61,13 @@ Per test file, current build model:
|
|||||||
| `cc` test .c → .o | 0.02s |
|
| `cc` test .c → .o | 0.02s |
|
||||||
| link | 0.02s |
|
| link | 0.02s |
|
||||||
|
|
||||||
|
> **STALE as of el #132 — re-measured 2026-08-16.** The `test_compiler` figure below was
|
||||||
|
> *entirely* the `strlen`-per-character quadratic, now fixed. Re-measured on the same host:
|
||||||
|
> **3.58s → 0.03s (119x)**, and the 422 KB compiler concatenation likewise compiles in 0.03s.
|
||||||
|
> The table is retained only as the historical record that motivated the gate. The remaining
|
||||||
|
> per-file cost is the redundant `el_runtime.c` rebuild, which §9's compile-once architecture
|
||||||
|
> addresses.
|
||||||
|
|
||||||
Per-file `elc` time across the existing suite:
|
Per-file `elc` time across the existing suite:
|
||||||
|
|
||||||
| File | Bytes | elc time |
|
| File | Bytes | elc time |
|
||||||
@@ -416,6 +423,48 @@ Wall-clock needs statistics. **Allocation counts do not.** They are perfectly de
|
|||||||
> a level. That is why the gate fits a curve across a sweep instead of comparing one number to a
|
> a level. That is why the gate fits a curve across a sweep instead of comparing one number to a
|
||||||
> threshold.
|
> threshold.
|
||||||
|
|
||||||
|
> **Second correction, same day — THE ALLOCATION GATE ALONE WOULD HAVE MISSED THE REAL BUG.**
|
||||||
|
>
|
||||||
|
> el #132 found the actual elc quadratic: `strlen()` called inside `str_char_code()` and
|
||||||
|
> `str_slice()`, so the lexer rescanned the remaining input on every character. Pure CPU.
|
||||||
|
> **Zero allocation.** `str_char_code` is a bounds check and an index — it allocates nothing.
|
||||||
|
>
|
||||||
|
> Measured on three controlled specimens (`lang/.work/fitprobe.el`), growth ratio per doubling of
|
||||||
|
> n across n = 200/400/800/1600:
|
||||||
|
>
|
||||||
|
> | specimen | allocs | bytes | time | what it proves |
|
||||||
|
> |---|---|---|---|---|
|
||||||
|
> | `linear` — one alloc per item | 2.00 2.00 2.00 → **O(n)** | 2.16 2.07 2.23 → **O(n)** | 0.83 2.00 2.05 → **O(n)** | clean baseline |
|
||||||
|
> | `accum` — rebuilds accumulator | 2.00 2.00 2.00 → **O(n)** | 3.97 3.99 3.99 → **O(n²)** | noisy | count misses, **bytes catches** |
|
||||||
|
> | `compute` — n scans over n chars | 0 → **FLAT** | 0 → **FLAT** | 3.93 4.01 3.96 → **O(n²)** | **both alloc signals blind; only time catches** |
|
||||||
|
>
|
||||||
|
> `compute` is el #132's shape exactly. A gate fitting only allocation count and bytes classifies
|
||||||
|
> it as FLAT and passes it. **The gate as originally specified would not have caught the defect it
|
||||||
|
> was created for.**
|
||||||
|
>
|
||||||
|
> Therefore the gate fits **THREE** signals and fails if ANY exceeds its declared curve:
|
||||||
|
>
|
||||||
|
> ```
|
||||||
|
> bench "elc_compile" over n in [...] expect time O(n) allocs O(n) bytes O(n) { ... }
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> - **allocs (count)** — deterministic, zero-noise. Catches per-item allocation growth.
|
||||||
|
> - **allocs (bytes)** — deterministic, zero-noise. Catches accumulator-rebuild quadratics that
|
||||||
|
> count cannot see.
|
||||||
|
> - **time** — noisy, needs the sweep and statistics. The ONLY signal that sees pure-compute
|
||||||
|
> complexity regressions. Gate on the fitted *exponent*, never on absolute duration, so CI
|
||||||
|
> hardware variance scales the coefficient and leaves the classification intact.
|
||||||
|
>
|
||||||
|
> The deterministic signals remain preferable where they apply — they need no statistics and are
|
||||||
|
> correct on the first run. They are simply not sufficient.
|
||||||
|
>
|
||||||
|
> **`black_box` is mandatory, and consuming the result is NOT enough.** The first version of
|
||||||
|
> `compute` accumulated `total + 1` in a nested loop and reported **0 µs at every n** while
|
||||||
|
> returning a numerically correct n². Clang recognised the idiom and closed the loop to a
|
||||||
|
> multiply. Feeding the result into output did not prevent it. Only making the inner operation an
|
||||||
|
> opaque external call restored the real curve. A benchmark harness that trusts the user to defeat
|
||||||
|
> the optimiser will silently measure nothing — and report success while doing it.
|
||||||
|
|
||||||
Instrument the runtime with allocation counters and fit *those* against n instead of time:
|
Instrument the runtime with allocation counters and fit *those* against n instead of time:
|
||||||
|
|
||||||
```el
|
```el
|
||||||
|
|||||||
@@ -0,0 +1,91 @@
|
|||||||
|
// fitprobe.el — controlled growth-curve specimens for validating the complexity fitter.
|
||||||
|
//
|
||||||
|
// Three deliberately-shaped workloads. None depends on a real defect existing,
|
||||||
|
// which is the point: the fitter must be provable against KNOWN curves.
|
||||||
|
//
|
||||||
|
// linear — one allocation per item. count O(n), bytes O(n), time O(n)
|
||||||
|
// accum — rebuilds its accumulator. count O(n), bytes O(n^2), time O(n^2)
|
||||||
|
// compute — nested arithmetic, no alloc. count O(1), bytes O(1), time O(n^2)
|
||||||
|
//
|
||||||
|
// `compute` is the specimen that matters. It is the shape of el #132
|
||||||
|
// (strlen-per-character inside str_char_code): pure CPU, zero allocation.
|
||||||
|
// An allocation-only gate is structurally blind to it.
|
||||||
|
//
|
||||||
|
// No imports — uses runtime builtins directly so nothing collides.
|
||||||
|
|
||||||
|
fn work_linear(n: Int) -> Int {
|
||||||
|
let parts: [String] = native_list_empty()
|
||||||
|
let i: Int = 0
|
||||||
|
while i < n {
|
||||||
|
let parts = native_list_append(parts, int_to_str(i))
|
||||||
|
let i = i + 1
|
||||||
|
}
|
||||||
|
return native_list_len(parts)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn work_accum(n: Int) -> Int {
|
||||||
|
let acc: String = ""
|
||||||
|
let i: Int = 0
|
||||||
|
while i < n {
|
||||||
|
let acc = acc + "x"
|
||||||
|
let i = i + 1
|
||||||
|
}
|
||||||
|
return str_len(acc)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn work_compute(n: Int) -> Int {
|
||||||
|
// str_char_code is an opaque external call, so the C optimiser cannot
|
||||||
|
// reduce this nest to a closed form the way it does with `total + 1`.
|
||||||
|
// This is the exact shape of el #132: n scans over n characters, pure
|
||||||
|
// CPU, ZERO allocation.
|
||||||
|
let s: String = "abcdefghij"
|
||||||
|
let total: Int = 0
|
||||||
|
let i: Int = 0
|
||||||
|
while i < n {
|
||||||
|
let j: Int = 0
|
||||||
|
while j < n {
|
||||||
|
let total = total + str_char_code(s, 0)
|
||||||
|
let j = j + 1
|
||||||
|
}
|
||||||
|
let i = i + 1
|
||||||
|
}
|
||||||
|
return total
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run_one(mode: String, n: Int) {
|
||||||
|
let c0: Int = el_alloc_count()
|
||||||
|
let b0: Int = el_alloc_bytes()
|
||||||
|
let t0: Int = el_now_instant()
|
||||||
|
|
||||||
|
let r: Int = 0
|
||||||
|
if str_eq(mode, "linear") { let r = work_linear(n) }
|
||||||
|
if str_eq(mode, "accum") { let r = work_accum(n) }
|
||||||
|
if str_eq(mode, "compute") { let r = work_compute(n) }
|
||||||
|
|
||||||
|
let t1: Int = el_now_instant()
|
||||||
|
let c1: Int = el_alloc_count()
|
||||||
|
let b1: Int = el_alloc_bytes()
|
||||||
|
|
||||||
|
println(mode + "\t" + int_to_str(n)
|
||||||
|
+ "\t" + int_to_str(c1 - c0)
|
||||||
|
+ "\t" + int_to_str(b1 - b0)
|
||||||
|
+ "\t" + int_to_str((t1 - t0) / 1000)
|
||||||
|
+ "\t" + int_to_str(r))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
fn sweep(mode: String) {
|
||||||
|
run_one(mode, 200)
|
||||||
|
run_one(mode, 400)
|
||||||
|
run_one(mode, 800)
|
||||||
|
run_one(mode, 1600)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
fn main() -> Int {
|
||||||
|
println("mode\tn\tallocs\tbytes\tusec\tsink")
|
||||||
|
sweep("linear")
|
||||||
|
sweep("accum")
|
||||||
|
sweep("compute")
|
||||||
|
return 0
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user