test framework phase 3/4: black_box barrier + three-signal complexity gate, armed #139

2026-08-16T03:01:51Z

will.anderson commented

2026-08-16 03:01:51 +00:00

Phase 4 is armed and proven in both directions — on live workloads, not replayed series:

scan us: 1821 3251 6007 10422    -> O(n)    PASS
quad us:  922 3667 13524 44792   -> O(n^2)  FAIL

The healthy per-character scan passes; a deliberately quadratic rescan-from-zero — el #132's exact shape — fires the gate. The quadratic specimen exists specifically because a gate only ever proven to pass is decoration.

Three signals, because allocation alone is blind. el #132's quadratic was pure CPU with zero allocation, so an allocation-only gate would have passed it clean. fitprobe.el carries three controlled specimens proving the classifier: linear (O(n) everywhere), accum (count linear, bytes quadratic — count alone misses it), and compute (both allocation signals FLAT, only time catches it — #132's shape). None depends on a real defect existing.

el_black_box is a genuine barrier (inline asm, "+r" constraint, memory clobber). Necessary: an earlier accumulator specimen reported 0 µs at every n while returning a numerically correct answer — clang had closed the loop to a multiply. A harness that trusts the author to defeat the optimiser measures nothing and reports success while doing it.

The classifier caught its own bad measurement. First run returned INDETERMINATE rather than passing an unwarmed sweep. The scan is linear — ratios 3.37, 2.92, 1.76, 1.65, head is cold cache, tail is truth. So elb_spread_ok was wrong: judging spread across the whole sweep rejected correct data. It now judges the asymptotic tail, which is what a complexity bound actually claims, with warmup taken and discarded per point. REFUSED/INDETERMINATE earned their place within minutes of existing — without them the gate would have reported a confident O(n²) on healthy code.

Independently verified: branch has merged dev and contains every compiler fix from #131–#138; the scan workload reproduces at flat ns/char=2 across a 16× range in n.

Purely additive: 707 insertions, no deletions.

Phase 4 is **armed and proven in both directions** — on live workloads, not replayed series: ``` scan us: 1821 3251 6007 10422 -> O(n) PASS quad us: 922 3667 13524 44792 -> O(n^2) FAIL ``` The healthy per-character scan passes; a deliberately quadratic rescan-from-zero — el #132's exact shape — fires the gate. The quadratic specimen exists specifically because **a gate only ever proven to pass is decoration.** **Three signals, because allocation alone is blind.** el #132's quadratic was pure CPU with zero allocation, so an allocation-only gate would have passed it clean. `fitprobe.el` carries three controlled specimens proving the classifier: `linear` (O(n) everywhere), `accum` (count linear, **bytes** quadratic — count alone misses it), and `compute` (both allocation signals FLAT, **only time** catches it — #132's shape). None depends on a real defect existing. **`el_black_box` is a genuine barrier** (inline asm, `"+r"` constraint, memory clobber). Necessary: an earlier accumulator specimen reported **0 µs at every n** while returning a numerically correct answer — clang had closed the loop to a multiply. A harness that trusts the author to defeat the optimiser measures nothing and reports success while doing it. **The classifier caught its own bad measurement.** First run returned INDETERMINATE rather than passing an unwarmed sweep. The scan *is* linear — ratios 3.37, 2.92, 1.76, 1.65, head is cold cache, tail is truth. So `elb_spread_ok` was wrong: judging spread across the whole sweep **rejected correct data**. It now judges the asymptotic tail, which is what a complexity bound actually claims, with warmup taken and discarded per point. REFUSED/INDETERMINATE earned their place within minutes of existing — without them the gate would have reported a confident O(n²) on healthy code. **Independently verified:** branch has merged dev and contains every compiler fix from #131–#138; the scan workload reproduces at flat `ns/char=2` across a 16× range in n. Purely additive: 707 insertions, no deletions.

will.anderson added 6 commits 2026-08-16 03:01:55 +00:00

Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile 4c3414072b

design: gate on THREE signals -- the alloc gate would have missed el #132 6291a35bb9

el #132's quadratic (strlen per character in str_char_code/str_slice) is
pure CPU and allocates NOTHING. Measured on three controlled specimens:

  specimen  allocs        bytes         time
  linear    2.00 -> O(n)  2.16 -> O(n)  2.05 -> O(n)
  accum     2.00 -> O(n)  3.99 -> O(n2) noisy
  compute   FLAT          FLAT          3.96 -> O(n2)

'compute' is #132's shape. A gate fitting only allocation count and bytes
classifies it FLAT and passes -- it would not have caught the defect it
was created for. The gate now fits time AND count AND bytes, failing if
any exceeds its declared curve.

Also: black_box is mandatory and consuming the result is NOT sufficient.
The first 'compute' reported 0us at every n while returning a correct n2 --
clang closed the loop to a multiply. Only an opaque call restored the curve.

Adds lang/tests/bench/fitprobe.el as the fitter's known-good/known-bad set,
so the classifier is provable without depending on a real bug existing.
Marks DESIGN.md 1.3 stale: test_compiler 3.58s -> 0.03s (119x).

Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile b5d1e53902

bench: real black_box barrier + three-signal growth-curve gate 6a6b589ba0

Adds el_black_box (inline asm, +r constraint, memory clobber) and
runtime/elbench.el: a growth-curve classifier that gates time AND
allocation-count AND allocation-bytes, failing if any exceeds its
declared curve.

Refusal is a first-class verdict. The classifier REFUSES rather than
classifying when the largest measurement is below the floor, or when a
series is hard-flat across an 8x input range -- the shape produced when
the optimiser deletes the work. Reporting O(1) there would be a
confident answer with nothing behind it. Disagreeing ratios report
INDETERMINATE rather than a guess.

Deviation from DESIGN.md 6.2, stated in the source: uses consecutive
ratios on a mandated geometric sweep rather than least-squares over
candidate curves. Ratios are directly interpretable on a doubling sweep
and need no floating point; the cost is weaker O(n) vs O(n log n)
separation, reported as an ambiguous band rather than guessed.

Documents the counter scope limit: engram_*.c and libcurl malloc are
NOT tracked, so a flat curve over engram/HTTP-dominated work is not
evidence of anything.

13 tests prove the classifier against real measured series from
fitprobe.el -- including that an accumulator's allocation COUNT is
linear while its bytes are quadratic, and that el #132's pure-CPU shape
reads FLAT on both allocation signals and is caught only by time.

Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile cf060adbfd

bench: arm the Phase 4 gate -- proven to pass clean AND fire on a quadratic

El SDK CI - dev / build-and-test (pull_request) Failing after 12m7s

Details

e0b2c0ea54

Adds tests/native/test_lexer_scaling.el, the regression gate for el #132.

Both directions are proven on LIVE workloads, not synthetic series:
  healthy per-character scan  1821 3251 6007 10422 us -> O(n)   PASS
  rescan-from-zero (the #132 shape)  922 3667 13524 44792 -> O(n^2) FAIL

A gate only proven to pass is decoration. The quadratic specimen exists so
the gate is proven to FIRE.

Also fixes elb_spread_ok to judge the ASYMPTOTIC TAIL (last three ratios)
rather than the whole sweep. Measured on a genuinely linear scan the ratios
ran 3.37 2.92 1.76 1.65 -- the head looks quadratic because it is cold
cache, the tail is the truth. Whole-sweep spread rejected correct data. A
complexity bound is an asymptotic claim and must be judged asymptotically.

That fix came from the classifier refusing to rubber-stamp my own bad
measurement: it reported INDETERMINATE on an unwarmed sweep rather than
passing it. Warmup is now taken and discarded at every sweep point.

Reverts the == workarounds in test_elbench.el now that el #137 has landed;
the natural form generates no str_eq and all 13 fitter tests stay green.
The  workaround remains -- the Plus arm is still open.

will.anderson merged commit 0832865952 into dev

2026-08-16 03:02:23 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#139