Files
el/docs/v1/experiments/findings/measurement-defects.md
T
bigmerge c6ba0677f0 log v1 experiments: nineteen cycles, organised by the method that produced them
cycles/    one file per Ishikawa -> scientific method -> Six Sigma loop, named
           for the DEFECT not the fix, carrying the commit record as written at
           the time
findings/  what the cycles produced, cross-cut: live bugs, architecture answers,
           and defects in my own measurement

The organising finding is that predictions which came back FALSE produced every
significant result. Eleven of sixty-one failed, and those eleven found: that the
arity table was not drifted but 40% incomplete; that the AST traversal is
irreducible and only rules and judgments move; that guards could refuse through
the seam after all; and that routing el_bin_lookup through the gate did NOT fix
the SIGSEGV, because the fallback strlen was the hazard -- a wrong fix I would
otherwise have shipped as verified.

One cycle was run without committing predictions first and had to be discarded
as rigged. It is kept, in full, as 18-async-half-expressible.md.
2026-08-17 10:52:17 -05:00

1.7 KiB

Defects in my own measurement

Recorded because the pattern is the point: four of these, all the same shape — searching by name or scope instead of by the operation itself. Each was caught by running something, never by reading.

1. Scoped the search to one file

Reported test as an inert keyword by checking only parser.el. codegen consumes it at 4135 for --test mode, and the tree has 408 uses. Removing it would have broken every test in the suite — including the ones used to verify the removal.

2. Searched by variable name, not by operation

Grepped for native_list_append(tokens to find direct token appends. interp_tokens_append_all calls its parameters dst/result, carries its own copy of the stride, and corrupted generation 2 — while generation 1 built fine, because the compiler's own source uses string interpolation.

3. Scoped to compiler sources; the stride had escaped into tests

tok_count in test_compiler.el computed len/2 independently. 21 tests failed after the token layout changed.

4. Read the wrong exit code

timeout 10 /tmp/leakrun 2>&1 | head -2; echo "exit=$?"     # reports head's exit

Reported exit=0 for a program that was returning 139 (SIGSEGV). I nearly recorded a segfault as a clean run.

And one that was not a measurement defect but a method defect

One cycle was run without committing predictions first — see cycles/18-async-half-expressible.md. The test joined the thread immediately after creating it and printed the word DEFERRED itself. A test authored by the party holding the conclusion, with nothing committed beforehand, cannot fail. It had to be discarded and re-run.