el-retired/docs/v2/experiments/evidence/ITERATION/CLAIMS.md

CLAIMS — ITERATION-level evidence for v2

Re-run 2026-08-17. This directory holds the evidence that is about iteration 2 as a whole rather than about any one cycle. Per-cycle evidence and verdicts are in ../cycles/<NN>-<slug>/CLAIMS.md.

Before this run, docs/ held 42 markdown files and zero data files. It now holds a captured artifact for every re-runnable claim, each one recording the command verbatim, stdout, stderr, the exit code captured directly and never through a pipe, the commit SHA, the tree state, UTC, and the sha256 of both streams.

The measurement window, pinned

Window start	a116710bacba679615cb84b4ea9ce98c3acaa380
Window end	a0cc95e3d… (see 0026-zz-head-at-end.out)
Commits in window	5
Commits touching lang/	0
Commits touching scripts/ or tools/evidence/capture.sh	0

0022-measurement-window-lang-untouched.out is the proof. A concurrent agent was landing v1 evidence documents into this repository throughout, so HEAD moved five times and git status read dirty for most of the run — but no compiler source, no runtime source, and no measuring instrument changed while any of this was measured. The system under test and the recorder were both stable.

Historical claims were measured inside detached worktrees pinned at their own commits, so those artifacts carry the system-under-test's SHA and a clean tree. verify-manifest.sh flags every dirty-tree artifact as provenance-weak; the load-bearing measurements are deliberately not among them.

What was captured at iteration level

Claim	Artifact	Result
the full gauntlet	0003-gauntlet-verify-sh	VERIFIED, exit 0
fixpoint gen2 == gen3	0004-fixpoint-gen2-eq-gen3	OK, by sha256
native suite	0005-native-suite	105 tests, 105 passed, 0 failed, 253 assertions, exit 0
every harness individually, own exit code	0006–0018	13 harnesses, all exit 0
assertion tally	0020-harness-assertion-tally	76 assertions across 13 harnesses
codegen.el total / comment / code	0019-codegen-total-vs-comment-vs-code	4434 total, 3197 code, 981 comment, 256 blank
unguarded-deref ratchet	0021-unguarded-deref-ratchet	11 unguarded casts, at baseline 11, exit 0
control over the whole record	0023-CONTROL-verify-manifest-over-all-v2-cycles	250 artifacts verified, 0 altered, 0 missing

The gauntlet

  fixpoint gen2 == gen3              ok
  native suite                       105 tests, 105 passed, 0 failed, 253 assertions in 7.115ms
  integration harnesses              13 green, 76 assertions
  unguarded-deref ratchet            at or below baseline

  VERIFIED

The fixpoint proof

gen1 sha256 = aa340a723e7b29e585e30c07d6dc9834e072ef15a07df5e1afb363c18e168c76
gen2 sha256 = 61210655962abd07ce395d66a9232582fd10c2a878c80f9b85fefc9081c5313e
gen3 sha256 = 61210655962abd07ce395d66a9232582fd10c2a878c80f9b85fefc9081c5313e
FIXPOINT gen2 == gen3   OK
note: gen1 != gen2, as expected whenever codegen output changes

The gen1 != gen2 line matters: comparing gen1 to gen2 would be the wrong test, and it is recorded here as an observed fact rather than as a design comment.

codegen.el composition at HEAD

total_lines = 4434
code        = 3197   72.1%
comment     =  981   22.1%
blank       =  256

Cycle 05 established that the metric worth quoting is code, not total. On that metric the compiler stands at 3197 code lines, against the 3589 baseline at 45325f7 — −392, −10.9% across v1 and v2 together.

v2's standing, with references

Verdicts across all eight cycles

Cycle	R	D	NR	NC	total
01 the-compiler-was-never-measured	26	5	5	1	37
02 seven-tables-and-a-shipped-defect	21	2	0	0	23
03 a-gate-with-no-detector	23	2	0	0	25
04 verification-was-a-convention	7	2	0	3	12
05 the-metric-counted-comments	12	1	7	0	20
06 form-is-half-bindable	12	0	0	1	13
07 the-caller-was-already-there	17	1	1	2	21
08 PREDICTIONS	15	0	0	7	22
total	133	13	13	14	173

R = REPRODUCED, D = DIVERGED, NR = NOT-REPRODUCIBLE, NC = NOT-CAPTURED.

133 of 173 assertions reproduced from raw data. Nothing was reconstructed.

The load-bearing results, re-verified

The index classifies v2 as one SUPPORTS pair plus a BOUNDS, and commit 9540f23 weights the whole 27-cycle record at four load-bearing results. Those that live in v2 were given the most care in this re-run:

Cycle 06 — a new suffix form by a two-line data edit, no rebuild. Fully reproduced. The compiler binary's sha256 and mtime are byte-identical before and after the edit and its mtime predates the edit by 37.4 s; git diff --stat reads 1 file changed, 2 insertions(+); the same binary then emits 3LL * 1209600000000000LL; the linked program prints 3628800000000000; and reverting the two lines reverts the behaviour. 12 REPRODUCED, 0 DIVERGED.

Cycle 07 — exactly three emission points, and a construct that sees its caller. Reproduced, but not by the instrument the cycle used. Commit fbb3a52 records that the cycle's central assertion is a grep of compiler source the harness never compiles; I confirmed that independently — appending a comment to codegen.el turns the assertion red while the binary's sha256 and the emitted C are unchanged. So the bound was re-established from the emitted C of the binary under test: three functions, entry wrap exit each, nine points, every one keyed on the function name, and zero for loops, call sites and field accesses. AUDIT save called by handle_request / AUDIT save called by migrate reproduce verbatim.

Cycle 02 — 49 builtin names moved to data. Reproduced exactly: 51 → 100 entries in signatures.rel, delta 49; six of seven tables collapse to 3–6 line wrappers; duration_unit_nanos alone keeps its 14 literals, which is P3.

The divergences that matter

Thirteen divergences across the record. Four are substantive:

1. Cycle 02, A17 — the shipped defect was worse than recorded. The doc says el_input_len returned 0 for "roughly 60%" of string literals. Measured at the buggy commit against 20 literals of lengths 1–20, checked by an independent hashlib oracle: 20 of 20, a 100% failure rate. Standalone alignment probes give 80% on the doc's own five literals and 90% on twenty. The rate is link-layout dependent and is not a stable quantity; the doc's 60% was a five-literal sample. That is the same error — a sample of five is not a measurement of a rate — inside the paragraph that teaches it.

2. Cycle 03, D-1 — the ratchet is not a ratchet. It compares a total against BASELINE=11 rather than a set of sites, so guarding one site while adding a new dereference passes: it printed a brand-new unguarded deref in its own output while reporting ok 11 unguarded casts, at baseline 11. Its regex also misses EL_CSTR, leaving 452 casts invisible to the detector. And let s: String = 42 — defect #1 in that cycle's own list — still segfaults at HEAD, exit 139.

3. Cycle 07, A15 — the timing numbers do not reproduce; the conclusion does. Seven interleaved rounds gave BASE mean 52.29 ms and STACK mean 52.29 ms, ratio 1.0000, spread 2 ms each. The doc's 0.24–0.25 s are cold-cache figures: measured cold/warm on this workload is 4.30×, which is larger than the 50% "regression" that was reported and retracted.

4. Cycle 04, A5 — P3 as worded is false. verify.sh contains no detector for an exit code read through a pipe; it avoids the defect by construction. Prevention is not detection, so the cycle's 5/5 is 4/5 as worded and 5/5 in substance.

The remaining nine are off-by-one line counts (4525 for 4524, 4662 for 4661, 4446 for 4445, 80 for 79, 3267 for 3266) and one misattribution in cycle 01 (capabilities.md names a different table than the doc claims). None changes a conclusion; all are recorded because a record about miscounting must not carry uncorrected miscounts.

What could not be reproduced, and why

• 13 NOT-REPRODUCIBLE. Almost all are the emit/table/other bucket split from cycles 01 and 05. The classifier that produced 57.2%, 2578 lines, 21 fns, 42.8%, 15.4% and 27.4% was never committed, and its criterion for "emission" versus "table" versus "other" was never written down. No replacement classifier was written and passed off as a reproduction. What could be checked was the arithmetic, and it holds: emit 1909 + table 506 + other 794 = 3209, exactly the independently measured code total at 8cb7613.

• 14 NOT-CAPTURED. Seven are cycle 08's predictions P1–P7, which are ungraded by construction: the mutation study was never run, and running it today would be a new experiment, not a reproduction. The seal is intact — P1–P7 are byte-identical to their form at d936bde, which makes this genuine pre-registration. The rest are prose assertions with no measurement behind them, including cycle 06's P3, which the document itself labels "NOT TESTED — an argument, not a measurement".

The instrument, and one defect in it

capture.sh writes its manifest line with printf '%s\n' "$*". A command containing a newline therefore splits one manifest record across several lines and desynchronises the artifact counter. This was hit independently on cycle 01 and on cycle 02. In both cases the fix was the same and it was not to edit the manifest: the evidence directory was deleted and every capture re-run from scratch with the logic moved into instrument files. Editing a manifest to match its artifacts inverts what a manifest is for.

verify-manifest.sh globs "$ROOT"/*/MANIFEST.tsv, so it must be given the parent of the cycle directories. Handed a cycle directory it prints 0 artifacts verified, 0 altered, 0 missing and exits 0 — a green verdict having checked nothing. That is the same shape as cycle 03's subject: a gate nothing is forced through. The correct invocation is recorded in 0023.

Failed runs are retained rather than deleted — a SIGPIPE-killed build in cycle 03, a malformed awk in cycles 03 and 07, and scripts/verify.sh returning exit 127 at abb0ab4 because it did not exist yet. Removing a failed attempt from an evidence record is itself a measurement defect.