Archived
will
d3495476f4
El SDK Release / build-and-release (push) Failing after 13m0s
The generated C, amalgams, vendored runtime pins, and compiled binaries from the Claude Code era are removed from the worktree. The El sources survive; this tree is now source-only for the first-principles rebuild. Per Principal direction 2026-08-19.
208 lines
10 KiB
Markdown
208 lines
10 KiB
Markdown
# CLAIMS — ITERATION-level evidence for v2
|
||
|
||
Re-run 2026-08-17. This directory holds the evidence that is about **iteration 2
|
||
as a whole** rather than about any one cycle. Per-cycle evidence and verdicts
|
||
are in `../cycles/<NN>-<slug>/CLAIMS.md`.
|
||
|
||
Before this run, `docs/` held 42 markdown files and **zero data files**. It now
|
||
holds a captured artifact for every re-runnable claim, each one recording the
|
||
command verbatim, stdout, stderr, the exit code captured directly and never
|
||
through a pipe, the commit SHA, the tree state, UTC, and the sha256 of both
|
||
streams.
|
||
|
||
## The measurement window, pinned
|
||
|
||
| | |
|
||
|---|---|
|
||
| Window start | `a116710bacba679615cb84b4ea9ce98c3acaa380` |
|
||
| Window end | `a0cc95e3d…` (see `0026-zz-head-at-end.out`) |
|
||
| Commits in window | **5** |
|
||
| Commits touching `lang/` | **0** |
|
||
| Commits touching `scripts/` or `tools/evidence/capture.sh` | **0** |
|
||
|
||
`0022-measurement-window-lang-untouched.out` is the proof. A concurrent agent
|
||
was landing v1 evidence documents into this repository throughout, so HEAD moved
|
||
five times and `git status` read dirty for most of the run — but **no compiler
|
||
source, no runtime source, and no measuring instrument changed while any of this
|
||
was measured.** The system under test and the recorder were both stable.
|
||
|
||
Historical claims were measured inside detached worktrees pinned at their own
|
||
commits, so those artifacts carry the system-under-test's SHA and a clean tree.
|
||
`verify-manifest.sh` flags every dirty-tree artifact as provenance-weak; the
|
||
load-bearing measurements are deliberately not among them.
|
||
|
||
## What was captured at iteration level
|
||
|
||
| Claim | Artifact | Result |
|
||
|---|---|---|
|
||
| the full gauntlet | `0003-gauntlet-verify-sh` | **VERIFIED, exit 0** |
|
||
| fixpoint gen2 == gen3 | `0004-fixpoint-gen2-eq-gen3` | **OK**, by sha256 |
|
||
| native suite | `0005-native-suite` | **105 tests, 105 passed, 0 failed, 253 assertions**, exit 0 |
|
||
| every harness individually, own exit code | `0006`–`0018` | **13 harnesses, all exit 0** |
|
||
| assertion tally | `0020-harness-assertion-tally` | **76 assertions across 13 harnesses** |
|
||
| `codegen.el` total / comment / code | `0019-codegen-total-vs-comment-vs-code` | **4434 total, 3197 code, 981 comment, 256 blank** |
|
||
| unguarded-deref ratchet | `0021-unguarded-deref-ratchet` | **11 unguarded casts, at baseline 11**, exit 0 |
|
||
| control over the whole record | `0023-CONTROL-verify-manifest-over-all-v2-cycles` | **250 artifacts verified, 0 altered, 0 missing** |
|
||
|
||
### The gauntlet
|
||
|
||
```
|
||
fixpoint gen2 == gen3 ok
|
||
native suite 105 tests, 105 passed, 0 failed, 253 assertions in 7.115ms
|
||
integration harnesses 13 green, 76 assertions
|
||
unguarded-deref ratchet at or below baseline
|
||
|
||
VERIFIED
|
||
```
|
||
|
||
### The fixpoint proof
|
||
|
||
```
|
||
gen1 sha256 = aa340a723e7b29e585e30c07d6dc9834e072ef15a07df5e1afb363c18e168c76
|
||
gen2 sha256 = 61210655962abd07ce395d66a9232582fd10c2a878c80f9b85fefc9081c5313e
|
||
gen3 sha256 = 61210655962abd07ce395d66a9232582fd10c2a878c80f9b85fefc9081c5313e
|
||
FIXPOINT gen2 == gen3 OK
|
||
note: gen1 != gen2, as expected whenever codegen output changes
|
||
```
|
||
|
||
The `gen1 != gen2` line matters: comparing gen1 to gen2 would be the wrong test,
|
||
and it is recorded here as an observed fact rather than as a design comment.
|
||
|
||
### `codegen.el` composition at HEAD
|
||
|
||
```
|
||
total_lines = 4434
|
||
code = 3197 72.1%
|
||
comment = 981 22.1%
|
||
blank = 256
|
||
```
|
||
|
||
Cycle 05 established that the metric worth quoting is code, not total. On that
|
||
metric the compiler stands at **3197 code lines**, against the **3589** baseline
|
||
at `45325f7` — **−392, −10.9%** across v1 and v2 together.
|
||
|
||
## v2's standing, with references
|
||
|
||
### Verdicts across all eight cycles
|
||
|
||
| Cycle | R | D | NR | NC | total |
|
||
|---|---|---|---|---|---|
|
||
| 01 the-compiler-was-never-measured | 26 | 5 | 5 | 1 | 37 |
|
||
| 02 seven-tables-and-a-shipped-defect | 21 | 2 | 0 | 0 | 23 |
|
||
| 03 a-gate-with-no-detector | 23 | 2 | 0 | 0 | 25 |
|
||
| 04 verification-was-a-convention | 7 | 2 | 0 | 3 | 12 |
|
||
| 05 the-metric-counted-comments | 12 | 1 | 7 | 0 | 20 |
|
||
| 06 form-is-half-bindable | 12 | 0 | 0 | 1 | 13 |
|
||
| 07 the-caller-was-already-there | 17 | 1 | 1 | 2 | 21 |
|
||
| 08 PREDICTIONS | 15 | 0 | 0 | 7 | 22 |
|
||
| **total** | **133** | **13** | **13** | **14** | **173** |
|
||
|
||
R = REPRODUCED, D = DIVERGED, NR = NOT-REPRODUCIBLE, NC = NOT-CAPTURED.
|
||
|
||
**133 of 173 assertions reproduced from raw data. Nothing was reconstructed.**
|
||
|
||
### The load-bearing results, re-verified
|
||
|
||
The index classifies v2 as one SUPPORTS pair plus a BOUNDS, and commit
|
||
`9540f23` weights the whole 27-cycle record at four load-bearing results. Those
|
||
that live in v2 were given the most care in this re-run:
|
||
|
||
**Cycle 06 — a new suffix form by a two-line data edit, no rebuild.** Fully
|
||
reproduced. The compiler binary's sha256 and mtime are byte-identical before and
|
||
after the edit and its mtime predates the edit by 37.4 s; `git diff --stat`
|
||
reads `1 file changed, 2 insertions(+)`; the same binary then emits
|
||
`3LL * 1209600000000000LL`; the linked program prints `3628800000000000`; and
|
||
reverting the two lines reverts the behaviour. **12 REPRODUCED, 0 DIVERGED.**
|
||
|
||
**Cycle 07 — exactly three emission points, and a construct that sees its
|
||
caller.** Reproduced, but *not* by the instrument the cycle used. Commit
|
||
`fbb3a52` records that the cycle's central assertion is a grep of compiler
|
||
source the harness never compiles; I confirmed that independently — appending a
|
||
**comment** to `codegen.el` turns the assertion red while the binary's sha256
|
||
and the emitted C are unchanged. So the bound was re-established from the
|
||
**emitted C of the binary under test**: three functions, `entry wrap exit` each,
|
||
nine points, every one keyed on the function name, and **zero** for loops, call
|
||
sites and field accesses. `AUDIT save called by handle_request` /
|
||
`AUDIT save called by migrate` reproduce verbatim.
|
||
|
||
**Cycle 02 — 49 builtin names moved to data.** Reproduced exactly: 51 → 100
|
||
entries in `signatures.rel`, delta 49; six of seven tables collapse to 3–6 line
|
||
wrappers; `duration_unit_nanos` alone keeps its 14 literals, which is P3.
|
||
|
||
### The divergences that matter
|
||
|
||
Thirteen divergences across the record. Four are substantive:
|
||
|
||
1. **Cycle 02, A17 — the shipped defect was worse than recorded.** The doc says
|
||
`el_input_len` returned 0 for "roughly 60%" of string literals. Measured at
|
||
the buggy commit against 20 literals of lengths 1–20, checked by an
|
||
independent `hashlib` oracle: **20 of 20, a 100% failure rate.** Standalone
|
||
alignment probes give 80% on the doc's own five literals and 90% on twenty.
|
||
The rate is link-layout dependent and is not a stable quantity; the doc's 60%
|
||
was a five-literal sample. That is the same error — *a sample of five is not a
|
||
measurement of a rate* — inside the paragraph that teaches it.
|
||
|
||
2. **Cycle 03, D-1 — the ratchet is not a ratchet.** It compares a *total*
|
||
against `BASELINE=11` rather than a *set of sites*, so guarding one site while
|
||
adding a new dereference passes: it printed a brand-new unguarded deref in its
|
||
own output while reporting `ok 11 unguarded casts, at baseline 11`. Its regex
|
||
also misses `EL_CSTR`, leaving **452 casts invisible to the detector**. And
|
||
`let s: String = 42` — defect #1 in that cycle's own list — **still segfaults
|
||
at HEAD, exit 139.**
|
||
|
||
3. **Cycle 07, A15 — the timing numbers do not reproduce; the conclusion does.**
|
||
Seven interleaved rounds gave BASE mean 52.29 ms and STACK mean 52.29 ms,
|
||
ratio 1.0000, spread 2 ms each. The doc's 0.24–0.25 s are cold-cache figures:
|
||
measured cold/warm on this workload is **4.30×**, which is larger than the
|
||
50% "regression" that was reported and retracted.
|
||
|
||
4. **Cycle 04, A5 — P3 as worded is false.** `verify.sh` contains no detector
|
||
for an exit code read through a pipe; it *avoids* the defect by construction.
|
||
Prevention is not detection, so the cycle's 5/5 is 4/5 as worded and 5/5 in
|
||
substance.
|
||
|
||
The remaining nine are off-by-one line counts (`4525` for 4524, `4662` for 4661,
|
||
`4446` for 4445, `80` for 79, `3267` for 3266) and one misattribution in cycle 01
|
||
(`capabilities.md` names a different table than the doc claims). None changes a
|
||
conclusion; all are recorded because a record about miscounting must not carry
|
||
uncorrected miscounts.
|
||
|
||
### What could not be reproduced, and why
|
||
|
||
- **13 NOT-REPRODUCIBLE.** Almost all are the emit/table/other bucket split from
|
||
cycles 01 and 05. **The classifier that produced 57.2%, 2578 lines, 21 fns,
|
||
42.8%, 15.4% and 27.4% was never committed**, and its criterion for
|
||
"emission" versus "table" versus "other" was never written down. No
|
||
replacement classifier was written and passed off as a reproduction. What could
|
||
be checked was the arithmetic, and it holds: emit 1909 + table 506 + other 794
|
||
= **3209**, exactly the independently measured code total at `8cb7613`.
|
||
|
||
- **14 NOT-CAPTURED.** Seven are cycle 08's predictions P1–P7, which are
|
||
ungraded by construction: the mutation study was never run, and running it
|
||
today would be a new experiment, not a reproduction. The seal is intact —
|
||
P1–P7 are byte-identical to their form at `d936bde`, which makes this genuine
|
||
pre-registration. The rest are prose assertions with no measurement behind
|
||
them, including cycle 06's P3, which the document itself labels
|
||
"NOT TESTED — an argument, not a measurement".
|
||
|
||
## The instrument, and one defect in it
|
||
|
||
`capture.sh` writes its manifest line with `printf '%s\n' "$*"`. **A command
|
||
containing a newline therefore splits one manifest record across several lines
|
||
and desynchronises the artifact counter.** This was hit independently on cycle 01
|
||
and on cycle 02. In both cases the fix was the same and it was not to edit the
|
||
manifest: the evidence directory was deleted and every capture re-run from
|
||
scratch with the logic moved into instrument files. Editing a manifest to match
|
||
its artifacts inverts what a manifest is for.
|
||
|
||
`verify-manifest.sh` globs `"$ROOT"/*/MANIFEST.tsv`, so it must be given the
|
||
**parent** of the cycle directories. Handed a cycle directory it prints
|
||
`0 artifacts verified, 0 altered, 0 missing` and **exits 0** — a green verdict
|
||
having checked nothing. That is the same shape as cycle 03's subject: a gate
|
||
nothing is forced through. The correct invocation is recorded in `0023`.
|
||
|
||
Failed runs are retained rather than deleted — a SIGPIPE-killed build in cycle 03,
|
||
a malformed `awk` in cycles 03 and 07, and `scripts/verify.sh` returning exit 127
|
||
at `abb0ab4` because it did not exist yet. Removing a failed attempt from an
|
||
evidence record is itself a measurement defect.
|