Measured on the 75-query extended gold set (iteration 8's held-out extension)
against the certified stack baseline results-stack-ext.json, on the embedded
corpus. Three runs of the candidate, zero drift.
A. CLAIM 24 WITHOUT THE THRESHOLD - net +4, NOT-SHOWN, kept in the tree.
fixed : q14, q25 (in-sample paraphrase), q43, q52, q63, q67 (held-out)
broken : q15 (paraphrase), q28 (associative)
8 discordant, McNemar exact p = 0.2891, floor is 6.
heldout_paraphrase 16.7% -> 30.0%, paraphrase 61.5% -> 69.2%.
Every regression guard held: exact_rare 6/6, phrase 7/7, nonsense 10/10,
superseded 2/3. Latency FLAT: p50 641 -> 632 ms.
The in-sample half (+q14 +q25 -q15 -q28 = 0) was already on record in
iteration 7's cmp-nogate.json, so only the held-out +4 is new.
B. ASYMMETRIC TASK PREFIXES ON THE EMBEDDER - net -5, REVERTED in this commit.
Rationale was sound and the prediction was wrong, which is why it was worth
measuring: nomic-embed-text is an asymmetric retrieval encoder and this file
embedded query and document bare on both sides. Prefixing does exactly what
the model card implies for the far-away cases - it rescued q42 (gold at
GLOBAL COSINE RANK 25,564) and q39 - but it re-ranks the whole space and
broke more than it fixed:
fixed : q24, q39, q42
broken : q18, q19, q22, q31, q43, q44, q52, q63
heldout_paraphrase 30.0% -> 23.3%, paraphrase 69.2% -> 53.8%.
The corpus and the reproducer are kept (embed-corpus-prefixed.py,
snapshot-pre-repair-20260806-embedded-prefixed.json) so nobody re-runs it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tim Lingo