Iteration 7 measured this harness's own ceiling at +3 gross / +1 net against a
decision floor of 6. An instrument whose ceiling sits below its floor cannot
certify or refute anything, so the gold set — not the retriever — was the
blocker. This iteration builds no retrieval mechanism; it fixes the instrument
and uses it once.
extend_gold_set.py appends 37 queries (q39-q75) WITHOUT touching q01-q38, so
every committed baseline and per-query id stays comparable. 30 held-out
paraphrases: targets sampled mechanically (seed 8080) from addressable
500-2600 char nodes outside the original answer space and outside any duplicate
cluster; queries authored from the node body alone, before any retrieval was
run, and each one re-proved at build time to share ZERO content words with its
target. 7 extra nonsense controls, fully mechanical.
Why this was needed: the original 13 paraphrase and 6 associative queries share
ONE answer space — the 13 `Self - Values (grounded)` children. 19 of 35 scored
queries tested retrieval against a single 13-node neighbourhood.
FIRST OUT-OF-SAMPLE RESULT (main vs the accumulated stack, embedded corpus):
held-out only : +5 / -0, p=0.0625 — one query short of the floor, NOT-SHOWN
original 38 : +15 / -0
full 75 : +20 / -0, p=0.0000, latency 0.46x
In-sample paraphrase 61.5% vs out-of-sample 16.7%: generalisation is real,
directional and 3.7x weaker than the headline number suggested.
Also recorded: 47.4% of this corpus is redundant and ONE record accounts for
46.6% of all 78,768 nodes (36,737 byte-identical copies under distinct ids).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tim Lingo
77e3c5fa26