The retrieval match test is a raw substring scan, so a query token matches
anywhere INSIDE a corpus word: "throom" matches "bathroom". Measured over the
38-query gold set on this corpus that is not a rare accident - q28's lexical
leg is 36,954 records of which only 13 contain a query token at a word start
(99.96% mid-word noise), six further queries carry ~20,500 mid-word-only
records each, and the nonsense control q35 returns 7 records ALL of which
match only mid-word.
istr_contains_wordstart() anchors a token to a word start (preceding char not
alphanumeric) while still matching suffixes, so "value" still hits "values".
That empties the lexical leg for gibberish, and the nhits==0 corpus-vocabulary
gate (iteration 6's mechanism, feat/claim24-unfloored-semantic) then makes the
whole query decline rather than let the semantic leg answer it.
Measured vs feat/bm25-lexical-leg on the embedded corpus, 2 runs each,
0 queries of run-to-run drift on both sides:
net +1 (nonsense:q35), 0 losses, McNemar p=1.0 -> NOT-SHOWN (floor is 6)
nonsense clean 2/3 -> 3/3; exact_rare 100%, phrase 100%, paraphrase 61.5%,
associative 66.7%, superseded 2/3 all UNCHANGED
latency p50 1184 -> 543 ms (0.46x)
Iteration 6 called q35 "a DEFECTIVE CONTROL ... cannot be cleaned without
breaking the lexical leg". It can: the defect was the match primitive, and
cleaning it cost nothing.
Also committed: results-wsclaim24.json + cmp-nogate.json, a measured negative
for bundling the claim-24 unfloored semantic leg on top (gains q14/q25, breaks
q15/q28/q33/q34, net -2) - it independently reproduces iteration 6's q15/q28
losses and shows unflooring REQUIRES the vocabulary gate.
Reproducers: legs.py (leg-level replica, reproduces baseline hit@5 exactly on
all 38 queries), policy2.py, ceiling.py, wb2.py.
Tim Lingo
4eb4c9e287