Tim Lingo
4eb4c9e287
The retrieval match test is a raw substring scan, so a query token matches anywhere INSIDE a corpus word: "throom" matches "bathroom". Measured over the 38-query gold set on this corpus that is not a rare accident - q28's lexical leg is 36,954 records of which only 13 contain a query token at a word start (99.96% mid-word noise), six further queries carry ~20,500 mid-word-only records each, and the nonsense control q35 returns 7 records ALL of which match only mid-word. istr_contains_wordstart() anchors a token to a word start (preceding char not alphanumeric) while still matching suffixes, so "value" still hits "values". That empties the lexical leg for gibberish, and the nhits==0 corpus-vocabulary gate (iteration 6's mechanism, feat/claim24-unfloored-semantic) then makes the whole query decline rather than let the semantic leg answer it. Measured vs feat/bm25-lexical-leg on the embedded corpus, 2 runs each, 0 queries of run-to-run drift on both sides: net +1 (nonsense:q35), 0 losses, McNemar p=1.0 -> NOT-SHOWN (floor is 6) nonsense clean 2/3 -> 3/3; exact_rare 100%, phrase 100%, paraphrase 61.5%, associative 66.7%, superseded 2/3 all UNCHANGED latency p50 1184 -> 543 ms (0.46x) Iteration 6 called q35 "a DEFECTIVE CONTROL ... cannot be cleaned without breaking the lexical leg". It can: the defect was the match primitive, and cleaning it cost nothing. Also committed: results-wsclaim24.json + cmp-nogate.json, a measured negative for bundling the claim-24 unfloored semantic leg on top (gains q14/q25, breaks q15/q28/q33/q34, net -2) - it independently reproduces iteration 6's q15/q28 losses and shows unflooring REQUIRES the vocabulary gate. Reproducers: legs.py (leg-level replica, reproduces baseline hit@5 exactly on all 38 queries), policy2.py, ceiling.py, wb2.py.
44 lines
1.5 KiB
Python
44 lines
1.5 KiB
Python
import json,sys,pickle,numpy as np,itertools
|
|
sys.path.insert(0,'.')
|
|
from policy2 import legs3,outcome,G,NODES,merge
|
|
# cache per-query leg id-lists, floored and unfloored
|
|
cache={}
|
|
for q in G['queries']:
|
|
Lf,Sf,Af=legs3(q['query'])
|
|
Lu,Su,Au=legs3(q['query'],unfloor=True)
|
|
cache[q['id']]=dict(L=Lf,Sf=Sf,A=Af,Su=Su,Au=Au)
|
|
pickle.dump(cache,open('ceil.pkl','wb'))
|
|
def mrg(pattern,L,S,A,lim=10):
|
|
out=[];p={'L':0,'S':0,'A':0};src={'L':L,'S':S,'A':A}
|
|
i=0
|
|
while len(out)<lim:
|
|
prog=False
|
|
for ch in pattern:
|
|
lst=src[ch]
|
|
if p[ch]<len(lst):
|
|
x=lst[p[ch]];p[ch]+=1;prog=True
|
|
if x not in out: out.append(x)
|
|
if len(out)>=lim: return out
|
|
if not prog: break
|
|
return out
|
|
def ev(pattern,unfl):
|
|
res={}
|
|
for q in G['queries']:
|
|
c=cache[q['id']]
|
|
S=c['Su'] if unfl else c['Sf']
|
|
ids=[NODES[i]['id'] for i in mrg(pattern,c['L'],S,c['A'],10)]
|
|
res[q['id']]=outcome(q,ids)
|
|
return res
|
|
base=ev('LSA',False)
|
|
print("baseline",sum(base.values()))
|
|
best=[]
|
|
pats=['LSA','LAS','SLA','ALS','SAL','ASL','LSSA','LSASA','LSAA','LSSAA','LSAS','SSLA','LLSA','SALSA','LSAAS']
|
|
for unfl in (False,True):
|
|
for p in pats:
|
|
r=ev(p,unfl)
|
|
g=sorted(k for k in base if r[k] and not base[k]);l=sorted(k for k in base if base[k] and not r[k])
|
|
best.append((len(g)-len(l),p,unfl,g,l))
|
|
best.sort(reverse=True)
|
|
for n,p,u,g,l in best[:10]:
|
|
print("net=%+d pat=%-6s unfloor=%s gains=%s losses=%s"%(n,p,u,g,l))
|