Tim Lingo
6f3a048f36
engram_assoc_leg previously took its seeds only from the top-3 LEXICAL hits. For a paraphrase query the lexical hits are noise by construction, so the walk never reached the neighbourhood that holds the answer. This adds the seeding pass Will documents at el_runtime.c l.6082 — "Semantic seeding (HippoRAG pattern, use similarity twice): the query is embedded, the top-K nodes by cosine join the seed set" — using his own ENGRAM_EMBED_SEED_K (8). Similarity is now used twice, coherently: cosine picks where to STAND in the graph, the structural-relation walk decides what is REACHABLE, and cosine orders what was reached (iteration 2's finding, unchanged). The seed list is deliberately NOT floored at ENGRAM_EMBED_SEED_MIN. Measured over all 38 gold queries: true paraphrase targets score cosine 0.46-0.66 and the three nonsense controls' own nearest neighbours score 0.55/0.60/0.62 — the distributions OVERLAP, so no absolute cosine floor separates signal from gibberish. The gate that works is reachability: gibberish's nearest neighbours carry no structural edge, so its graph leg is empty and the controls hold. The raw top-K is selected inside the existing scoring pass, so the cosine is computed exactly once per node: no extra corpus pass, no extra embed round-trip, latency flat (p50 1220 -> 1227 ms, 1.01x). Measured vs the certified baseline feat/hybrid-semantic-recall, embedded corpus, 2 runs each, zero run-to-run drift on both sides: hit@5 51.4% -> 68.6% MRR@10 0.387 -> 0.461 paraphrase 38.5% -> 61.5% associative 0% -> 66.7% exact_rare 100% held, nonsense 2/3 held, superseded 2/3 held phrase 85.7% -> 71.4% (q11, the known rank-5 rotation tax) net +6 queries (7 fixed / 1 broken), McNemar p=0.0703
32 lines
1.6 KiB
Python
32 lines
1.6 KiB
Python
import numpy as np, json, urllib.request
|
|
SP="/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad"
|
|
M=np.load(SP+'/emb.npy'); ids=open(SP+'/ids.txt',encoding='utf-8',errors='surrogateescape').read().split('\n')
|
|
np.seterr(all='ignore')
|
|
bad=~np.isfinite(M).all(axis=1)
|
|
M[bad]=0.0
|
|
print("non-finite rows zeroed:",int(bad.sum()))
|
|
idx={k:i for i,k in enumerate(ids)}
|
|
gold=json.load(open("/Users/timlingo/Development/neuron-technologies/_wt-assoc-leg/tools/retrieval-eval/gold_set.json"))['queries']
|
|
def emb(t):
|
|
b=json.dumps({"model":"nomic-embed-text","prompt":t}).encode()
|
|
r=urllib.request.Request("http://127.0.0.1:11434/api/embeddings",data=b,headers={"Content-Type":"application/json"})
|
|
v=np.array(json.load(urllib.request.urlopen(r,timeout=60))["embedding"],dtype=np.float32)
|
|
return v/(np.linalg.norm(v)+1e-9)
|
|
out={}
|
|
for q in gold:
|
|
v=emb(q['query']); s=M@v
|
|
s=s[np.isfinite(s)]
|
|
mu=float(s.mean()); sd=float(s.std())
|
|
top=np.sort(s)[::-1][:10]
|
|
z=[(float(t)-mu)/sd for t in top]
|
|
grank=[]
|
|
for rel in q['relevant']:
|
|
if rel in idx:
|
|
j=idx[rel]; grank.append((int((M@v > (M@v)[j]).sum())+1, round(float((M@v)[j]),3)))
|
|
grank.sort()
|
|
out[q['id']]=dict(cat=q['category'],mu=round(mu,3),sd=round(sd,4),top1=round(float(top[0]),3),
|
|
z1=round(z[0],2),z3=round(z[2],2),z5=round(z[4],2),gold=grank[:1])
|
|
print("%s %-11s mu=%.3f sd=%.4f top1=%.3f z1=%5.2f z3=%5.2f z5=%5.2f gold=%s"%(
|
|
q['id'],q['category'],mu,sd,top[0],z[0],z[2],z[4],grank[:1]))
|
|
json.dump(out,open(SP+'/zprobe.json','w'),indent=1)
|