Neuron
635453b936
Replaces the score-fusion first cut with rank fusion, which is what the data called for. nomic's cosine scale is compressed (true matches 0.55-0.70, unrelated pairs 0.35-0.50), so an additive blend of cosine onto token-coverage is dominated by whichever leg has the wider spread. Alternation is invariant to both scales: L1, S1, L2, S2, ... deduped, capped at limit Lexical ranking is left byte-identical; the semantic ranking is computed beside it and admitted only above ENGRAM_EMBED_SEED_MIN (0.60) — Will's existing seed floor, no new tuning constant. That floor is what keeps the nonsense controls clean: a query with no real match must not be answered with its neighbours. embed-corpus.py / merge-corpus.py produce the derived corpus the semantic leg needs (76,986 vectors, nomic-embed-text, 0 failures, 11 min). Zero of 78,791 nodes carried an embedding before this; the field round-tripped through the snapshot but nothing ever wrote it. MEASURED, 38-query gold set, paired against the SAME derived corpus so the comparison isolates the code change: hit@5 34.3% -> 51.4% paraphrase 0.0% -> 38.5% MRR@10 0.294 -> 0.387 superseded 1/3 -> 2/3 outranks recall@10 33.3% -> 50.5% latency p50 1146 -> 1220ms (1.06x) exact_rare 100% -> 100% phrase 85.7% -> 85.7% nonsense 2/3 -> 2/3 6 queries fixed, 0 broken, McNemar exact p=0.0312, 0 drift across repeats. Regression guards all held. Contrast PR #135, which swapped the read path to spreading activation wholesale: phrase 85.7 -> 28.6, latency 2.81x. Correct mechanism, wrong substrate. The substrate is now present. Restores engram claim 24 (previously 0% honoured). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
44 lines
2.0 KiB
Python
44 lines
2.0 KiB
Python
import json,sys,time,urllib.request,threading,queue
|
|
SRC="/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json"
|
|
OUT=sys.argv[1]
|
|
URL="http://127.0.0.1:11434/api/embeddings"; MODEL="nomic-embed-text"
|
|
MAXB=2000 # ENGRAM_EMBED_MAX_CHARS, applied to bytes as the C code does
|
|
d=json.load(open(SRC,encoding='utf-8',errors='surrogateescape'))
|
|
tasks=[]
|
|
for n in d["nodes"]:
|
|
c=n.get("content") or ""; t=n.get("node_type") or ""
|
|
if len(c)<8: continue # eg_embed_eligible
|
|
if t in ("InternalStateEvent","Tag"): continue
|
|
b=c.encode('utf-8',errors='surrogateescape')[:MAXB]
|
|
tasks.append((n.get("id") or "", b.decode('utf-8',errors='replace')))
|
|
del d
|
|
print("tasks",len(tasks),flush=True)
|
|
q=queue.Queue(); [q.put(t) for t in tasks]
|
|
lock=threading.Lock(); f=open(OUT,"w",encoding="utf-8",errors="surrogateescape"); done=[0]; t0=time.time(); fails=[0]
|
|
def work():
|
|
while True:
|
|
try: nid,txt=q.get_nowait()
|
|
except queue.Empty: return
|
|
v=None
|
|
for attempt in range(3):
|
|
try:
|
|
body=json.dumps({"model":MODEL,"prompt":txt}).encode()
|
|
r=urllib.request.Request(URL,data=body,headers={"Content-Type":"application/json"})
|
|
with urllib.request.urlopen(r,timeout=120) as fh: v=json.load(fh)["embedding"]
|
|
break
|
|
except Exception as e:
|
|
if attempt==2:
|
|
with lock: fails[0]+=1
|
|
time.sleep(0.5)
|
|
with lock:
|
|
if v: f.write(nid+"\t"+",".join("%.5g"%x for x in v)+"\n")
|
|
done[0]+=1
|
|
if done[0]%2000==0:
|
|
el=time.time()-t0
|
|
print("%d/%d %.1f/s eta %.1fmin fails=%d"%(done[0],len(tasks),done[0]/el,(len(tasks)-done[0])/(done[0]/el)/60,fails[0]),flush=True)
|
|
f.flush()
|
|
ths=[threading.Thread(target=work) for _ in range(8)]
|
|
[t.start() for t in ths]; [t.join() for t in ths]
|
|
f.close()
|
|
print("DONE",done[0],"fails",fails[0],"secs %.1f"%(time.time()-t0),flush=True)
|