Neuron
635453b936
Replaces the score-fusion first cut with rank fusion, which is what the data called for. nomic's cosine scale is compressed (true matches 0.55-0.70, unrelated pairs 0.35-0.50), so an additive blend of cosine onto token-coverage is dominated by whichever leg has the wider spread. Alternation is invariant to both scales: L1, S1, L2, S2, ... deduped, capped at limit Lexical ranking is left byte-identical; the semantic ranking is computed beside it and admitted only above ENGRAM_EMBED_SEED_MIN (0.60) — Will's existing seed floor, no new tuning constant. That floor is what keeps the nonsense controls clean: a query with no real match must not be answered with its neighbours. embed-corpus.py / merge-corpus.py produce the derived corpus the semantic leg needs (76,986 vectors, nomic-embed-text, 0 failures, 11 min). Zero of 78,791 nodes carried an embedding before this; the field round-tripped through the snapshot but nothing ever wrote it. MEASURED, 38-query gold set, paired against the SAME derived corpus so the comparison isolates the code change: hit@5 34.3% -> 51.4% paraphrase 0.0% -> 38.5% MRR@10 0.294 -> 0.387 superseded 1/3 -> 2/3 outranks recall@10 33.3% -> 50.5% latency p50 1146 -> 1220ms (1.06x) exact_rare 100% -> 100% phrase 85.7% -> 85.7% nonsense 2/3 -> 2/3 6 queries fixed, 0 broken, McNemar exact p=0.0312, 0 drift across repeats. Regression guards all held. Contrast PR #135, which swapped the read path to spreading activation wholesale: phrase 85.7 -> 28.6, latency 2.81x. Correct mechanism, wrong substrate. The substrate is now present. Restores engram claim 24 (previously 0% honoured). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
18 lines
674 B
Python
18 lines
674 B
Python
import json,sys
|
|
SRC="/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json"
|
|
TSV,OUT=sys.argv[1],sys.argv[2]
|
|
emb={}
|
|
for line in open(TSV,encoding='utf-8',errors='surrogateescape'):
|
|
p=line.rstrip("\n").rsplit("\t",1)
|
|
if len(p)==2 and p[1].count(",")>100: emb[p[0]]=p[1]
|
|
print("vectors",len(emb),flush=True)
|
|
d=json.load(open(SRC,encoding='utf-8',errors='surrogateescape'))
|
|
hit=0
|
|
for n in d["nodes"]:
|
|
v=emb.get(n.get("id") or "")
|
|
if v: n["emb"]=v; hit+=1
|
|
print("attached",hit,"of",len(d["nodes"]),flush=True)
|
|
with open(OUT,"w",encoding='utf-8',errors='surrogateescape') as f:
|
|
json.dump(d,f,ensure_ascii=False)
|
|
print("wrote",OUT,flush=True)
|