Tim Lingo
b9ef66cae9
The accumulated retrieval stack (iterations 1-9) put the claim-24 semantic
leg and the claim-10 associative leg on engram_search_json — the function
~40 internal .el call sites already used as a KEYED read. Seven of those
sites delete every record that comes back ("prune all existing X nodes,
keep exactly one"): memory.el:176, sessions.el:250/268/444/523,
soul.el:359.
mem_boot_count_inc() calls engram_search_json("soul:boot_count", 50) and
engram_forget()s all 50 results. With a lexical leg that returned 1 record.
With a semantic leg it returns 50 — the 49 nearest neighbours of the STRING
"soul:boot_count" — and the soul deletes them.
MEASURED on the harness corpus, isolated, read-only, zero writes from any
caller: 234 node records destroyed in a single boot. The deletion list is
the soul's own lookup result list, in rank order. Casualties include 6
Knowledge nodes, a layer-1 "CORE IDENTITY - GENESIS, LINEAGE" Memory, the
value node kn-58874a74, and the gold answers to 8 of the 75 gold-set
queries. After the fix: 1 deletion, which is the one the code intends.
THE BOUNDARY, from Will. Claim 24 authorises the vector index "to respond
to EMBEDDING SEARCH QUERIES by returning the node records whose embedding
vectors have the highest cosine similarity to a query vector". A keyed
state read is not an embedding search query; it is the identifier-keyed
retrieval of claim 23 ("node records are stored under a key encoding the
node identifier"). One function served both, so a nearest neighbour of
"soul:boot_count" was treated as a boot counter.
So: engram_search_json returns to its lexical contract, and the legs move
to engram_recall_json, which is what /api/neuron/recall reaches — the route
the MCP wrapper, the app, and this harness all call. Retrieval quality on
that route is unchanged by construction.
MEASURED, 75-query extended gold set, embedded corpus, vs the iteration-9
baseline: +3 / -0 (q15, q28, q60), p=0.2500, hit@5 53.8 -> 58.5%, latency
1.02x, every regression guard held, nonsense 10/10. Net +3 against a floor
of 6 is NOT-SHOWN and I am not calling it an improvement. The deliverable
is the defect.
Diagnostics kept, env-gated (EG_DIAG / EG_DIAG_ID), zero cost when unset:
node/embedding census at load, per-query leg dump, and a FORGET log — the
last is the regression detector for exactly this class of bug.
LIMIT, stated: handle_api_search_knowledge still uses the lexical function.
It is a retrieval surface and arguably wants the legs, but nothing in this
harness measures it, so I did not change unmeasured behaviour.
149 lines
3.3 KiB
JSON
149 lines
3.3 KiB
JSON
{
|
|
"baseline": "unfloor-clean",
|
|
"candidate": "splitfix",
|
|
"n_shared_queries": 75,
|
|
"fixed_by_candidate": [
|
|
"q15",
|
|
"q28",
|
|
"q60"
|
|
],
|
|
"broken_by_candidate": [],
|
|
"discordant": 3,
|
|
"net_queries": 3,
|
|
"mcnemar_exact_p": 0.25,
|
|
"min_detectable_swing_queries": 6,
|
|
"observed_run_to_run_drift_queries": 0,
|
|
"noise_floor_queries": 6,
|
|
"verdict": "no measurable difference",
|
|
"baseline_aggregate": {
|
|
"n_queries": 75,
|
|
"n_scored": 65,
|
|
"hit@5": 0.5384615384615384,
|
|
"recall@5": 0.44907176157176154,
|
|
"recall@10": 0.5380300255300255,
|
|
"precision@5": 0.13230769230769232,
|
|
"mrr@10": 0.32437728937728944,
|
|
"nonsense_clean": "10/10",
|
|
"superseded_outranks": "2/3",
|
|
"latency_ms_p50": 632.5,
|
|
"latency_ms_p95": 992.5,
|
|
"latency_ms_max": 1177.8,
|
|
"errors": 0,
|
|
"by_category": {
|
|
"associative": {
|
|
"n": 6,
|
|
"hit@5": 0.5,
|
|
"recall@5": 0.07575757575757576,
|
|
"recall@10": 0.13636363636363635,
|
|
"mrr@10": 0.23214285714285712
|
|
},
|
|
"exact_rare": {
|
|
"n": 6,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"mrr@10": 1.0
|
|
},
|
|
"heldout_paraphrase": {
|
|
"n": 30,
|
|
"hit@5": 0.3,
|
|
"recall@5": 0.3,
|
|
"recall@10": 0.4,
|
|
"mrr@10": 0.11638888888888889
|
|
},
|
|
"nonsense": {
|
|
"n": 10,
|
|
"clean": 10,
|
|
"avg_false_positives": 0.0
|
|
},
|
|
"paraphrase": {
|
|
"n": 13,
|
|
"hit@5": 0.6923076923076923,
|
|
"recall@5": 0.6923076923076923,
|
|
"recall@10": 0.7692307692307693,
|
|
"mrr@10": 0.29423076923076924
|
|
},
|
|
"phrase": {
|
|
"n": 7,
|
|
"hit@5": 1.0,
|
|
"recall@5": 0.5335884353741497,
|
|
"recall@10": 0.5933956916099773,
|
|
"mrr@10": 0.8214285714285714
|
|
},
|
|
"superseded": {
|
|
"n": 3,
|
|
"hit@5": 0.3333333333333333,
|
|
"recall@5": 0.3333333333333333,
|
|
"recall@10": 0.6666666666666666,
|
|
"mrr@10": 0.20833333333333334,
|
|
"outranks": 2
|
|
}
|
|
}
|
|
},
|
|
"candidate_aggregate": {
|
|
"n_queries": 75,
|
|
"n_scored": 65,
|
|
"hit@5": 0.5846153846153846,
|
|
"recall@5": 0.48102442429365505,
|
|
"recall@10": 0.5692415490492414,
|
|
"precision@5": 0.14153846153846153,
|
|
"mrr@10": 0.3351709401709402,
|
|
"nonsense_clean": "10/10",
|
|
"superseded_outranks": "2/3",
|
|
"latency_ms_p50": 646.9,
|
|
"latency_ms_p95": 1028.5,
|
|
"latency_ms_max": 1197.8,
|
|
"errors": 0,
|
|
"by_category": {
|
|
"associative": {
|
|
"n": 6,
|
|
"hit@5": 0.6666666666666666,
|
|
"recall@5": 0.08857808857808858,
|
|
"recall@10": 0.15967365967365968,
|
|
"mrr@10": 0.24166666666666667
|
|
},
|
|
"exact_rare": {
|
|
"n": 6,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"mrr@10": 1.0
|
|
},
|
|
"heldout_paraphrase": {
|
|
"n": 30,
|
|
"hit@5": 0.3333333333333333,
|
|
"recall@5": 0.3333333333333333,
|
|
"recall@10": 0.43333333333333335,
|
|
"mrr@10": 0.12120370370370372
|
|
},
|
|
"nonsense": {
|
|
"n": 10,
|
|
"clean": 10,
|
|
"avg_false_positives": 0.0
|
|
},
|
|
"paraphrase": {
|
|
"n": 13,
|
|
"hit@5": 0.7692307692307693,
|
|
"recall@5": 0.7692307692307693,
|
|
"recall@10": 0.8461538461538461,
|
|
"mrr@10": 0.33269230769230773
|
|
},
|
|
"phrase": {
|
|
"n": 7,
|
|
"hit@5": 1.0,
|
|
"recall@5": 0.5335884353741497,
|
|
"recall@10": 0.5775226757369615,
|
|
"mrr@10": 0.8214285714285714
|
|
},
|
|
"superseded": {
|
|
"n": 3,
|
|
"hit@5": 0.3333333333333333,
|
|
"recall@5": 0.3333333333333333,
|
|
"recall@10": 0.6666666666666666,
|
|
"mrr@10": 0.20833333333333334,
|
|
"outranks": 2
|
|
}
|
|
}
|
|
},
|
|
"repeat_variance": {}
|
|
} |