Tim Lingo
b9ef66cae9
The accumulated retrieval stack (iterations 1-9) put the claim-24 semantic
leg and the claim-10 associative leg on engram_search_json — the function
~40 internal .el call sites already used as a KEYED read. Seven of those
sites delete every record that comes back ("prune all existing X nodes,
keep exactly one"): memory.el:176, sessions.el:250/268/444/523,
soul.el:359.
mem_boot_count_inc() calls engram_search_json("soul:boot_count", 50) and
engram_forget()s all 50 results. With a lexical leg that returned 1 record.
With a semantic leg it returns 50 — the 49 nearest neighbours of the STRING
"soul:boot_count" — and the soul deletes them.
MEASURED on the harness corpus, isolated, read-only, zero writes from any
caller: 234 node records destroyed in a single boot. The deletion list is
the soul's own lookup result list, in rank order. Casualties include 6
Knowledge nodes, a layer-1 "CORE IDENTITY - GENESIS, LINEAGE" Memory, the
value node kn-58874a74, and the gold answers to 8 of the 75 gold-set
queries. After the fix: 1 deletion, which is the one the code intends.
THE BOUNDARY, from Will. Claim 24 authorises the vector index "to respond
to EMBEDDING SEARCH QUERIES by returning the node records whose embedding
vectors have the highest cosine similarity to a query vector". A keyed
state read is not an embedding search query; it is the identifier-keyed
retrieval of claim 23 ("node records are stored under a key encoding the
node identifier"). One function served both, so a nearest neighbour of
"soul:boot_count" was treated as a boot counter.
So: engram_search_json returns to its lexical contract, and the legs move
to engram_recall_json, which is what /api/neuron/recall reaches — the route
the MCP wrapper, the app, and this harness all call. Retrieval quality on
that route is unchanged by construction.
MEASURED, 75-query extended gold set, embedded corpus, vs the iteration-9
baseline: +3 / -0 (q15, q28, q60), p=0.2500, hit@5 53.8 -> 58.5%, latency
1.02x, every regression guard held, nonsense 10/10. Net +3 against a floor
of 6 is NOT-SHOWN and I am not calling it an improvement. The deliverable
is the defect.
Diagnostics kept, env-gated (EG_DIAG / EG_DIAG_ID), zero cost when unset:
node/embedding census at load, per-query leg dump, and a FORGET log — the
last is the regression detector for exactly this class of bug.
LIMIT, stated: handle_api_search_knowledge still uses the lexical function.
It is a retrieval surface and arguably wants the legs, but nothing in this
harness measures it, so I did not change unmeasured behaviour.
Retrieval eval harness
Measures Neuron's memory retrieval so a change can be shown to help before it is believed to help. Nothing else on the memory roadmap should ship without a run through this.
tools/retrieval-eval/run_comparison.sh --baseline main --candidate <branch>
That builds a soul from each ref, boots each in isolation on a fixed corpus, runs the gold set three times per ref, and prints a table plus a verdict that refuses to call a difference real if it is inside the noise band.
What was reused
This is not a new idea, it is the missing third of an existing one.
| Prior work | What it gave | What was missing |
|---|---|---|
docs/research/graphrag_eval/ (collect.py, score.py, 2026-06-08) |
The three-retriever comparison that produced the numbers everyone quotes: substring 1.7% P@5, graph 21.7%, BM25 55%. Per-query relevant-id scoring, fixed-denominator precision@5, unique-relevant analysis. | 13 hand-written queries, judged by an LLM after the fact; measured the live soul on the live engram. |
docs/research-archive/p0-prototypes/eval_pinned_40q_20260715.py |
The pinned-query discipline: ground truth committed as regexes so every run judges alike, plus a --check winnability gate. 40 queries in 5 bands including a deliberate paraphrase-hard band. |
Scored offline replicas of substring/BM25 — it never ran the real retrieval path. |
docs/research-archive/p0-prototypes/stage0_eval_20260714.py |
The hit@5 metric and the substring/BM25 reference implementations. |
Same: offline only. |
scripts/verify-soul-contract.sh |
The isolation recipe, verbatim: throwaway port, throwaway HOME, SOUL_ENGRAM_PATH, and the non-obvious SOUL_ISE_URL pin that stops an "isolated" soul silently syncing the operator's live brain. |
It is a contract gate, not a measurement. |
_engine-liveness-91/gen-soul-amalgam.sh + .gitea/workflows/ci.yaml |
The build recipe (elc --target=c with every .elh on the import chain removed) and CI's exact compile flags. |
— |
Reused directly: the isolation recipe, the build recipe, fixed-denominator precision@5, the pinned-ground-truth and winnability ideas. New here: ids rather than regexes as ground truth, an associative category derived from real graph edges, a superseded/contradicted category scored on ranking, a machine-checked zero-lexical-overlap guarantee on paraphrases, paired significance testing, and — the point — measurement against the real compiled soul rather than an offline replica of one leg of it.
Design fit
The thing under measurement is Will's designed retrieval: spreading activation
over the weighted directed graph, four-factor multiplicative scoring (parent
strength x edge weight x target salience x query/target cosine). A Python
re-implementation would measure my reading of the design. So the harness
compiles the actual soul.el amalgam and asks it over HTTP on
/api/neuron/recall, exactly as the MCP wrapper and the app do.
Files
| File | Does |
|---|---|
build_gold_set.py |
Derives and validates the gold set from the corpus. --check re-validates and exits non-zero if a query became unwinnable or a paraphrase leaked a word. |
gold_set.json |
38 queries. Every one carries a derivation string. |
run_eval.py |
Boots one soul in isolation, runs the gold set, writes metrics. Kills and confirms dead its child; records the confirmation in the results file. |
compare.py |
Paired diff of two result files with McNemar's exact test and a stated noise floor. |
build-soul.sh |
Compiles a soul binary from a plain source tree. |
run_comparison.sh |
All of the above, end to end, from two git refs. |
The gold set — 38 queries
Built from the real corpus (snapshot-pre-repair-20260806.json, 78,768 nodes /
14,214 edges) so it reflects one person's accumulating memory, not document QA.
| Category | n | Expected answer derived by |
|---|---|---|
exact_rare |
6 | Mined. Tokens with document frequency 1 across all 78,768 nodes, whose single containing node is a 300–6000 char Memory/Knowledge/Belief. That node is the only possible answer. Re-verified every build. |
phrase |
7 | Mined. Case-insensitive verbatim scan; the matching set is the answer key. Phrases matching >25 nodes are rejected as too diffuse. |
paraphrase |
13 | Hand-selected, machine-checked. Target locked by id; the build then proves that zero content words of the query appear anywhere in the target's label, content, or tags. A leak fails the build — the category cannot quietly decay into lexical matching. |
associative |
6 | Derived from edges. Query built from one value node's distinctive vocabulary; expected answers are its siblings on the Self - Values (grounded) hub. Siblings sharing any query word are dropped, so the only route from query to answer is seed -> hub -> sibling. |
nonsense |
3 | Control. Verified that no token occurs anywhere in the corpus. Correct behaviour is to return nothing. |
superseded |
3 | Derived. Correction/stale pairs located by regex scan, kept only when both sides resolve to different surviving nodes. Scored on ranking: the correction must be returned and rank above the stale node. |
Metrics
hit@5, recall@5, recall@10, precision@5 (fixed denominator 5, so an
empty result is punished like a page of junk), MRR@10, and wall-clock latency
per query (p50/p95/max). Output is a table plus a machine-readable JSON per run
so runs can be diffed.
Honesty about noise
- Minimum detectable swing on this 38-query set: 6 queries. If every query
that changes changes the same way,
p = 2 x 0.5^n, which first drops under 0.05 at n=6. Any net change smaller than that is inside the noise band andcompare.pysays so in those words. - Run-to-run drift is measured, not assumed. Activation is a stateful read
by design (traversal reinforces what it touches), so identical inputs need not
give identical outputs. Observed:
main0 queries of drift across 3 runs (fully deterministic); the activation branch 1 query. - The noise floor used for the verdict is
max(6, observed_drift + 1). - This gold set is underpowered for small effects. A genuine 3-query improvement would not clear the bar. Growing the set is the fix; until then, a small positive delta means "not shown", not "no effect".
First result: main vs feat/recall-through-activation
Corpus and gold set identical, three runs each, fresh corpus copy per run.
| main | recall-through-activation | delta | |
|---|---|---|---|
| hit@5 | 34.3% | 22.9% | -11.4pp |
| recall@5 | 26.9% | 19.1% | -7.9pp |
| recall@10 | 33.3% | 24.3% | -9.1pp |
| precision@5 | 12.0% | 7.4% | -4.6pp |
| MRR@10 | 0.294 | 0.242 | -0.053 |
| latency p50 | 1140 ms | 3209 ms | 2.81x |
| latency p95 | 1584 ms | 4852 ms | 3.06x |
| nonsense clean | 2/3 | 2/3 | — |
| superseded outranks | 1/3 | 0/3 | -1 |
By category (hit@5):
| category | main | activation |
|---|---|---|
| exact_rare | 100% | 100% |
| phrase | 85.7% | 28.6% |
| paraphrase | 0% | 0% |
| associative | 0% | 0% |
| superseded | 0% | 0% |
Verdict: directionally worse, one query short of significant. 5 discordant pairs, all 5 against the candidate, 0 for it. McNemar exact p = 0.0625 — under the stated rule that is inside the noise band, so the harness reports "no measurable difference" on accuracy and the honest summary is "5 for 5 the wrong way, needs a 6th or a larger gold set to call".
Latency is a different story: 2.8x at p50 is deterministic and far outside any noise band. That regression is real.
The result the branch was written for did not appear. Its stated purpose was to
recover sibling nodes one hub-hop away — the associative category — and that
category is 0/6 on both builds. Probing directly: for the query
Marines hernia sepsis medical ward, the activation build returns the lexical
seed node itself at rank 8, and none of its 12 hub siblings anywhere in the top
10. The traversal is running; it is not reaching siblings.
Two corpus facts likely explain it, and both are measurable rather than speculative:
- The graph is nearly edgeless. Only 4,060 of 78,768 nodes (5.2%) carry any edge at all — 14,214 edges total, 0.18 per node. Spreading activation over a graph with no edges is an expensive way to do lexical matching, which is roughly what the numbers show.
- No embeddings. No node in this snapshot has an embedding field, so the fourth factor of the four-factor product — query/target cosine similarity — has nothing to compute from, and the semantic seeding pass is inert.
That is the harness earning its keep on its first job: the change would have felt like progress (it is the designed mechanism, and it does run) and measures as a regression on phrase queries plus a 2.8x latency cost, with its intended benefit unrealised because the corpus lacks the structure it needs.