feat(engine): recall retrieval stack — 18.5%→58.5% hit@5, plus the fix for a defect that deletes 234 nodes per boot #141

2026-08-08T00:29:15Z

tim.lingo commented

2026-08-08 00:29:15 +00:00

Ten measured iterations on recall. Two cleared the bar. One of the eight that didn't found a defect that would have destroyed memories in production.

This branch is the accumulated keep-stack: iterations 1, 2, 3, 4, 7, 9, 10. Every number below was measured on a copy of the live graph (78,768 nodes) with a committed harness, two runs per candidate, zero run-to-run drift. Nothing here has run on the live brain.

Read this part first

Iteration 10 found that the changes stacked on this branch caused the engine to delete 234 node records on every boot — including identity records, a core-identity node, and the gold answers to 8 of our own test queries.

Cause: the semantic leg was added to engram_search_json, which ~40 other call sites already use for a different purpose, and seven of those callers delete what it returns. It has been dormant only because the live graph has no embeddings, so the new leg returns nothing there.

The moment embeddings are populated in production — the step every prior iteration recommended — every restart begins shredding real memories. Fixed on this branch (soul-splitfix); records destroyed per boot back to 1, which is the pre-existing boot-counter node the code intends to replace. Any rollout of embeddings must be gated on this fix.

What is proven

Bar: a change must move ≥6 of the gold queries, above the measured noise floor.

Iteration	Change	Result
1	Embed the corpus (76,986 vectors) + a semantic leg in `engram_search_json`, rank-fused by strict alternation	hit@5 34.3% → 51.4%, paraphrase 0% → 38.5%, McNemar p=0.0312 · KEEP
3	Semantic seeding of the graph leg at Will's own `ENGRAM_EMBED_SEED_K=8`	hit@5 51.4% → 68.6%, associative 0% → 66.7% (first non-zero ever) · KEEP

Restores engram claim 24 (06-claims.md l.148), previously 0% honoured — a semantic read path independent of traversal, which is what distinguishes it from #135 — and the HippoRAG seeding pattern documented verbatim at el_runtime.c l.6082-6083.

Original vs final, same 75 queries, one measurement each side

	main	this branch
hit@5	18.5%	58.5%
recall@10	17.9%	56.9%
latency p50	1,380 ms	647 ms
nonsense correctly refused	9/10	10/10
records destroyed per boot	1	1

Out of sample it is a third of that. On the 30 held-out queries — written from the records before any search was run against them, machine-checked to share no content words with their answers — it went 0/30 → 10/30. In-sample 61.5% vs held-out 16.7% is the measured size of the flattery. Reported both ways deliberately.

Measured dead ends — do not re-attempt

Each cost a build; none needs another.

Score-fusion of cosine into the lexical score. nomic cosine is compressed (true 0.55-0.70, unrelated 0.35-0.50) so an additive blend is dominated by whichever leg has wider spread. Alternation is scale-invariant. Do not blend scores from two rankings with different distributions.
The Layer-2 executive filter (claims 44/45), iteration 5. Ported faithfully, moved 0 queries: all 13 nodes that answer 19 of 35 queries are identical on every field the filter reads. A filter cannot sort identical things.
Importance-weighting the score, iteration 7. Lost 3-11 queries in every variant, gained 0. Correct answers are not the high-importance nodes.
Asymmetric query/document embedding prefixes, iteration 9. Net −5. Rescues far-fetched matches, breaks obvious ones.
15 leg-interleaving orders × 2 directions, iteration 7. All 30 identical. Re-ordering is exhausted; the remaining misses need better finding, not better sorting.
Restricting the semantic leg to well-connected nodes, iteration 6. Scored +4 with no losses and was deliberately refused: it works because 19 of 35 gold answers sit among 219 well-connected nodes out of 78,791. Shipping it caps production recall at those 219. That is fitting the test, and it is recorded rather than banked.

Three production defects found and NOT fixed here

Save-side, outside this branch's scope, each needs its own decision:

1,032 nodes have unaddressable identifiers — 453 outright binary garbage, one of them 1.3M characters, which was outranking real answers.
46.6% of the graph — 36,737 nodes — are byte-identical copies of one single record.
Result lists were returning the same node repeatedly.

Reviewing this

Harness, gold sets, every results JSON and every comparison are committed under tools/retrieval-eval/. compare.py prints its own verdict against the noise floor. The embedded corpus is preserved outside soul-managed directories at ~/neuron-eval-corpora/ (642MB) — the soul rotates everything it owns.

Branches for the two discarded iterations are pushed and left unmerged as evidence: feat/executive-filter-recall, feat/claim24-unfloored-semantic.

🤖 Generated with Claude Code

**Ten measured iterations on recall. Two cleared the bar. One of the eight that didn't found a defect that would have destroyed memories in production.** This branch is the accumulated keep-stack: iterations 1, 2, 3, 4, 7, 9, 10. Every number below was measured on a copy of the live graph (78,768 nodes) with a committed harness, two runs per candidate, zero run-to-run drift. Nothing here has run on the live brain. ## Read this part first **Iteration 10 found that the changes stacked on this branch caused the engine to delete 234 node records on every boot** — including identity records, a core-identity node, and the gold answers to 8 of our own test queries. Cause: the semantic leg was added to `engram_search_json`, which ~40 other call sites already use for a different purpose, and seven of those callers *delete* what it returns. It has been dormant only because the live graph has no embeddings, so the new leg returns nothing there. **The moment embeddings are populated in production — the step every prior iteration recommended — every restart begins shredding real memories.** Fixed on this branch (`soul-splitfix`); records destroyed per boot back to 1, which is the pre-existing boot-counter node the code intends to replace. Any rollout of embeddings must be gated on this fix. ## What is proven Bar: a change must move ≥6 of the gold queries, above the measured noise floor. | Iteration | Change | Result | |---|---|---| | 1 | Embed the corpus (76,986 vectors) + a semantic leg in `engram_search_json`, rank-fused by strict alternation | hit@5 34.3% → 51.4%, paraphrase 0% → 38.5%, McNemar p=0.0312 · **KEEP** | | 3 | Semantic seeding of the graph leg at Will's own `ENGRAM_EMBED_SEED_K=8` | hit@5 51.4% → 68.6%, associative 0% → 66.7% (first non-zero ever) · **KEEP** | Restores engram claim 24 (`06-claims.md` l.148), previously 0% honoured — a semantic read path *independent of* traversal, which is what distinguishes it from #135 — and the HippoRAG seeding pattern documented verbatim at `el_runtime.c` l.6082-6083. ## Original vs final, same 75 queries, one measurement each side | | main | this branch | |---|---|---| | hit@5 | 18.5% | 58.5% | | recall@10 | 17.9% | 56.9% | | latency p50 | 1,380 ms | 647 ms | | nonsense correctly refused | 9/10 | 10/10 | | records destroyed per boot | 1 | 1 | **Out of sample it is a third of that.** On the 30 held-out queries — written from the records before any search was run against them, machine-checked to share no content words with their answers — it went 0/30 → 10/30. In-sample 61.5% vs held-out 16.7% is the measured size of the flattery. Reported both ways deliberately. ## Measured dead ends — do not re-attempt Each cost a build; none needs another. - **Score-fusion of cosine into the lexical score.** `nomic` cosine is compressed (true 0.55-0.70, unrelated 0.35-0.50) so an additive blend is dominated by whichever leg has wider spread. Alternation is scale-invariant. Do not blend scores from two rankings with different distributions. - **The Layer-2 executive filter (claims 44/45), iteration 5.** Ported faithfully, moved 0 queries: all 13 nodes that answer 19 of 35 queries are *identical* on every field the filter reads. A filter cannot sort identical things. - **Importance-weighting the score, iteration 7.** Lost 3-11 queries in every variant, gained 0. Correct answers are not the high-importance nodes. - **Asymmetric query/document embedding prefixes, iteration 9.** Net −5. Rescues far-fetched matches, breaks obvious ones. - **15 leg-interleaving orders × 2 directions, iteration 7.** All 30 identical. Re-ordering is exhausted; the remaining misses need better *finding*, not better *sorting*. - **Restricting the semantic leg to well-connected nodes, iteration 6.** Scored +4 with no losses and was **deliberately refused**: it works because 19 of 35 gold answers sit among 219 well-connected nodes out of 78,791. Shipping it caps production recall at those 219. That is fitting the test, and it is recorded rather than banked. ## Three production defects found and NOT fixed here Save-side, outside this branch's scope, each needs its own decision: 1. **1,032 nodes have unaddressable identifiers** — 453 outright binary garbage, one of them 1.3M characters, which was outranking real answers. 2. **46.6% of the graph — 36,737 nodes — are byte-identical copies of one single record.** 3. Result lists were returning the same node repeatedly. ## Reviewing this Harness, gold sets, every results JSON and every comparison are committed under `tools/retrieval-eval/`. `compare.py` prints its own verdict against the noise floor. The embedded corpus is preserved outside soul-managed directories at `~/neuron-eval-corpora/` (642MB) — the soul rotates everything it owns. Branches for the two discarded iterations are pushed and left unmerged as evidence: `feat/executive-filter-recall`, `feat/claim24-unfloored-semantic`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tim.lingo added 11 commits 2026-08-08 00:29:18 +00:00

test(retrieval): a measurement harness for memory recall, and its first verdict

Neuron Soul CI / build (pull_request) Failing after 14m41s

Details

Neuron Soul CI / deploy (pull_request) Failing after 14m45s

Details

cf41d12d22

Nothing else on the memory roadmap should be built until a change can be shown
to help. Right now we judge by feel, and the benchmark literature is full of
systems that felt better and measured worse. This is the missing gate.

WHAT IT MEASURES, AND WHY IT BOOTS A REAL SOUL
The subject is Will's designed retrieval — spreading activation over the
weighted directed graph, four-factor multiplicative scoring — not a proxy for
it. A Python re-implementation would measure my reading of the design, so the
harness compiles the actual soul.el amalgam from a git ref and asks it over
HTTP on /api/neuron/recall, exactly as the MCP wrapper and the app do.

BUILT ON WHAT WAS ALREADY HERE, NOT AROUND IT
  docs/research/graphrag_eval/{collect,score}.py  — per-query relevant-id
    scoring and fixed-denominator precision@5 (kept verbatim: an empty result
    should be punished like a page of junk).
  docs/research-archive/p0-prototypes/eval_pinned_40q_20260715.py — the pinned
    ground truth + --check winnability gate, so every run judges alike.
  scripts/verify-soul-contract.sh — the isolation recipe, including the
    non-obvious SOUL_ISE_URL pin without which an "isolated" soul silently
    syncs the operator's live brain.
  gen-soul-amalgam.sh + .gitea/workflows/ci.yaml — the build recipe and flags.
New here: ids rather than regexes as ground truth, an associative category
derived from real edges, a superseded category scored on ranking, a
machine-checked zero-lexical-overlap guarantee on paraphrases, paired
significance testing, and measurement of the real compiled soul rather than an
offline replica of one leg of it.

THE GOLD SET IS AUDITABLE, NOT VIBES
38 queries over the real 78,768-node corpus, each carrying a `derivation`
string, each re-validated by `build_gold_set.py --check`. exact_rare is mined
(document frequency 1). phrase is mined (verbatim scan; >25 matches rejected as
too diffuse). paraphrase is hand-selected then PROVEN to share zero content
words with its target — a leak fails the build, so the category cannot decay
into lexical matching. associative is derived from real hub edges with
lexically-reachable siblings dropped. nonsense is verified absent. superseded
pairs are kept only when both sides survive as distinct nodes.

HONEST ABOUT NOISE
Minimum detectable swing on 38 queries is 6: if every changed query moves the
same way, p = 2*0.5^n first clears 0.05 at n=6. Run-to-run drift is measured,
not assumed — activation is a stateful read, and it shows: main is fully
deterministic across 3 runs, the candidate drifts by 1 query. compare.py
reports "no measurable difference" for anything inside max(6, drift+1).

FIRST VERDICT — feat/recall-through-activation
hit@5 34.3% -> 22.9%, phrase 85.7% -> 28.6%, latency p50 2.81x. Five discordant
pairs, all five against the candidate, none for it; McNemar exact p = 0.0625,
so by the stated rule this is one query short of significant and is reported as
such rather than as a win for main. The latency regression is deterministic and
not in any noise band.

The benefit the branch was written for is absent: associative recall is 0/6 on
BOTH builds. Probed directly, the traversal returns the lexical seed at rank 8
and none of its 12 hub siblings. Two measured corpus facts explain it — only
4,060 of 78,768 nodes (5.2%) carry any edge, and no node has an embedding, so
the fourth factor of the four-factor product has nothing to compute from. The
mechanism runs; the corpus lacks the structure it needs.

SAFETY
Throwaway port, throwaway HOME, disposable per-run copy of the corpus; live
ports refused by name. Every soul started is killed AND confirmed dead by pid
probe, with the confirmation written into the results file; run_comparison.sh
sweeps for strays and exits non-zero if any survive. Nothing under ~/.neuron,
/Applications/Neuron*, or ~/neuron-dev-stack is read, written, or restarted.

Rung: E2E-VERIFIED — 6 full runs (3 per config) against the real compiled
binaries on the real corpus; numbers above are measured, not projected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(engram): fuse cosine similarity into the recall read path (claim 24) 315b2eff00

engram_search_json — the function /api/neuron/recall actually reaches — ranked
only by distinct-token match count, so the embedding field on every node record
was inert. Add the semantic leg as a UNION beside the lexical one, not a
replacement for it:

  fused = (distinct_tokens_matched / query_tokens) + 0.90 * sem
  sem   = clamp01((cos(q,n) - 0.60) / (1 - 0.60))     ; 0 when not comparable

Holding the semantic weight strictly below 1.0 means a node matching every
query token can never be displaced by semantics alone — the regression guard
that PR #135 lacked when it swapped the read path to spreading activation and
took phrase recall from 85.7% to 28.6%.

No query embedding (embedder down, circuit breaker open) => sem == 0 for all
nodes => fused == sc/ntok, a monotone map of the old integer score, so the
ordering degrades to the historical behaviour exactly.

Restores engram claim 24: 'maintain a vector similarity index over the semantic
embedding vectors of all stored node records, and ... respond to embedding
search queries by returning the node records whose embedding vectors have the
highest cosine similarity to a query vector, independently of the spreading
activation traversal.'

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(engram): rank-interleave the semantic leg into recall; embed the corpus 635453b936

Replaces the score-fusion first cut with rank fusion, which is what the data
called for. nomic's cosine scale is compressed (true matches 0.55-0.70,
unrelated pairs 0.35-0.50), so an additive blend of cosine onto token-coverage
is dominated by whichever leg has the wider spread. Alternation is invariant to
both scales:

  L1, S1, L2, S2, ...  deduped, capped at limit

Lexical ranking is left byte-identical; the semantic ranking is computed beside
it and admitted only above ENGRAM_EMBED_SEED_MIN (0.60) — Will's existing seed
floor, no new tuning constant. That floor is what keeps the nonsense controls
clean: a query with no real match must not be answered with its neighbours.

embed-corpus.py / merge-corpus.py produce the derived corpus the semantic leg
needs (76,986 vectors, nomic-embed-text, 0 failures, 11 min). Zero of 78,791
nodes carried an embedding before this; the field round-tripped through the
snapshot but nothing ever wrote it.

MEASURED, 38-query gold set, paired against the SAME derived corpus so the
comparison isolates the code change:

  hit@5      34.3% -> 51.4%     paraphrase   0.0% -> 38.5%
  MRR@10     0.294 -> 0.387     superseded   1/3  -> 2/3 outranks
  recall@10  33.3% -> 50.5%     latency p50  1146 -> 1220ms (1.06x)

  exact_rare 100% -> 100%   phrase 85.7% -> 85.7%   nonsense 2/3 -> 2/3

  6 queries fixed, 0 broken, McNemar exact p=0.0312, 0 drift across repeats.

Regression guards all held. Contrast PR #135, which swapped the read path to
spreading activation wholesale: phrase 85.7 -> 28.6, latency 2.81x. Correct
mechanism, wrong substrate. The substrate is now present.

Restores engram claim 24 (previously 0% honoured).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(engram): an associative leg on the recall path (claim 10 typed relations) 059ce02003

The recall route had no way to reach a node that shares no token and no
embedding neighbourhood with the query. The design reserves that case for the
graph, and nothing on the read path consulted an edge.

This adds a third ranked leg beside the lexical and semantic ones: expand the
top 3 lexical hits along STRUCTURAL relations only (claim 10 — identity,
contains, superseded_by, references, ...), two hops, both directions, pruned
at the same 0.02 firing threshold engram_activate uses; order what was reached
by query similarity. Merged by strict rotation, never by score blending.

Not PR #135. That wired recall wholesale to engram_activate and lost 57 points
of phrase accuracy. The failure there was RANK, not reach — a 2-hop associate
at strength 0.06 cannot outrank thousands of 1-hop neighbours of strong
lexical seeds. Here the lexical leg is untouched and the associative list is
empty for most queries, because a node whose only edges are `tagged` and
`related` expands to nothing.

MEASURED, hybrid-semantic baseline -> this, 38-query gold set, embedded corpus:
  associative  0.0% -> 66.7%   (first non-zero ever recorded on that category)
  hit@5       51.4% -> 62.9%
  exact_rare, phrase, paraphrase, nonsense, superseded: all unchanged
  latency p50 1.01x
  4 queries moved, all gains, 0 losses, McNemar p=0.125
  deterministic: two runs of the same binary differ on 0 of 38 rows

VERDICT: NOT-SHOWN. The harness needs 6 queries to clear p<0.05 and the whole
associative category is only 6 queries, so even 4/6 fixed cannot reach the
floor. The mechanism is confirmed to work; the gold set cannot certify it.

feat(engram): semantically seed the graph leg (Will's HippoRAG pass, SEED_K=8) 6f3a048f36

engram_assoc_leg previously took its seeds only from the top-3 LEXICAL hits.
For a paraphrase query the lexical hits are noise by construction, so the walk
never reached the neighbourhood that holds the answer. This adds the seeding
pass Will documents at el_runtime.c l.6082 — "Semantic seeding (HippoRAG
pattern, use similarity twice): the query is embedded, the top-K nodes by
cosine join the seed set" — using his own ENGRAM_EMBED_SEED_K (8).

Similarity is now used twice, coherently: cosine picks where to STAND in the
graph, the structural-relation walk decides what is REACHABLE, and cosine
orders what was reached (iteration 2's finding, unchanged).

The seed list is deliberately NOT floored at ENGRAM_EMBED_SEED_MIN. Measured
over all 38 gold queries: true paraphrase targets score cosine 0.46-0.66 and
the three nonsense controls' own nearest neighbours score 0.55/0.60/0.62 —
the distributions OVERLAP, so no absolute cosine floor separates signal from
gibberish. The gate that works is reachability: gibberish's nearest neighbours
carry no structural edge, so its graph leg is empty and the controls hold.

The raw top-K is selected inside the existing scoring pass, so the cosine is
computed exactly once per node: no extra corpus pass, no extra embed
round-trip, latency flat (p50 1220 -> 1227 ms, 1.01x).

Measured vs the certified baseline feat/hybrid-semantic-recall, embedded
corpus, 2 runs each, zero run-to-run drift on both sides:
  hit@5 51.4% -> 68.6%   MRR@10 0.387 -> 0.461
  paraphrase 38.5% -> 61.5%   associative 0% -> 66.7%
  exact_rare 100% held, nonsense 2/3 held, superseded 2/3 held
  phrase 85.7% -> 71.4% (q11, the known rank-5 rotation tax)
  net +6 queries (7 fixed / 1 broken), McNemar p=0.0703

feat(engram): BM25-shaped lexical leg + addressability guard on the read path 9c39084e60

engram_search_json ranked its lexical leg by raw distinct-token coverage with
salience as tiebreak: a token in 30,000 nodes counted the same as a token in 1,
and a 1.3 MB record matched nearly every query token by surface area alone.
Score it BM25-shaped instead - Lucene-form IDF and length normalisation over
the corpus mean - with per-token document frequency accumulated in the SAME
corpus pass that finds the hits (no extra scan, no extra round-trip).

Also refuse to return records whose identifier is not printable ASCII. This
corpus carries 1,032 such records (453 by the printable test) from a save-side
corruption; they occupy 125 of 303 returned slots on main. Claims 12, 23 and 27
all key on the node identifier, so such a record is unfetchable by any caller.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

measure: BM25 lexical leg vs semseed baseline - net +2 (q10,q11), NOT-SHOWN 55f9ee3cb0

hit@5 68.6% -> 74.3%, phrase 71.4% -> 100%, MRR@10 0.461 -> 0.502, latency
p50 0.97x. Zero losses, zero run-to-run drift on both sides. 2 queries moved
against a 6-query noise floor: NO MEASURABLE DIFFERENCE by the harness's own
test (McNemar exact p=0.50). Unaddressable records in returned slots: 57 -> 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(engram): word-start match primitive + corpus-vocabulary gate on recall 4eb4c9e287

The retrieval match test is a raw substring scan, so a query token matches
anywhere INSIDE a corpus word: "throom" matches "bathroom". Measured over the
38-query gold set on this corpus that is not a rare accident - q28's lexical
leg is 36,954 records of which only 13 contain a query token at a word start
(99.96% mid-word noise), six further queries carry ~20,500 mid-word-only
records each, and the nonsense control q35 returns 7 records ALL of which
match only mid-word.

istr_contains_wordstart() anchors a token to a word start (preceding char not
alphanumeric) while still matching suffixes, so "value" still hits "values".
That empties the lexical leg for gibberish, and the nhits==0 corpus-vocabulary
gate (iteration 6's mechanism, feat/claim24-unfloored-semantic) then makes the
whole query decline rather than let the semantic leg answer it.

Measured vs feat/bm25-lexical-leg on the embedded corpus, 2 runs each,
0 queries of run-to-run drift on both sides:
  net +1 (nonsense:q35), 0 losses, McNemar p=1.0 -> NOT-SHOWN (floor is 6)
  nonsense clean 2/3 -> 3/3; exact_rare 100%, phrase 100%, paraphrase 61.5%,
  associative 66.7%, superseded 2/3 all UNCHANGED
  latency p50 1184 -> 543 ms (0.46x)

Iteration 6 called q35 "a DEFECTIVE CONTROL ... cannot be cleaned without
breaking the lexical leg". It can: the defect was the match primitive, and
cleaning it cost nothing.

Also committed: results-wsclaim24.json + cmp-nogate.json, a measured negative
for bundling the claim-24 unfloored semantic leg on top (gains q14/q25, breaks
q15/q28/q33/q34, net -2) - it independently reproduces iteration 6's q15/q28
losses and shows unflooring REQUIRES the vocabulary gate.

Reproducers: legs.py (leg-level replica, reproduces baseline hit@5 exactly on
all 38 queries), policy2.py, ceiling.py, wb2.py.

feat(engram): claim 24 without a threshold, and an embedding substrate that knows query from document 9790d9342d

Two changes to the ONE leg that generalises. Iteration 8 measured that out of
sample the semantic leg contributes 100% of the stack's gain and the graph leg
contributes nothing, so this is where the remaining headroom is.

1. THE 0.60 FLOOR IS A PER-QUERY LOTTERY, AND CLAIM 24 HAS NO THRESHOLD IN IT.
   06-claims.md l.148: "respond to embedding search queries by returning the
   node records whose embedding vectors have the HIGHEST COSINE SIMILARITY to a
   query vector, independently of the spreading activation traversal." A
   ranking. ENGRAM_EMBED_SEED_MIN is defined at el_runtime.c l.6094 as the
   HippoRAG seed-JOIN threshold and l.6102 admits the read-path leg merely
   "reuses" it. Measured on the 30 held-out paraphrases: the query's own top-1
   cosine ranges 0.564-0.680, so the constant keeps a rank-1 answer for one
   query and discards a rank-1 answer for the next. Six golds sit at global
   cosine rank 1-2 scoring 0.564-0.589 - discarded by nothing but the constant.
   What holds the nonsense controls is the corpus-vocabulary gate (nhits == 0),
   not this floor. Cosine clamped to [0,1] per 05-detailed-description l.69.

2. THE VECTORS THEMSELVES ANSWER THE WRONG QUESTION. EL_EMBED_MODEL defaults to
   nomic-embed-text, an ASYMMETRIC retrieval encoder trained with task prefixes.
   Embedding query and document bare - as this file did on both sides - measures
   topical similarity rather than answer-hood. eg_embed_fetch now takes the task
   prefix: EL_EMBED_QUERY_PREFIX on the three query call sites, EL_EMBED_DOC_PREFIX
   on the two backfill sites. Restores no claim, and says so: Will specifies only
   "computed by an embedding model over the node's content" (l.17), so the model
   is his and its correct use is ours. It is the substrate under claim 24 -
   the index is only as good as the vectors in it.

Reproducer for the derived corpus: tools/retrieval-eval/embed-corpus-prefixed.py.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

measure: claim 24 unflooring is +4 (NOT-SHOWN); asymmetric embedding prefixes are -5 (discarded) 9717a4eeaf

Measured on the 75-query extended gold set (iteration 8's held-out extension)
against the certified stack baseline results-stack-ext.json, on the embedded
corpus. Three runs of the candidate, zero drift.

A. CLAIM 24 WITHOUT THE THRESHOLD - net +4, NOT-SHOWN, kept in the tree.
   fixed  : q14, q25 (in-sample paraphrase), q43, q52, q63, q67 (held-out)
   broken : q15 (paraphrase), q28 (associative)
   8 discordant, McNemar exact p = 0.2891, floor is 6.
   heldout_paraphrase 16.7% -> 30.0%, paraphrase 61.5% -> 69.2%.
   Every regression guard held: exact_rare 6/6, phrase 7/7, nonsense 10/10,
   superseded 2/3. Latency FLAT: p50 641 -> 632 ms.
   The in-sample half (+q14 +q25 -q15 -q28 = 0) was already on record in
   iteration 7's cmp-nogate.json, so only the held-out +4 is new.

B. ASYMMETRIC TASK PREFIXES ON THE EMBEDDER - net -5, REVERTED in this commit.
   Rationale was sound and the prediction was wrong, which is why it was worth
   measuring: nomic-embed-text is an asymmetric retrieval encoder and this file
   embedded query and document bare on both sides. Prefixing does exactly what
   the model card implies for the far-away cases - it rescued q42 (gold at
   GLOBAL COSINE RANK 25,564) and q39 - but it re-ranks the whole space and
   broke more than it fixed:
   fixed  : q24, q39, q42
   broken : q18, q19, q22, q31, q43, q44, q52, q63
   heldout_paraphrase 30.0% -> 23.3%, paraphrase 69.2% -> 53.8%.
   The corpus and the reproducer are kept (embed-corpus-prefixed.py,
   snapshot-pre-repair-20260806-embedded-prefixed.json) so nobody re-runs it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fix(engram): the semantic leg was deleting 234 memory records per boot

Neuron Soul CI / build (pull_request) Failing after 14m34s

Details

Neuron Soul CI / deploy (pull_request) Has been skipped

Details

b9ef66cae9

The accumulated retrieval stack (iterations 1-9) put the claim-24 semantic
leg and the claim-10 associative leg on engram_search_json — the function
~40 internal .el call sites already used as a KEYED read. Seven of those
sites delete every record that comes back ("prune all existing X nodes,
keep exactly one"): memory.el:176, sessions.el:250/268/444/523,
soul.el:359.

mem_boot_count_inc() calls engram_search_json("soul:boot_count", 50) and
engram_forget()s all 50 results. With a lexical leg that returned 1 record.
With a semantic leg it returns 50 — the 49 nearest neighbours of the STRING
"soul:boot_count" — and the soul deletes them.

MEASURED on the harness corpus, isolated, read-only, zero writes from any
caller: 234 node records destroyed in a single boot. The deletion list is
the soul's own lookup result list, in rank order. Casualties include 6
Knowledge nodes, a layer-1 "CORE IDENTITY - GENESIS, LINEAGE" Memory, the
value node kn-58874a74, and the gold answers to 8 of the 75 gold-set
queries. After the fix: 1 deletion, which is the one the code intends.

THE BOUNDARY, from Will. Claim 24 authorises the vector index "to respond
to EMBEDDING SEARCH QUERIES by returning the node records whose embedding
vectors have the highest cosine similarity to a query vector". A keyed
state read is not an embedding search query; it is the identifier-keyed
retrieval of claim 23 ("node records are stored under a key encoding the
node identifier"). One function served both, so a nearest neighbour of
"soul:boot_count" was treated as a boot counter.

So: engram_search_json returns to its lexical contract, and the legs move
to engram_recall_json, which is what /api/neuron/recall reaches — the route
the MCP wrapper, the app, and this harness all call. Retrieval quality on
that route is unchanged by construction.

MEASURED, 75-query extended gold set, embedded corpus, vs the iteration-9
baseline: +3 / -0 (q15, q28, q60), p=0.2500, hit@5 53.8 -> 58.5%, latency
1.02x, every regression guard held, nonsense 10/10. Net +3 against a floor
of 6 is NOT-SHOWN and I am not calling it an improvement. The deliverable
is the defect.

Diagnostics kept, env-gated (EG_DIAG / EG_DIAG_ID), zero cost when unset:
node/embedding census at load, per-query leg dump, and a FORGET log — the
last is the regression detector for exactly this class of bug.

LIMIT, stated: handle_api_search_knowledge still uses the lexical function.
It is a retrieval surface and arguably wants the legs, but nothing in this
harness measures it, so I did not change unmeasured behaviour.

tim.lingo referenced this pull request

2026-08-08 00:33:27 +00:00

Save-side: 46.6% of the graph (36,737 nodes) are byte-identical copies of one record #144

tim.lingo referenced this pull request

2026-08-08 00:34:04 +00:00

Save-side: 1,032 nodes have unaddressable identifiers (453 binary garbage, one 1.3M chars outranking real answers) #145

tim.lingo referenced this pull request

2026-08-08 00:34:15 +00:00

Changing a shared function turned retrieval into deletion — 234 nodes destroyed per boot, dormant in prod only because embeddings are empty #146

tim.lingo referenced this pull request

2026-08-08 00:36:25 +00:00

feat(engine): recall now spreads activation through the graph instead of matching substrings #135