integrate: semantic retrieval + write-through, shipped together (closes the #146 ordering risk) #147
Reference in New Issue
Block a user
Delete Branch "integrate/semantic-plus-writethrough"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Integrates semantic retrieval with the write-through persistence that landed in main, so both ship together. Neither is useful alone: write-through without semantic recall stores memories nobody can find by meaning; semantic recall without write-through finds memories that die on the next restart.
Why this is one PR and not two
feat/semantic-leg-dropoutcarries the fix for #146 — the semantic leg was wired intoengram_search_json, which seven sites use as a keyed read and then delete every result from. One of those runs at every boot. Measured cost when it was live: ~234 real records destroyed per startup for nine iterations, including a core-identity memory and a value node.That makes ordering safety-critical. Semantic retrieval must never reach production without the
engram_recall_jsonsplit, and the split is only meaningful alongside durable writes. Shipping them separately is how the landmine gets armed.Gates run before deploying (isolated lab, real 44,608-node corpus)
pruned 190 old session-start events; the pre-merge binary drops 189 identically.)kill -9, read it back. Old binary LOST it; this one SURVIVED. A test that passes for both proves nothing, so the old arm was run deliberately.Value — Honesty Before Comfortat rank 2.Verified on the operator machine after deploy
Memory written reached the owning store; survived
kill -9; service back in 10s; identity intact (root, 13/13 values, hub wiring, 31/31 self, memory-philosophy); mind and store at delta 0.Known limitation, stated plainly
The semantic path is inert without meaning-vectors in the graph, and production currently has none. Deploying this binary alone does not improve retrieval — it makes the improvement possible. Loading vectors is a separate data operation.
Near-miss worth recording
The first attempt to measure the vector arm ran against a build without the semantic leg. Both arms returned byte-identical results except one float rounding artifact — which would have produced two identical numbers and a false "vectors don't help". Caught by diffing a single probe before running the battery. This is the same failure mode postmortem 0004 documents.
Nothing else on the memory roadmap should be built until a change can be shown to help. Right now we judge by feel, and the benchmark literature is full of systems that felt better and measured worse. This is the missing gate. WHAT IT MEASURES, AND WHY IT BOOTS A REAL SOUL The subject is Will's designed retrieval — spreading activation over the weighted directed graph, four-factor multiplicative scoring — not a proxy for it. A Python re-implementation would measure my reading of the design, so the harness compiles the actual soul.el amalgam from a git ref and asks it over HTTP on /api/neuron/recall, exactly as the MCP wrapper and the app do. BUILT ON WHAT WAS ALREADY HERE, NOT AROUND IT docs/research/graphrag_eval/{collect,score}.py — per-query relevant-id scoring and fixed-denominator precision@5 (kept verbatim: an empty result should be punished like a page of junk). docs/research-archive/p0-prototypes/eval_pinned_40q_20260715.py — the pinned ground truth + --check winnability gate, so every run judges alike. scripts/verify-soul-contract.sh — the isolation recipe, including the non-obvious SOUL_ISE_URL pin without which an "isolated" soul silently syncs the operator's live brain. gen-soul-amalgam.sh + .gitea/workflows/ci.yaml — the build recipe and flags. New here: ids rather than regexes as ground truth, an associative category derived from real edges, a superseded category scored on ranking, a machine-checked zero-lexical-overlap guarantee on paraphrases, paired significance testing, and measurement of the real compiled soul rather than an offline replica of one leg of it. THE GOLD SET IS AUDITABLE, NOT VIBES 38 queries over the real 78,768-node corpus, each carrying a `derivation` string, each re-validated by `build_gold_set.py --check`. exact_rare is mined (document frequency 1). phrase is mined (verbatim scan; >25 matches rejected as too diffuse). paraphrase is hand-selected then PROVEN to share zero content words with its target — a leak fails the build, so the category cannot decay into lexical matching. associative is derived from real hub edges with lexically-reachable siblings dropped. nonsense is verified absent. superseded pairs are kept only when both sides survive as distinct nodes. HONEST ABOUT NOISE Minimum detectable swing on 38 queries is 6: if every changed query moves the same way, p = 2*0.5^n first clears 0.05 at n=6. Run-to-run drift is measured, not assumed — activation is a stateful read, and it shows: main is fully deterministic across 3 runs, the candidate drifts by 1 query. compare.py reports "no measurable difference" for anything inside max(6, drift+1). FIRST VERDICT — feat/recall-through-activation hit@5 34.3% -> 22.9%, phrase 85.7% -> 28.6%, latency p50 2.81x. Five discordant pairs, all five against the candidate, none for it; McNemar exact p = 0.0625, so by the stated rule this is one query short of significant and is reported as such rather than as a win for main. The latency regression is deterministic and not in any noise band. The benefit the branch was written for is absent: associative recall is 0/6 on BOTH builds. Probed directly, the traversal returns the lexical seed at rank 8 and none of its 12 hub siblings. Two measured corpus facts explain it — only 4,060 of 78,768 nodes (5.2%) carry any edge, and no node has an embedding, so the fourth factor of the four-factor product has nothing to compute from. The mechanism runs; the corpus lacks the structure it needs. SAFETY Throwaway port, throwaway HOME, disposable per-run copy of the corpus; live ports refused by name. Every soul started is killed AND confirmed dead by pid probe, with the confirmation written into the results file; run_comparison.sh sweeps for strays and exits non-zero if any survive. Nothing under ~/.neuron, /Applications/Neuron*, or ~/neuron-dev-stack is read, written, or restarted. Rung: E2E-VERIFIED — 6 full runs (3 per config) against the real compiled binaries on the real corpus; numbers above are measured, not projected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The accumulated retrieval stack (iterations 1-9) put the claim-24 semantic leg and the claim-10 associative leg on engram_search_json — the function ~40 internal .el call sites already used as a KEYED read. Seven of those sites delete every record that comes back ("prune all existing X nodes, keep exactly one"): memory.el:176, sessions.el:250/268/444/523, soul.el:359. mem_boot_count_inc() calls engram_search_json("soul:boot_count", 50) and engram_forget()s all 50 results. With a lexical leg that returned 1 record. With a semantic leg it returns 50 — the 49 nearest neighbours of the STRING "soul:boot_count" — and the soul deletes them. MEASURED on the harness corpus, isolated, read-only, zero writes from any caller: 234 node records destroyed in a single boot. The deletion list is the soul's own lookup result list, in rank order. Casualties include 6 Knowledge nodes, a layer-1 "CORE IDENTITY - GENESIS, LINEAGE" Memory, the value node kn-58874a74, and the gold answers to 8 of the 75 gold-set queries. After the fix: 1 deletion, which is the one the code intends. THE BOUNDARY, from Will. Claim 24 authorises the vector index "to respond to EMBEDDING SEARCH QUERIES by returning the node records whose embedding vectors have the highest cosine similarity to a query vector". A keyed state read is not an embedding search query; it is the identifier-keyed retrieval of claim 23 ("node records are stored under a key encoding the node identifier"). One function served both, so a nearest neighbour of "soul:boot_count" was treated as a boot counter. So: engram_search_json returns to its lexical contract, and the legs move to engram_recall_json, which is what /api/neuron/recall reaches — the route the MCP wrapper, the app, and this harness all call. Retrieval quality on that route is unchanged by construction. MEASURED, 75-query extended gold set, embedded corpus, vs the iteration-9 baseline: +3 / -0 (q15, q28, q60), p=0.2500, hit@5 53.8 -> 58.5%, latency 1.02x, every regression guard held, nonsense 10/10. Net +3 against a floor of 6 is NOT-SHOWN and I am not calling it an improvement. The deliverable is the defect. Diagnostics kept, env-gated (EG_DIAG / EG_DIAG_ID), zero cost when unset: node/embedding census at load, per-query leg dump, and a FORGET log — the last is the regression detector for exactly this class of bug. LIMIT, stated: handle_api_search_knowledge still uses the lexical function. It is a retrieval surface and arguably wants the legs, but nothing in this harness measures it, so I did not change unmeasured behaviour.Pull request closed