neuron-technologies/neuron
3
0
Fork 0
Code Issues 44 Pull Requests 7 Actions Packages Projects Releases Wiki Activity

feat(engine): recall now spreads activation through the graph instead of matching substrings #135

Closed
tim.lingo wants to merge 1 commits from feat/recall-through-activation into main
pull from: feat/recall-through-activation
merge into: :main
Conversation 1 Commits 1 Files Changed 1
+134 -10
tim.lingo commented 2026-08-07 18:24:48 +00:00
Member
Copy Link
Copy Source

Neuron's spreading-activation retrieval was fully built, compiled into the shipped binary, and never called by the two tools everything uses. This wires it up.

What was wrong

handle_api_recall ended with engram_search_json(...) — a single-token case-insensitive substring matcher, scored by counting query tokens found in content/label/tags, sorted by coverage then raw salience. It never touches an edge.

Meanwhile engram_activate / engram_activate_json are real BFS spreading activation over the graph, fully implemented, present in the shipped binary (nm confirms T _engram_activate_json) — and called from only four places, none of them recall or searchKnowledge.

Measured consequence: recall("inherited") returned substring hits including a TCO evaluation framework and an unrelated whitepaper, and did not return the 12 sibling Value nodes one hop off the same hub.

The design being restored

Engram provisional 64/064,260, claim 1: "no data is retrieved from the weighted directed graph except through the spreading activation traversal." Scoring is four-factor and multiplicative — "the multiplication of all four factors enforces a conjunctive property… addition would allow many weak associations to accumulate into false relevance." This is a restoration, not an invention.

Seeding

engram_activate already takes query text, not seed ids, and seeds itself: lexically (any node matching ≥1 token, initial activation = salience × temporal_decay × dampening × token_coverage) then a semantic top-K cosine supplement. All four existing call sites pass text the same way, so this follows the runtime's own convention.

Exact lookup is protected by a seeded hybrid, not a bolted-on union: activation's collector drops any reached node below bg × confidence < 0.1 unless promoted to working memory, so a rare token on a dormant node can seed the traversal and go unreported. The lexical seed list is appended after the activated ranking, deduped, until limit fills — the same seed set the traversal already computed, restored to the tail.

Measured, two builds of this tree, same 79,250-node graph, cold start

query main@18714e6 this branch
inherited 10 results — ntn-genesis, 1, a Disneyland fragment. Value node rank 23, hub rank 32 Value node rank 3, Self — Values hub rank 4
Structure is not inherited Value rank 31, plus a corrupt node Value rank 16; returns voice/imprint/VBD material
volatility-based decomposition 1 of 10 relevant 6 of 10 relevant
searchKnowledge, same query 2 of 5 relevant 5 of 5
HNSW, Fayetteville (rare exact) 1, 1 1, 1 — no regression
nonsense control 0 0

searchKnowledge got the same treatment because its "activate fallback" was unreachable dead code — it fired only if engram_search_json returned a string not starting with [ or {, which it never does.

Correction to an earlier claim of ours: multi-word queries returning zero is true of the live soul binary but not of main — main's matcher is already tokenized, so it returns 10 results, just poor ones. The defect on main is relevance, not emptiness. The live soul is running an older build.

Rung

E2E-VERIFIED. Amalgam 1,164,103 → 1,167,337 bytes (1226 → 1229 bodies); binaries 903,096 and 903,272 bytes, 0 errors. Response shape unchanged (bare node array) so the MCP wrapper, tools/telegram-gateway.sh and cli/neuron_mcp.py keep working. Live :7770 never written to.

Known limits, stated

  1. The 12 sibling Values still don't surface. A ~58-day-dormant seed scores 0.7 × 0.05 decay floor × 0.34 dampen ≈ 0.012, below the runtime's 0.02 firing threshold, so it cannot propagate at all. That is tuning inside the vendored runtime, not this wiring. Do not fix it by lowering the threshold — el_runtime.c:5720 documents the invariant BREAKTHROUGH_WEIGHT < min(type_thresholds), and breaking it re-opens the 2026-08-02 WM-thrashing pathology (661-903 breakthroughs per 60s heartbeat).
  2. Recall is now a stateful read — it reinforces promoted nodes and folds the query into the context centroid. Design-fit per claim 29 (retrieval-as-access) and already true of every chat turn, but results warm and shift across repeated calls.
  3. engram_activate_json has no internal cap, so it materialises full node JSON for every reported node before projection. Bounded in practice by the collector floor; a runtime limit param would be the clean fix.

🤖 Generated with Claude Code

**Neuron's spreading-activation retrieval was fully built, compiled into the shipped binary, and never called by the two tools everything uses.** This wires it up. ## What was wrong `handle_api_recall` ended with `engram_search_json(...)` — a **single-token case-insensitive substring matcher**, scored by counting query tokens found in content/label/tags, sorted by coverage then raw salience. It never touches an edge. Meanwhile `engram_activate` / `engram_activate_json` are real BFS spreading activation over the graph, fully implemented, present in the shipped binary (`nm` confirms `T _engram_activate_json`) — and called from only four places, **none of them `recall` or `searchKnowledge`**. Measured consequence: `recall("inherited")` returned substring hits including a TCO evaluation framework and an unrelated whitepaper, and did **not** return the 12 sibling Value nodes one hop off the same hub. ## The design being restored Engram provisional 64/064,260, **claim 1**: *"no data is retrieved from the weighted directed graph except through the spreading activation traversal."* Scoring is four-factor and multiplicative — *"the multiplication of all four factors enforces a conjunctive property… addition would allow many weak associations to accumulate into false relevance."* This is a restoration, not an invention. ## Seeding `engram_activate` already takes **query text, not seed ids**, and seeds itself: lexically (any node matching ≥1 token, initial activation = `salience × temporal_decay × dampening × token_coverage`) then a semantic top-K cosine supplement. All four existing call sites pass text the same way, so this follows the runtime's own convention. Exact lookup is protected by a **seeded hybrid**, not a bolted-on union: activation's collector drops any reached node below `bg × confidence < 0.1` unless promoted to working memory, so a rare token on a dormant node can seed the traversal and go unreported. The lexical seed list is appended *after* the activated ranking, deduped, until `limit` fills — the same seed set the traversal already computed, restored to the tail. ## Measured, two builds of this tree, same 79,250-node graph, cold start | query | main@18714e6 | this branch | |---|---|---| | `inherited` | 10 results — `ntn-genesis`, `1`, a Disneyland fragment. Value node rank **23**, hub rank **32** | Value node rank **3**, `Self — Values` hub rank **4** | | `Structure is not inherited` | Value rank **31**, plus a corrupt node | Value rank **16**; returns voice/imprint/VBD material | | `volatility-based decomposition` | 1 of 10 relevant | **6 of 10** relevant | | `searchKnowledge`, same query | 2 of 5 relevant | **5 of 5** | | `HNSW`, `Fayetteville` (rare exact) | 1, 1 | **1, 1 — no regression** | | nonsense control | 0 | **0** | `searchKnowledge` got the same treatment because its "activate fallback" was **unreachable dead code** — it fired only if `engram_search_json` returned a string not starting with `[` or `{`, which it never does. **Correction to an earlier claim of ours:** multi-word queries returning *zero* is true of the live soul binary but **not** of `main` — main's matcher is already tokenized, so it returns 10 results, just poor ones. The defect on main is relevance, not emptiness. The live soul is running an older build. ## Rung **E2E-VERIFIED.** Amalgam 1,164,103 → 1,167,337 bytes (1226 → 1229 bodies); binaries 903,096 and 903,272 bytes, 0 errors. Response shape unchanged (bare node array) so the MCP wrapper, `tools/telegram-gateway.sh` and `cli/neuron_mcp.py` keep working. Live `:7770` never written to. ## Known limits, stated 1. **The 12 sibling Values still don't surface.** A ~58-day-dormant seed scores `0.7 × 0.05 decay floor × 0.34 dampen ≈ 0.012`, below the runtime's `0.02` firing threshold, so it cannot propagate at all. That is tuning inside the vendored runtime, not this wiring. **Do not fix it by lowering the threshold** — `el_runtime.c:5720` documents the invariant `BREAKTHROUGH_WEIGHT < min(type_thresholds)`, and breaking it re-opens the 2026-08-02 WM-thrashing pathology (661-903 breakthroughs per 60s heartbeat). 2. Recall is now a **stateful read** — it reinforces promoted nodes and folds the query into the context centroid. Design-fit per claim 29 (retrieval-as-access) and already true of every chat turn, but results warm and shift across repeated calls. 3. `engram_activate_json` has no internal cap, so it materialises full node JSON for every reported node before projection. Bounded in practice by the collector floor; a runtime `limit` param would be the clean fix. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
tim.lingo added 1 commit 2026-08-07 18:24:50 +00:00
feat(recall): retrieve through spreading activation, not substring matching
Neuron Soul CI / build (pull_request) Failing after 14m29s
Details
Neuron Soul CI / deploy (pull_request) Has been skipped
Details
bca7d8ac99 
recall and searchKnowledge both ended at engram_search_json — a case-
insensitive substring matcher scored by how many distinct query tokens appear
in a node's content/label/tags, tie-broken by raw salience. It never read a
single edge. Meanwhile engram_activate / engram_activate_json — real BFS
spreading activation over the weighted directed graph, four-factor
multiplicative scoring, two-layer background/working-memory filter — has been
implemented and compiled into the shipped runtime the whole time, called from
four places, none of them retrieval.

This wires retrieval to the traversal, restoring the designed mechanism:
Engram provisional 64/064,260 claim 1, "no data is retrieved from the weighted
directed graph except through the spreading activation traversal."

Seeding follows the runtime's own convention (all four existing call sites pass
query TEXT, not seed ids): engram_activate seeds lexically — every node
matching >=1 query token, initial activation = salience x temporal_decay x
dampening x token_coverage — then supplements with the top-K nodes by cosine
against the query embedding. So the lexical surface recall used to RETURN is
now the SEED SET of the traversal, and what comes back is what those seeds
activate.

Exact lookup is not regressed. engram_activate's collector drops any reached
node whose background_activation x confidence < 0.1 unless it was promoted to
working memory, so a rare token on a dormant node can seed and still go
unreported. Retrieval therefore appends the lexical seed list after the
activated ranking, deduped by id, until `limit` is filled — the same seed set
the traversal already computed, restored to the tail, not a parallel search.

searchKnowledge gets the identical path. Its existing "activate fallback" was
unreachable dead code: it fired only when engram_search_json's return did not
start with '[' or '{', and that function always emits a '['-prefixed array.

Response shape is unchanged — a bare array of full engram node objects, so the
MCP wrapper, tools/telegram-gateway.sh (.value.content) and cli/neuron_mcp.py
keep working. Activation strength is a ranking input here, not a payload change.

Measured, cold-start, two builds of this tree against the same 79,250-node /
14,214-edge graph (main @ 18714e6 vs this branch):

  "volatility-based decomposition"  before: 1 of 10 results relevant
                                    after:  6 of 10, incl. architecture/styles/
                                    vbd/glossary.md and project-design
                                    foundations
  "Structure is not inherited"      before: persona boilerplate, "1", a
                                    Disneyland fragment, a corrupted node
                                    after:  self/voice registers, neuron/
                                    user-imprint/boundary-definition,
                                    diagrams/vbd.md
  "inherited"                       Value - Structure Is Not Inherited:
                                    rank 23 -> rank 3; Self - Values hub:
                                    rank 32 -> rank 4
  searchKnowledge, same query       2 of 5 relevant -> 5 of 5
  "HNSW" / "Fayetteville" (rare)    1 result both builds - no regression
  nonsense control                  0 results both builds

Known limit, unchanged by this commit: the 12 sibling Value nodes still do not
surface. A ~58-day-dormant seed's activation (0.7 salience x 0.05 decay floor x
0.34 dampening ~= 0.012) lands below the runtime's 0.02 firing threshold, so it
cannot propagate to its neighbours at all. That is runtime tuning inside the
vendored el_runtime.c, not the wiring.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tim.lingo commented 2026-08-08 00:36:24 +00:00
Author
Member
Copy Link
Copy Source

Measured against this branch, and the result argues against merging it as a retrieval fix.

The harness ran main versus this branch on 38 gold queries over a 78,768-node copy of the live graph, two runs each:

  • net -5 queries (5 discordant, all in the losing direction)
  • McNemar exact p = 0.0625, against a measured noise floor of 6 queries
  • compare.py verdict: no measurable difference
  • hit@5 unchanged at 34.3%; associative stayed 0/6

Full data: tools/retrieval-eval/comparison-main-vs-recall-through-activation.json, on the stack branch in #141.

Why it does not help, measured rather than argued: claim 1's traversal cannot reach the answers, because 95.8% of nodes have zero edges and the last edge was created 2026-07-19. Routing recall through activation on a graph with almost no edges routes it through nothing. The gold associative targets sit at global cosine ranks 9,966 / 30,473 / 12,854 / 17,371 / 966 - semantics is anti-correlated there too. It is an edge problem.

What did work is in #141: a semantic leg independent of traversal, which is engram claim 24 (06-claims.md l.148) rather than claim 1 - 18.5% -> 58.5% hit@5, and associative off zero for the first time once the graph leg is seeded semantically.

This is not a request to close #135. The diagnosis in it stands and the dead-code finding it rests on is real. It is a request that it not be merged as a retrieval fix on the strength of the reasoning alone, because the reasoning was tested and did not hold.

**Measured against this branch, and the result argues against merging it as a retrieval fix.** The harness ran `main` versus this branch on 38 gold queries over a 78,768-node copy of the live graph, two runs each: - **net -5 queries** (5 discordant, all in the losing direction) - McNemar exact p = 0.0625, against a measured noise floor of 6 queries - `compare.py` verdict: **no measurable difference** - hit@5 unchanged at 34.3%; associative stayed 0/6 Full data: `tools/retrieval-eval/comparison-main-vs-recall-through-activation.json`, on the stack branch in #141. **Why it does not help, measured rather than argued:** claim 1's traversal cannot reach the answers, because 95.8% of nodes have zero edges and the last edge was created 2026-07-19. Routing recall through activation on a graph with almost no edges routes it through nothing. The gold associative targets sit at global cosine ranks 9,966 / 30,473 / 12,854 / 17,371 / 966 - semantics is anti-correlated there too. It is an edge problem. **What did work** is in #141: a semantic leg *independent of* traversal, which is engram claim 24 (`06-claims.md` l.148) rather than claim 1 - 18.5% -> 58.5% hit@5, and associative off zero for the first time once the graph leg is seeded semantically. This is not a request to close #135. The diagnosis in it stands and the dead-code finding it rests on is real. It is a request that it not be merged as a retrieval *fix* on the strength of the reasoning alone, because the reasoning was tested and did not hold.
tim.lingo closed this pull request 2026-08-09 16:46:27 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
BETA-CRITICAL
On the beta critical path — must resolve before ship
 blocks-public-beta
Must be fixed before any public beta
 HELD
Blocked pending an external decision (counsel / dual sign-off)
 ORTHOGONAL
Parallel workstream (testing/hardening) — safe to assign off critical path
 P0
Drop-everything severity
 POST-BETA
Important but not blocking the beta ship
 security
Security vulnerability
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
andre.almeida (Andre Almeida) tim.lingo (Tim Lingo) will.anderson (Will Anderson)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/neuron#135
Delete Branch "feat/recall-through-activation"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?