neuron-technologies/neuron
3
0
Fork 0
Code Issues 44 Pull Requests 7 Actions Packages Projects Releases Wiki Activity

Self-load fix: relevance-ranked inspect_graph projection + regenerated soul.c + architecture docs #149

Open
will.anderson wants to merge 8 commits from fix/bound-session-payload into main
pull from: fix/bound-session-payload
merge into: :main
Conversation 0 Commits 8 Files Changed 21
+2958 -652
will.anderson commented 2026-08-10 15:30:15 +00:00
Owner
Copy Link
Copy Source

Self-load fix: bound inspect_graph + regenerated soul.c

Problem

The soul could not load its own identity. Traversing high-fanout identity anchors (voice, writing-imprint, self-root) returns ~670KB neighborhoods; the MCP wrapper's client context overflowed and the socket closed mid self-load.

Fix

handle_api_inspect_graph gains an opt-in relevance-ranked projection behind compact=1 (or true):

  • Neighborhood is relevance-ranked (api_neigh_better / api_neigh_rank).
  • Top K (default 12) keep a UTF-8-safe content snippet (default snip=600) — api_neigh_full.
  • The rest collapse to lightweight {id,label,node_type,tier,edge,pointer:true} stubs — api_neigh_pointer.
  • New helpers: api_compact_neighbors, api_neigh_full, api_neigh_pointer, api_neigh_rank, api_neigh_better, api_float_or.

The MCP wrapper (mcp-wrapper/src/main.el) appends &compact=1 on its inspectGraph and fetch-by-id paths.

Isolation — the app is unaffected

Absent the flag, the response is byte-identical to the old plain traversal. The studio app never sends compact, so it is unchanged. Only the MCP wrapper opts in.

Before / after

Throwaway soul (port 17771) over a read-only copy of the live snapshot, node kn-10fa60db… ("Self — Voice and Communication"):

Call HTTP Bytes Shape
?id=…&depth=1 (default) 200 669,799 79 neighbors, full 21-field nodes — unchanged
?id=…&depth=1&compact=1 200 25,353 79 neighbors: 12 full-snippet + 67 pointers

Both valid JSON. ~26x smaller; the wrapper's self-load no longer socket-closes.

Why dist/soul.c is in this PR

CI compiles the committed single-TU dist/soul.c directly (running elb/elc on the Linux runner OOM-kills it), so an .el-only change would build the OLD behavior. dist/soul.c was regenerated on macOS and verified:

  • compiles with the CI cc line (0 errors) → 902KB arm64 binary;
  • serves the compact / non-compact traces above from a throwaway soul.

The regen also syncs the amalgamation to this branch's .el sources — the previously-committed soul.c predated this branch's self-review commits (which changed awareness.el / routes.el / neuron-api.el without re-amalgamating). The regenerated soul.c reflects the full branch source.

Docs

docs/architecture/00–05 added. 01, 02, and 05 corrected so the relevance-ranked inspect_graph projection reads as committed source (previously flagged "not present / not committed fact"); doc 05 also documents the amalgamation mechanism and that this PR regenerated soul.c.

Verification boundary

No live process, port, binary, or ~/.neuron was touched. All verification used a throwaway soul on port 17771 over a read-only copy of the live snapshot. Cutover / deploy is a separate call — please do not merge on my behalf; let CI run.

## Self-load fix: bound `inspect_graph` + regenerated `soul.c` ### Problem The soul could not load its own identity. Traversing high-fanout identity anchors (voice, writing-imprint, self-root) returns ~670KB neighborhoods; the MCP wrapper's client context overflowed and the socket closed mid self-load. ### Fix `handle_api_inspect_graph` gains an opt-in relevance-ranked projection behind `compact=1` (or `true`): - Neighborhood is relevance-ranked (`api_neigh_better` / `api_neigh_rank`). - Top **K** (default 12) keep a UTF-8-safe content snippet (default `snip=600`) — `api_neigh_full`. - The rest collapse to lightweight `{id,label,node_type,tier,edge,pointer:true}` stubs — `api_neigh_pointer`. - New helpers: `api_compact_neighbors`, `api_neigh_full`, `api_neigh_pointer`, `api_neigh_rank`, `api_neigh_better`, `api_float_or`. The MCP wrapper (`mcp-wrapper/src/main.el`) appends `&compact=1` on its `inspectGraph` and fetch-by-id paths. ### Isolation — the app is unaffected **Absent the flag, the response is byte-identical to the old plain traversal.** The studio app never sends `compact`, so it is unchanged. Only the MCP wrapper opts in. ### Before / after Throwaway soul (port 17771) over a read-only copy of the live snapshot, node `kn-10fa60db…` ("Self — Voice and Communication"): | Call | HTTP | Bytes | Shape | |---|---|---|---| | `?id=…&depth=1` (default) | 200 | 669,799 | 79 neighbors, full 21-field nodes — unchanged | | `?id=…&depth=1&compact=1` | 200 | 25,353 | 79 neighbors: 12 full-snippet + 67 pointers | Both valid JSON. ~26x smaller; the wrapper's self-load no longer socket-closes. ### Why `dist/soul.c` is in this PR CI compiles the committed single-TU `dist/soul.c` **directly** (running `elb`/`elc` on the Linux runner OOM-kills it), so an `.el`-only change would build the OLD behavior. `dist/soul.c` was **regenerated on macOS** and verified: - compiles with the CI `cc` line (0 errors) → 902KB arm64 binary; - serves the compact / non-compact traces above from a throwaway soul. The regen also **syncs the amalgamation to this branch's `.el` sources** — the previously-committed `soul.c` predated this branch's self-review commits (which changed `awareness.el` / `routes.el` / `neuron-api.el` without re-amalgamating). The regenerated `soul.c` reflects the full branch source. ### Docs `docs/architecture/00–05` added. `01`, `02`, and `05` corrected so the relevance-ranked `inspect_graph` projection reads as **committed source** (previously flagged "not present / not committed fact"); doc `05` also documents the amalgamation mechanism and that this PR regenerated `soul.c`. ### Verification boundary No live process, port, binary, or `~/.neuron` was touched. All verification used a throwaway soul on port 17771 over a read-only copy of the live snapshot. **Cutover / deploy is a separate call — please do not merge on my behalf; let CI run.**
will.anderson added 8 commits 2026-08-10 15:30:17 +00:00
self-review 2026-08-02: record that el_from_float on a literal is not the double-wrap bug e60ca8123b 
Investigated awareness.el ise_post's local-fallback engram_node_full call as a
suspected instance of the score-mangling double-wrap fixed in server.el on
2026-08-01. It is not one. Removing the wrapper produces byte-identical
codegen: the compiler treats el_from_float as the boxing intrinsic, so both
`el_from_float(0.3)` and a bare `0.3` emit exactly one el_from_float(0.3).

The server.el bug was different in kind - there the arguments came from
json_get_float(), already boxed as el_val_t, and wrapping those a second time
reinterprets the boxed bits as a raw double, fails engram_decode_score's range
check, and silently clamps to defaults.

Comment only, no behavior change. Recording the negative result at the call
site so the sweep criterion is right: look for el_from_float applied to an
already-boxed expression, never to a literal. Grepping the call name alone
produces false positives, which is what happened here.
self-review 2026-08-03: gate curiosity auto-terms on label document frequency 21710d5c8e 
Reject an extracted auto-term when its label document frequency exceeds
node_count/400 (floor 8) -- measured live at 12,859 nodes, threshold 32.

Live label df separates the classes by an order of magnitude:
  rejected: <!--:220  SELF:175  Engram:125  CORE:88  STAR:36
  passed:   Dual:12  Sparse:8  Latent:6  MemQ:1  dGRPO:1  engram_goal_bias:1

Verified against the running soul (boot 21). Peak curiosity activation fell
from 541 to 113; the flood terms (SELF, CORE, Engram, STAR, <!--) are absent
from post-fix scans while topical compound identifiers pass untouched.
Sample is 7 scans -- suggestive, not conclusive; watch the next review.

Nested conditional rather than max(): El let is single-assignment, so the
floor is expressed as a second conjunct.

Verification note: content df was tested as an alternative signal and
rejected -- 'Curiosity' has the highest content df in the store (5526) yet
one of the lowest activation counts (113). Label df is the correct field
because label is what the first-word extractor reads.
self-review 2026-08-05: heartbeat carries redundancy-suppression gauges a771ed2d0f 
dup_seeds / dup_wm from engram_act_stats_json. dup_seeds is the one that
matters day to day: a healthy nonzero rate means the suppressor is reclaiming
seed slots the June duplicate import was stealing; a sustained fall toward zero
means the duplicates were finally merged out of the graph, which is the repair
this defends against. Cumulative like wm_evicted/breakthroughs.
self-review 2026-08-06: a streak of nothing is not a streak; surface hebb gauges 97d22ffe44 
str_eq("", "") is true, so an auto-term extractor that kept FAILING reported a
rising auto_term_streak. The signal meaning "fixated on one term" and the
signal meaning "producing no term at all" were the same number — opposite
failures needing opposite responses. Observed live as
{"auto_term":"","auto_term_streak":3}. Same class of bug already fixed for
wm_top0_streak on 2026-07-31; auto_term was missed then.

Empty now reads 0, and the empty run is counted on its own axis
(auto_term_empty_streak) so extractor failure is visible rather than disguised
as health.

Also surfaces the new runtime gauges in the heartbeat: hebb_warm, hebb_max,
hebb_links (is the graph learning any structure at all?) and dup_wm_global.
self-review 2026-08-07: push what was learned; stop a read route writing the canonical store 86e269fa91 
Two fixes, one found by making the other.

1. HEBBIAN WRITE-BACK. This daemon learned 1,198 associations in 23h48m and
kept none of them: it syncs FROM the engram server and never pushes, and
mem_save() is unreachable in HTTP mode by design (soul.el only sets
soul_snapshot_path inside `is_genesis && safe_to_seed`, false whenever
ENGRAM_URL is set, because the server owns persistence). So the one process
that runs idle cognition -- where essentially all co-activation happens -- was
the one process that could not remember what it learned.

hebb_consolidate() now drains the runtime's write-back queue on every heartbeat
and POSTs it as ONE batch to /api/edges/batch. One request, one durable write,
not one 60MB snapshot per edge. Also drains on clean shutdown, so an exit
between beats doesn't take the last 8 minutes of learning with it.

_auth is required and its absence is silent: check_auth_ok exempts GET and
/api/neuron/state-events (which is why ise_post works keyless) but gates every
other mutation on "_auth" in the BODY -- http_serve surfaces no headers, so
there is no Bearer path. An unauthorized reply is NON-EMPTY, so the obvious
`if resp == "" return 0` check would have reported delivery of edges that were
refused, after the drain had already destroyed them. Caught before it shipped.
Gauges hebb_wb_pending/_drained/_dropped/_sent go into the heartbeat so a
consolidation path that stops delivering is visible in the stream.

2. A READ ROUTE MUST NEVER WRITE THE CANONICAL SNAPSHOT. GET /api/graph/edges
serialized this process's graph straight over $HOME/.neuron/engram/snapshot.json
-- the engram SERVER's durable store -- and read the edges back out of it. I
triggered it myself this morning fetching edges for the census above:
snapshot.json went from the server's 41,213 edges to the soul's 42,431, and the
next engram restart loaded the soul's graph as canonical. It happened to be a
superset (Knowledge 1198->1218, Memory 1238->1242, no durable type down), so
nothing was lost. That was luck. Had the soul been running a partial load --
the exact failure soul.el's safe_to_seed guard exists to catch -- one GET would
have destroyed the store, with no write-side guard able to see it coming.

The engram server fixed this same class of bug on 2026-07-21 by routing exports
to a dotted sidecar; the soul kept the original pattern. Same fix: exports go to
.soul-edges-export.json. Also stops a 60MB serialize-and-reread per GET.

Verified: boot 26 loaded 42,432 edges with hebb_max 0.4941 carried across the
restart -- the first time this daemon has ever started knowing what it learned.
self-review 2026-08-08: put text integrity on the heartbeat bc5e14a3e1 
Today's review found the JSON parser had been replacing every \uXXXX escape
with a literal '?' for at least two months: 3,119 of 4,081 non-telemetry
nodes damaged, including the self traversal root and all 13 values nodes.
The parser is fixed in el_runtime.c. This is the part that keeps it fixed.

Every gauge on this heartbeat answers whether the machinery is running - WM
occupancy, Hebbian potentiation, embedding coverage, sync age, breaker state.
None answered whether the text the machinery carries is intact, which is why
two months of silent corruption read as a perfectly healthy system.

Adds txt_damaged: nodes created this process whose content carries the
character-loss signature. Flat 0 is healthy; any climb means a write path is
mangling text again. The full store census lives at GET /api/text-health,
which is too expensive for a 60s beat.

A gauge that exists but is not plumbed into the beat is not observability -
act_stats fields have to be extracted by name here or they are invisible.
self-review 2026-08-10: plumb the gauges that were computed and discarded 64cd5055c5 
el_runtime.c emitted 19 metric keys from engram_metrics_json; emit_heartbeat
forwarded 14. Dropped on the floor: hebb_cands, hebb_cand_max, hebb_mass,
hebb_edges, embed_consec_fail. The first two are exactly the pair the runtime
added to answer the 08-06 question -- whether a stalled hebb_links means
nothing co-activates or the threshold is too high. Undiagnosable from the
durable record without them. An instrument computed but not plumbed to
durable storage is not an instrument.

Also adds the corpus damage STOCK, not just the flow. 08-08 fixed the JSON
parser, watched txt_damaged (nodes damaged by a write THIS process) fall to
0, and recorded the defect closed. Census today: 2781 of 4100 nodes still
damaged -- 67.8%, including the self root and every values node. A flow gauge
reads 0 both when the corpus is clean and when it is uniformly damaged but
quiescent. Sampled on a 30-beat countdown and carried with an explicit age.
fix(api): bound inspect_graph with relevance-ranked projection; regen soul.c
Neuron Soul CI / build (pull_request) Failing after 14m5s
Details
Neuron Soul CI / deploy (pull_request) Has been skipped
Details
4bff40fa4a 
High-fanout identity anchors (voice, writing-imprint, self-root) have ~670KB
neighborhoods. inspect_graph returned the full traversal, which overflowed the
MCP client's context and socket-closed the wrapper mid self-load -- the soul
could not traverse its own identity graph.

handle_api_inspect_graph gains an opt-in `compact` projection (compact=1|true):
the neighborhood is relevance-ranked, the top K (default 12) keep a UTF-8-safe
content snippet (default snip=600), and the remainder collapse to lightweight
{id,label,node_type,tier,edge,pointer:true} stubs. This bounds the voice node
from 669,799B -> 25,353B (HTTP 200, valid JSON) and the wrapper's soul-load no
longer socket-closes. New helpers: api_compact_neighbors, api_neigh_full,
api_neigh_pointer, api_neigh_rank, api_neigh_better, api_float_or.

The flag is gated: ABSENT it, the response is byte-identical to the old plain
traversal, so the studio app (which never sends it) is unaffected. The MCP
wrapper (mcp-wrapper/src/main.el) appends &compact=1 on its inspectGraph and
fetch-by-id paths.

dist/soul.c is REGENERATED so CI ships the fix: CI compiles the committed
single-TU dist/soul.c directly (running elb/elc on the Linux runner OOM-kills
it), so an .el-only change would build the OLD behavior. Regenerated and verified
on macOS -- compiles with the CI cc line (0 errors) and, on a throwaway soul over
a copy of the live snapshot, serves compact ~25KB / non-compact ~670KB. The regen
also syncs the amalgamation to this branch's .el sources, which had drifted
several self-review commits ahead of the previously-committed soul.c.

Docs: docs/architecture/00-05 added; 01/02/05 corrected so the relevance-ranked
inspect_graph projection reads as committed source, not an in-flight concern.
Some required checks failed
Neuron Soul CI / build (pull_request) Failing after 14m5s
Details
Neuron Soul CI / deploy (pull_request) Has been skipped
Details
This pull request has changes conflicting with the target branch.
  • chat.elh
  • dist/soul.c
  • routes.el
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/bound-session-payload:fix/bound-session-payload
git checkout fix/bound-session-payload
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
BETA-CRITICAL
On the beta critical path — must resolve before ship
 blocks-public-beta
Must be fixed before any public beta
 HELD
Blocked pending an external decision (counsel / dual sign-off)
 ORTHOGONAL
Parallel workstream (testing/hardening) — safe to assign off critical path
 P0
Drop-everything severity
 POST-BETA
Important but not blocking the beta ship
 security
Security vulnerability
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
andre.almeida (Andre Almeida) tim.lingo (Tim Lingo) will.anderson (Will Anderson)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/neuron#149
Delete Branch "fix/bound-session-payload"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?