Self-load fix: relevance-ranked graph projection + architecture docs (clean re-merge) #155

2026-08-15T23:40:47Z

will.anderson commented

2026-08-15 23:40:47 +00:00

Re-does PR #149's merge clean off current main. Real conflicts resolved:

chat.elh, routes.el, mcp-wrapper/src/main.el: main had evolved past #149's base on the same code paths (operator_identity_block decl, the read-route-clobbering-snapshot fix, graph_bound_params/compact_flag) — kept main's newer versions in each case, #149 didn't add anything not already superseded.
dist/soul.c: left as-is (stale relative to these merged .el sources) rather than hand-regenerating — the real regen tooling is PR #154, landing next, which will do one correct regen incorporating everything instead of two fragile ones.

Re-does PR #149's merge clean off current main. Real conflicts resolved: - chat.elh, routes.el, mcp-wrapper/src/main.el: main had evolved past #149's base on the same code paths (operator_identity_block decl, the read-route-clobbering-snapshot fix, graph_bound_params/compact_flag) — kept main's newer versions in each case, #149 didn't add anything not already superseded. - dist/soul.c: left as-is (stale relative to these merged .el sources) rather than hand-regenerating — the real regen tooling is PR #154, landing next, which will do one correct regen incorporating everything instead of two fragile ones.

will.anderson added 9 commits 2026-08-15 23:40:48 +00:00

self-review 2026-08-02: record that el_from_float on a literal is not the double-wrap bug e60ca8123b

Investigated awareness.el ise_post's local-fallback engram_node_full call as a
suspected instance of the score-mangling double-wrap fixed in server.el on
2026-08-01. It is not one. Removing the wrapper produces byte-identical
codegen: the compiler treats el_from_float as the boxing intrinsic, so both
`el_from_float(0.3)` and a bare `0.3` emit exactly one el_from_float(0.3).

The server.el bug was different in kind - there the arguments came from
json_get_float(), already boxed as el_val_t, and wrapping those a second time
reinterprets the boxed bits as a raw double, fails engram_decode_score's range
check, and silently clamps to defaults.

Comment only, no behavior change. Recording the negative result at the call
site so the sweep criterion is right: look for el_from_float applied to an
already-boxed expression, never to a literal. Grepping the call name alone
produces false positives, which is what happened here.

self-review 2026-08-03: gate curiosity auto-terms on label document frequency 21710d5c8e

Reject an extracted auto-term when its label document frequency exceeds
node_count/400 (floor 8) -- measured live at 12,859 nodes, threshold 32.

Live label df separates the classes by an order of magnitude:
  rejected: <!--:220  SELF:175  Engram:125  CORE:88  STAR:36
  passed:   Dual:12  Sparse:8  Latent:6  MemQ:1  dGRPO:1  engram_goal_bias:1

Verified against the running soul (boot 21). Peak curiosity activation fell
from 541 to 113; the flood terms (SELF, CORE, Engram, STAR, <!--) are absent
from post-fix scans while topical compound identifiers pass untouched.
Sample is 7 scans -- suggestive, not conclusive; watch the next review.

Nested conditional rather than max(): El let is single-assignment, so the
floor is expressed as a second conjunct.

Verification note: content df was tested as an alternative signal and
rejected -- 'Curiosity' has the highest content df in the store (5526) yet
one of the lowest activation counts (113). Label df is the correct field
because label is what the first-word extractor reads.

self-review 2026-08-05: heartbeat carries redundancy-suppression gauges a771ed2d0f

dup_seeds / dup_wm from engram_act_stats_json. dup_seeds is the one that
matters day to day: a healthy nonzero rate means the suppressor is reclaiming
seed slots the June duplicate import was stealing; a sustained fall toward zero
means the duplicates were finally merged out of the graph, which is the repair
this defends against. Cumulative like wm_evicted/breakthroughs.

self-review 2026-08-06: a streak of nothing is not a streak; surface hebb gauges 97d22ffe44

str_eq("", "") is true, so an auto-term extractor that kept FAILING reported a
rising auto_term_streak. The signal meaning "fixated on one term" and the
signal meaning "producing no term at all" were the same number — opposite
failures needing opposite responses. Observed live as
{"auto_term":"","auto_term_streak":3}. Same class of bug already fixed for
wm_top0_streak on 2026-07-31; auto_term was missed then.

Empty now reads 0, and the empty run is counted on its own axis
(auto_term_empty_streak) so extractor failure is visible rather than disguised
as health.

Also surfaces the new runtime gauges in the heartbeat: hebb_warm, hebb_max,
hebb_links (is the graph learning any structure at all?) and dup_wm_global.

self-review 2026-08-07: push what was learned; stop a read route writing the canonical store 86e269fa91

Two fixes, one found by making the other.

1. HEBBIAN WRITE-BACK. This daemon learned 1,198 associations in 23h48m and
kept none of them: it syncs FROM the engram server and never pushes, and
mem_save() is unreachable in HTTP mode by design (soul.el only sets
soul_snapshot_path inside `is_genesis && safe_to_seed`, false whenever
ENGRAM_URL is set, because the server owns persistence). So the one process
that runs idle cognition -- where essentially all co-activation happens -- was
the one process that could not remember what it learned.

hebb_consolidate() now drains the runtime's write-back queue on every heartbeat
and POSTs it as ONE batch to /api/edges/batch. One request, one durable write,
not one 60MB snapshot per edge. Also drains on clean shutdown, so an exit
between beats doesn't take the last 8 minutes of learning with it.

_auth is required and its absence is silent: check_auth_ok exempts GET and
/api/neuron/state-events (which is why ise_post works keyless) but gates every
other mutation on "_auth" in the BODY -- http_serve surfaces no headers, so
there is no Bearer path. An unauthorized reply is NON-EMPTY, so the obvious
`if resp == "" return 0` check would have reported delivery of edges that were
refused, after the drain had already destroyed them. Caught before it shipped.
Gauges hebb_wb_pending/_drained/_dropped/_sent go into the heartbeat so a
consolidation path that stops delivering is visible in the stream.

2. A READ ROUTE MUST NEVER WRITE THE CANONICAL SNAPSHOT. GET /api/graph/edges
serialized this process's graph straight over $HOME/.neuron/engram/snapshot.json
-- the engram SERVER's durable store -- and read the edges back out of it. I
triggered it myself this morning fetching edges for the census above:
snapshot.json went from the server's 41,213 edges to the soul's 42,431, and the
next engram restart loaded the soul's graph as canonical. It happened to be a
superset (Knowledge 1198->1218, Memory 1238->1242, no durable type down), so
nothing was lost. That was luck. Had the soul been running a partial load --
the exact failure soul.el's safe_to_seed guard exists to catch -- one GET would
have destroyed the store, with no write-side guard able to see it coming.

The engram server fixed this same class of bug on 2026-07-21 by routing exports
to a dotted sidecar; the soul kept the original pattern. Same fix: exports go to
.soul-edges-export.json. Also stops a 60MB serialize-and-reread per GET.

Verified: boot 26 loaded 42,432 edges with hebb_max 0.4941 carried across the
restart -- the first time this daemon has ever started knowing what it learned.

self-review 2026-08-08: put text integrity on the heartbeat bc5e14a3e1

Today's review found the JSON parser had been replacing every \uXXXX escape
with a literal '?' for at least two months: 3,119 of 4,081 non-telemetry
nodes damaged, including the self traversal root and all 13 values nodes.
The parser is fixed in el_runtime.c. This is the part that keeps it fixed.

Every gauge on this heartbeat answers whether the machinery is running - WM
occupancy, Hebbian potentiation, embedding coverage, sync age, breaker state.
None answered whether the text the machinery carries is intact, which is why
two months of silent corruption read as a perfectly healthy system.

Adds txt_damaged: nodes created this process whose content carries the
character-loss signature. Flat 0 is healthy; any climb means a write path is
mangling text again. The full store census lives at GET /api/text-health,
which is too expensive for a 60s beat.

A gauge that exists but is not plumbed into the beat is not observability -
act_stats fields have to be extracted by name here or they are invisible.

self-review 2026-08-10: plumb the gauges that were computed and discarded 64cd5055c5

el_runtime.c emitted 19 metric keys from engram_metrics_json; emit_heartbeat
forwarded 14. Dropped on the floor: hebb_cands, hebb_cand_max, hebb_mass,
hebb_edges, embed_consec_fail. The first two are exactly the pair the runtime
added to answer the 08-06 question -- whether a stalled hebb_links means
nothing co-activates or the threshold is too high. Undiagnosable from the
durable record without them. An instrument computed but not plumbed to
durable storage is not an instrument.

Also adds the corpus damage STOCK, not just the flow. 08-08 fixed the JSON
parser, watched txt_damaged (nodes damaged by a write THIS process) fall to
0, and recorded the defect closed. Census today: 2781 of 4100 nodes still
damaged -- 67.8%, including the self root and every values node. A flow gauge
reads 0 both when the corpus is clean and when it is uniformly damaged but
quiescent. Sampled on a 30-beat countdown and carried with an explicit age.

fix(api): bound inspect_graph with relevance-ranked projection; regen soul.c

Neuron Soul CI / build (pull_request) Failing after 14m5s

Details

Neuron Soul CI / deploy (pull_request) Has been skipped

Details

4bff40fa4a

High-fanout identity anchors (voice, writing-imprint, self-root) have ~670KB
neighborhoods. inspect_graph returned the full traversal, which overflowed the
MCP client's context and socket-closed the wrapper mid self-load -- the soul
could not traverse its own identity graph.

handle_api_inspect_graph gains an opt-in `compact` projection (compact=1|true):
the neighborhood is relevance-ranked, the top K (default 12) keep a UTF-8-safe
content snippet (default snip=600), and the remainder collapse to lightweight
{id,label,node_type,tier,edge,pointer:true} stubs. This bounds the voice node
from 669,799B -> 25,353B (HTTP 200, valid JSON) and the wrapper's soul-load no
longer socket-closes. New helpers: api_compact_neighbors, api_neigh_full,
api_neigh_pointer, api_neigh_rank, api_neigh_better, api_float_or.

The flag is gated: ABSENT it, the response is byte-identical to the old plain
traversal, so the studio app (which never sends it) is unaffected. The MCP
wrapper (mcp-wrapper/src/main.el) appends &compact=1 on its inspectGraph and
fetch-by-id paths.

dist/soul.c is REGENERATED so CI ships the fix: CI compiles the committed
single-TU dist/soul.c directly (running elb/elc on the Linux runner OOM-kills
it), so an .el-only change would build the OLD behavior. Regenerated and verified
on macOS -- compiles with the CI cc line (0 errors) and, on a throwaway soul over
a copy of the live snapshot, serves compact ~25KB / non-compact ~670KB. The regen
also syncs the amalgamation to this branch's .el sources, which had drifted
several self-review commits ahead of the previously-committed soul.c.

Docs: docs/architecture/00-05 added; 01/02/05 corrected so the relevance-ranked
inspect_graph projection reads as committed source, not an in-flight concern.

Merge remote-tracking branch 'origin/pr/149' into HEAD

Neuron Soul CI / build (pull_request) Failing after 13m22s

Details

Neuron Soul CI / deploy (pull_request) Has been skipped

Details

c3762ec352

# Conflicts:
#	chat.elh
#	dist/soul.c
#	mcp-wrapper/src/main.el
#	routes.el

will.anderson merged commit 8355426526 into main

2026-08-15 23:41:00 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/neuron#155