Compare commits

...

5 Commits

Author SHA1 Message Date
will.anderson b6ed9340bf soul: the engram is the canonical store, not a fallback
Neuron Soul CI / build (pull_request) Failing after 4m20s
Neuron Soul CI / deploy (pull_request) Has been skipped
The soul preferred its own local snapshot over the engram:

    // Always try local snapshot first ... HTTP Engram is only used for the
    // very first boot (empty/absent snapshot).

The copy outranked the store. Every one of these is a cost of that inversion,
and all of them were live tonight:

  - the graphs drifted: 31,795 nodes / 75,241 edges in the soul against
    13,439 / 37,670 in the engram — more than twice the edges, silently
  - write-through exists only to reconcile them, and had never once run
  - /api/graph/edges serialized 128 MB to answer a read, because the soul's
    copy was not the engram's
  - a read route overwrote the engram's canonical snapshot.json with the
    soul's divergent copy
  - three resident copies of one graph (soul, engram, Neuron.app) — about
    7.2 GB of RAM for a store that is 2.2 GB on disk, which is what pushed the
    host into swap

None of those are features. They are reconciliation debt from one decision.

The engram had already reached this conclusion for its own boot path — "the
durable owner is the paged store (neuron.egm + neuron.wal) ... snapshot.json is
never read again as the ongoing store. This closes the 'restart reverted to a
17h-old snapshot' data-loss window." The soul kept booting the legacy way the
engram had abandoned, and inherited exactly that data-loss window.

So in HTTP-engram mode the soul now seeds from the engram on EVERY boot and
never reads a local snapshot, present or not — a stale copy that outranks the
store is the bug, not a fallback. It already never wrote one in this mode
(gated behind is_genesis && safe_to_seed, and safe_to_seed requires
!using_http_engram), so this supplies the missing half.

It also refuses to boot on an empty seed rather than silently rebuilding a
divergent graph from nothing. launchd KeepAlive with ThrottleInterval=10 turns
that into a retry every 10s until the engram is up — self-healing, no spin.

File mode (no ENGRAM_URL) is untouched: there the soul genuinely is the owner.

Verified before deploy: with a deliberately empty local snapshot planted, the
soul booted in ~30s reporting 13,446 nodes / 37,675 edges — the engram's
contents, not the empty local file.
2026-08-15 21:16:22 -05:00
will.anderson 97bf91739e Merge pull request 'fix(routes): /api/graph/edges must not write the canonical snapshot' (#161) from fix/graph-edges-no-canonical-clobber into main
Neuron Soul CI / build (push) Failing after 14m2s
Neuron Soul CI / deploy (push) Has been skipped
2026-08-16 01:44:57 +00:00
will.anderson cf154387ce fix(routes): /api/graph/edges must not write the canonical snapshot
Neuron Soul CI / build (pull_request) Failing after 13m44s
Neuron Soul CI / deploy (pull_request) Has been skipped
This route called engram_save() over ~/.neuron/engram/snapshot.json — the
engram server's CANONICAL store — then fs_read it back, to answer a READ
query. A read route overwriting the persistence owner's file.

This defect was fixed once before (export redirected to a scratch path). It
came back tonight in the @route dispatch conversion: the hand-written dispatch
block held the FIXED version, the @route-decorated copy held the unfixed one,
and the merge kept the decorated copy. Calling the endpoint afterward
overwrote the canonical snapshot and immediately preceded an engram crash.

Now calls engram_edges_json(limit, offset) — the builtin the route's own TODO
asked for — which reads g->edges directly. No file is written or read.
Bounded: limit defaults to 1000, offset supported, so the whole-graph read
that fell over is not reachable by default.

Verified: same request that previously rewrote snapshot.json now leaves it
byte-identical (sha256 unchanged before/after), and returns real edge records
with every persisted field.
2026-08-15 20:13:46 -05:00
will.anderson cfdf312cb3 Merge pull request 'docs(architecture): record the 2026-08-14 deep-night sessions' (#160) from docs/architecture-2026-08-14-deep-night into main
Neuron Soul CI / build (push) Failing after 4m4s
Neuron Soul CI / deploy (push) Has been skipped
2026-08-16 00:40:36 +00:00
will.anderson e8b1af83fd Merge pull request 'fix(mcp-wrapper): route agentic ops to the engram, stop fabricating a cause' (#159) from fix/mcp-wrapper-agentic-routing into main
Neuron Soul CI / build (push) Has been cancelled
Neuron Soul CI / deploy (push) Has been cancelled
2026-08-16 00:37:13 +00:00
2 changed files with 60 additions and 21 deletions
+17 -8
View File
@@ -420,14 +420,23 @@ fn r_api_graph_nodes(method: String, path: String, body: String) -> String {
@route("/api/graph/edges", "GET", "exact") @manager
fn r_api_graph_edges(method: String, path: String, body: String) -> String {
// TODO(reliability #8): engram_save races with awareness loop mem_save().
// Both now use atomic write-to-temp+rename (el_runtime.c). Serialised
// by engram_global_mu. Future: add engram_edges_json() builtin.
let snap_path: String = env("HOME") + "/.neuron/engram/snapshot.json"
engram_save(snap_path)
let snap: String = fs_read(snap_path)
let edges_raw: String = json_get_raw(snap, "edges")
return if str_eq(edges_raw, "") { "[]" } else { edges_raw }
// Reads edges straight from the store. No file is written or read.
//
// This route used to engram_save() the ENTIRE graph over
// ~/.neuron/engram/snapshot.json the engram server's CANONICAL store
// and then fs_read it back, just to answer a read query. Two defects in
// one line: a READ route clobbering the persistence owner's canonical
// file (the defect fixed once already, then reintroduced when the
// hand-written dispatch block was replaced by @route dispatch and the
// unfixed copy is the one that survived), and a 128 MB serialize +
// reread + parse per request. Calling it on 2026-08-15 overwrote the
// canonical snapshot and preceded an engram crash loop.
//
// engram_edges_json is the builtin the old TODO here asked for. Bounded
// by default (1000) the unbounded whole-graph read is what fell over.
let lim: Int = api_query_int(path, "limit", 1000)
let off: Int = api_query_int(path, "offset", 0)
return engram_edges_json(lim, off)
}
// GET /api/chat legacy probe interface; body may be empty ───────────────
+43 -13
View File
@@ -584,17 +584,35 @@ println("[soul] boot - cgi=" + soul_cgi_id + " port=" + int_to_str(port))
let using_http_engram: Bool = !str_eq(engram_url_raw, "")
// Always try local snapshot first. If it has content (>50 nodes) it was
// previously seeded from HTTP Engram and is kept up-to-date by the awareness
// loop use it. This preserves sessions and memories across restarts.
// HTTP Engram is only used for the very first boot (empty/absent snapshot).
engram_load(snapshot)
let local_node_count: Int = engram_node_count()
let snapshot_usable: Bool = local_node_count > 50
if using_http_engram && !snapshot_usable {
// First boot or empty/corrupt snapshot: seed from HTTP Engram.
println("[soul] engram -> HTTP " + engram_url_raw + " (no local snapshot, first boot)")
// THE ENGRAM IS THE CANONICAL STORE. The soul's resident graph is a working
// copy of it, never a rival source of truth.
//
// This used to be inverted: "always try local snapshot first... HTTP Engram is
// only used for the very first boot." The copy outranked the store. Everything
// that followed is a cost of that one inversion:
// - the two graphs drifted (13,479 nodes/74,563 edges in the soul vs
// 13,425/37,658 in the engram nearly 2x the edges, silently)
// - write-through exists solely to reconcile them, and had never once run
// - /api/graph/edges serialized 128 MB to answer a read, because the soul's
// copy was not the engram's
// - a read route overwrote the engram's canonical snapshot.json with the
// soul's divergent copy
// - three copies of the same memory: neuron.egm, snapshot.json, soul RAM
// None of those are features. They are all reconciliation debt.
//
// The engram itself already reached this conclusion for its own boot path:
// "the durable owner is the paged store (neuron.egm + neuron.wal) ...
// snapshot.json is never read again as the ongoing store. This closes the
// 'restart reverted to a 17h-old snapshot' data-loss window." The soul kept
// booting the legacy way the engram had already abandoned, and inherited
// exactly the data-loss window that comment describes.
//
// So in HTTP-engram mode the soul seeds from the engram, EVERY boot, and never
// consults or writes a local snapshot. The local file is not read even when
// present a stale copy that outranks the store is the bug, not a fallback.
// (File mode, no ENGRAM_URL, is unchanged: there the soul IS the owner.)
if using_http_engram {
println("[soul] engram -> HTTP " + engram_url_raw + " (canonical store; local snapshot ignored)")
let nodes_json: String = http_get(engram_url_raw + "/api/nodes?limit=10000")
let edges_json: String = http_get(engram_url_raw + "/api/edges")
let nodes_part: String = if str_eq(nodes_json, "") { "[]" } else { nodes_json }
@@ -603,9 +621,21 @@ if using_http_engram && !snapshot_usable {
let tmp_path: String = "/tmp/soul-engram-" + soul_cgi_id + ".json"
fs_write(tmp_path, snapshot_data)
engram_load(tmp_path)
println("[soul] loaded from HTTP Engram - nodes=" + int_to_str(engram_node_count()) + " edges=" + int_to_str(engram_edge_count()))
let seeded: Int = engram_node_count()
if seeded < 50 {
// Refuse to run blind. An empty seed in HTTP mode means the canonical
// store was unreachable or empty; continuing would let the soul rebuild
// a divergent graph from nothing, which is how the copies split before.
println("[soul] FATAL: engram at " + engram_url_raw + " returned " + int_to_str(seeded)
+ " nodes. The canonical store is unreachable or empty; refusing to boot on a"
+ " local copy. Fix the engram, then restart.")
exit_program(1)
}
println("[soul] loaded from engram - nodes=" + int_to_str(seeded) + " edges=" + int_to_str(engram_edge_count()))
} else {
println("[soul] loaded from local snapshot - nodes=" + int_to_str(local_node_count) + " edges=" + int_to_str(engram_edge_count()))
engram_load(snapshot)
println("[soul] file mode (no ENGRAM_URL) - soul owns the store - nodes="
+ int_to_str(engram_node_count()) + " edges=" + int_to_str(engram_edge_count()))
}
load_identity_context()