feat(soul): write-through to the persistence owner — memories survive restart (#117)
The soul obeys half of its own ownership rule. soul.el:571-573 says "when
ENGRAM_URL is set the HTTP Engram owns persistence — the soul must NEVER write
to the local snapshot", and it doesn't. But nothing was ever built to hand the
soul's writes TO that owner: sync is pull-only (/api/sync -> engram_load_merge),
so every node created inside the soul lived in process RAM and was shed on
restart. Measured live 2026-08-07: soul node_count=102184, engram 79197.
SCOPE CORRECTION vs the earlier internal spec: engram provisional claim 17's
"pull-then-push" is a PEER-ENGRAM to PEER-ENGRAM protocol (claims 15-18 say so
explicitly). The soul is a CALLER of the database API, not a peer. Claim 17 is
NOT authority for a soul<->engram contract and is no longer cited as such. The
design here follows from the ownership rule alone.
Mechanism: a new Accessor, persist.el, is the single boundary. Writes stage a
delta to a filesystem spool and are pushed to the owner via POST /api/load-merge
— NOT POST /api/nodes, which mints a new server-side id (breaking dedup and
edges) and drops label/tier/tags/importance/confidence (verified in a sandbox:
a tier "Canonical" probe came back "Working"). load-merge preserves the id and
every field, dedups nodes by id and edges by (from,to,relation) so retries are
no-ops, and calls persist_canonical() so THE OWNER writes its own file — the
ownership rule is honoured rather than worked around.
Spool-and-drain rather than push-per-write: measured ~0.38s per load-merge at
live scale (79k nodes/176MB), and a chat turn writes 5-7 nodes. The spool is on
disk, not in process state, because the soul serves each connection on its own
pthread and a shared buffer would lose entries to a read-modify-write race. That
also buys crash recovery: writes orphaned by kill -9 are drained on next boot.
Honesty: api_persisted (the gate all 10 MCP write handlers pass through) and
mem_store now assert AT THE OWNER instead of reading back the soul's own RAM.
With the owner down a write returns {"ok":false,"error":"write_not_persisted"}
and the delta is queued — where main returns {"ok":true} for a write that dies.
Coverage: 35 node sites + 9 edge sites routed through the boundary. Deliberately
excluded, with reasons in persist.el: 4 InternalStateEvent sites (Will's own
telemetry carve-out), the boot counter and the persona (both already have
bespoke owner-side write-backs), and soul.el's 54 genesis identity edges
(file-mode only). engram_strengthen and engram_forget are NOT propagated —
load-merge cannot update or delete, and hard-deleting at the owner would fail
verify-soul-contract.sh section B.
Also fixed here:
- routes.el GET /api/graph/edges engram_save()'d straight over the owner's
canonical snapshot.json — a read route, in a non-owner process, clobbering the
canonical on every call. Same defect class Will removed from the engram in el
dc39a61. Now exports to a scratch path. With this gone the soul writes nothing
at all in HTTP mode.
- persist.el must clear the runtime's _tl_fs_read_len hint after every fs_read.
In vendored runtime v1.0.0-20260501 that hint becomes the NEXT response's
Content-Length, so reading a spool file mid-request made an 86-byte reply go
out as 497 bytes with 411 bytes of adjacent heap trailing it. Caught and fixed
at our boundary; the runtime class was fixed upstream in el 43636ae, which is
not the pinned runtime here.
Rung: E2E-VERIFIED, discriminating. Same harness, same engram binary:
write-through: LEG 1 PRESENT at owner, LEG 2 SURVIVED kill -9 + restart
main: LEG 1 ABSENT at owner, LEG 2 LOST
verify-soul-contract.sh: GATE PASS on both builds (27/27 routes, immutability).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -186,7 +186,7 @@ fn route_imprint_contextual(body: String) -> String {
|
||||
return "{\"ok\":false,\"error\":\"empty body\"}"
|
||||
}
|
||||
let tags: String = "[\"imprint\",\"contextual\"]"
|
||||
let id: String = engram_node_full(
|
||||
let id: String = wt_node(
|
||||
body,
|
||||
"Entity",
|
||||
"imprint:contextual",
|
||||
@@ -208,7 +208,7 @@ fn route_imprint_user(body: String) -> String {
|
||||
return "{\"ok\":false,\"error\":\"empty body\"}"
|
||||
}
|
||||
let tags: String = "[\"imprint\",\"user\"]"
|
||||
let id: String = engram_node_full(
|
||||
let id: String = wt_node(
|
||||
body,
|
||||
"Entity",
|
||||
"imprint:user",
|
||||
@@ -239,7 +239,7 @@ fn route_synthesize(body: String) -> String {
|
||||
}
|
||||
let req: String = "synthesize " + parent_a + " " + parent_b
|
||||
let tags: String = "[\"soul-inbox-pending\",\"synthesis-request\"]"
|
||||
engram_node_full(
|
||||
wt_node(
|
||||
req,
|
||||
"Entity",
|
||||
"synthesis-request",
|
||||
@@ -395,7 +395,28 @@ fn handle_connectors(method: String, clean: String, body: String) -> String {
|
||||
return "{\"ok\":false,\"error\":\"unknown connectors route\"}"
|
||||
}
|
||||
|
||||
// handle_request — the soul's HTTP entry point.
|
||||
//
|
||||
// NOTE ON THE NAME (neuron#117): the el runtime resolves this handler by NAME
|
||||
// via dlsym(RTLD_DEFAULT, "handle_request") — that is why the Linux build must
|
||||
// link -rdynamic. So the dispatcher body moved to route_dispatch and the name
|
||||
// `handle_request` stays put as a thin wrapper. Do not rename it back.
|
||||
//
|
||||
// The wrapper exists to give the write-through boundary a guaranteed flush
|
||||
// point. route_dispatch returns from ~60 places; a per-branch flush would be
|
||||
// forgotten on the 61st. Draining here means EVERY request that staged a write
|
||||
// pushes it before the connection closes, whatever route produced it, including
|
||||
// routes added later that know nothing about persistence.
|
||||
//
|
||||
// wt_drain is a no-op (no HTTP, no cost) when nothing is staged and when the
|
||||
// soul is not in HTTP-engram mode, so this is free on read traffic.
|
||||
fn handle_request(method: String, path: String, body: String) -> String {
|
||||
let resp: String = route_dispatch(method, path, body)
|
||||
let flushed: Int = wt_drain()
|
||||
return resp
|
||||
}
|
||||
|
||||
fn route_dispatch(method: String, path: String, body: String) -> String {
|
||||
let clean: String = strip_query(path)
|
||||
|
||||
// ACTIVITY STAMP (2026-07-30 self-review): every inbound HTTP request —
|
||||
@@ -432,10 +453,27 @@ fn handle_request(method: String, path: String, body: String) -> String {
|
||||
return engram_scan_nodes_json(9999, 0)
|
||||
}
|
||||
if str_eq(clean, "/api/graph/edges") {
|
||||
// TODO(reliability #8): engram_save races with awareness loop mem_save().
|
||||
// Both now use atomic write-to-temp+rename (el_runtime.c). Serialised
|
||||
// by engram_global_mu. Future: add engram_edges_json() builtin.
|
||||
let snap_path: String = env("HOME") + "/.neuron/engram/snapshot.json"
|
||||
// FIXED (neuron#117): this GET used to engram_save() straight over
|
||||
// ~/.neuron/engram/snapshot.json — a READ route, in a process that is
|
||||
// NOT the persistence owner, overwriting the owner's canonical file
|
||||
// on every call. It broke soul.el:571-573 ("the soul must NEVER write
|
||||
// to the local snapshot") and it is the same defect class Will removed
|
||||
// from the engram itself in el `dc39a61` ("stop read routes clobbering
|
||||
// canonical snapshot"), where route_scan_edges/route_sync were moved
|
||||
// to scratch paths for exactly this reason. It was also the race the
|
||||
// old TODO(reliability #8) admitted to.
|
||||
//
|
||||
// Export to a scratch path instead. Same response, no canonical write.
|
||||
// The soul's own snapshot writes are otherwise already gated behind
|
||||
// state key "soul_snapshot_path", which is set ONLY in the genesis
|
||||
// file-mode branch (soul.el: is_genesis && safe_to_seed, and
|
||||
// safe_to_seed is unconditionally false when ENGRAM_URL is set) — so
|
||||
// after this change the soul writes nothing at all in HTTP mode.
|
||||
// Future: add an engram_edges_json() builtin and drop the file round
|
||||
// trip entirely.
|
||||
let scratch_dir: String = env("TMPDIR")
|
||||
let scratch_base: String = if str_eq(scratch_dir, "") { "/tmp" } else { scratch_dir }
|
||||
let snap_path: String = scratch_base + "/soul-edges-export-" + state_get("soul_cgi_id") + ".json"
|
||||
engram_save(snap_path)
|
||||
let snap: String = fs_read(snap_path)
|
||||
let edges_raw: String = json_get_raw(snap, "edges")
|
||||
|
||||
Reference in New Issue
Block a user