dd952c0e468f4820d49a7fdbbb7941277af8f31c
1 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Tim Lingo
|
dd952c0e46 |
feat(soul): write-through to the persistence owner — memories survive restart (#117)
The soul obeys half of its own ownership rule. soul.el:571-573 says "when
ENGRAM_URL is set the HTTP Engram owns persistence — the soul must NEVER write
to the local snapshot", and it doesn't. But nothing was ever built to hand the
soul's writes TO that owner: sync is pull-only (/api/sync -> engram_load_merge),
so every node created inside the soul lived in process RAM and was shed on
restart. Measured live 2026-08-07: soul node_count=102184, engram 79197.
SCOPE CORRECTION vs the earlier internal spec: engram provisional claim 17's
"pull-then-push" is a PEER-ENGRAM to PEER-ENGRAM protocol (claims 15-18 say so
explicitly). The soul is a CALLER of the database API, not a peer. Claim 17 is
NOT authority for a soul<->engram contract and is no longer cited as such. The
design here follows from the ownership rule alone.
Mechanism: a new Accessor, persist.el, is the single boundary. Writes stage a
delta to a filesystem spool and are pushed to the owner via POST /api/load-merge
— NOT POST /api/nodes, which mints a new server-side id (breaking dedup and
edges) and drops label/tier/tags/importance/confidence (verified in a sandbox:
a tier "Canonical" probe came back "Working"). load-merge preserves the id and
every field, dedups nodes by id and edges by (from,to,relation) so retries are
no-ops, and calls persist_canonical() so THE OWNER writes its own file — the
ownership rule is honoured rather than worked around.
Spool-and-drain rather than push-per-write: measured ~0.38s per load-merge at
live scale (79k nodes/176MB), and a chat turn writes 5-7 nodes. The spool is on
disk, not in process state, because the soul serves each connection on its own
pthread and a shared buffer would lose entries to a read-modify-write race. That
also buys crash recovery: writes orphaned by kill -9 are drained on next boot.
Honesty: api_persisted (the gate all 10 MCP write handlers pass through) and
mem_store now assert AT THE OWNER instead of reading back the soul's own RAM.
With the owner down a write returns {"ok":false,"error":"write_not_persisted"}
and the delta is queued — where main returns {"ok":true} for a write that dies.
Coverage: 35 node sites + 9 edge sites routed through the boundary. Deliberately
excluded, with reasons in persist.el: 4 InternalStateEvent sites (Will's own
telemetry carve-out), the boot counter and the persona (both already have
bespoke owner-side write-backs), and soul.el's 54 genesis identity edges
(file-mode only). engram_strengthen and engram_forget are NOT propagated —
load-merge cannot update or delete, and hard-deleting at the owner would fail
verify-soul-contract.sh section B.
Also fixed here:
- routes.el GET /api/graph/edges engram_save()'d straight over the owner's
canonical snapshot.json — a read route, in a non-owner process, clobbering the
canonical on every call. Same defect class Will removed from the engram in el
dc39a61. Now exports to a scratch path. With this gone the soul writes nothing
at all in HTTP mode.
- persist.el must clear the runtime's _tl_fs_read_len hint after every fs_read.
In vendored runtime v1.0.0-20260501 that hint becomes the NEXT response's
Content-Length, so reading a spool file mid-request made an 86-byte reply go
out as 497 bytes with 411 bytes of adjacent heap trailing it. Caught and fixed
at our boundary; the runtime class was fixed upstream in el 43636ae, which is
not the pinned runtime here.
Rung: E2E-VERIFIED, discriminating. Same harness, same engram binary:
write-through: LEG 1 PRESENT at owner, LEG 2 SURVIVED kill -9 + restart
main: LEG 1 ABSENT at owner, LEG 2 LOST
verify-soul-contract.sh: GATE PASS on both builds (27/27 routes, immutability).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|