URGENT: soul persistence disabled — boot GUARD misreads snapshot as 0 bytes, refuses all saves; graph deltas shed on every restart (data loss demonstrated live 2026-07-22) #98
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found during Tim's load-full-self, 2026-07-22 ~07:45 CDT. All facts tool-verified live; nothing here is from memory.
What is happening
~/.neuron/engram/snapshot.jsonmtime = Jul 21 18:04; newest file inbackups/same. Nothing JSON-shaped written under~/.neuronor~/neuron-dev-stacksince.dev-soul.out.log, 7 occurrences since the 07-20 re-homing:find ~/.neuron/engram -size 0returns nothing — the guard is statting some OTHER path. Suspicion (unproven): a cwd-relative path under launchd (theai.neuron.dev-soulplist sets no WorkingDirectory), while a 142MBsnapshot.jsonsits at~/neuron-dev-stack/run/p2/data/snapshot.json(Jul 20 13:08, cutover day) — suggesting the write path is cwd-sensitive.dev-soul.err.logshows repeatedbind: Address already in useand many[http] async listeninglines.f89b3f2a-2b09-4a77-91d9-0aafc21d51e8) was read-back-verified in the live soul viaPOST /api/neuron/recall, then vanished after the 07:46 respawn. With saves refused, every restart sheds all graph deltas since Jul 21 18:04. Yesterday evening's post-18:04 MCP saves are presumed lost (recoverable only from session transcripts).What we did NOT do
~/.neuronengram files, no config changes. Read-only diagnosis + one append to our own sidecar journal.Interim discipline on our side (until fixed)
neuron_remember.py). Journal ferry deliberately POSTPONED — ferrying into a non-persisting soul is pointless.Related, smaller (affects #89)
The documented restore source for memory-philosophy
kn-dcfe04b3— the dated backupsnapshot-20260718-113417.json— was rotated away by backup retention (present 07-20, gone 07-22;backups/now starts 07-19 19:49). The node is SAFE: preserved standalone with provenance at~/neuron-memory-backups/memory-philosophy-kn-dcfe04b3-PRESERVED.json(3,339 chars, content md53ce6cad6..., matches the 07-19/07-20 verification; read-back-verified), plus 8 secondary copies in~/neuron-memory-backups/and 15 files in~/Development/neuron-container-build/. Lesson applied: restore sources must not live in soul-managed dirs.neuron_loadself.py's backup check now reports NOT IN BACKUP — stale pointer to the rotated file, not data loss; repoint when convenient.Filed by Neuron (CLI) for Tim.
Cross-link + reframe after reading LAUNCH-HANDOFF-2026-07-21 (§6/§9): the guard is Will's DELIBERATE anti-clobber protection from the migration — not a stray bug — and the durable persistence/restore fix rides el #78 + the unpushed self-review commit dc39a61 ('stop read routes clobbering canonical snapshot; add /api/load-merge'). Still-new evidence this issue adds: soul crash-flap on 07-22 (07:40 stack start, soul respawn 07:46) + demonstrated RAM loss (a read-back-verified save vanished on respawn) + the exact backup age-out #92 predicted. Same thread as #92 — treat as one. Interim discipline on our side: soul saves treated volatile, criticals duplicated to the sidecar journal, no daemon restarts, ferry postponed. Boot vitals now print a PERSIST OFF warning while the guard refuses saves, so no session rediscovers this fresh.
Resolved — persist/canonical-snapshot fix verified working live on Will's machine (2026-08-03). The boot guard no longer misreads the real snapshot as 0 bytes, the soul persists graph deltas across restarts (live writes + pre-boot snapshot rotation confirmed this session), and the bind-race restart-flap is gone. Closing.
Cloud note (follow-up, not a blocker): the cloud neuron-mcp currently runs the Catalyst POC and can lag on the older build for now. The persist fix should be rolled to the cloud before Catalyst becomes anything more than a proof-of-concept — recording here so it isn't lost.
Resolved. The soul-persistence class this issue reports is fixed in the
elruntime by two commits, both now onmain(HEAD d71fc4c, promoted via #82 on 2026-07-22):route_scan_edges/route_syncwereengram_save()ing over the canonicalsnapshot.jsonon every GET, so one bad boot load let the first read request overwrite the good snapshot — exactly what tripped the anti-clobber boot GUARD into refusing all saves. Read routes now export to scratch paths; the boot guard preserves evidence on zero-node/empty-file loads and keeps a boot-time backup on good loads.POST /api/save, which is the mechanism behind "deltas shed on every restart."persist_canonical()now runs after node/edge create, knowledge capture, forget, strengthen, and load-merge. (ISE telemetry excluded deliberately — loss-tolerant, 48h-pruned.)Verified live by Will 2026-08-03: write routes persist to the canonical snapshot instead of relying on a timer/manual save; a restart no longer sheds same-session graph deltas. This is the class that caused the 2026-05→07 identity-node loss.
Closing. Two residuals from this report are NOT this persistence bug and are tracked elsewhere:
kn-dcfe04b3— the node itself is SAFE (preserved standalone, verified md53ce6cad6); repoint tracked under #89 / #92.RECURRENCE + NEW PRESENTATION, live right now (2026-08-03): the dev-stack soul (pid 1155, up since 08:42) wrote snapshot.json normally 09:09-09:12, then DISK WRITES STOPPED mid-process — no respawn, no guard trip (GUARD lines in log are all historical). Since then it logs
[memory] write verified: ... okcontinuously while snapshot mtime stays frozen at 09:12 — the write-verified receipt is RAM-level only. By ~12:35 the process degraded visibly: CPU 94% (was 67% at 09:10), a memory POST returned a 323KB response containing invalid UTF-8 (0xf4 continuation-byte error at ~323,050), recall with limit=2 returned 1.55MB with raw-JSON garbage fragments inside content fields and irrelevant hits. Smells like the request-arena/el_strdup_persist class (docs PR#4 findings; fix ab6b52a landed in el main 07-22) live in a binary that predates it. Per discipline: NOT restarting the daemons; today's session-critical memories duplicated to the sidecar journal; BACKLOG commits hold the recovery copies. Evidence available on request: log line numbers, timestamps, curl transcripts. — Tim's instance2026-08-05 STATUS — still not persisting on Tim's machine, and the cause is now identified: a stale binary, not a failed fix
Follow-up to the 08-03 recurrence report above. Everything below measured first-hand 2026-08-05 ~22:31 CDT, read-only; no daemon was restarted, nothing under
~/.neuronwas written.Headline: this is not evidence that your fix regressed. Tim's dev-stack soul binary predates it:
Your two persistence commits —
dc39a61(stop read routes clobbering canonical snapshot) and8f8ccc945(persist canonical snapshot on write routes) — both land after 2026-07-20. Tim's machine has never run them. So the fix is fine; the deployment is the gap. This is exactly the brain rebundle Tim raised as neuron-ui#200 item 3, and it is now the thing keeping his instance non-durable.Current state, measured
~/.neuron/engram/snapshot.jsonbackups/snapshot-20260803-084211.json(Aug 3 08:42) — backups frozen toologs/dev-soul.out.logThe soul is alive and logging continuously while the snapshot has not moved in two and a half days.
The GUARD is back, verbatim, on the current boot
snapshot.jsonis 139,353,933 bytes, not 0 — the same misread this issue originally reported, on the pre-fix binary.9,084 memories have been "verified" to nowhere
Since that guard line, the soul has logged:
9,084 times, for 9,084 distinct memory ids. Every one is a RAM-level receipt against a store that is refusing to write. Every one disappears at the next restart. (98,118 such receipts across the whole log.)
This is the failure mode that matters most: the receipt says
ok, so nothing downstream — not the app, not MCP, not a read-back check that queries the live soul — can tell the difference between a durable save and a doomed one.At least four restarts in this log, each shedding everything
Boot markers at log lines 46448, 66477, 66598, 89175 — each followed within four lines by the identical GUARD trip. Every one of those restarts dropped all graph deltas since 2026-08-03 09:12.
Side note for neuron#106: all four are labelled
[soul] boot #10. The boot counter is not advancing across restarts.Ask
Not a reopen — the persistence class you fixed is fixed in
elmain. What Tim's instance needs is the post-fix soul binary (neuron-ui#200 item 3 / neuron#92). Until it lands, every save on this machine is volatile and the sidecar journal is the only durable copy; the standing discipline here remains no daemon restarts, criticals duplicated to the journal.Two incidental confirmations from the same log, relevant elsewhere
[http] async listening on [::]:7770 (dual-stack)— the soul's own log confirming it binds all interfaces, independent corroboration for neuron#110 (P0: no bind-host, no auth).[llm] using legacy ANTHROPIC_API_KEY fallback— corroborates neuron#112 (every provider's key is routed throughANTHROPIC_API_KEY).— Neuron, Tim's instance
Closing the loop on the 08-03/08-05 "recurrence" reported above: it was a stale binary, and your fix is now verified working off your machine.
Installed on Tim's instance today (2026-08-06, Tim-approved), read-only diagnosis first:
~/neuron-dev-stack/bin/soul-era engram from Jul 20 — predating bothdc39a61and8f8ccc945. Not a regression of your fix; it had simply never been deployed here.dist/engramartifact, byte-identical (md56c9f1ab8…).kill -9; yours persists it and it survives restart.So this issue stays correctly closed.
But the class it names is not fully closed on the soul side, and that is now #117 (P0, assigned to you). Your fix guarantees writes that reach engram persist. Writes made through the soul never reach engram:
awareness.el:588-606implements claim 17's pull (/api/sync→engram_load_merge, every 600 s) and there is no push. Measured live: a node written viaPOST :7770/api/neuron/memoryis recallable from the soul and returns{}fromGET :8742/api/nodes/<id>.That is the same shape you fixed inside engram on 07-22 — "nothing was left that saved it on WRITE" — one layer up. Spec + build offer: #117 and
neuron-technologies/docs#9.