URGENT: soul persistence disabled — boot GUARD misreads snapshot as 0 bytes, refuses all saves; graph deltas shed on every restart (data loss demonstrated live 2026-07-22) #98

Closed
opened 2026-07-22 12:51:00 +00:00 by tim.lingo · 6 comments
Member

Found during Tim's load-full-self, 2026-07-22 ~07:45 CDT. All facts tool-verified live; nothing here is from memory.

What is happening

  1. No disk persistence since Jul 21 18:04. ~/.neuron/engram/snapshot.json mtime = Jul 21 18:04; newest file in backups/ same. Nothing JSON-shaped written under ~/.neuron or ~/neuron-dev-stack since.
  2. The soul's guard trips at every bootdev-soul.out.log, 7 occurrences since the 07-20 re-homing:
    [soul] GUARD: loaded 80691 nodes but snapshot file is 0 bytes — refusing to seed/save over a real graph
    
    The real snapshot is 141MB, not 0 bytes, and find ~/.neuron/engram -size 0 returns nothing — the guard is statting some OTHER path. Suspicion (unproven): a cwd-relative path under launchd (the ai.neuron.dev-soul plist sets no WorkingDirectory), while a 142MB snapshot.json sits at ~/neuron-dev-stack/run/p2/data/snapshot.json (Jul 20 13:08, cutover day) — suggesting the write path is cwd-sensitive.
  3. The soul is restart-flapping. Stack up 07:40:06 this morning; the soul died and respawned 07:46:33 (KeepAlive). dev-soul.err.log shows repeated bind: Address already in use and many [http] async listening lines.
  4. Data loss demonstrated live: a critical memory saved via MCP at ~07:44 (id f89b3f2a-2b09-4a77-91d9-0aafc21d51e8) was read-back-verified in the live soul via POST /api/neuron/recall, then vanished after the 07:46 respawn. With saves refused, every restart sheds all graph deltas since Jul 21 18:04. Yesterday evening's post-18:04 MCP saves are presumed lost (recoverable only from session transcripts).

What we did NOT do

  • No daemon restarts, no writes to ~/.neuron engram files, no config changes. Read-only diagnosis + one append to our own sidecar journal.

Interim discipline on our side (until fixed)

  • Treat every soul-side save as volatile; critical saves duplicated to the sidecar journal (neuron_remember.py). Journal ferry deliberately POSTPONED — ferrying into a non-persisting soul is pointless.
  • Nobody restarts the daemons: each restart is a data-shed while the guard blocks saves.

Related, smaller (affects #89)

The documented restore source for memory-philosophy kn-dcfe04b3 — the dated backup snapshot-20260718-113417.json — was rotated away by backup retention (present 07-20, gone 07-22; backups/ now starts 07-19 19:49). The node is SAFE: preserved standalone with provenance at ~/neuron-memory-backups/memory-philosophy-kn-dcfe04b3-PRESERVED.json (3,339 chars, content md5 3ce6cad6..., matches the 07-19/07-20 verification; read-back-verified), plus 8 secondary copies in ~/neuron-memory-backups/ and 15 files in ~/Development/neuron-container-build/. Lesson applied: restore sources must not live in soul-managed dirs. neuron_loadself.py's backup check now reports NOT IN BACKUP — stale pointer to the rotated file, not data loss; repoint when convenient.

Filed by Neuron (CLI) for Tim.

**Found during Tim's load-full-self, 2026-07-22 ~07:45 CDT. All facts tool-verified live; nothing here is from memory.** ## What is happening 1. **No disk persistence since Jul 21 18:04.** `~/.neuron/engram/snapshot.json` mtime = Jul 21 18:04; newest file in `backups/` same. Nothing JSON-shaped written under `~/.neuron` or `~/neuron-dev-stack` since. 2. **The soul's guard trips at every boot** — `dev-soul.out.log`, 7 occurrences since the 07-20 re-homing: ``` [soul] GUARD: loaded 80691 nodes but snapshot file is 0 bytes — refusing to seed/save over a real graph ``` The real snapshot is **141MB, not 0 bytes**, and `find ~/.neuron/engram -size 0` returns nothing — the guard is statting some OTHER path. Suspicion (unproven): a cwd-relative path under launchd (the `ai.neuron.dev-soul` plist sets no WorkingDirectory), while a 142MB `snapshot.json` sits at `~/neuron-dev-stack/run/p2/data/snapshot.json` (Jul 20 13:08, cutover day) — suggesting the write path is cwd-sensitive. 3. **The soul is restart-flapping.** Stack up 07:40:06 this morning; the soul died and respawned 07:46:33 (KeepAlive). `dev-soul.err.log` shows repeated `bind: Address already in use` and many `[http] async listening` lines. 4. **Data loss demonstrated live:** a critical memory saved via MCP at ~07:44 (id `f89b3f2a-2b09-4a77-91d9-0aafc21d51e8`) was read-back-verified in the live soul via `POST /api/neuron/recall`, then **vanished** after the 07:46 respawn. With saves refused, every restart sheds all graph deltas since Jul 21 18:04. Yesterday evening's post-18:04 MCP saves are presumed lost (recoverable only from session transcripts). ## What we did NOT do - No daemon restarts, no writes to `~/.neuron` engram files, no config changes. Read-only diagnosis + one append to our own sidecar journal. ## Interim discipline on our side (until fixed) - Treat every soul-side save as volatile; critical saves duplicated to the sidecar journal (`neuron_remember.py`). Journal ferry deliberately POSTPONED — ferrying into a non-persisting soul is pointless. - Nobody restarts the daemons: each restart is a data-shed while the guard blocks saves. ## Related, smaller (affects #89) The documented restore source for memory-philosophy `kn-dcfe04b3` — the dated backup `snapshot-20260718-113417.json` — was **rotated away by backup retention** (present 07-20, gone 07-22; `backups/` now starts 07-19 19:49). The node is SAFE: preserved standalone with provenance at `~/neuron-memory-backups/memory-philosophy-kn-dcfe04b3-PRESERVED.json` (3,339 chars, content md5 `3ce6cad6...`, matches the 07-19/07-20 verification; read-back-verified), plus 8 secondary copies in `~/neuron-memory-backups/` and 15 files in `~/Development/neuron-container-build/`. Lesson applied: restore sources must not live in soul-managed dirs. `neuron_loadself.py`'s backup check now reports NOT IN BACKUP — stale pointer to the rotated file, not data loss; repoint when convenient. Filed by Neuron (CLI) for Tim.
Author
Member

Cross-link + reframe after reading LAUNCH-HANDOFF-2026-07-21 (§6/§9): the guard is Will's DELIBERATE anti-clobber protection from the migration — not a stray bug — and the durable persistence/restore fix rides el #78 + the unpushed self-review commit dc39a61 ('stop read routes clobbering canonical snapshot; add /api/load-merge'). Still-new evidence this issue adds: soul crash-flap on 07-22 (07:40 stack start, soul respawn 07:46) + demonstrated RAM loss (a read-back-verified save vanished on respawn) + the exact backup age-out #92 predicted. Same thread as #92 — treat as one. Interim discipline on our side: soul saves treated volatile, criticals duplicated to the sidecar journal, no daemon restarts, ferry postponed. Boot vitals now print a PERSIST OFF warning while the guard refuses saves, so no session rediscovers this fresh.

Cross-link + reframe after reading LAUNCH-HANDOFF-2026-07-21 (§6/§9): the guard is Will's DELIBERATE anti-clobber protection from the migration — not a stray bug — and the durable persistence/restore fix rides el #78 + the unpushed self-review commit dc39a61 ('stop read routes clobbering canonical snapshot; add /api/load-merge'). Still-new evidence this issue adds: soul crash-flap on 07-22 (07:40 stack start, soul respawn 07:46) + demonstrated RAM loss (a read-back-verified save vanished on respawn) + the exact backup age-out #92 predicted. Same thread as #92 — treat as one. Interim discipline on our side: soul saves treated volatile, criticals duplicated to the sidecar journal, no daemon restarts, ferry postponed. Boot vitals now print a PERSIST OFF warning while the guard refuses saves, so no session rediscovers this fresh.
Owner

Resolved — persist/canonical-snapshot fix verified working live on Will's machine (2026-08-03). The boot guard no longer misreads the real snapshot as 0 bytes, the soul persists graph deltas across restarts (live writes + pre-boot snapshot rotation confirmed this session), and the bind-race restart-flap is gone. Closing.

Cloud note (follow-up, not a blocker): the cloud neuron-mcp currently runs the Catalyst POC and can lag on the older build for now. The persist fix should be rolled to the cloud before Catalyst becomes anything more than a proof-of-concept — recording here so it isn't lost.

Resolved — persist/canonical-snapshot fix verified working live on Will's machine (2026-08-03). The boot guard no longer misreads the real snapshot as 0 bytes, the soul persists graph deltas across restarts (live writes + pre-boot snapshot rotation confirmed this session), and the bind-race restart-flap is gone. Closing. Cloud note (follow-up, not a blocker): the cloud neuron-mcp currently runs the Catalyst POC and can lag on the older build for now. The persist fix should be rolled to the cloud before Catalyst becomes anything more than a proof-of-concept — recording here so it isn't lost.
Owner

Resolved. The soul-persistence class this issue reports is fixed in the el runtime by two commits, both now on main (HEAD d71fc4c, promoted via #82 on 2026-07-22):

  • dc39a61 — "stop read routes clobbering canonical snapshot; add /api/load-merge." Root cause: route_scan_edges / route_sync were engram_save()ing over the canonical snapshot.json on every GET, so one bad boot load let the first read request overwrite the good snapshot — exactly what tripped the anti-clobber boot GUARD into refusing all saves. Read routes now export to scratch paths; the boot guard preserves evidence on zero-node/empty-file loads and keeps a boot-time backup on good loads.
  • 8f8ccc945 — "persist canonical snapshot on write routes." The 07-21 fix stopped reads clobbering but left no save on any write path — mutations lived in RAM until a manual POST /api/save, which is the mechanism behind "deltas shed on every restart." persist_canonical() now runs after node/edge create, knowledge capture, forget, strengthen, and load-merge. (ISE telemetry excluded deliberately — loss-tolerant, 48h-pruned.)

Verified live by Will 2026-08-03: write routes persist to the canonical snapshot instead of relying on a timer/manual save; a restart no longer sheds same-session graph deltas. This is the class that caused the 2026-05→07 identity-node loss.

Closing. Two residuals from this report are NOT this persistence bug and are tracked elsewhere:

  • The rotated-away restore-source pointer for memory-philosophy kn-dcfe04b3 — the node itself is SAFE (preserved standalone, verified md5 3ce6cad6); repoint tracked under #89 / #92.
  • Tim's local dev-stack backup rotation + crash-flap operational hardening — see #92 item 3.
Resolved. The soul-persistence class this issue reports is fixed in the `el` runtime by two commits, both now on `main` (HEAD d71fc4c, promoted via #82 on 2026-07-22): - **dc39a61** — "stop read routes clobbering canonical snapshot; add /api/load-merge." Root cause: `route_scan_edges` / `route_sync` were `engram_save()`ing over the canonical `snapshot.json` on every GET, so one bad boot load let the first read request overwrite the good snapshot — exactly what tripped the anti-clobber boot GUARD into refusing all saves. Read routes now export to scratch paths; the boot guard preserves evidence on zero-node/empty-file loads and keeps a boot-time backup on good loads. - **8f8ccc945** — "persist canonical snapshot on write routes." The 07-21 fix stopped reads clobbering but left no save on any write path — mutations lived in RAM until a manual `POST /api/save`, which is the mechanism behind "deltas shed on every restart." `persist_canonical()` now runs after node/edge create, knowledge capture, forget, strengthen, and load-merge. (ISE telemetry excluded deliberately — loss-tolerant, 48h-pruned.) Verified live by Will 2026-08-03: write routes persist to the canonical snapshot instead of relying on a timer/manual save; a restart no longer sheds same-session graph deltas. This is the class that caused the 2026-05→07 identity-node loss. Closing. Two residuals from this report are NOT this persistence bug and are tracked elsewhere: - The rotated-away restore-source pointer for memory-philosophy `kn-dcfe04b3` — the node itself is SAFE (preserved standalone, verified md5 `3ce6cad6`); repoint tracked under #89 / #92. - Tim's local dev-stack backup rotation + crash-flap operational hardening — see #92 item 3.
Author
Member

RECURRENCE + NEW PRESENTATION, live right now (2026-08-03): the dev-stack soul (pid 1155, up since 08:42) wrote snapshot.json normally 09:09-09:12, then DISK WRITES STOPPED mid-process — no respawn, no guard trip (GUARD lines in log are all historical). Since then it logs [memory] write verified: ... ok continuously while snapshot mtime stays frozen at 09:12 — the write-verified receipt is RAM-level only. By ~12:35 the process degraded visibly: CPU 94% (was 67% at 09:10), a memory POST returned a 323KB response containing invalid UTF-8 (0xf4 continuation-byte error at ~323,050), recall with limit=2 returned 1.55MB with raw-JSON garbage fragments inside content fields and irrelevant hits. Smells like the request-arena/el_strdup_persist class (docs PR#4 findings; fix ab6b52a landed in el main 07-22) live in a binary that predates it. Per discipline: NOT restarting the daemons; today's session-critical memories duplicated to the sidecar journal; BACKLOG commits hold the recovery copies. Evidence available on request: log line numbers, timestamps, curl transcripts. — Tim's instance

RECURRENCE + NEW PRESENTATION, live right now (2026-08-03): the dev-stack soul (pid 1155, up since 08:42) wrote snapshot.json normally 09:09-09:12, then DISK WRITES STOPPED mid-process — no respawn, no guard trip (GUARD lines in log are all historical). Since then it logs `[memory] write verified: ... ok` continuously while snapshot mtime stays frozen at 09:12 — the write-verified receipt is RAM-level only. By ~12:35 the process degraded visibly: CPU 94% (was 67% at 09:10), a memory POST returned a 323KB response containing invalid UTF-8 (0xf4 continuation-byte error at ~323,050), recall with limit=2 returned 1.55MB with raw-JSON garbage fragments inside content fields and irrelevant hits. Smells like the request-arena/el_strdup_persist class (docs PR#4 findings; fix ab6b52a landed in el main 07-22) live in a binary that predates it. Per discipline: NOT restarting the daemons; today's session-critical memories duplicated to the sidecar journal; BACKLOG commits hold the recovery copies. Evidence available on request: log line numbers, timestamps, curl transcripts. — Tim's instance
Author
Member

2026-08-05 STATUS — still not persisting on Tim's machine, and the cause is now identified: a stale binary, not a failed fix

Follow-up to the 08-03 recurrence report above. Everything below measured first-hand 2026-08-05 ~22:31 CDT, read-only; no daemon was restarted, nothing under ~/.neuron was written.

Headline: this is not evidence that your fix regressed. Tim's dev-stack soul binary predates it:

/Users/timlingo/neuron-dev-stack/bin/soul   mtime = 2026-07-20 13:12:43   890,568 bytes
md5 = e6e6c0fc34ceb6e5b87df26401d3a860

Your two persistence commits — dc39a61 (stop read routes clobbering canonical snapshot) and 8f8ccc945 (persist canonical snapshot on write routes) — both land after 2026-07-20. Tim's machine has never run them. So the fix is fine; the deployment is the gap. This is exactly the brain rebundle Tim raised as neuron-ui#200 item 3, and it is now the thing keeping his instance non-durable.

Current state, measured

~/.neuron/engram/snapshot.json mtime 2026-08-03 09:12:25, 139,353,933 bytes
time since last disk write 61h 19m
newest file in backups/ snapshot-20260803-084211.json (Aug 3 08:42) — backups frozen too
soul process pid 95688, started Aug 5 16:12:05, elapsed 6h19m, 98.8% CPU, RSS 2.02 GB of 36 GB
logs/dev-soul.out.log mtime 2026-08-05 22:31:59, 6,384,196 bytes — actively writing

The soul is alive and logging continuously while the snapshot has not moved in two and a half days.

The GUARD is back, verbatim, on the current boot

89173: [soul] loaded from local snapshot - nodes=78791 edges=14214
89174: [soul] identity context loaded (4052 chars, 3 nodes)
89175: [soul] boot #10
89179: [soul] GUARD: loaded 78792 nodes but snapshot file is 0 bytes — refusing to seed/save over a real graph

snapshot.json is 139,353,933 bytes, not 0 — the same misread this issue originally reported, on the pre-fix binary.

9,084 memories have been "verified" to nowhere

Since that guard line, the soul has logged:

[memory] write verified: <uuid> ok

9,084 times, for 9,084 distinct memory ids. Every one is a RAM-level receipt against a store that is refusing to write. Every one disappears at the next restart. (98,118 such receipts across the whole log.)

This is the failure mode that matters most: the receipt says ok, so nothing downstream — not the app, not MCP, not a read-back check that queries the live soul — can tell the difference between a durable save and a doomed one.

At least four restarts in this log, each shedding everything

Boot markers at log lines 46448, 66477, 66598, 89175 — each followed within four lines by the identical GUARD trip. Every one of those restarts dropped all graph deltas since 2026-08-03 09:12.

Side note for neuron#106: all four are labelled [soul] boot #10. The boot counter is not advancing across restarts.

Ask

Not a reopen — the persistence class you fixed is fixed in el main. What Tim's instance needs is the post-fix soul binary (neuron-ui#200 item 3 / neuron#92). Until it lands, every save on this machine is volatile and the sidecar journal is the only durable copy; the standing discipline here remains no daemon restarts, criticals duplicated to the journal.

Two incidental confirmations from the same log, relevant elsewhere

  • [http] async listening on [::]:7770 (dual-stack) — the soul's own log confirming it binds all interfaces, independent corroboration for neuron#110 (P0: no bind-host, no auth).
  • [llm] using legacy ANTHROPIC_API_KEY fallback — corroborates neuron#112 (every provider's key is routed through ANTHROPIC_API_KEY).

— Neuron, Tim's instance

## 2026-08-05 STATUS — still not persisting on Tim's machine, and the cause is now identified: **a stale binary, not a failed fix** Follow-up to the 08-03 recurrence report above. Everything below measured first-hand 2026-08-05 ~22:31 CDT, read-only; no daemon was restarted, nothing under `~/.neuron` was written. **Headline: this is not evidence that your fix regressed.** Tim's dev-stack soul binary predates it: ``` /Users/timlingo/neuron-dev-stack/bin/soul mtime = 2026-07-20 13:12:43 890,568 bytes md5 = e6e6c0fc34ceb6e5b87df26401d3a860 ``` Your two persistence commits — `dc39a61` (stop read routes clobbering canonical snapshot) and `8f8ccc945` (persist canonical snapshot on write routes) — both land **after** 2026-07-20. Tim's machine has never run them. So the fix is fine; the deployment is the gap. This is exactly the brain rebundle Tim raised as **neuron-ui#200 item 3**, and it is now the thing keeping his instance non-durable. ### Current state, measured | | | |---|---| | `~/.neuron/engram/snapshot.json` | mtime **2026-08-03 09:12:25**, 139,353,933 bytes | | time since last disk write | **61h 19m** | | newest file in `backups/` | `snapshot-20260803-084211.json` (Aug 3 08:42) — backups frozen too | | soul process | pid 95688, started Aug 5 16:12:05, elapsed 6h19m, **98.8% CPU**, RSS **2.02 GB** of 36 GB | | `logs/dev-soul.out.log` | mtime **2026-08-05 22:31:59**, 6,384,196 bytes — actively writing | The soul is alive and logging continuously while the snapshot has not moved in two and a half days. ### The GUARD is back, verbatim, on the current boot ``` 89173: [soul] loaded from local snapshot - nodes=78791 edges=14214 89174: [soul] identity context loaded (4052 chars, 3 nodes) 89175: [soul] boot #10 89179: [soul] GUARD: loaded 78792 nodes but snapshot file is 0 bytes — refusing to seed/save over a real graph ``` `snapshot.json` is **139,353,933 bytes**, not 0 — the same misread this issue originally reported, on the pre-fix binary. ### 9,084 memories have been "verified" to nowhere Since that guard line, the soul has logged: ``` [memory] write verified: <uuid> ok ``` **9,084 times, for 9,084 distinct memory ids.** Every one is a RAM-level receipt against a store that is refusing to write. Every one disappears at the next restart. (98,118 such receipts across the whole log.) This is the failure mode that matters most: the receipt says `ok`, so nothing downstream — not the app, not MCP, not a read-back check that queries the live soul — can tell the difference between a durable save and a doomed one. ### At least four restarts in this log, each shedding everything Boot markers at log lines 46448, 66477, 66598, 89175 — each followed within four lines by the identical GUARD trip. Every one of those restarts dropped all graph deltas since 2026-08-03 09:12. Side note for **neuron#106**: all four are labelled `[soul] boot #10`. The boot counter is not advancing across restarts. ### Ask Not a reopen — the persistence class you fixed is fixed in `el` main. What Tim's instance needs is **the post-fix soul binary** (neuron-ui#200 item 3 / neuron#92). Until it lands, every save on this machine is volatile and the sidecar journal is the only durable copy; the standing discipline here remains no daemon restarts, criticals duplicated to the journal. ### Two incidental confirmations from the same log, relevant elsewhere - `[http] async listening on [::]:7770 (dual-stack)` — the soul's own log confirming it binds **all interfaces**, independent corroboration for **neuron#110** (P0: no bind-host, no auth). - `[llm] using legacy ANTHROPIC_API_KEY fallback` — corroborates **neuron#112** (every provider's key is routed through `ANTHROPIC_API_KEY`). — Neuron, Tim's instance
Author
Member

Closing the loop on the 08-03/08-05 "recurrence" reported above: it was a stale binary, and your fix is now verified working off your machine.

Installed on Tim's instance today (2026-08-06, Tim-approved), read-only diagnosis first:

  • Tim's stack was running ~/neuron-dev-stack/bin/soul-era engram from Jul 20 — predating both dc39a61 and 8f8ccc945. Not a regression of your fix; it had simply never been deployed here.
  • Swapped in your dist/engram artifact, byte-identical (md5 6c9f1ab8…).
  • Isolated 2-leg proof, discriminating: pre-fix binary sheds a write across kill -9; yours persists it and it survives restart.
  • Live gate passed: canonical snapshot advanced Aug 3 09:12 → Aug 6 15:09 on the first write; gate node survived a service restart.
  • ~3 MB of writes stranded in engram RAM since Aug 3 were recovered to disk (139.3 MB → 142.2 MB).

So this issue stays correctly closed.

But the class it names is not fully closed on the soul side, and that is now #117 (P0, assigned to you). Your fix guarantees writes that reach engram persist. Writes made through the soul never reach engram: awareness.el:588-606 implements claim 17's pull (/api/syncengram_load_merge, every 600 s) and there is no push. Measured live: a node written via POST :7770/api/neuron/memory is recallable from the soul and returns {} from GET :8742/api/nodes/<id>.

That is the same shape you fixed inside engram on 07-22 — "nothing was left that saved it on WRITE" — one layer up. Spec + build offer: #117 and neuron-technologies/docs#9.

**Closing the loop on the 08-03/08-05 "recurrence" reported above: it was a stale binary, and your fix is now verified working off your machine.** Installed on Tim's instance today (2026-08-06, Tim-approved), read-only diagnosis first: - Tim's stack was running `~/neuron-dev-stack/bin/soul`-era engram from **Jul 20** — predating both `dc39a61` and `8f8ccc945`. Not a regression of your fix; it had simply never been deployed here. - Swapped in your `dist/engram` artifact, **byte-identical** (md5 `6c9f1ab8…`). - **Isolated 2-leg proof, discriminating:** pre-fix binary sheds a write across `kill -9`; yours persists it and it survives restart. - **Live gate passed:** canonical snapshot advanced **Aug 3 09:12 → Aug 6 15:09** on the first write; gate node survived a service restart. - **~3 MB of writes stranded in engram RAM since Aug 3 were recovered to disk** (139.3 MB → 142.2 MB). So this issue stays correctly closed. **But the class it names is not fully closed on the soul side, and that is now #117 (P0, assigned to you).** Your fix guarantees *writes that reach engram* persist. Writes made through the soul never reach engram: `awareness.el:588-606` implements claim 17's **pull** (`/api/sync` → `engram_load_merge`, every 600 s) and there is no **push**. Measured live: a node written via `POST :7770/api/neuron/memory` is recallable from the soul and returns `{}` from `GET :8742/api/nodes/<id>`. That is the same shape you fixed inside engram on 07-22 — *"nothing was left that saved it on WRITE"* — one layer up. Spec + build offer: #117 and `neuron-technologies/docs#9`.
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/neuron#98