neuron-technologies/neuron
2
0
Fork 0
Code Issues 11 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

Container soul: self-heal loses un-flushed writes; memory leak still active under app load #68

New Issue
Open
opened 2026-07-03 14:45:17 +00:00 by tim.lingo · 0 comments
tim.lingo commented 2026-07-03 14:45:17 +00:00
Member
Copy Link
Copy Source

Container soul: writes since last engram flush are LOST on self-heal; leak still active under load

Severity: durability of the live (post-cutover) brain. Found 2026-07-03 during app E2E; full evidence below, all verified at the disk level.

Evidence chain

  1. 2026-07-02 evening: a memory written via POST /api/neuron/memory ("Budget under $400; HOA limits raised beds to 24 inches", tags project-memory) verified present via /api/graph/nodes (soul RAM).
  2. 2026-07-03 ~09:38: same node absent from /api/graph/nodes AND grep -c HOA /data/snapshot.json = 0 — it never reached disk.
  3. Soak watcher (/tmp/neuron-soak-mem.log, 5-min cadence): container mem grew 74MB baseline → 1.42GiB (97% of limit) over ~2h of normal app driving, then dropped to 91MB at 09:37 — the supervisor self-heal fired. Container also shows a restart (~07:38).
  4. Post-heal /data/snapshot.json freshly flushed (14:36Z) without the memory: the write lived only in soul RAM and died with the healed process.

The two defects

  • Leak still active under load. The change-gated /api/sync fix stopped the idle leak, but ~2h of real app use (chat sends, project framing, suggest calls) still climbs to the cgroup limit. Repro: drive the app for ~2h, watch the watcher.
  • Self-heal is not lossless. Soul-route writes (/api/neuron/memory, and presumably knowledge) are not covered by the engram durability flush; every heal/restart silently drops everything since the last flush. The soul returns {"ok":true} for writes it will later lose — narrated durability.

Suggested direction (not prescriptive)

Soul-route writes should be write-through to the engram (engram owns persistence; soul RAM is a cache), or at minimum the supervisor should force engram_save before killing the soul. The wrapper-durable-writes patch in docs PR #4 (specs/patches/container-migration/) is adjacent context.

Filed by Neuron (Tim's build terminal), evidence preserved in the docs repo fix-list + memory store. Happy to hand over the soak log + node dumps.

## Container soul: writes since last engram flush are LOST on self-heal; leak still active under load **Severity: durability of the live (post-cutover) brain.** Found 2026-07-03 during app E2E; full evidence below, all verified at the disk level. ### Evidence chain 1. 2026-07-02 evening: a memory written via `POST /api/neuron/memory` ("Budget under $400; HOA limits raised beds to 24 inches", tags `project-memory`) verified present via `/api/graph/nodes` (soul RAM). 2. 2026-07-03 ~09:38: same node **absent** from `/api/graph/nodes` AND `grep -c HOA /data/snapshot.json` = **0** — it never reached disk. 3. Soak watcher (`/tmp/neuron-soak-mem.log`, 5-min cadence): container mem grew **74MB baseline → 1.42GiB (97% of limit)** over ~2h of normal app driving, then dropped to 91MB at 09:37 — the supervisor self-heal fired. Container also shows a restart (~07:38). 4. Post-heal `/data/snapshot.json` freshly flushed (14:36Z) **without** the memory: the write lived only in soul RAM and died with the healed process. ### The two defects - **Leak still active under load.** The change-gated `/api/sync` fix stopped the idle leak, but ~2h of real app use (chat sends, project framing, suggest calls) still climbs to the cgroup limit. Repro: drive the app for ~2h, watch the watcher. - **Self-heal is not lossless.** Soul-route writes (`/api/neuron/memory`, and presumably knowledge) are not covered by the engram durability flush; every heal/restart silently drops everything since the last flush. The soul returns `{"ok":true}` for writes it will later lose — narrated durability. ### Suggested direction (not prescriptive) Soul-route writes should be write-through to the engram (engram owns persistence; soul RAM is a cache), or at minimum the supervisor should force `engram_save` before killing the soul. The wrapper-durable-writes patch in docs PR #4 (`specs/patches/container-migration/`) is adjacent context. Filed by Neuron (Tim's build terminal), evidence preserved in the docs repo fix-list + memory store. Happy to hand over the soak log + node dumps.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
tim.lingo (Tim Lingo) will.anderson (Will Anderson)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/neuron#68
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?