Engram Recovery, Cutover & Build — Decisions & Reversal Runbook

Date: 2026-08-12 · Owner: Neuron (for Will) · Status: LIVING (finalized with actuals after cutover)

Per Will's standing rule: every change ships with what's happening, the decisions + rationale, and an exact path for reversal. Advance authorization (Will, 2026-08-12): promote to prod for Will's own testing — NOT the website, no customer may see any of this. Customer-facing surfaces stay frozen.

1. Scope & guardrails

In scope (his testing env): the local engram service :8742 on Will's Mac, and the engram-tiered-storage build. Internal testing only.
FROZEN — do NOT touch: the marketing website, any customer-facing Cloud Run service, any public deploy. No customer exposure. Broader prod promotion (beyond Will's local testing) requires an explicit, separate go.

2. What's changing (and why)

#	Change	Rationale	Reversal (see §5)
1	Crash-loop stopped (`launchctl bootout ai.neuron.engram`)	42 crashes/day; re-running a crashing WAL-replay over the store is the only corruption risk	R1
2	btree fix committed `9e28def` (branch `engram-tiered-storage`)	Root cause: `int_max_keys` /8 vs /16 → node overflow → stack smash	R2
3	Rebuild `:8742` store from `:7770` working-store fresh export + fixed binary; cut over	Working store (~12,825, incl. today) is the truth; bloated egm lost ~1,500 nodes	R3
4	Tag `engram-tiered-m8` after green cutover	The fix makes M8 boot real data	R4
5	(Forthcoming) M9 / M10 / M-INTEROCEPTION build	The cognitive architecture; each staged + tagged separately	per-milestone

3. Key decisions

Recover from :7770 (truth), NOT the bloated :8742 egm. The egm recovers only 11,532 nodes (dropped ~1,500 during the crash-loop). The working :7770 store has the full, current set.
NOT the stale snapshot.json (06:10). It predates today's ~30 design memories. Using it would silently lose today's work.
HARD durability gate: the recovery does NOT cut over until a fresh :7770 export is verified to contain today's memories (node IDs 5a649121, 47be987f, fcce29d0, d02ad0f6).
Delete nothing. All prior stores/binaries/snapshots retained as reversal assets.
:7770 is read/export-only during recovery — never modified, never killed.

4. Reversal assets (backups)

~/.neuron/engram-incident-backup-20260812-180043/ — pre-fix egm(458MB)+wal(44MB)+snapshot(65MB), APFS-cloned.
~/.neuron/engram-recovery-export-<ts>.json — fresh :7770 export (created Phase 1; the durable truth).
Moved-aside originals: neuron.egm/neuron.wal renamed (kept) during cutover.
~/.neuron/engram/snapshot.json (06:10) · snapshot.golden.json (Aug 3) · .sync-export.json (16:50).
Git: branch engram-tiered-storage, fix 9e28def; prior ~/.neuron/bin/engram binary retained.

5. Reversal paths (exact)

R1 — undo "crash-loop stopped": launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.neuron.engram.plist. (NOT recommended with the unfixed binary — it will crash-loop again.)

R2 — revert the code fix: git -C /tmp/engram-tiered-wt revert 9e28def. (NOT recommended — reintroduces the overflow crash. The fix is defensive and correct.)

R3 — roll back the live cutover to a safe state:

launchctl bootout gui/$(id -u)/ai.neuron.engram
Restore the prior store: move the freshly-imported store aside; restore the moved-aside originals OR the backup dir contents into ~/.neuron/engram/.
Restore the prior binary if it was replaced: copy the retained ~/.neuron/bin/engram back.
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.neuron.engram.plist; verify.

Note: restoring the pre-fix binary reintroduces the crash. The genuinely safe rollback state is fixed binary + the recovery export (which is the target state). To fully abort: bootout and leave :8742 DOWN — your memory is safe and served by :7770 regardless.

R4 — undo the tag: git tag -d engram-tiered-m8 (and delete remote tag if pushed).

If :7770 is ever affected (it should not be — export/read-only): it holds the truth; if needed, rebuild from ~/.neuron/engram-recovery-export-<ts>.json.

6. Validation (how Will tests "here")

After cutover: :8742 listens; node_count ≈ 12,825; today's memories findable by-id AND by-search; no crash-loop over ≥30s; write-survives-restart. Then Will can exercise retrieval/writes on his machine.

7. Customer-facing status

UNTOUCHED. No website, no customer service, no public deploy changed by any step here.

8. ACTUALS — recovery COMPLETE & VERIFIED (2026-08-12 ~19:18)

Durability gate PASSED. Proof the stale files were unusable: the 16:50 .sync-export.json and 06:10 snapshot.json contained zero of the 4 canary IDs. Fresh export path used: GET :7770/api/graph/edges → sidecar (never touched canonical snapshot).
Durable truth exports (sha256-verified): ~/.neuron/engram-recovery-export-20260812-190712.json (12,734 nodes, all canaries) and ~/.neuron/engram-recovery-export-precutover-20260812-191634.json (12,704 nodes, all canaries).
Fixed binary: container-capped fold of current server.el + fixed engram_store.c (9e28def), arm64, sha256 feafd0c9…, 0 errors.
Cutover: binary+plist backed up to ~/.neuron/backups/pre-recovery-cutover-20260812-191844; bloated originals renamed *.pre-recovery-* (NOT deleted); clean egm placed; fixed binary deployed; launchctl bootstrap.
Live state (independently verified): :8742 up (pid 31277), /health = ok, node_count 12,679 / edges 43,466 / embedded 4,290, all 4 design canaries findable by-id AND by-search, write-survives-restart PASS, no crash-loop (no crash reports post-cutover).
:7770 truth daemon: untouched (read-only GETs only), still serving.
Tag: engram-tiered-m8 created on 9e28def (LOCAL only — not pushed).
Reversal state: all backups + the moved-aside bloated egm retained. Safe abort at any time: launchctl bootout → :8742 down → memory still served by :7770. Full restore per §5.

Status: incident CLOSED. Memory recovered, durable, live. Customer-facing: untouched.

6.3 KiB Raw Blame History