docs(engram): cognitive architecture design, prior-art scan, M10 reification, M8 perf profile, cutover-reversal runbooks, and 2026-08-12 session record
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# Engram Recovery, Cutover & Build — Decisions & Reversal Runbook
|
||||
|
||||
**Date:** 2026-08-12 · **Owner:** Neuron (for Will) · **Status:** LIVING (finalized with actuals after cutover)
|
||||
|
||||
Per Will's standing rule: every change ships with what's happening, the decisions + rationale, and an
|
||||
exact path for reversal. **Advance authorization (Will, 2026-08-12):** promote to prod for Will's
|
||||
own testing — **NOT the website, no customer may see any of this.** Customer-facing surfaces stay frozen.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope & guardrails
|
||||
- **In scope (his testing env):** the local engram service `:8742` on Will's Mac, and the
|
||||
`engram-tiered-storage` build. Internal testing only.
|
||||
- **FROZEN — do NOT touch:** the marketing website, any customer-facing Cloud Run service, any public
|
||||
deploy. No customer exposure. Broader prod promotion (beyond Will's local testing) requires an
|
||||
explicit, separate go.
|
||||
|
||||
## 2. What's changing (and why)
|
||||
| # | Change | Rationale | Reversal (see §5) |
|
||||
|---|--------|-----------|-------------------|
|
||||
| 1 | Crash-loop stopped (`launchctl bootout ai.neuron.engram`) | 42 crashes/day; re-running a crashing WAL-replay over the store is the only corruption risk | R1 |
|
||||
| 2 | btree fix committed `9e28def` (branch `engram-tiered-storage`) | Root cause: `int_max_keys` /8 vs /16 → node overflow → stack smash | R2 |
|
||||
| 3 | Rebuild `:8742` store from `:7770` working-store fresh export + fixed binary; cut over | Working store (~12,825, incl. today) is the truth; bloated egm lost ~1,500 nodes | R3 |
|
||||
| 4 | Tag `engram-tiered-m8` after green cutover | The fix makes M8 boot real data | R4 |
|
||||
| 5 | (Forthcoming) M9 / M10 / M-INTEROCEPTION build | The cognitive architecture; each staged + tagged separately | per-milestone |
|
||||
|
||||
## 3. Key decisions
|
||||
- **Recover from `:7770` (truth), NOT the bloated `:8742` egm.** The egm recovers only 11,532 nodes
|
||||
(dropped ~1,500 during the crash-loop). The working `:7770` store has the full, current set.
|
||||
- **NOT the stale `snapshot.json` (06:10).** It predates today's ~30 design memories. Using it would
|
||||
silently lose today's work.
|
||||
- **HARD durability gate:** the recovery does NOT cut over until a fresh `:7770` export is verified to
|
||||
contain today's memories (node IDs `5a649121`, `47be987f`, `fcce29d0`, `d02ad0f6`).
|
||||
- **Delete nothing.** All prior stores/binaries/snapshots retained as reversal assets.
|
||||
- **`:7770` is read/export-only** during recovery — never modified, never killed.
|
||||
|
||||
## 4. Reversal assets (backups)
|
||||
- `~/.neuron/engram-incident-backup-20260812-180043/` — pre-fix egm(458MB)+wal(44MB)+snapshot(65MB), APFS-cloned.
|
||||
- `~/.neuron/engram-recovery-export-<ts>.json` — fresh `:7770` export (created Phase 1; the durable truth).
|
||||
- Moved-aside originals: `neuron.egm`/`neuron.wal` renamed (kept) during cutover.
|
||||
- `~/.neuron/engram/snapshot.json` (06:10) · `snapshot.golden.json` (Aug 3) · `.sync-export.json` (16:50).
|
||||
- Git: branch `engram-tiered-storage`, fix `9e28def`; prior `~/.neuron/bin/engram` binary retained.
|
||||
|
||||
## 5. Reversal paths (exact)
|
||||
|
||||
**R1 — undo "crash-loop stopped":** `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.neuron.engram.plist`.
|
||||
(NOT recommended with the *unfixed* binary — it will crash-loop again.)
|
||||
|
||||
**R2 — revert the code fix:** `git -C /tmp/engram-tiered-wt revert 9e28def`.
|
||||
(NOT recommended — reintroduces the overflow crash. The fix is defensive and correct.)
|
||||
|
||||
**R3 — roll back the live cutover to a safe state:**
|
||||
1. `launchctl bootout gui/$(id -u)/ai.neuron.engram`
|
||||
2. Restore the prior store: move the freshly-imported store aside; restore the moved-aside originals
|
||||
OR the backup dir contents into `~/.neuron/engram/`.
|
||||
3. Restore the prior binary if it was replaced: copy the retained `~/.neuron/bin/engram` back.
|
||||
4. `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.neuron.engram.plist`; verify.
|
||||
- **Note:** restoring the *pre-fix* binary reintroduces the crash. The genuinely safe rollback state is
|
||||
**fixed binary + the recovery export** (which is the target state). To fully abort: `bootout` and
|
||||
leave `:8742` DOWN — **your memory is safe and served by `:7770` regardless.**
|
||||
|
||||
**R4 — undo the tag:** `git tag -d engram-tiered-m8` (and delete remote tag if pushed).
|
||||
|
||||
**If `:7770` is ever affected** (it should not be — export/read-only): it holds the truth; if needed,
|
||||
rebuild from `~/.neuron/engram-recovery-export-<ts>.json`.
|
||||
|
||||
## 6. Validation (how Will tests "here")
|
||||
After cutover: `:8742` listens; node_count ≈ 12,825; today's memories findable by-id AND by-search;
|
||||
no crash-loop over ≥30s; write-survives-restart. Then Will can exercise retrieval/writes on his machine.
|
||||
|
||||
## 7. Customer-facing status
|
||||
**UNTOUCHED.** No website, no customer service, no public deploy changed by any step here.
|
||||
|
||||
---
|
||||
|
||||
## 8. ACTUALS — recovery COMPLETE & VERIFIED (2026-08-12 ~19:18)
|
||||
|
||||
- **Durability gate PASSED.** Proof the stale files were unusable: the 16:50 `.sync-export.json` and
|
||||
06:10 `snapshot.json` contained **zero** of the 4 canary IDs. Fresh export path used:
|
||||
`GET :7770/api/graph/edges` → sidecar (never touched canonical snapshot).
|
||||
- **Durable truth exports (sha256-verified):**
|
||||
`~/.neuron/engram-recovery-export-20260812-190712.json` (12,734 nodes, all canaries) and
|
||||
`~/.neuron/engram-recovery-export-precutover-20260812-191634.json` (12,704 nodes, all canaries).
|
||||
- **Fixed binary:** container-capped fold of current `server.el` + fixed `engram_store.c` (`9e28def`),
|
||||
arm64, sha256 `feafd0c9…`, 0 errors.
|
||||
- **Cutover:** binary+plist backed up to `~/.neuron/backups/pre-recovery-cutover-20260812-191844`;
|
||||
bloated originals renamed `*.pre-recovery-*` (NOT deleted); clean egm placed; fixed binary deployed;
|
||||
`launchctl bootstrap`.
|
||||
- **Live state (independently verified):** `:8742` up (pid 31277), `/health` = ok,
|
||||
**node_count 12,679 / edges 43,466 / embedded 4,290**, all 4 design canaries findable by-id AND
|
||||
by-search, **write-survives-restart PASS**, **no crash-loop** (no crash reports post-cutover).
|
||||
- **`:7770` truth daemon:** untouched (read-only GETs only), still serving.
|
||||
- **Tag:** `engram-tiered-m8` created on `9e28def` (LOCAL only — not pushed).
|
||||
- **Reversal state:** all backups + the moved-aside bloated egm retained. Safe abort at any time:
|
||||
`launchctl bootout` → `:8742` down → memory still served by `:7770`. Full restore per §5.
|
||||
|
||||
**Status: incident CLOSED. Memory recovered, durable, live. Customer-facing: untouched.**
|
||||
|
||||
Reference in New Issue
Block a user