docs(engram): cognitive architecture design, prior-art scan, M10 reification, M8 perf profile, cutover-reversal runbooks, and 2026-08-12 session record
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# Engram Recovery, Cutover & Build — Decisions & Reversal Runbook
|
||||
|
||||
**Date:** 2026-08-12 · **Owner:** Neuron (for Will) · **Status:** LIVING (finalized with actuals after cutover)
|
||||
|
||||
Per Will's standing rule: every change ships with what's happening, the decisions + rationale, and an
|
||||
exact path for reversal. **Advance authorization (Will, 2026-08-12):** promote to prod for Will's
|
||||
own testing — **NOT the website, no customer may see any of this.** Customer-facing surfaces stay frozen.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope & guardrails
|
||||
- **In scope (his testing env):** the local engram service `:8742` on Will's Mac, and the
|
||||
`engram-tiered-storage` build. Internal testing only.
|
||||
- **FROZEN — do NOT touch:** the marketing website, any customer-facing Cloud Run service, any public
|
||||
deploy. No customer exposure. Broader prod promotion (beyond Will's local testing) requires an
|
||||
explicit, separate go.
|
||||
|
||||
## 2. What's changing (and why)
|
||||
| # | Change | Rationale | Reversal (see §5) |
|
||||
|---|--------|-----------|-------------------|
|
||||
| 1 | Crash-loop stopped (`launchctl bootout ai.neuron.engram`) | 42 crashes/day; re-running a crashing WAL-replay over the store is the only corruption risk | R1 |
|
||||
| 2 | btree fix committed `9e28def` (branch `engram-tiered-storage`) | Root cause: `int_max_keys` /8 vs /16 → node overflow → stack smash | R2 |
|
||||
| 3 | Rebuild `:8742` store from `:7770` working-store fresh export + fixed binary; cut over | Working store (~12,825, incl. today) is the truth; bloated egm lost ~1,500 nodes | R3 |
|
||||
| 4 | Tag `engram-tiered-m8` after green cutover | The fix makes M8 boot real data | R4 |
|
||||
| 5 | (Forthcoming) M9 / M10 / M-INTEROCEPTION build | The cognitive architecture; each staged + tagged separately | per-milestone |
|
||||
|
||||
## 3. Key decisions
|
||||
- **Recover from `:7770` (truth), NOT the bloated `:8742` egm.** The egm recovers only 11,532 nodes
|
||||
(dropped ~1,500 during the crash-loop). The working `:7770` store has the full, current set.
|
||||
- **NOT the stale `snapshot.json` (06:10).** It predates today's ~30 design memories. Using it would
|
||||
silently lose today's work.
|
||||
- **HARD durability gate:** the recovery does NOT cut over until a fresh `:7770` export is verified to
|
||||
contain today's memories (node IDs `5a649121`, `47be987f`, `fcce29d0`, `d02ad0f6`).
|
||||
- **Delete nothing.** All prior stores/binaries/snapshots retained as reversal assets.
|
||||
- **`:7770` is read/export-only** during recovery — never modified, never killed.
|
||||
|
||||
## 4. Reversal assets (backups)
|
||||
- `~/.neuron/engram-incident-backup-20260812-180043/` — pre-fix egm(458MB)+wal(44MB)+snapshot(65MB), APFS-cloned.
|
||||
- `~/.neuron/engram-recovery-export-<ts>.json` — fresh `:7770` export (created Phase 1; the durable truth).
|
||||
- Moved-aside originals: `neuron.egm`/`neuron.wal` renamed (kept) during cutover.
|
||||
- `~/.neuron/engram/snapshot.json` (06:10) · `snapshot.golden.json` (Aug 3) · `.sync-export.json` (16:50).
|
||||
- Git: branch `engram-tiered-storage`, fix `9e28def`; prior `~/.neuron/bin/engram` binary retained.
|
||||
|
||||
## 5. Reversal paths (exact)
|
||||
|
||||
**R1 — undo "crash-loop stopped":** `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.neuron.engram.plist`.
|
||||
(NOT recommended with the *unfixed* binary — it will crash-loop again.)
|
||||
|
||||
**R2 — revert the code fix:** `git -C /tmp/engram-tiered-wt revert 9e28def`.
|
||||
(NOT recommended — reintroduces the overflow crash. The fix is defensive and correct.)
|
||||
|
||||
**R3 — roll back the live cutover to a safe state:**
|
||||
1. `launchctl bootout gui/$(id -u)/ai.neuron.engram`
|
||||
2. Restore the prior store: move the freshly-imported store aside; restore the moved-aside originals
|
||||
OR the backup dir contents into `~/.neuron/engram/`.
|
||||
3. Restore the prior binary if it was replaced: copy the retained `~/.neuron/bin/engram` back.
|
||||
4. `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.neuron.engram.plist`; verify.
|
||||
- **Note:** restoring the *pre-fix* binary reintroduces the crash. The genuinely safe rollback state is
|
||||
**fixed binary + the recovery export** (which is the target state). To fully abort: `bootout` and
|
||||
leave `:8742` DOWN — **your memory is safe and served by `:7770` regardless.**
|
||||
|
||||
**R4 — undo the tag:** `git tag -d engram-tiered-m8` (and delete remote tag if pushed).
|
||||
|
||||
**If `:7770` is ever affected** (it should not be — export/read-only): it holds the truth; if needed,
|
||||
rebuild from `~/.neuron/engram-recovery-export-<ts>.json`.
|
||||
|
||||
## 6. Validation (how Will tests "here")
|
||||
After cutover: `:8742` listens; node_count ≈ 12,825; today's memories findable by-id AND by-search;
|
||||
no crash-loop over ≥30s; write-survives-restart. Then Will can exercise retrieval/writes on his machine.
|
||||
|
||||
## 7. Customer-facing status
|
||||
**UNTOUCHED.** No website, no customer service, no public deploy changed by any step here.
|
||||
|
||||
---
|
||||
|
||||
## 8. ACTUALS — recovery COMPLETE & VERIFIED (2026-08-12 ~19:18)
|
||||
|
||||
- **Durability gate PASSED.** Proof the stale files were unusable: the 16:50 `.sync-export.json` and
|
||||
06:10 `snapshot.json` contained **zero** of the 4 canary IDs. Fresh export path used:
|
||||
`GET :7770/api/graph/edges` → sidecar (never touched canonical snapshot).
|
||||
- **Durable truth exports (sha256-verified):**
|
||||
`~/.neuron/engram-recovery-export-20260812-190712.json` (12,734 nodes, all canaries) and
|
||||
`~/.neuron/engram-recovery-export-precutover-20260812-191634.json` (12,704 nodes, all canaries).
|
||||
- **Fixed binary:** container-capped fold of current `server.el` + fixed `engram_store.c` (`9e28def`),
|
||||
arm64, sha256 `feafd0c9…`, 0 errors.
|
||||
- **Cutover:** binary+plist backed up to `~/.neuron/backups/pre-recovery-cutover-20260812-191844`;
|
||||
bloated originals renamed `*.pre-recovery-*` (NOT deleted); clean egm placed; fixed binary deployed;
|
||||
`launchctl bootstrap`.
|
||||
- **Live state (independently verified):** `:8742` up (pid 31277), `/health` = ok,
|
||||
**node_count 12,679 / edges 43,466 / embedded 4,290**, all 4 design canaries findable by-id AND
|
||||
by-search, **write-survives-restart PASS**, **no crash-loop** (no crash reports post-cutover).
|
||||
- **`:7770` truth daemon:** untouched (read-only GETs only), still serving.
|
||||
- **Tag:** `engram-tiered-m8` created on `9e28def` (LOCAL only — not pushed).
|
||||
- **Reversal state:** all backups + the moved-aside bloated egm retained. Safe abort at any time:
|
||||
`launchctl bootout` → `:8742` down → memory still served by `:7770`. Full restore per §5.
|
||||
|
||||
**Status: incident CLOSED. Memory recovered, durable, live. Customer-facing: untouched.**
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
# M-INTEROCEPTION — flags, defaults, and reversal runbook
|
||||
|
||||
Branch `engram-tiered-storage` (worktree `/tmp/engram-tiered-wt`), on trunk
|
||||
`f6a0777`. Six faces, each its own commit. Every behavior-changing feature is
|
||||
behind an env flag **default OFF = byte-identical to trunk** (proven per face);
|
||||
the read-only builtins are purely additive. NOT pushed, NOT tagged. The live
|
||||
`:8742` daemon, `~/.neuron/engram`, and launchctl were never touched — all
|
||||
verification ran on copies with throwaway HOME + /tmp dirs.
|
||||
|
||||
The server binary was rebuilt from the **byte-unchanged** `engram/dist/engram.c`
|
||||
plus the modified runtime and links cleanly, so these changes integrate into the
|
||||
real server without regenerating dist. Two HTTP routes are **deferred to cutover**
|
||||
because regenerating dist from `server.el` drifts ~285 lines with no source
|
||||
change (the prebuilt elc is a Linux x86-64 binary; a locally-built elc is a
|
||||
different compiler revision). The C builtins behind those routes are complete
|
||||
and tested via pure-C harnesses.
|
||||
|
||||
## Flags
|
||||
|
||||
| Flag | Default | Face | Effect when set |
|
||||
|------|---------|------|-----------------|
|
||||
| `ENGRAM_CONSOLIDATION` | `0` (off) | P1 | Enables the two-threshold promotion layer (ISE connection edges + permanence marking). |
|
||||
| `ENGRAM_CONSOL_CONN_MIN` | `0.6` | P1 | ISE salience needed to form connection edges. |
|
||||
| `ENGRAM_CONSOL_PERM_MIN` | `0.9` | P1 | ACT-R base-level needed to mark a node durable. |
|
||||
| `ENGRAM_CONSOL_WM_TOPK` | `5` | P1 | Max wm_top nodes a strong ISE wires to. |
|
||||
| `ENGRAM_CHRONOCEPTION` | `0` (off) | P2 | Enables field aging (`engram_age_field`) + reboot catch-up. |
|
||||
| `ENGRAM_CHRONO_TC` | `3600` (s) | P2 | Field cooling time-constant for `exp(-dt/TC)`. |
|
||||
| (none) | — | P0, P3, P4, P5 | Additive read-only builtins / observability; no flag. |
|
||||
|
||||
With all flags unset the runtime is byte-identical to trunk except for P4, which
|
||||
adds five backward-compatible fields to `/api/act-stats` (pure observability).
|
||||
|
||||
## Faces, commits, and how to disable / revert
|
||||
|
||||
| Face | Commit | Disable (no revert) | Revert |
|
||||
|------|--------|---------------------|--------|
|
||||
| P0 embeddings builtin `engram_scan_nodes_emb_json` | `c20cb3b` | n/a (additive, unused until route wired) | `git revert c20cb3b` |
|
||||
| P1 two-threshold consolidation | `5f6ce5c` | leave `ENGRAM_CONSOLIDATION` unset | `git revert 5f6ce5c` |
|
||||
| P2 chronoception field aging | `0af39df` | leave `ENGRAM_CHRONOCEPTION` unset | `git revert 0af39df` |
|
||||
| P3 drift-sensor primitive `engram_geo_displacement` | `816b258` | n/a (pure fn, only called if wired) | `git revert 816b258` |
|
||||
| P4 afferent counters in act-stats | `65ca0a3` | n/a (always on; observability only) | `git revert 65ca0a3` |
|
||||
| P5 dream-recall builtin `engram_dreams_json` | `77a4bc9` | n/a (additive, unused until route wired) | `git revert 77a4bc9` |
|
||||
|
||||
Reverts are independent and can be applied in any order (no cross-face code
|
||||
dependencies; each touches distinct functions).
|
||||
|
||||
## Data-side reversibility
|
||||
|
||||
- **P1 connection edges** carry `relation="hebbian-associate"`, `metadata`
|
||||
`{"origin":"consolidated-from-ISE"}`. Remove all with one query over that
|
||||
marker. They are also swept automatically with their ISE at the 48h prune
|
||||
unless the ISE was promoted to permanence.
|
||||
- **P1 permanence** marks a node durable via the metadata marker
|
||||
`consolidated-from-ISE`. Demote by clearing the marker; the node then becomes
|
||||
prunable again. No struct/schema change — the marker rides in existing
|
||||
metadata and survives the store round-trip.
|
||||
- **P2 last-tick** persists to a sidecar file `chrono_last_tick` in the data
|
||||
dir. Delete it to reset catch-up; it is written only when the flag is set.
|
||||
|
||||
## Deferred to cutover (elc-drift blocker)
|
||||
|
||||
- `GET /api/embeddings` and `GET /api/graph/dump` → back onto
|
||||
`engram_scan_nodes_emb_json` (P0).
|
||||
- `GET /api/dreams?since=` → back onto `engram_dreams_json` (P5).
|
||||
|
||||
Wire by hand-patching `engram/dist/engram.c` surgically (mirror an existing
|
||||
route like `route_scan_nodes`), leaving all other dist lines byte-identical, and
|
||||
editing `server.el` as source of truth. Do NOT full-regenerate dist.
|
||||
|
||||
## Known follow-up (P3, honestly flagged)
|
||||
|
||||
The drift sensor primitive is complete and tested, but a **live** self-drift
|
||||
reading needs a persisted `SelfAnchor` baseline descriptor to compare "now"
|
||||
against, and **no persisted self node / anchored self-neighborhood exists yet**.
|
||||
A self was NOT fabricated. Capturing a durable SelfAnchor snapshot and wiring an
|
||||
`ENGRAM_DRIFT_SENSOR` live reading is the remaining work before P3 goes live.
|
||||
Reference in New Issue
Block a user