will.anderson
4965600d65
Reversal runbook for the ENGRAM_GEOMETRY_PRIMING cutover (default OFF, reversible flag flip; exact rollback) and the A/B perf profile: default-OFF binary GO (byte-identical to M8), enabling the flag NO-GO on latency (3.2x/13x) with no demonstrated recall benefit; safety/sanitizer clean.
131 lines
7.4 KiB
Markdown
131 lines
7.4 KiB
Markdown
# Runbook — M9 Geometry Priming: Cutover & Reversal
|
||
|
||
**Date:** 2026-08-12
|
||
**Component:** engram activation (`lang/runtime/el_runtime.c` → `engram_activate`)
|
||
**Branch:** `engram-tiered-storage`
|
||
**Flag:** `ENGRAM_GEOMETRY_PRIMING` (env, **default OFF = current M8 behavior, byte-identical**)
|
||
**Blast radius if wrong:** the core recall path of Will's live memory. Treat with according care.
|
||
|
||
---
|
||
|
||
## 1. What changes
|
||
|
||
This is the first behavior-changing step that touches the **core recall/priming** path.
|
||
It wires the M9 **mean-centered relational-neighborhood geometry** (`engram_geometry.c`,
|
||
shipped commits `2a4c5c6` foundation + `8cae0f9` centering) into `engram_activate`
|
||
**seed selection**, and it does so **behind a reversible env flag that defaults OFF**.
|
||
|
||
- **Flag OFF (default):** `engram_activate` runs the exact M8 code path. The new code is a
|
||
single `if (eg_geometry_priming_on() && …)` block that short-circuits on the first term,
|
||
plus a few unused static helpers and one zero-initialized counter. **No behavioral change.**
|
||
- **Flag ON (`ENGRAM_GEOMETRY_PRIMING=1`):** after M8 produces its ANN seed set, the
|
||
**centered** geometry of that neighborhood is computed and used to, **composing with**
|
||
(never replacing) M8's ANN candidate generation:
|
||
1. **Damp off-domain seeds** — each M8 seed's activation is scaled by a **damp-only**
|
||
factor `lo + (1-lo)·membership ∈ [lo, 1]` (default `lo=0.5`). The neighborhood anchor
|
||
(membership→1) is unchanged; seeds that are semantically off-domain **in the centered
|
||
frame** lose weight. This is the disambiguation win. It can only *sharpen*, never amplify.
|
||
2. **Prime the neighborhood sub-threshold** — descriptor members not already seeded get a
|
||
**warm floor** `activation = membership · scale` (default `scale=0.08`, strictly below the
|
||
WM promotion gate `0.15`), capped at `ENGRAM_GEO_PRIME_MAX` (default 32), ISE nodes skipped.
|
||
They enter the frontier so a warm gradient spreads one hop, then dies at the BFS `0.02`
|
||
cutoff. **Safe because the BFS keeps the max** (`el_runtime.c` `if (!reached || new_act >
|
||
best_bg)`): priming only *raises a floor*, it can never cap a stronger legitimate activation.
|
||
|
||
### Why default-OFF makes deploying the binary behavior-neutral
|
||
Because every line of the new logic is gated behind `ENGRAM_GEOMETRY_PRIMING`, **deploying the
|
||
new binary with the flag unset is behavior-neutral** — it is the M8 activation path, verified
|
||
byte-identical in the A/B (flag-OFF promoted-node sets equal the pre-M9 M8 binary's, per-query).
|
||
Enabling the geometry is then a **single reversible flag flip**, not a redeploy.
|
||
|
||
---
|
||
|
||
## 2. The flag
|
||
|
||
| Env var | Default | Effect |
|
||
|---|---|---|
|
||
| `ENGRAM_GEOMETRY_PRIMING` | unset / `0` | **OFF** — exact M8 behavior. |
|
||
| `ENGRAM_GEOMETRY_PRIMING=1` | — | **ON** — centered-geometry seed damping + sub-threshold priming. |
|
||
| `ENGRAM_GEO_SEED_LO` | `0.5` | Seed damp floor (factor ∈ [LO,1]). `1.0` disables damping. |
|
||
| `ENGRAM_GEO_PRIME_SCALE` | `0.08` | Warm-floor scale; clamped `(0, WM_gate=0.15)`. |
|
||
| `ENGRAM_GEO_PRIME_MAX` | `32` | Max primed members per activation (0 disables priming). |
|
||
|
||
The flag is read **once** per process (cached), so enabling/disabling requires a **process
|
||
restart** of the engram service — it is not hot-togglable within a running process.
|
||
|
||
---
|
||
|
||
## 3. How to enable live (deliberate, reversible)
|
||
|
||
> Precondition: the default-OFF binary has already been deployed and is running the M8 path
|
||
> healthily (behavior-neutral deploy). Do this only with Will present, per the standing rails.
|
||
|
||
1. **Snapshot first** (always, before any activation-behavior change):
|
||
`~/.neuron/backups/pre-geometry-priming-<ts>/` ← copy `neuron.egm`, `neuron.wal`,
|
||
the current `engram` binary, and `ai.neuron.engram.plist`.
|
||
2. Add `ENGRAM_GEOMETRY_PRIMING=1` to the engram service environment
|
||
(`ai.neuron.engram.plist` `EnvironmentVariables`).
|
||
3. `launchctl bootout gui/$(id -u)/ai.neuron.engram` → `launchctl bootstrap …` (restart so the
|
||
flag is re-read).
|
||
4. **Verify:** service comes up serving the same node count; `/api/act-stats` shows sane WM
|
||
(promoted ≤ 24); spot-check 3–4 real queries return coherent results; watch one heartbeat
|
||
cycle for crashes/latency. The `geo_primed` counter (if surfaced) should be > 0.
|
||
|
||
---
|
||
|
||
## 4. Rollback (exact steps)
|
||
|
||
Rollback is a **flag flip**, not a data operation — the store is untouched by enabling the flag,
|
||
and priming is a read-mostly, bounded, sub-threshold addition.
|
||
|
||
**Fast path (preferred) — disable the flag:**
|
||
1. Remove `ENGRAM_GEOMETRY_PRIMING` (or set `=0`) from `ai.neuron.engram.plist`.
|
||
2. `launchctl bootout … && launchctl bootstrap …`.
|
||
3. Verify: service healthy, activation is the M8 path again. **Done** — no data change to undo.
|
||
|
||
**Full path (only if the binary itself is suspect) — redeploy prior binary:**
|
||
1. `launchctl bootout gui/$(id -u)/ai.neuron.engram`.
|
||
2. Restore the prior `engram` binary from `~/.neuron/backups/pre-geometry-priming-<ts>/`.
|
||
3. Restore `ai.neuron.engram.plist` from the same backup (flag absent).
|
||
4. `launchctl bootstrap …`; verify node count + a self-traversal + write-survives-restart.
|
||
5. If (and only if) the store was somehow mutated: restore `neuron.egm` + `neuron.wal` from the
|
||
backup. **Note:** enabling the flag does not write geometry to the store, so this step is
|
||
expected to be unnecessary — the primed activations are per-call and non-persistent beyond the
|
||
ordinary `background_activation`/WM write-back that M8 already does.
|
||
|
||
**Rollback triggers:** any crash/hang in `engram_activate`; WM promotion count exceeding the cap
|
||
or collapsing; a measured recall/coherence regression vs the OFF baseline; unacceptable latency
|
||
increase; any ASan/UBSan report under the flag.
|
||
|
||
---
|
||
|
||
## 5. Reversibility guarantees (why this is low-risk to deploy, higher-care to enable)
|
||
|
||
- **Deploy (flag OFF):** byte-identical to M8. Verified in A/B. Zero-risk redeploy.
|
||
- **Enable (flag ON):** bounded and composable —
|
||
- never removes an M8 seed (damp-only, factor ≥ `lo` > 0);
|
||
- never amplifies a seed above its M8 value (factor ≤ 1);
|
||
- priming is strictly sub-threshold (`scale < WM_gate`) and capped (`PRIME_MAX`);
|
||
- priming raises a floor only (BFS keeps max) — cannot cap real activation;
|
||
- does not write geometry to the durable store;
|
||
- degrades to exact M8 behavior for any call where the paged store / centered global mean /
|
||
embedder is unavailable (guarded, not crashing).
|
||
- **Disable:** one env removal + restart; no data to reconcile.
|
||
|
||
---
|
||
|
||
## 6. Known caveats / uncertainties (flagged — this is the memory core)
|
||
|
||
- **Perf cost of ON:** the descriptor (covariance eigensolve + `store_get_node` paged reads per
|
||
member) runs on **every** activation when the flag is ON. See
|
||
`docs/architecture/design/perf/engram-geometry-priming-profile.md` for the measured OFF-vs-ON
|
||
latency. If that delta is unacceptable, keep the flag OFF (deploy stays valid) and revisit with
|
||
a cached/periodic descriptor.
|
||
- **Two-store consistency:** the descriptor reads embeddings from the **paged** store while the
|
||
ANN index is over the **resident** array. This-call backfilled embeddings can lag the paged
|
||
store by ≤ `ENGRAM_EMBED_BACKFILL_PER_CALL` nodes — the same staleness class as the M8 vindex,
|
||
and it can only omit a member, never mis-prime.
|
||
- **Damp tuning:** `lo=0.5` can at most halve an off-domain seed. If a coherence regression is
|
||
observed, raise `ENGRAM_GEO_SEED_LO` toward `1.0` (→ priming-only, no damping) before disabling
|
||
entirely.
|