Files
neuron/docs/runbooks/2026-08-12-geometry-priming-cutover-reversal.md
T
will.anderson 4965600d65 docs(engram): M9 geometry-priming reversal runbook + A/B perf profile
Reversal runbook for the ENGRAM_GEOMETRY_PRIMING cutover (default OFF, reversible
flag flip; exact rollback) and the A/B perf profile: default-OFF binary GO
(byte-identical to M8), enabling the flag NO-GO on latency (3.2x/13x) with no
demonstrated recall benefit; safety/sanitizer clean.
2026-08-12 20:29:16 -05:00

7.4 KiB
Raw Blame History

Runbook — M9 Geometry Priming: Cutover & Reversal

Date: 2026-08-12 Component: engram activation (lang/runtime/el_runtime.cengram_activate) Branch: engram-tiered-storage Flag: ENGRAM_GEOMETRY_PRIMING (env, default OFF = current M8 behavior, byte-identical) Blast radius if wrong: the core recall path of Will's live memory. Treat with according care.


1. What changes

This is the first behavior-changing step that touches the core recall/priming path. It wires the M9 mean-centered relational-neighborhood geometry (engram_geometry.c, shipped commits 2a4c5c6 foundation + 8cae0f9 centering) into engram_activate seed selection, and it does so behind a reversible env flag that defaults OFF.

  • Flag OFF (default): engram_activate runs the exact M8 code path. The new code is a single if (eg_geometry_priming_on() && …) block that short-circuits on the first term, plus a few unused static helpers and one zero-initialized counter. No behavioral change.
  • Flag ON (ENGRAM_GEOMETRY_PRIMING=1): after M8 produces its ANN seed set, the centered geometry of that neighborhood is computed and used to, composing with (never replacing) M8's ANN candidate generation:
    1. Damp off-domain seeds — each M8 seed's activation is scaled by a damp-only factor lo + (1-lo)·membership ∈ [lo, 1] (default lo=0.5). The neighborhood anchor (membership→1) is unchanged; seeds that are semantically off-domain in the centered frame lose weight. This is the disambiguation win. It can only sharpen, never amplify.
    2. Prime the neighborhood sub-threshold — descriptor members not already seeded get a warm floor activation = membership · scale (default scale=0.08, strictly below the WM promotion gate 0.15), capped at ENGRAM_GEO_PRIME_MAX (default 32), ISE nodes skipped. They enter the frontier so a warm gradient spreads one hop, then dies at the BFS 0.02 cutoff. Safe because the BFS keeps the max (el_runtime.c if (!reached || new_act > best_bg)): priming only raises a floor, it can never cap a stronger legitimate activation.

Why default-OFF makes deploying the binary behavior-neutral

Because every line of the new logic is gated behind ENGRAM_GEOMETRY_PRIMING, deploying the new binary with the flag unset is behavior-neutral — it is the M8 activation path, verified byte-identical in the A/B (flag-OFF promoted-node sets equal the pre-M9 M8 binary's, per-query). Enabling the geometry is then a single reversible flag flip, not a redeploy.


2. The flag

Env var Default Effect
ENGRAM_GEOMETRY_PRIMING unset / 0 OFF — exact M8 behavior.
ENGRAM_GEOMETRY_PRIMING=1 ON — centered-geometry seed damping + sub-threshold priming.
ENGRAM_GEO_SEED_LO 0.5 Seed damp floor (factor ∈ [LO,1]). 1.0 disables damping.
ENGRAM_GEO_PRIME_SCALE 0.08 Warm-floor scale; clamped (0, WM_gate=0.15).
ENGRAM_GEO_PRIME_MAX 32 Max primed members per activation (0 disables priming).

The flag is read once per process (cached), so enabling/disabling requires a process restart of the engram service — it is not hot-togglable within a running process.


3. How to enable live (deliberate, reversible)

Precondition: the default-OFF binary has already been deployed and is running the M8 path healthily (behavior-neutral deploy). Do this only with Will present, per the standing rails.

  1. Snapshot first (always, before any activation-behavior change): ~/.neuron/backups/pre-geometry-priming-<ts>/ ← copy neuron.egm, neuron.wal, the current engram binary, and ai.neuron.engram.plist.
  2. Add ENGRAM_GEOMETRY_PRIMING=1 to the engram service environment (ai.neuron.engram.plist EnvironmentVariables).
  3. launchctl bootout gui/$(id -u)/ai.neuron.engramlaunchctl bootstrap … (restart so the flag is re-read).
  4. Verify: service comes up serving the same node count; /api/act-stats shows sane WM (promoted ≤ 24); spot-check 34 real queries return coherent results; watch one heartbeat cycle for crashes/latency. The geo_primed counter (if surfaced) should be > 0.

4. Rollback (exact steps)

Rollback is a flag flip, not a data operation — the store is untouched by enabling the flag, and priming is a read-mostly, bounded, sub-threshold addition.

Fast path (preferred) — disable the flag:

  1. Remove ENGRAM_GEOMETRY_PRIMING (or set =0) from ai.neuron.engram.plist.
  2. launchctl bootout … && launchctl bootstrap ….
  3. Verify: service healthy, activation is the M8 path again. Done — no data change to undo.

Full path (only if the binary itself is suspect) — redeploy prior binary:

  1. launchctl bootout gui/$(id -u)/ai.neuron.engram.
  2. Restore the prior engram binary from ~/.neuron/backups/pre-geometry-priming-<ts>/.
  3. Restore ai.neuron.engram.plist from the same backup (flag absent).
  4. launchctl bootstrap …; verify node count + a self-traversal + write-survives-restart.
  5. If (and only if) the store was somehow mutated: restore neuron.egm + neuron.wal from the backup. Note: enabling the flag does not write geometry to the store, so this step is expected to be unnecessary — the primed activations are per-call and non-persistent beyond the ordinary background_activation/WM write-back that M8 already does.

Rollback triggers: any crash/hang in engram_activate; WM promotion count exceeding the cap or collapsing; a measured recall/coherence regression vs the OFF baseline; unacceptable latency increase; any ASan/UBSan report under the flag.


5. Reversibility guarantees (why this is low-risk to deploy, higher-care to enable)

  • Deploy (flag OFF): byte-identical to M8. Verified in A/B. Zero-risk redeploy.
  • Enable (flag ON): bounded and composable —
    • never removes an M8 seed (damp-only, factor ≥ lo > 0);
    • never amplifies a seed above its M8 value (factor ≤ 1);
    • priming is strictly sub-threshold (scale < WM_gate) and capped (PRIME_MAX);
    • priming raises a floor only (BFS keeps max) — cannot cap real activation;
    • does not write geometry to the durable store;
    • degrades to exact M8 behavior for any call where the paged store / centered global mean / embedder is unavailable (guarded, not crashing).
  • Disable: one env removal + restart; no data to reconcile.

6. Known caveats / uncertainties (flagged — this is the memory core)

  • Perf cost of ON: the descriptor (covariance eigensolve + store_get_node paged reads per member) runs on every activation when the flag is ON. See docs/architecture/design/perf/engram-geometry-priming-profile.md for the measured OFF-vs-ON latency. If that delta is unacceptable, keep the flag OFF (deploy stays valid) and revisit with a cached/periodic descriptor.
  • Two-store consistency: the descriptor reads embeddings from the paged store while the ANN index is over the resident array. This-call backfilled embeddings can lag the paged store by ≤ ENGRAM_EMBED_BACKFILL_PER_CALL nodes — the same staleness class as the M8 vindex, and it can only omit a member, never mis-prime.
  • Damp tuning: lo=0.5 can at most halve an off-domain seed. If a coherence regression is observed, raise ENGRAM_GEO_SEED_LO toward 1.0 (→ priming-only, no damping) before disabling entirely.