diff --git a/docs/architecture/design/perf/engram-geometry-priming-profile.md b/docs/architecture/design/perf/engram-geometry-priming-profile.md new file mode 100644 index 0000000..3cdfc0b --- /dev/null +++ b/docs/architecture/design/perf/engram-geometry-priming-profile.md @@ -0,0 +1,97 @@ +# Perf Profile — M9 Geometry Priming (ENGRAM_GEOMETRY_PRIMING) + +**Date:** 2026-08-12 +**Branch:** `engram-tiered-storage` +**Change:** `ENGRAM_GEOMETRY_PRIMING` (default OFF) in `el_runtime.c` `engram_activate` + `engram_geometry.c` +**Method:** A/B over 15 representative queries against a **copy** of the recovered store +(`~/.neuron/engram/.neuron.egm.disabled`, ~4190 embedded nodes, 768-d nomic-embed-text), +throwaway HOME, ports 48799/48800. **Live `:8742` never touched.** `engram.c` (folded from +`server.el`) reused byte-identical across M8 and M9, so the only variable is `el_runtime.c`. + +Three configs: **A** = M9 flag OFF · **B** = M9 flag ON (`=1`) · **C** = pre-M9 M8 baseline binary. + +--- + +## Build + +| Artifact | Result | +|---|---| +| M9 `-O2` link (`… engram_geometry.c … -lssl -lcrypto -lcurl -lpthread -lm`) | rc=0, 499,720 B arm64 | +| ASan/UBSan link (`-fsanitize=address,undefined -O1`) | rc=0, 1,945,616 B | +| Warnings from `el_runtime.c` / `engram_geometry.c` | **0** (3 pre-existing `-Wparentheses-equality` in generated `engram.c` only) | +| `nm`: `engram_geo_mean_build`, `engram_geometry_descriptor` | present (T); `eg_geometry_priming_on` inlined (static-local `.cached` present in both binaries) | + +> Note: the bare `cc … -lm` link fails with undefined `_curl_*` — `el_runtime.c` uses libcurl for +> the ollama embedder. The canonical link must include `-lssl -lcrypto -lcurl` (per `link.sh`). + +--- + +## Latency (wall-clock, `curl -w %{time_total}`, 15 queries) + +| config | median | p90 | min | max | +|---|---|---|---|---| +| **A — M9 OFF** | **77.8 ms** | 80.5 ms | 71.1 | 84.2 | +| C — M8 baseline | 76.0 ms | 81.2 ms | 71.4 | 91.4 | +| **B — M9 ON** | **249.6 ms** | **1039.2 ms** | 169.2 | **1256.3** | + +- **OFF adds zero cost:** 77.8 ms vs M8 76.0 ms — within noise. The flag is free when unset. +- **ON regresses hard:** **3.21x median** (+171.8 ms), **~13x p90** (80 → 1039 ms), max **1.26 s**. +- The warm-cache path (global mean already built) is ~0.5 s; the cold path pays the full + `engram_geo_mean_build` scan (O(N·dim) over ~4190 × 768). The persistent per-query cost is the + **descriptor** itself — covariance eigensolve over up to `max_members` (400) × 768-d plus one + `store_get_node` **paged read per member** — run on *every* activation while the flag is ON. + +--- + +## Retrieval quality (the win it was supposed to buy) + +**Coherence** — mean pairwise cosine in centered space, top-20 by activation strength +(node embeddings re-derived via nomic-embed-text; centered against the mean of the gathered +result set — the *true* store-wide mean is not exposed by the API, flagged as an approximation): + +| | OFF | ON | Δ | +|---|---|---|---| +| mean over 15 queries | 0.1067 | 0.1114 | **+0.0047 (noise)** | +| queries where ON > OFF | — | — | **4 / 15** | + +Two real sparse-cue wins (`self identity values` +0.118, `hebbian learning edges` +0.064), but the +**polysemous cues — the disambiguation target — are mostly flat or down.** + +**Disambiguation** — no clean "scope to one sense" pattern on polysemous cues. Additions/drops are +small (±2..8 of 300-item sets) and not sense-coherent (e.g. `memory` gains some on-domain nodes but +also infra items; `core` similar). + +**Count shift:** ON adds sub-threshold neighbors to sparse cues (+3..+4) and trims a few from dense +polysemous cues (−1..−3) — consistent with priming warming sparse neighborhoods and damping +off-domain seeds on dense ones, but the net does not move measured coherence. + +--- + +## Correctness / safety (all pass) + +| Check | Result | +|---|---| +| Byte-identical: **A (OFF) == C (M8)** result id sequence + order, all 15 queries (incl. 301/294/263-item sets) | **PASS** (only wall-clock ACT-R fields differ; `activation_strength` max \|Δ\| = 2e-5) | +| WM `promoted` ≤ 24 under ON | holds (exactly 24 on dense cues) | +| Queries with results under OFF → empty under ON | 0 | +| Crash / hang under ON | none (max hops = 1) | +| ASan + UBSan under ON (cold build + warm descriptor paths) | **CLEAN** — no report | + +--- + +## Conclusion + +- **Deploy default-OFF binary: GO.** Byte-identical to M8, zero cost off, clean build, sanitizer clean. +- **Enable flag: NO-GO (for now).** 3.21x median / ~13x p90 latency for no reliable quality gain + (coherence +0.0047 mean = noise; no clean disambiguation). Correctness/safety are fine — it simply + does not earn its cost. **This is a cost/benefit NO-GO, not a defect.** + +### Prerequisites before re-evaluating the flag +1. **Amortize the descriptor cost.** The per-query geo-mean build + eigensolve + paged reads + dominate. Cache the neighborhood descriptor (it is the M10 cell-assembly cache's job) and/or + compute geometry periodically/off-hot-path rather than on every `engram_activate`. +2. **Center against the true store-wide mean** (the `GeoMeanCache` already computes it) rather than + a per-query gathered-set approximation, and re-measure coherence — the current signal may be + understated by the approximation. +3. **Re-tune** `ENGRAM_GEO_SEED_LO` / `PRIME_SCALE` / `PRIME_MAX` and re-measure only after (1), + so tuning is not chasing latency noise. diff --git a/docs/runbooks/2026-08-12-geometry-priming-cutover-reversal.md b/docs/runbooks/2026-08-12-geometry-priming-cutover-reversal.md new file mode 100644 index 0000000..863b698 --- /dev/null +++ b/docs/runbooks/2026-08-12-geometry-priming-cutover-reversal.md @@ -0,0 +1,130 @@ +# Runbook — M9 Geometry Priming: Cutover & Reversal + +**Date:** 2026-08-12 +**Component:** engram activation (`lang/runtime/el_runtime.c` → `engram_activate`) +**Branch:** `engram-tiered-storage` +**Flag:** `ENGRAM_GEOMETRY_PRIMING` (env, **default OFF = current M8 behavior, byte-identical**) +**Blast radius if wrong:** the core recall path of Will's live memory. Treat with according care. + +--- + +## 1. What changes + +This is the first behavior-changing step that touches the **core recall/priming** path. +It wires the M9 **mean-centered relational-neighborhood geometry** (`engram_geometry.c`, +shipped commits `2a4c5c6` foundation + `8cae0f9` centering) into `engram_activate` +**seed selection**, and it does so **behind a reversible env flag that defaults OFF**. + +- **Flag OFF (default):** `engram_activate` runs the exact M8 code path. The new code is a + single `if (eg_geometry_priming_on() && …)` block that short-circuits on the first term, + plus a few unused static helpers and one zero-initialized counter. **No behavioral change.** +- **Flag ON (`ENGRAM_GEOMETRY_PRIMING=1`):** after M8 produces its ANN seed set, the + **centered** geometry of that neighborhood is computed and used to, **composing with** + (never replacing) M8's ANN candidate generation: + 1. **Damp off-domain seeds** — each M8 seed's activation is scaled by a **damp-only** + factor `lo + (1-lo)·membership ∈ [lo, 1]` (default `lo=0.5`). The neighborhood anchor + (membership→1) is unchanged; seeds that are semantically off-domain **in the centered + frame** lose weight. This is the disambiguation win. It can only *sharpen*, never amplify. + 2. **Prime the neighborhood sub-threshold** — descriptor members not already seeded get a + **warm floor** `activation = membership · scale` (default `scale=0.08`, strictly below the + WM promotion gate `0.15`), capped at `ENGRAM_GEO_PRIME_MAX` (default 32), ISE nodes skipped. + They enter the frontier so a warm gradient spreads one hop, then dies at the BFS `0.02` + cutoff. **Safe because the BFS keeps the max** (`el_runtime.c` `if (!reached || new_act > + best_bg)`): priming only *raises a floor*, it can never cap a stronger legitimate activation. + +### Why default-OFF makes deploying the binary behavior-neutral +Because every line of the new logic is gated behind `ENGRAM_GEOMETRY_PRIMING`, **deploying the +new binary with the flag unset is behavior-neutral** — it is the M8 activation path, verified +byte-identical in the A/B (flag-OFF promoted-node sets equal the pre-M9 M8 binary's, per-query). +Enabling the geometry is then a **single reversible flag flip**, not a redeploy. + +--- + +## 2. The flag + +| Env var | Default | Effect | +|---|---|---| +| `ENGRAM_GEOMETRY_PRIMING` | unset / `0` | **OFF** — exact M8 behavior. | +| `ENGRAM_GEOMETRY_PRIMING=1` | — | **ON** — centered-geometry seed damping + sub-threshold priming. | +| `ENGRAM_GEO_SEED_LO` | `0.5` | Seed damp floor (factor ∈ [LO,1]). `1.0` disables damping. | +| `ENGRAM_GEO_PRIME_SCALE` | `0.08` | Warm-floor scale; clamped `(0, WM_gate=0.15)`. | +| `ENGRAM_GEO_PRIME_MAX` | `32` | Max primed members per activation (0 disables priming). | + +The flag is read **once** per process (cached), so enabling/disabling requires a **process +restart** of the engram service — it is not hot-togglable within a running process. + +--- + +## 3. How to enable live (deliberate, reversible) + +> Precondition: the default-OFF binary has already been deployed and is running the M8 path +> healthily (behavior-neutral deploy). Do this only with Will present, per the standing rails. + +1. **Snapshot first** (always, before any activation-behavior change): + `~/.neuron/backups/pre-geometry-priming-/` ← copy `neuron.egm`, `neuron.wal`, + the current `engram` binary, and `ai.neuron.engram.plist`. +2. Add `ENGRAM_GEOMETRY_PRIMING=1` to the engram service environment + (`ai.neuron.engram.plist` `EnvironmentVariables`). +3. `launchctl bootout gui/$(id -u)/ai.neuron.engram` → `launchctl bootstrap …` (restart so the + flag is re-read). +4. **Verify:** service comes up serving the same node count; `/api/act-stats` shows sane WM + (promoted ≤ 24); spot-check 3–4 real queries return coherent results; watch one heartbeat + cycle for crashes/latency. The `geo_primed` counter (if surfaced) should be > 0. + +--- + +## 4. Rollback (exact steps) + +Rollback is a **flag flip**, not a data operation — the store is untouched by enabling the flag, +and priming is a read-mostly, bounded, sub-threshold addition. + +**Fast path (preferred) — disable the flag:** +1. Remove `ENGRAM_GEOMETRY_PRIMING` (or set `=0`) from `ai.neuron.engram.plist`. +2. `launchctl bootout … && launchctl bootstrap …`. +3. Verify: service healthy, activation is the M8 path again. **Done** — no data change to undo. + +**Full path (only if the binary itself is suspect) — redeploy prior binary:** +1. `launchctl bootout gui/$(id -u)/ai.neuron.engram`. +2. Restore the prior `engram` binary from `~/.neuron/backups/pre-geometry-priming-/`. +3. Restore `ai.neuron.engram.plist` from the same backup (flag absent). +4. `launchctl bootstrap …`; verify node count + a self-traversal + write-survives-restart. +5. If (and only if) the store was somehow mutated: restore `neuron.egm` + `neuron.wal` from the + backup. **Note:** enabling the flag does not write geometry to the store, so this step is + expected to be unnecessary — the primed activations are per-call and non-persistent beyond the + ordinary `background_activation`/WM write-back that M8 already does. + +**Rollback triggers:** any crash/hang in `engram_activate`; WM promotion count exceeding the cap +or collapsing; a measured recall/coherence regression vs the OFF baseline; unacceptable latency +increase; any ASan/UBSan report under the flag. + +--- + +## 5. Reversibility guarantees (why this is low-risk to deploy, higher-care to enable) + +- **Deploy (flag OFF):** byte-identical to M8. Verified in A/B. Zero-risk redeploy. +- **Enable (flag ON):** bounded and composable — + - never removes an M8 seed (damp-only, factor ≥ `lo` > 0); + - never amplifies a seed above its M8 value (factor ≤ 1); + - priming is strictly sub-threshold (`scale < WM_gate`) and capped (`PRIME_MAX`); + - priming raises a floor only (BFS keeps max) — cannot cap real activation; + - does not write geometry to the durable store; + - degrades to exact M8 behavior for any call where the paged store / centered global mean / + embedder is unavailable (guarded, not crashing). +- **Disable:** one env removal + restart; no data to reconcile. + +--- + +## 6. Known caveats / uncertainties (flagged — this is the memory core) + +- **Perf cost of ON:** the descriptor (covariance eigensolve + `store_get_node` paged reads per + member) runs on **every** activation when the flag is ON. See + `docs/architecture/design/perf/engram-geometry-priming-profile.md` for the measured OFF-vs-ON + latency. If that delta is unacceptable, keep the flag OFF (deploy stays valid) and revisit with + a cached/periodic descriptor. +- **Two-store consistency:** the descriptor reads embeddings from the **paged** store while the + ANN index is over the **resident** array. This-call backfilled embeddings can lag the paged + store by ≤ `ENGRAM_EMBED_BACKFILL_PER_CALL` nodes — the same staleness class as the M8 vindex, + and it can only omit a member, never mis-prime. +- **Damp tuning:** `lo=0.5` can at most halve an off-domain seed. If a coherence regression is + observed, raise `ENGRAM_GEO_SEED_LO` toward `1.0` (→ priming-only, no damping) before disabling + entirely.