149 lines
9.4 KiB
Markdown
149 lines
9.4 KiB
Markdown
# M10 — Reification: Persisted First-Class Neighborhood Geometry
|
||
|
||
**Date:** 2026-08-12
|
||
**Branch:** `engram-tiered-storage` (worktree `/tmp/engram-tiered-wt`)
|
||
**Status:** Reification substrate = **GO** (additive, geometry-priming stays default-OFF).
|
||
Enabling geometry-priming = **NO-GO** (latency blocker removed; recall-quality benefit still absent).
|
||
**Live `:8742` never touched. Not pushed. Not tagged.**
|
||
|
||
This is the sequel to [M9 geometry priming](perf/engram-geometry-priming-profile.md), whose A/B
|
||
showed enabling the flag cost **3.2× median / 13× p90** latency for **no reliable quality gain**.
|
||
The M9 profile named two prerequisites before re-evaluating: **(1) amortize the per-query
|
||
descriptor cost** (eigensolve + paged reads on every `engram_activate`) and **(2) center recall
|
||
quality against the true store-wide mean**, not a per-query gathered-set approximation. M10 does both.
|
||
|
||
## What changed vs the original brief (important)
|
||
|
||
The task began as "reify the geometry into a durable **cache**." Will corrected this mid-flight, and
|
||
the correction is the design: **do not build a cache — reify densely co-wired neighborhoods into
|
||
first-class, PERSISTED store records** that survive restart, load on boot, and evolve via
|
||
supersede+provenance. *"A cache that lies is worse than a slow lookup."* This matches
|
||
[the cognitive-architecture design §2](engram-cognitive-architecture.md) (reification = durable
|
||
structure, not a fragile derived shortcut) and the memory-core discipline (evolve or forget, never
|
||
a stale canonical). Two modes, **no general cache layer**:
|
||
|
||
1. **Persist first-class** — reified/crystallized neighborhoods (the self; stable topology). The
|
||
geometry-priming **hot path reads these**. Never compute geometry on the activation path.
|
||
2. **Compute on the fly** — ad-hoc/transient domain geometries (viz, exploration). Uses the
|
||
existing M9 `engram_geometry_descriptor`, fresh each call, no storage. Occasional, so its cost
|
||
is acceptable.
|
||
|
||
## Storage schema (first-class, on the existing TLV node store — no new on-disk format)
|
||
|
||
A reified neighborhood is an ordinary store **node**, so it inherits durability, boot-load,
|
||
`store_supersede`, and adjacency for free (design §2: *structure all the way down under one rule*).
|
||
|
||
| Record | `node_type` | `emb` | `metadata` (`GEO1` text schema) | id |
|
||
|---|---|---|---|---|
|
||
| Centering frame | `GeoMeanFrame` | the true store-wide mean vector (persisted **once**) | `{}` | `geo-meanframe` |
|
||
| Neighborhood | `Neighborhood` | the **raw centroid** (prototype; centroid-ANN-able; `centered = emb − meanframe`) | hub id · meanframe ref · scalars (`radius`, `total_variance`, `k_core`, `co_registration`, `n_embedded`) · axis **extents** · member list `{id → membership, centrality, core}` | `nbhd-<hub>-<built_at_ms>` |
|
||
|
||
Member links are also persisted as edges `relation="member"` (`nbhd → member`). The membership
|
||
`{id → w}` in the metadata is what priming reads; it is computed **centered against the persisted
|
||
true mean**, which is what resolves the M9 quality caveat.
|
||
|
||
**Detection (v1, honest).** Hub-anchored neighborhoods over the strong-edge hebb-weighted graph:
|
||
compute per-node weighted degree `Σ eff_w` (`eff_w = weight·(1+0.5·hebb)`, non-tombstoned /
|
||
non-inhibitory, ≥ `edge_min_weight`) over from+to edges; rank descending; **greedy non-redundant
|
||
cover** — reify each hub's descriptor once, skip a hub already a member (`w ≥ cover_membership`) of an
|
||
accepted neighborhood, stop at `max_neighborhoods` (default 128). Env-tunable (`min_weighted_degree`,
|
||
`max_neighborhoods`, refresh frac) without rebuild.
|
||
|
||
**Honest caveat — hebb ≈ 0 today.** On the current store there is ~no Hebbian potentiation, so
|
||
`eff_w` reduces to the **authored** edge weight; the detected neighborhoods currently reflect
|
||
authored graph structure, not learned co-activation. The design is unchanged and self-correcting
|
||
once hebb accrues (degree ranking and skeleton shift automatically).
|
||
|
||
## Boot isolation — why priming-OFF stays byte-identical
|
||
|
||
The persisted records carry embeddings (centroid / mean) but are **structure, not corpus content**.
|
||
On boot they are routed **out** of the resident activation graph into a dedicated resident reify
|
||
index, and member edges are skipped from adjacency (matched by the `nbhd-…` id convention, so no real
|
||
edge of any relation is affected). Consequences, all verified:
|
||
|
||
- store-wide mean, hub-degree scan, and the descriptor never admit a structural record as a member
|
||
(`geo_is_structural_id` / `node_type` guards);
|
||
- vindex / seed selection / results / embedding backfill are **identical** to a store that was never
|
||
reified ⇒ `ENGRAM_GEOMETRY_PRIMING` OFF is **byte-identical to M8/M9**.
|
||
|
||
The resident index is the **loaded form** of the durable records (like the resident node array is the
|
||
loaded form of node records, or adjacency of edges). Geometry is computed **once**, offline, and
|
||
**persisted**; boot only *parses* — it never recomputes geometry.
|
||
|
||
## Hot path
|
||
|
||
`ENGRAM_GEOMETRY_PRIMING=1` resolves the M8 seed set to the best persisted neighborhood — O(seeds)
|
||
membership hash lookup; miss → centroid-nearest against the loaded centered centroids — and applies
|
||
the M9 damp+prime logic from the persisted membership. **No geometry computed on activation.** Set
|
||
`ENGRAM_GEO_PRIMING_NOCACHE=1` to fall back to the M9 per-query descriptor (ad-hoc / A/B control).
|
||
|
||
## Results (A/B on COPIES; `store_reified` = 128 neighborhoods; live untouched)
|
||
|
||
**Correctness / durability**
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| Reified records inert: `A_off` (reified store) vs `C_clean` (no records), id sequence all 15 queries | **MATCH** (byte-identical) |
|
||
| Restart survival: reify → checkpoint+close → fresh-process reopen | 128 `Neighborhood` + 1 `GeoMeanFrame` present; index loads **28 ms**; hub-seed lookup HIT **~1 µs** |
|
||
| Supersede/provenance: re-reify | prior same-hub record superseded (new timestamped id, old tombstoned); live count stable |
|
||
| Build warnings from `el_runtime.c` / `engram_geometry.c` (−O2) | **0** (3 pre-existing in generated `engram.c`) |
|
||
| ASan/UBSan — module (write/load/lookup) + full server hot path, priming ON, 6 queries | **CLEAN** (all http 200, no trap) |
|
||
|
||
**Latency (median / p90, 15 queries)**
|
||
|
||
| config | median | p90 | vs A_off |
|
||
|---|---|---|---|
|
||
| C_clean (no records, OFF) | 78.6 ms | 82.7 ms | — |
|
||
| **A_off** (reified, OFF) | **77.4 ms** | **83.1 ms** | 1.00× |
|
||
| **B_on** (reified, **hot path / persisted**) | **81.7 ms** | **85.7 ms** | **1.06× / 1.03× — FLAT** |
|
||
| D_nocache (reified, ON, M9 per-query) | 255.3 ms | 872.3 ms | **3.30× / 10.5×** |
|
||
|
||
Reading a persisted first-class record instead of computing a per-query descriptor **removes the M9
|
||
latency blocker** (3.3×/10.5× → 1.06×/1.03×). There is no cold first-query penalty — the index loads
|
||
at boot.
|
||
|
||
**Recall quality (mean pairwise cosine, top-20, TRUE store-wide centered frame, stored vectors)**
|
||
|
||
| | OFF | ON | Δ |
|
||
|---|---|---|---|
|
||
| mean over 15 queries | 0.1721 | 0.1555 | **−0.0166** |
|
||
| polysemous cues (9) | 0.1442 | 0.1441 | −0.0001 |
|
||
| queries where ON > OFF | — | — | **3 / 15** |
|
||
|
||
Even with the true mean and persisted structure there is **no reliable coherence gain** — slightly
|
||
negative on average, one sparse win (`self identity values` +0.078) and one notable regression
|
||
(`precision over brute force` −0.215). On dense polysemous cues the top-20 head is unchanged
|
||
(sub-threshold priming does not reorder it), so their coherence is flat.
|
||
|
||
## Verdict
|
||
|
||
- **Reification substrate: GO** (merge additive, geometry-priming default-OFF). It is the durable
|
||
first-class structure the memory core needs — persisted, boot-loaded, restart-surviving,
|
||
supersede-able, provably inert when OFF — and it turns priming into a flat-latency lookup.
|
||
Groundwork for the self-as-structure, multi-scale neighborhoods, and the §5 operators.
|
||
- **Enable geometry-priming: NO-GO (still).** The M9 *latency* objection is resolved; the *quality*
|
||
objection is not. Keep the flag default-OFF.
|
||
|
||
### Uncertainties / limits (memory core — flagged)
|
||
|
||
1. **hebb ≈ 0** ⇒ neighborhoods reflect authored edges, not learned co-activation. The quality result
|
||
is likely **understated**; re-evaluate after hebb accrues (or seed hebb from usage).
|
||
2. **Hub-anchored detection** can resolve a sparse query to a semantically mismatched neighborhood
|
||
(the −0.215 regression). Semantic-aware / co-registration-gated detection is future work.
|
||
3. **Top-20 coherence is insensitive** to sub-threshold priming on dense cues — it may be the wrong
|
||
metric for what priming does (it warms a floor, it does not reorder the head). A retrieval-utility
|
||
or disambiguation-accuracy metric would measure the intended effect better.
|
||
4. **v1 simplifications:** axis **direction** vectors are not persisted (extents only; directions
|
||
recomputable on the fly); member edges are persisted but inert to activation.
|
||
|
||
## Reversal
|
||
|
||
Fully additive and reversible.
|
||
|
||
- The binary is **byte-identical to M8/M9 when `ENGRAM_GEOMETRY_PRIMING` is unset/0** — the default.
|
||
- Reified records live only in stores you explicitly run the reify step against; a store that was
|
||
never reified behaves exactly as before (the resident index is empty ⇒ priming no-ops).
|
||
- To remove reified structure from a store: tombstone the `Neighborhood` / `GeoMeanFrame` records and
|
||
compact (they are ordinary nodes). No format change to undo.
|
||
- `ENGRAM_GEO_PRIMING_NOCACHE=1` restores the M9 per-query path for ad-hoc geometries / comparison.
|