neuron/docs/architecture/design/engram-m10-reification.md

M10 — Reification: Persisted First-Class Neighborhood Geometry

Date: 2026-08-12 Branch: engram-tiered-storage (worktree /tmp/engram-tiered-wt) Status: Reification substrate = GO (additive, geometry-priming stays default-OFF). Enabling geometry-priming = NO-GO (latency blocker removed; recall-quality benefit still absent). Live :8742 never touched. Not pushed. Not tagged.

This is the sequel to M9 geometry priming, whose A/B showed enabling the flag cost 3.2× median / 13× p90 latency for no reliable quality gain. The M9 profile named two prerequisites before re-evaluating: (1) amortize the per-query descriptor cost (eigensolve + paged reads on every engram_activate) and (2) center recall quality against the true store-wide mean, not a per-query gathered-set approximation. M10 does both.

What changed vs the original brief (important)

The task began as "reify the geometry into a durable cache." Will corrected this mid-flight, and the correction is the design: do not build a cache — reify densely co-wired neighborhoods into first-class, PERSISTED store records that survive restart, load on boot, and evolve via supersede+provenance. "A cache that lies is worse than a slow lookup." This matches the cognitive-architecture design §2 (reification = durable structure, not a fragile derived shortcut) and the memory-core discipline (evolve or forget, never a stale canonical). Two modes, no general cache layer:

1. Persist first-class — reified/crystallized neighborhoods (the self; stable topology). The geometry-priming hot path reads these. Never compute geometry on the activation path.
2. Compute on the fly — ad-hoc/transient domain geometries (viz, exploration). Uses the existing M9 engram_geometry_descriptor, fresh each call, no storage. Occasional, so its cost is acceptable.

Storage schema (first-class, on the existing TLV node store — no new on-disk format)

A reified neighborhood is an ordinary store node, so it inherits durability, boot-load, store_supersede, and adjacency for free (design §2: structure all the way down under one rule).

Record	node_type	emb	metadata (GEO1 text schema)	id
Centering frame	GeoMeanFrame	the true store-wide mean vector (persisted once)	{}	geo-meanframe
Neighborhood	Neighborhood	the raw centroid (prototype; centroid-ANN-able; centered = emb − meanframe)	hub id · meanframe ref · scalars (radius, total_variance, k_core, co_registration, n_embedded) · axis extents · member list {id → membership, centrality, core}	nbhd-<hub>-<built_at_ms>

Member links are also persisted as edges relation="member" (nbhd → member). The membership {id → w} in the metadata is what priming reads; it is computed centered against the persisted true mean, which is what resolves the M9 quality caveat.

Detection (v1, honest). Hub-anchored neighborhoods over the strong-edge hebb-weighted graph: compute per-node weighted degree Σ eff_w (eff_w = weight·(1+0.5·hebb), non-tombstoned / non-inhibitory, ≥ edge_min_weight) over from+to edges; rank descending; greedy non-redundant cover — reify each hub's descriptor once, skip a hub already a member (w ≥ cover_membership) of an accepted neighborhood, stop at max_neighborhoods (default 128). Env-tunable (min_weighted_degree, max_neighborhoods, refresh frac) without rebuild.

Honest caveat — hebb ≈ 0 today. On the current store there is ~no Hebbian potentiation, so eff_w reduces to the authored edge weight; the detected neighborhoods currently reflect authored graph structure, not learned co-activation. The design is unchanged and self-correcting once hebb accrues (degree ranking and skeleton shift automatically).

Boot isolation — why priming-OFF stays byte-identical

The persisted records carry embeddings (centroid / mean) but are structure, not corpus content. On boot they are routed out of the resident activation graph into a dedicated resident reify index, and member edges are skipped from adjacency (matched by the nbhd-… id convention, so no real edge of any relation is affected). Consequences, all verified:

• store-wide mean, hub-degree scan, and the descriptor never admit a structural record as a member (geo_is_structural_id / node_type guards);
• vindex / seed selection / results / embedding backfill are identical to a store that was never reified ⇒ ENGRAM_GEOMETRY_PRIMING OFF is byte-identical to M8/M9.

The resident index is the loaded form of the durable records (like the resident node array is the loaded form of node records, or adjacency of edges). Geometry is computed once, offline, and persisted; boot only parses — it never recomputes geometry.

Hot path

ENGRAM_GEOMETRY_PRIMING=1 resolves the M8 seed set to the best persisted neighborhood — O(seeds) membership hash lookup; miss → centroid-nearest against the loaded centered centroids — and applies the M9 damp+prime logic from the persisted membership. No geometry computed on activation. Set ENGRAM_GEO_PRIMING_NOCACHE=1 to fall back to the M9 per-query descriptor (ad-hoc / A/B control).

Results (A/B on COPIES; store_reified = 128 neighborhoods; live untouched)

Correctness / durability

Check	Result
Reified records inert: A_off (reified store) vs C_clean (no records), id sequence all 15 queries	MATCH (byte-identical)
Restart survival: reify → checkpoint+close → fresh-process reopen	128 Neighborhood + 1 GeoMeanFrame present; index loads 28 ms; hub-seed lookup HIT ~1 µs
Supersede/provenance: re-reify	prior same-hub record superseded (new timestamped id, old tombstoned); live count stable
Build warnings from el_runtime.c / engram_geometry.c (−O2)	0 (3 pre-existing in generated engram.c)
ASan/UBSan — module (write/load/lookup) + full server hot path, priming ON, 6 queries	CLEAN (all http 200, no trap)

Latency (median / p90, 15 queries)

config	median	p90	vs A_off
C_clean (no records, OFF)	78.6 ms	82.7 ms	—
A_off (reified, OFF)	77.4 ms	83.1 ms	1.00×
B_on (reified, hot path / persisted)	81.7 ms	85.7 ms	1.06× / 1.03× — FLAT
D_nocache (reified, ON, M9 per-query)	255.3 ms	872.3 ms	3.30× / 10.5×

Reading a persisted first-class record instead of computing a per-query descriptor removes the M9 latency blocker (3.3×/10.5× → 1.06×/1.03×). There is no cold first-query penalty — the index loads at boot.

Recall quality (mean pairwise cosine, top-20, TRUE store-wide centered frame, stored vectors)

	OFF	ON	Δ
mean over 15 queries	0.1721	0.1555	−0.0166
polysemous cues (9)	0.1442	0.1441	−0.0001
queries where ON > OFF	—	—	3 / 15

Even with the true mean and persisted structure there is no reliable coherence gain — slightly negative on average, one sparse win (self identity values +0.078) and one notable regression (precision over brute force −0.215). On dense polysemous cues the top-20 head is unchanged (sub-threshold priming does not reorder it), so their coherence is flat.

Verdict

• Reification substrate: GO (merge additive, geometry-priming default-OFF). It is the durable first-class structure the memory core needs — persisted, boot-loaded, restart-surviving, supersede-able, provably inert when OFF — and it turns priming into a flat-latency lookup. Groundwork for the self-as-structure, multi-scale neighborhoods, and the §5 operators.
• Enable geometry-priming: NO-GO (still). The M9 latency objection is resolved; the quality objection is not. Keep the flag default-OFF.

Uncertainties / limits (memory core — flagged)

1. hebb ≈ 0 ⇒ neighborhoods reflect authored edges, not learned co-activation. The quality result is likely understated; re-evaluate after hebb accrues (or seed hebb from usage).
2. Hub-anchored detection can resolve a sparse query to a semantically mismatched neighborhood (the −0.215 regression). Semantic-aware / co-registration-gated detection is future work.
3. Top-20 coherence is insensitive to sub-threshold priming on dense cues — it may be the wrong metric for what priming does (it warms a floor, it does not reorder the head). A retrieval-utility or disambiguation-accuracy metric would measure the intended effect better.
4. v1 simplifications: axis direction vectors are not persisted (extents only; directions recomputable on the fly); member edges are persisted but inert to activation.

Reversal

Fully additive and reversible.

• The binary is byte-identical to M8/M9 when ENGRAM_GEOMETRY_PRIMING is unset/0 — the default.
• Reified records live only in stores you explicitly run the reify step against; a store that was never reified behaves exactly as before (the resident index is empty ⇒ priming no-ops).
• To remove reified structure from a store: tombstone the Neighborhood / GeoMeanFrame records and compact (they are ordinary nodes). No format change to undo.
• ENGRAM_GEO_PRIMING_NOCACHE=1 restores the M9 per-query path for ad-hoc geometries / comparison.