Bring the relational-neighborhood geometry OPERATORS from the viz proxy
(engram-geometry-proxy.py §5) into the C runtime as reusable primitives, and
expose each as an EL builtin so any CGI app / el program can use them — not just
the engram service internals.
engram_geometry.{h,c} (pure, libm-only, read-only over descriptors):
- engram_geo_overlap : shared-member Jaccard + centroid/scale proximity
score + intersection centroid.
- engram_geo_subtract : orthogonal-complement residual (project A onto
I - V_B V_Bᵀ), closed-form variance_explained_by_B,
residual ellipsoid + centroid-diff; set-diff variant.
- engram_geo_combine : pooled descriptor (exact law-of-total-variance mean +
covariance), re-eigendecomposed.
- engram_geo_distance : centroid L2 + cosine + closed-form Wasserstein-2
(Bures) — mirrors the proxy _wasserstein2.
- engram_geo_analogy : orthogonal Procrustes R = UVᵀ (SVD) aligning A's
principal frame to B's + apply helper.
The C descriptor is full-dim/centered with a low-rank covariance from its top
axes; operators mirror the proxy FORMULAS and do the Wasserstein/combine eigen
work inside the small joint-axis subspace (exact there). Reuses jacobi_sym.
EL builtins (el_runtime.{h,c}, el_seed.c native wrappers):
engram_geo_{descriptor,overlap,subtract,combine,distance,analogy}_json —
take comma-separated seed-id set(s), build the CENTERED descriptor against the
true store-wide mean (ad-hoc path), run the operator, return JSON. Additive:
no flag, no effect on activation/retrieval. Surfacing via engram.el + the elc
fold is a cutover step (same elc-drift deferral as the P0/P5 builtins); the C
table wiring is registered now.
Tested: synthetic closed-form unit suite (20/20) — Wasserstein, Jaccard/score,
orthogonal residual, set-diff, pooled combine, Procrustes recovery. ASan+UBSan
clean; 0 leaks. Read-only; no activation/retrieval behavior change.
Adds engram_geo_displacement(A, B, core_frac) — a read-only interoceptive
primitive that measures how far a neighborhood descriptor B has drifted from a
baseline A and decomposes it into GROWTH (periphery extends, core fixed) vs
CORRUPTION (the invariant core displaces). The core is the top core_frac of A's
members by centrality; per shared member (matched by id) the displacement is the
change in radial position (dist_centroid). Centroid separation (L2 + cosine) and
radius delta give the aggregate move. Pure function, no store mutation, no flag.
HONESTLY PARTIAL: a live self-drift reading needs a persisted SelfAnchor
baseline to compare "now" against, and no persisted self node / anchored
self-neighborhood exists in this store yet. The primitive takes an EXPLICIT
baseline so it is real and testable today; capturing a durable SelfAnchor and
wiring the ENGRAM_DRIFT_SENSOR live reading is a flagged follow-up. We do not
fabricate a self silently.
MEASURED on synthetic descriptors:
- GROWTH (periphery 0.50->0.90, core fixed): core_disp=0.000, periph_disp=0.400,
centroid_sep=0.000, radius_delta=0.400.
- CORRUPTION (core 0.10->0.60, periphery fixed): core_disp=0.500,
periph_disp=0.000, centroid_sep=0.566.
- Identity A vs A: zero drift.
The sensor discriminates cleanly (corruption core_disp >> growth core_disp).
ASan+UBSan clean.
Reification, not a cache. Densely co-wired relational neighborhoods are crystallized
into DURABLE first-class store records that survive restart, load on boot, and evolve
via supersede+provenance -- so the geometry-priming hot path READS persisted structure
instead of computing a per-query descriptor (the M9 3.2x/13x latency blocker).
engram_geometry.{h,c}:
- engram_geo_reify_store(): detect hub-anchored neighborhoods on the hebb-weighted
graph (greedy non-redundant cover), compute each centered descriptor ONCE against
the true store-wide mean, persist as node_type="Neighborhood" (raw centroid in emb,
membership+scalars+axis-extents in a compact GEO1 metadata schema) + member edges,
superseding any prior same-hub record. The mean is persisted once as "GeoMeanFrame".
- resident loaded form (index_new/add/finalize/lookup): parses the durable records at
boot (never recomputes geometry); O(seeds) membership lookup, miss -> centroid-nearest.
- descriptor: skip the Jacobi eigensolve when top_axes==0; reject structural records as
members (id-convention + node_type guards) so re-reify/ad-hoc stay clean.
el_runtime.c:
- boot routes Neighborhood/GeoMeanFrame records OUT of the activation graph into the
reify index, and skips member edges from adjacency -> ENGRAM_GEOMETRY_PRIMING OFF is
byte-identical to M8/M9 (verified across 15 queries).
- priming hot path reads the persisted membership; ENGRAM_GEO_PRIMING_NOCACHE=1 keeps
the M9 per-query descriptor for ad-hoc geometries / A/B control.
A/B on a COPY (128 neighborhoods): priming ON is now FLAT latency (1.06x median / 1.03x
p90 vs OFF) where the M9 per-query path is 3.30x/10.5x. Reified records provably inert
when OFF. Restart survival + supersede verified. Build 0 warnings (my code); ASan/UBSan
clean on module and full server hot path. Recall quality re-eval against the TRUE mean
still shows no reliable gain (mean coherence -0.017), so geometry-priming STAYS default-OFF
-- but the latency blocker is removed and the durable structure now exists. See
docs/architecture/design/engram-m10-reification.md.
The nomic-embed-text space over the corpus is strongly anisotropic (mean
pairwise cosine ~0.55), which compresses cosine-based domain separation almost
to nothing so the design-doc s5 operators (distance/overlap/Wasserstein) cannot
discriminate. Subtracting the global mean of the normalized embeddings restores
isotropy (mean pairwise cosine ~0) and sharpens the operators.
- add GeoMeanCache (engram_geo_mean_build / _maybe_refresh / _vec / _free): a
store-derived centering offset over the embed-eligible set, cached and
refreshed on significant drift; lives in geometry.c, not the store.
- engram_geometry_descriptor gains an optional global_mean: when supplied the
centroid, per-member cosine distance, and co-registration run in centered
space (GM=zeros reproduces the legacy raw path exactly).
- co-registration choice (b): the ANN query stays in raw unit space (index
unchanged) since centering is a rigid translation that ~preserves neighborhood
membership; only the descriptor statistics move to the centered frame.
Covariance/axes/radius are translation-invariant and therefore unchanged.
- test: synthetic ground-truth suite stays green (PERF + ASan/UBSan), plus new
centered/raw/mean-cache assertions.
- add bench_discrimination.c (env-gated, read-only, skips in CI): on a copy of
the real store the two-domain overlap operator drops 1.13 -> 0.008 and
cross-centroid cosine 0.899 -> 0.003 after centering, Euclid distance
unchanged (translation-invariant control).
No change to activation/retrieval behavior; wiring geometry into retrieval is a
separate, behavior-changing cutover.
The stone the operator/drift/occupation work stands on: express a relational
neighborhood as the compact joint geometry Will specified (design §3/§5, node
e94371bd) — semantic side (centroid, principal-axis ellipsoid via dual-PCA,
radius) braided with the relational side (k-core skeleton, hub->periphery
centrality gradient), plus soft membership and a co-registration diagnostic
(corr of hebb strength vs semantic proximity — >0 reifies, <0 flags dreams).
Built ONLY on the two standalone modules — engram_vindex (ANN, the cloud) and
engram_store (embeddings + hebb adjacency, the skeleton). Pure C11 + libm; does
not link or touch el_runtime.c. Strictly READ-ONLY: never mutates nodes, edges,
activation, the index, or any retrieval path. Not yet wired into retrieval —
foundation only.
Self-contained test (test_geometry.c) synthesizes two known embedding clusters
with intra-cluster hebb edges and verifies the descriptor recovers the shape:
centroid on the seeded cluster, hub = relational center, skeleton = the strong
intra-cluster wiring, positive co-registration, sorted axis extents. PERF +
ASan/UBSan passes both green; needs no live data.
will.anderson