Files
el/lang/runtime/engram_geometry.h
T
will.anderson 5336cfe0a6 M9 §5: geometry OPERATORS as C functions + EL builtins (read-only, staged)
Bring the relational-neighborhood geometry OPERATORS from the viz proxy
(engram-geometry-proxy.py §5) into the C runtime as reusable primitives, and
expose each as an EL builtin so any CGI app / el program can use them — not just
the engram service internals.

engram_geometry.{h,c} (pure, libm-only, read-only over descriptors):
  - engram_geo_overlap   : shared-member Jaccard + centroid/scale proximity
                           score + intersection centroid.
  - engram_geo_subtract  : orthogonal-complement residual (project A onto
                           I - V_B V_Bᵀ), closed-form variance_explained_by_B,
                           residual ellipsoid + centroid-diff; set-diff variant.
  - engram_geo_combine   : pooled descriptor (exact law-of-total-variance mean +
                           covariance), re-eigendecomposed.
  - engram_geo_distance  : centroid L2 + cosine + closed-form Wasserstein-2
                           (Bures) — mirrors the proxy _wasserstein2.
  - engram_geo_analogy   : orthogonal Procrustes R = UVᵀ (SVD) aligning A's
                           principal frame to B's + apply helper.
The C descriptor is full-dim/centered with a low-rank covariance from its top
axes; operators mirror the proxy FORMULAS and do the Wasserstein/combine eigen
work inside the small joint-axis subspace (exact there). Reuses jacobi_sym.

EL builtins (el_runtime.{h,c}, el_seed.c native wrappers):
  engram_geo_{descriptor,overlap,subtract,combine,distance,analogy}_json —
  take comma-separated seed-id set(s), build the CENTERED descriptor against the
  true store-wide mean (ad-hoc path), run the operator, return JSON. Additive:
  no flag, no effect on activation/retrieval. Surfacing via engram.el + the elc
  fold is a cutover step (same elc-drift deferral as the P0/P5 builtins); the C
  table wiring is registered now.

Tested: synthetic closed-form unit suite (20/20) — Wasserstein, Jaccard/score,
orthogonal residual, set-diff, pooled combine, Procrustes recovery. ASan+UBSan
clean; 0 leaks. Read-only; no activation/retrieval behavior change.
2026-08-13 00:26:51 -05:00

387 lines
23 KiB
C
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/* engram_geometry.h — M9 FOUNDATION: the relational-neighborhood GEOMETRY
* DESCRIPTOR (design doc §3, §5; memory node e94371bd).
*
* Computes, for a relational neighborhood grown from a seed set, the compact
* (KB-not-MB) joint geometry Will specified: the SEMANTIC geometry (centroid,
* covariance / principal axes, radius) braided with the RELATIONAL geometry
* (k-core skeleton, hub->periphery centrality gradient), plus soft membership.
*
* Two coordinate systems, one shape — "a constellation: bright prototype at the
* center, a cloud of members at varying distance, the strongest edges as a
* backbone, fading at the edges."
*
* Built ON the two standalone M-era modules only:
* - engram_vindex : semantic neighbors (the cloud) via ANN.
* - engram_store : node embeddings + hebb adjacency (the skeleton), read-only.
* It does NOT link or touch el_runtime.c, and it is a pure READ over the graph:
* it never modifies nodes, edges, activation, the index, or any retrieval path.
*
* Pure C11, stdlib + libm only. The descriptor is a foundation object; it is NOT
* wired into retrieval/priming yet (that is the next M9 step).
*/
#ifndef ENGRAM_GEOMETRY_H
#define ENGRAM_GEOMETRY_H
#include <stddef.h>
#include <stdint.h>
#include "engram_store.h"
#include "engram_vindex.h"
/* One member of the neighborhood + its place in the gradient. */
typedef struct {
char* id;
double membership; /* soft membership in [0,1] (semantic+relational blend) */
double centrality; /* skeleton weighted-degree — relational salience */
double salience; /* the node's own stored salience */
int core; /* k-core number (0 = fringe / not in any core) */
double dist_centroid; /* cosine distance of member emb to centroid (semantic)*/
int embedded; /* 1 if the member carried an emb vector */
} GeoMember;
/* One skeleton edge (indices into members[]). eff_weight = weight*(1+0.5*hebb),
* clamped to 1.0 — the effective propagation strength eg_edge_eff_weight uses. */
typedef struct { uint32_t a, b; double eff_weight; double hebb; } GeoEdge;
/* A compact principal axis of the ellipsoid: unit direction in R^dim + extent
* (sqrt of the covariance eigenvalue = the ellipsoid's half-width along it). */
typedef struct { float* axis; double extent; } GeoAxis;
typedef struct {
int dim;
/* ── anchor ── */
char* hub_id; /* highest-centrality member: the relational hub */
float* centroid; /* v̄ ∈ R^dim: mean of the member embeddings in the
* frame the descriptor operated in. When centered
* (global_mean != NULL) this is the CENTERED
* centroid (mean of L2-normalized embs minus the
* global mean): the neighborhood's location in the
* isotropic/whitened frame. Add global_mean back to
* recover the raw prototype point. When uncentered
* it is the raw mean of L2-normalized member embs. */
float* global_mean; /* the centering offset actually applied (dim floats),
* or NULL if the descriptor ran in raw space. The §5
* operators (distance/overlap/Wasserstein) are only
* discriminative in the centered frame — see notes. */
/* ── shape (compact covariance): top principal axes + extents ── */
int n_axes;
GeoAxis* axes; /* orientation + extents of the ellipsoid */
double total_variance; /* trace(Σ) = mean squared member dist to centroid*/
/* ── scale ── */
double radius; /* sqrt(total_variance) — the neighborhood breadth*/
/* ── members + gradient ── */
int n_members;
GeoMember* members; /* soft membership {id->weight} + centrality/salience */
/* ── skeleton ── */
int n_edges;
GeoEdge* edges; /* strong internal hebb edges = the backbone */
int k_core; /* the maximum core number present in the skeleton*/
/* ── diagnostics ── */
double co_registration;/* corr(hebb strength, semantic proximity) over */
/* internal edges: >0 = geometries agree (reify); */
/* <0 = disagree (surprising links / dream cands). */
int n_embedded; /* members that carried an emb vector */
} GeoDescriptor;
typedef struct {
int ann_k; /* semantic expansion: ANN neighbors per seed (0=off) */
int hop_relational; /* 1 = include seeds' hebb neighbors as members */
double edge_min_weight; /* skeleton: ignore internal edges below this eff wt */
int kcore_k; /* target k for the reported k-core (0 = auto/max) */
int top_axes; /* principal axes to retain (default 8) */
int max_members; /* cap neighborhood size (guards the eigensolve cost) */
} GeoParams;
/* Fill p with sane defaults: ann_k=24, hop_relational=1, edge_min_weight=0.05,
* kcore_k=0 (auto), top_axes=8, max_members=400. */
void engram_geo_default_params(GeoParams* p);
/* ── Global-mean cache (mean-centering / whitening the anisotropic emb space) ──
* The nomic-embed-text space over the engram corpus is strongly ANISOTROPIC:
* every embedding sits in a narrow cone (mean pairwise cosine ~0.55), which
* compresses cosine-based domain separation almost to nothing. Subtracting the
* GLOBAL MEAN of the (L2-normalized) embeddings recenters the cloud on the
* origin (mean pairwise cosine -> ~0), restoring isotropy so the §5 operators
* discriminate. The mean is a store-level derived quantity, like the ANN index:
* built once from the paged store, cached, and refreshed when the embedded set
* drifts. It lives here (not in the store) so this stays a contained, read-only
* addition; a runtime owns one GeoMeanCache per open store alongside its VIndex. */
typedef struct GeoMeanCache GeoMeanCache;
/* Scan every live node in `store` and compute the mean of the L2-normalized
* embeddings over the embed-eligible set (nodes carrying an emb vector; the
* unembedded telemetry/system nodes are skipped). Returns a malloc'd cache, or
* NULL on error / no embedded nodes. The offset vector is NOT renormalized — it
* is a translation, applied by subtraction. */
GeoMeanCache* engram_geo_mean_build(EngramPagedStore* store);
/* The cached offset (dim floats) — pass to engram_geometry_descriptor as
* global_mean. Valid until the cache is freed/refreshed. */
const float* engram_geo_mean_vec(const GeoMeanCache* c);
int engram_geo_mean_dim(const GeoMeanCache* c);
uint64_t engram_geo_mean_count(const GeoMeanCache* c); /* #embedded nodes used */
/* Recompute the mean IN PLACE iff the embedded-node count has drifted by more
* than `frac` (e.g. 0.10 = 10%) since the cache was built — "recompute on
* significant change". Returns 1 if it rebuilt, 0 if unchanged, <0 on error. */
int engram_geo_mean_maybe_refresh(GeoMeanCache* c, EngramPagedStore* store,
double frac);
void engram_geo_mean_free(GeoMeanCache* c);
/* Compute the geometry descriptor of the neighborhood grown from seed_ids.
* READ-ONLY over store + vindex.
* store — an opened store (borrowed; not modified).
* vindex — optional ANN index for semantic expansion; NULL disables it.
* vids — the ordinal->store-id map returned by vindex_build_from_store
* (vids[node_id] == store id). Required iff vindex != NULL.
* n_vids — length of vids.
* params — NULL to use engram_geo_default_params.
* global_mean — optional centering offset (dim floats, from engram_geo_mean_*).
* When non-NULL the SEMANTIC geometry is computed in mean-centered
* (isotropic) space: every normalized member emb has global_mean
* subtracted before the centroid / cosine-distance / co-registration
* math, so those operators discriminate. NULL = raw space (legacy).
* NOTE: the ANN neighbor query still runs in RAW unit-vector space —
* centering is a rigid translation that ~preserves neighborhood
* MEMBERSHIP, so the index needs no rebuild; only the descriptor
* STATISTICS move to the centered frame (co-registration choice (b)).
* The eigen/covariance shape (axes, radius) is translation-invariant
* and therefore identical in either frame.
* Returns a malloc'd descriptor (free with engram_geo_free), or NULL on error
* (no seeds resolvable, OOM). */
GeoDescriptor* engram_geometry_descriptor(
EngramPagedStore* store, VIndex* vindex,
char** vids, int n_vids,
const char* const* seed_ids, size_t n_seeds,
const GeoParams* params,
const float* global_mean);
void engram_geo_free(GeoDescriptor* g);
/* ── M-INTEROCEPTION P3: drift-sensor primitive (descriptor displacement) ────
* Read-only. GROWTH vs CORRUPTION split of how far B drifted from baseline A.
* See engram_geometry.c for the honesty note on the missing SelfAnchor. */
typedef struct {
double centroid_sep; /* L2 distance between centroids (same frame) */
double centroid_cos; /* 1 - cosine(centroidA, centroidB) */
double radius_delta; /* |radiusA - radiusB| — neighborhood scale change */
double core_disp; /* mean radial displacement of the invariant core */
double periph_disp; /* mean radial displacement of the periphery */
int core_matched; /* # core members matched by id across A,B */
int periph_matched; /* # periphery members matched by id across A,B */
} GeoDisplacement;
void engram_geo_displacement(const GeoDescriptor* a, const GeoDescriptor* b,
double core_frac, GeoDisplacement* out);
/* ═══════════════════════════════════════════════════════════════════════════
* §5 GEOMETRY OPERATORS — a relational ALGEBRA over neighborhood descriptors.
* These are the reusable primitives Will specified: "primitives any CGI
* application should be able to use." READ-ONLY and PURE (stdlib + libm only) —
* they consume GeoDescriptor(s) and never touch the store, index, or activation.
*
* FRAME CONTRACT: both inputs MUST have been built in the SAME frame — identical
* emb `dim` and identical `global_mean` (centered against the one true store-wide
* mean). The reify path builds every neighborhood that way, so descriptors are
* directly comparable. An operator returns <0 / NULL if the dims disagree.
*
* REPRESENTATION: the C descriptor lives in the FULL emb dim with a LOW-RANK
* covariance Σ = Σ_k extent_k² · a_k a_kᵀ over its retained principal axes
* (top_axes; the discarded tail variance is not modeled). Every operator mirrors
* the viz-proxy (engram-geometry-proxy.py §5) FORMULA exactly, but evaluates it on
* this representation — so semantics match the proxy while absolute numbers differ
* (proxy works in a 24-dim global-PCA reduced dense frame; C in full-dim low-rank).
* The Wasserstein / combine eigen-work is done inside the small JOINT axis subspace
* (dimension ≤ nA+nB+1), which is EXACT for the low-rank covariances there.
* Each result struct is released by its engram_geo_*_free.
* ═══════════════════════════════════════════════════════════════════════════ */
/* overlap(A,B): shared-member set + Jaccard + centroid/scale proximity score. */
typedef struct {
char** shared_ids; /* ids present in BOTH neighborhoods (owned) */
int n_shared;
int n_union; /* |A B| by id */
double jaccard; /* |A∩B| / |AB| */
double centroid_distance; /* L2 between the (centered) centroids */
double overlap_score; /* jacc*0.5 + max(0,1d/(rA+rB))*0.5 (proxy form)*/
float* intersection_centroid; /* midpoint of the two centroids (dim, owned) */
int dim;
} GeoOverlap;
int engram_geo_overlap(const GeoDescriptor* a, const GeoDescriptor* b, GeoOverlap* out);
void engram_geo_overlap_free(GeoOverlap* o);
/* subtract(A,B) — ORTHOGONAL-COMPLEMENT residual: project A onto I V_B V_Bᵀ
* (V_B = B's top `b_dims` principal axes) — "A with B's framing removed". Returns
* A's residual centroid + residual ellipsoid, the fraction of A's energy that lives
* inside B's subspace, and the centroid-difference vector. b_dims<=0 → min(3,nB). */
typedef struct {
int dim;
float* residual_centroid; /* P⊥ c_A (owned) */
float* centroid_diff; /* c_A c_B (owned) */
double centroid_diff_mag;
double variance_explained_by_B; /* (‖Qc_A‖²+Tr(QΣ_A)) / (‖c_A‖²+Tr Σ_A) ∈[0,1]*/
int removed_dims; /* # of B axes used as V_B */
double residual_scale; /* sqrt(Tr(P⊥ Σ_A P⊥)) */
int n_axes; /* residual principal axes (owned) */
GeoAxis* axes;
} GeoResidual;
int engram_geo_subtract(const GeoDescriptor* a, const GeoDescriptor* b,
int b_dims, GeoResidual* out);
void engram_geo_residual_free(GeoResidual* r);
/* set-diff variant of subtract: members in A but not in B + the centroid arrow. */
typedef struct {
char** only_ids; /* member ids in A and not in B (owned) */
int n_only;
int removed; /* |A ∩ B| (dropped) */
float* centroid_diff; /* c_A c_B (dim, owned) */
double centroid_diff_mag;
int dim;
} GeoSetDiff;
int engram_geo_setdiff(const GeoDescriptor* a, const GeoDescriptor* b, GeoSetDiff* out);
void engram_geo_setdiff_free(GeoSetDiff* s);
/* combine(A,B): a merged descriptor — POOLED centroid + POOLED covariance
* (exact law-of-total-variance: the covariance you'd get by concatenating the two
* member clouds), re-eigendecomposed for its principal axes. Members = id-union
* (membership = max). top_axes<=0 → 8. Returns a malloc'd GeoDescriptor (free with
* engram_geo_free) in the same frame as A, or NULL on error. */
GeoDescriptor* engram_geo_combine(const GeoDescriptor* a, const GeoDescriptor* b,
int top_axes);
/* distance(A,B): centroid L2 + centroid cosine + closed-form Wasserstein-2
* (Bures metric) between the two Gaussians — mirrors the proxy's _wasserstein2. */
typedef struct {
double centroid_distance;
double centroid_cosine;
double wasserstein2;
int dim;
} GeoDistance;
int engram_geo_distance(const GeoDescriptor* a, const GeoDescriptor* b, GeoDistance* out);
/* analogy(A,B): orthogonal PROCRUSTES transform min_R ‖A B R‖_F, RᵀR=I (SVD)
* aligning A's principal frame to B's (extent-scaled axes, paired by rank). R is
* returned COMPACTLY as an r×r rotation within the joint axis subspace `basis`
* (r vectors of dim floats); it acts as the identity on the orthogonal complement.
* Apply it to a vector with engram_geo_analogy_apply. */
typedef struct {
int dim;
int r; /* subspace rank; R is r×r */
float* basis; /* r×dim row-major orthonormal basis Q (owned) */
double* R; /* r×r rotation in Q-coords, row-major (owned) */
double residual; /* ‖A B R‖_F over the extent-scaled frames */
} GeoAnalogy;
int engram_geo_analogy(const GeoDescriptor* a, const GeoDescriptor* b, GeoAnalogy* out);
/* out_vec = R·v for v ∈ R^dim: v + Σ_i (R̂c c)_i q_i, c_i = q_i·v. dim floats. */
void engram_geo_analogy_apply(const GeoAnalogy* an, const float* v, float* out_vec);
void engram_geo_analogy_free(GeoAnalogy* an);
/* ═══════════════════════════════════════════════════════════════════════════
* M10 — REIFICATION: densely co-wired relational neighborhoods crystallized into
* FIRST-CLASS, PERSISTED store records (design doc §2; memory 885f5945). This is
* NOT a cache — it is durable structure. A reified neighborhood is a real store
* NODE (node_type "Neighborhood") that survives restart, is loaded on boot, and
* EVOLVES via supersede+provenance when the pattern shifts. The geometry-priming
* HOT PATH reads these persisted records (never computes geometry on the
* activation path). Ad-hoc/transient geometries still use the on-the-fly
* engram_geometry_descriptor above.
*
* Two record types, both ordinary TLV store nodes (no new on-disk format):
* - "GeoMeanFrame" : the store-wide centering mean, persisted ONCE (emb = mean
* vector, id ENGRAM_GEO_MEANFRAME_ID). Referenced by every
* neighborhood so priming centers against the SAME true mean.
* - "Neighborhood" : one reified neighborhood. emb = the RAW centroid (prototype
* point, so it stays centroid-ANN-able; centered_centroid =
* emb - meanframe). metadata = the compact "GEO1" schema:
* hub id, meanframe ref, scalar shape (radius, total_variance,
* k_core, co_registration, n_embedded), axis EXTENTS (ellipsoid
* half-widths), and the MEMBER list {id -> membership, centrality,
* core}. Member links are also persisted as edges relation="member".
*
* v1 honest simplifications (documented; extensible without migration): axis
* DIRECTION vectors are not persisted (extents capture the ellipsoid scale; the
* directions are recomputable via the on-the-fly descriptor for viz/operators);
* with hebb potentiation ~0 on today's store the "hebb-weighted" degree reduces to
* AUTHORED edge weight, so detected neighborhoods currently reflect authored edges —
* the design is unchanged and self-correcting once hebb accrues.
* ═══════════════════════════════════════════════════════════════════════════ */
#define ENGRAM_GEO_NBHD_TYPE "Neighborhood"
#define ENGRAM_GEO_MEANFRAME_TYPE "GeoMeanFrame"
#define ENGRAM_GEO_MEANFRAME_ID "geo-meanframe" /* stable id of the singleton */
#define ENGRAM_GEO_NBHD_ID_PREFIX "nbhd-" /* id = nbhd-<hub>-<built_at> */
#define ENGRAM_GEO_MEMBER_RELATION "member"
typedef struct {
int min_weighted_degree; /* hub qualifies iff strong-edge weighted degree >= this
* (0 = no floor: just rank + take top max_neighborhoods) */
int max_neighborhoods; /* homeostatic budget cap (default 128) */
double cover_membership; /* skip a hub already a member (w>=this) of an accepted
* neighborhood — greedy non-redundant cover (default 0.5) */
int persist_member_edges; /* 1 = also write relation="member" edges (default 1) */
GeoParams descriptor; /* per-neighborhood params (top_axes may be 0 = skip eigensolve) */
} GeoReifyParams;
/* Defaults: min_weighted_degree=0, max_neighborhoods=128, cover_membership=0.5,
* persist_member_edges=1, descriptor = engram_geo_default_params but top_axes=4,
* max_members=256 (reified neighborhoods stay compact). */
void engram_geo_reify_default_params(GeoReifyParams* p);
/* WRITE PATH (offline / consolidation — NEVER the activation hot path).
* Detect dense hub neighborhoods on the hebb-weighted graph, compute each one's
* CENTERED descriptor ONCE against the true store-wide mean, and PERSIST them as
* first-class records: the GeoMeanFrame (once) + one Neighborhood node per detected
* neighborhood (+ member edges), superseding any prior same-hub record with
* provenance. Read-then-write over `store`. Returns #neighborhoods persisted, or <0.
* Skips existing Neighborhood/GeoMeanFrame nodes when detecting (idempotent re-reify). */
int engram_geo_reify_store(EngramPagedStore* store, VIndex* vindex,
char** vids, int n_vids,
const GeoReifyParams* params);
/* ── Resident loaded form of the persisted records (boot-time; READ-ONLY) ─────
* The durable Neighborhood/GeoMeanFrame records are the source of truth; this
* index is their LOADED form (like the resident node array is the loaded form of
* the node records, or adjacency the loaded form of edges). It never recomputes
* geometry — it parses. Build it by feeding the runtime's boot node scan, or in
* one pass with engram_geo_reify_load. */
typedef struct GeoReifyIndex GeoReifyIndex;
GeoReifyIndex* engram_geo_reify_index_new(void);
/* Feed one store node; if it is a Neighborhood or GeoMeanFrame record it is parsed
* and absorbed (else ignored). The node is BORROWED (copied as needed). 0/<0. */
int engram_geo_reify_index_add(GeoReifyIndex* ix, const StoreNode* n);
/* Build the member->neighborhood hash after all adds. Call once. 0/<0. */
int engram_geo_reify_index_finalize(GeoReifyIndex* ix);
/* One-pass convenience: scan the store and build the finalized index. NULL if the
* store holds no reified records. */
GeoReifyIndex* engram_geo_reify_load(EngramPagedStore* store);
/* A borrowed view of one persisted neighborhood (owned by the index). */
typedef struct {
const char* id;
const char* hub_id;
int n_members;
char* const* member_ids; /* parallel arrays, length n_members */
const double* member_w; /* membership in [0,1] */
double radius;
double co_registration;
int k_core;
int n_embedded;
} GeoNeighborhood;
/* HOT-PATH LOOKUP (no geometry compute): resolve the seed set to the best
* persisted neighborhood — the one with the greatest summed seed membership; on a
* miss (no seed is a member of any neighborhood) fall back to the centroid nearest
* the query embedding (centered by the loaded mean frame). q_emb may be NULL (then
* a miss returns NULL). Returns a BORROWED handle (do NOT free) or NULL. */
const GeoNeighborhood* engram_geo_reify_lookup(
const GeoReifyIndex* ix,
const char* const* seed_ids, size_t n_seeds,
const float* q_emb, int q_dim);
int engram_geo_reify_count(const GeoReifyIndex* ix);
const float* engram_geo_reify_mean(const GeoReifyIndex* ix, int* dim); /* loaded true mean or NULL */
void engram_geo_reify_index_free(GeoReifyIndex* ix);
#endif /* ENGRAM_GEOMETRY_H */