neuron-technologies/el
3
0
Fork 0
Code Issues 4 Pull Requests 4 Actions Packages Projects Releases 5 Wiki Activity

M10: reify dense neighborhoods into first-class persisted records; geometry-priming reads them (default OFF)

Browse Source 
Reification, not a cache. Densely co-wired relational neighborhoods are crystallized
into DURABLE first-class store records that survive restart, load on boot, and evolve
via supersede+provenance -- so the geometry-priming hot path READS persisted structure
instead of computing a per-query descriptor (the M9 3.2x/13x latency blocker).

engram_geometry.{h,c}:
  - engram_geo_reify_store(): detect hub-anchored neighborhoods on the hebb-weighted
    graph (greedy non-redundant cover), compute each centered descriptor ONCE against
    the true store-wide mean, persist as node_type="Neighborhood" (raw centroid in emb,
    membership+scalars+axis-extents in a compact GEO1 metadata schema) + member edges,
    superseding any prior same-hub record. The mean is persisted once as "GeoMeanFrame".
  - resident loaded form (index_new/add/finalize/lookup): parses the durable records at
    boot (never recomputes geometry); O(seeds) membership lookup, miss -> centroid-nearest.
  - descriptor: skip the Jacobi eigensolve when top_axes==0; reject structural records as
    members (id-convention + node_type guards) so re-reify/ad-hoc stay clean.

el_runtime.c:
  - boot routes Neighborhood/GeoMeanFrame records OUT of the activation graph into the
    reify index, and skips member edges from adjacency -> ENGRAM_GEOMETRY_PRIMING OFF is
    byte-identical to M8/M9 (verified across 15 queries).
  - priming hot path reads the persisted membership; ENGRAM_GEO_PRIMING_NOCACHE=1 keeps
    the M9 per-query descriptor for ad-hoc geometries / A/B control.

A/B on a COPY (128 neighborhoods): priming ON is now FLAT latency (1.06x median / 1.03x
p90 vs OFF) where the M9 per-query path is 3.30x/10.5x. Reified records provably inert
when OFF. Restart survival + supersede verified. Build 0 warnings (my code); ASan/UBSan
clean on module and full server hot path. Recall quality re-eval against the TRUE mean
still shows no reliable gain (mean coherence -0.017), so geometry-priming STAYS default-OFF
-- but the latency blocker is removed and the durable structure now exists. See
docs/architecture/design/engram-m10-reification.md.
This commit is contained in:
Will Anderson
2026-08-12 23:06:49 -05:00
parent 7946b98d3d
commit f6a0777f90
3 changed files with 700 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lang/runtime/el_runtime.c
+145 -85
View File
@@ -7438,6 +7438,13 @@ static char* engram_first_n_chars(const char* s, size_t n) {
#include "engram_vindex.h" /* M8: ANN (HNSW) index for activation seed selection */
#include "engram_geometry.h" /* M9: centered relational-neighborhood geometry (priming) */
/* M10 REIFICATION: resident loaded form of the first-class persisted neighborhood
* records (Neighborhood + GeoMeanFrame). Built once at boot from the durable store
* (it PARSES persisted structure, never recomputes geometry). The geometry-priming
* hot path reads THIS instead of computing a per-query descriptor. NULL until boot;
* empty (count 0) on a store that has not been reified — priming then no-ops. */
static GeoReifyIndex* _eg_reify = NULL;
static EngramPagedStore* g_engram_store = NULL;
int engram_store_enabled(void) {
@@ -7597,6 +7604,16 @@ static void eg_store_put_edge(const EngramEdge* e) {
* so the store-on boot behaves byte-identically to the JSON path (M3.5 parity). */
static void eg_load_node_cb(const StoreNode* sn, void* ctx) {
EngramStore* g = (EngramStore*)ctx;
/* M10: first-class reified records (Neighborhood / GeoMeanFrame) are DURABLE
* STRUCTURE, not corpus content. Absorb them into the reify index and keep them
* OUT of the resident activation graph, so seed selection / vindex / results /
* embedding backfill are byte-identical to a store that was never reified. */
if (sn->node_type &&
(strcmp(sn->node_type, ENGRAM_GEO_NBHD_TYPE) == 0 ||
strcmp(sn->node_type, ENGRAM_GEO_MEANFRAME_TYPE) == 0)) {
if (_eg_reify) engram_geo_reify_index_add(_eg_reify, sn);
return;
}
engram_grow_nodes();
EngramNode* n = &g->nodes[g->node_count];
memset(n, 0, sizeof *n);
@@ -7628,6 +7645,12 @@ static void eg_load_node_cb(const StoreNode* sn, void* ctx) {
}
static void eg_load_edge_cb(const StoreEdge* se, void* ctx) {
EngramStore* g = (EngramStore*)ctx;
/* M10: skip the persisted member links (from a "nbhd-…" record). They are
* durable structure joining a neighborhood to its members, but inert to
* activation adjacency — dropping them here keeps spreading activation
* byte-identical to pre-reify. (Matched by id convention, so no real edge,
* whatever its relation string, is ever affected.) */
if (se->from_id && strncmp(se->from_id, ENGRAM_GEO_NBHD_ID_PREFIX, 5) == 0) return;
engram_grow_edges();
EngramEdge* e = &g->edges[g->edge_count];
memset(e, 0, sizeof *e);
@@ -7697,8 +7720,12 @@ el_val_t engram_store_boot(el_val_t data_dir) {
if (!g_engram_store) return (el_val_t)0;
EngramStore* g = engram_get();
eg_reset_resident(g);
/* M10: build the resident reify index alongside the graph load — eg_load_node_cb
* feeds the first-class Neighborhood/GeoMeanFrame records into it. */
if (!_eg_reify) _eg_reify = engram_geo_reify_index_new();
store_scan_nodes(g_engram_store, eg_load_node_cb, g);
store_scan_edges(g_engram_store, eg_load_edge_cb, g);
if (_eg_reify) engram_geo_reify_index_finalize(_eg_reify);
StoreLayer* ls = NULL; size_t ln = 0;
if (store_list_layers(g_engram_store, &ls, &ln) == 0) {
for (size_t i = 0; i < ln; i++) eg_load_layer_cb(g, &ls[i]);
@@ -9369,97 +9396,130 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
}
free(seed_dup);
/* ── M9 GEOMETRY PRIMING (ENGRAM_GEOMETRY_PRIMING, default OFF) ──────
* COMPOSES with the M8 seed set above: uses the CENTERED geometry of
* the seed neighborhood to (a) damp off-domain seeds by centered
* membership (disambiguation) and (b) prime nearby members sub-
* threshold (a warm floor). Flag OFF → this whole block is skipped and
* the seed set/activation are exactly what M8 produced. Read-only over
* the graph except for the bounded, sub-threshold seed additions here. */
if (eg_geometry_priming_on() && g_engram_store && q_emb && q_dim > 0 && nsel > 0) {
const float* gmean = eg_geo_mean_sync(q_dim);
if (gmean) {
/* Seed ids = the M8-selected semantic seeds; vids maps the
* resident-array VIndex ordinals (== g->nodes[] index) to store
* ids so the descriptor's ANN expansion resolves to paged nodes. */
const char** seed_ids = malloc((size_t)nsel * sizeof(char*));
char** vids = malloc((size_t)g->node_count * sizeof(char*));
if (seed_ids && vids) {
for (int s = 0; s < nsel; s++) seed_ids[s] = g->nodes[sel[s]].id;
for (int64_t i = 0; i < g->node_count; i++) vids[i] = g->nodes[i].id;
GeoDescriptor* geo = engram_geometry_descriptor(
g_engram_store, _eg_vindex, vids, (int)g->node_count,
seed_ids, (size_t)nsel, NULL, gmean);
/* ── GEOMETRY PRIMING (ENGRAM_GEOMETRY_PRIMING, default OFF) ─────────
* COMPOSES with the M8 seed set above: resolves the seed neighborhood's
* CENTERED geometry to (a) damp off-domain seeds by membership
* (disambiguation) and (b) prime nearby members sub-threshold (warm floor).
* Flag OFF → this whole block is skipped and the seed set/activation are
* exactly what M8 produced (byte-identical). Read-only over the graph
* except the bounded, sub-threshold seed additions here.
*
* M10: the neighborhood is READ from the PERSISTED first-class reify index
* (a durable Neighborhood record's membership map) — O(seeds) hash lookup,
* miss → centroid-nearest, NO geometry computed on the activation path. The
* membership is centered against the true store-wide mean persisted in the
* GeoMeanFrame record. Set ENGRAM_GEO_PRIMING_NOCACHE=1 to instead compute
* the descriptor fresh per query (the M9 on-the-fly path — kept for ad-hoc
* geometries and as the A/B latency control). */
if (eg_geometry_priming_on() && nsel > 0) {
static int _nocache = -1;
if (_nocache < 0) { const char* s = getenv("ENGRAM_GEO_PRIMING_NOCACHE");
_nocache = (s && s[0] && s[0] != '0') ? 1 : 0; }
const char** seed_ids = malloc((size_t)nsel * sizeof(char*));
if (seed_ids) {
for (int s = 0; s < nsel; s++) seed_ids[s] = g->nodes[sel[s]].id;
char* const* mids = NULL; /* resolved neighborhood member ids */
const double* mw = NULL; /* their centered membership in [0,1] */
int mn = 0;
GeoDescriptor* geo = NULL; /* on-the-fly path only (freed below) */
char** tmid = NULL; double* tmw = NULL;
if (!_nocache && _eg_reify && engram_geo_reify_count(_eg_reify) > 0) {
/* HOT PATH — read persisted structure, no compute. */
const GeoNeighborhood* nb = engram_geo_reify_lookup(
_eg_reify, seed_ids, (size_t)nsel, q_emb, q_dim);
if (nb && nb->n_members > 0) {
mids = nb->member_ids; mw = nb->member_w; mn = nb->n_members;
}
} else if (g_engram_store && q_emb && q_dim > 0) {
/* AD-HOC / NOCACHE control — compute the descriptor fresh. */
const float* gmean = eg_geo_mean_sync(q_dim);
char** vids = malloc((size_t)g->node_count * sizeof(char*));
if (gmean && vids) {
for (int64_t i = 0; i < g->node_count; i++) vids[i] = g->nodes[i].id;
geo = engram_geometry_descriptor(
g_engram_store, _eg_vindex, vids, (int)g->node_count,
seed_ids, (size_t)nsel, NULL, gmean);
}
free(vids);
if (geo && geo->n_members > 0) {
const double lo = eg_geo_seed_lo();
const double pscl = eg_geo_prime_scale();
const int pmax = eg_geo_prime_max();
/* Centered membership per resident idx (-1 = not in the
* geometry → left untouched by the reweight). */
double* geo_m = malloc((size_t)g->node_count * sizeof(double));
if (geo_m) {
for (int64_t i = 0; i < g->node_count; i++) geo_m[i] = -1.0;
tmid = malloc((size_t)geo->n_members * sizeof(char*));
tmw = malloc((size_t)geo->n_members * sizeof(double));
if (tmid && tmw) {
for (int m = 0; m < geo->n_members; m++) {
int64_t ri = engram_find_node_index(geo->members[m].id);
if (ri >= 0 && ri < g->node_count) {
double mv = geo->members[m].membership;
if (mv < 0.0) mv = 0.0; else if (mv > 1.0) mv = 1.0;
geo_m[ri] = mv;
}
tmid[m] = geo->members[m].id; tmw[m] = geo->members[m].membership;
}
/* (a) DAMP-ONLY seed reweight: factor = lo+(1-lo)*memb
* ∈ [lo,1]. Off-domain seeds (low centered membership)
* lose weight; the neighborhood anchor (memb→1) is
* unchanged. Never amplifies. Updates both the frontier
* act (drives propagation) and best_bg (drives this
* node's own WM weight). */
for (int64_t s = 0; s < seed_count; s++) {
int64_t si = seeds[s].idx;
if (si < 0 || si >= g->node_count) continue;
double mv = geo_m[si];
if (mv < 0.0) continue; /* not in geometry */
double factor = lo + (1.0 - lo) * mv;
seeds[s].act *= factor;
best_bg[si] *= factor;
}
/* (b) PRIME sub-threshold: descriptor members not
* already reached get a warm floor act=memb*pscl
* (pscl < WM gate ⇒ cannot self-promote) and enter the
* frontier so a warm gradient spreads one hop then dies
* at the 0.02 BFS cutoff. Capped at pmax; ISE skipped.
* Safe: BFS keeps max, so this only RAISES a floor and
* never caps a stronger legitimate activation. */
int primed = 0;
for (int m = 0; m < geo->n_members && primed < pmax; m++) {
int64_t ri = engram_find_node_index(geo->members[m].id);
if (ri < 0 || ri >= g->node_count) continue;
if (reached[ri]) continue; /* already a seed */
EngramNode* pn = &g->nodes[ri];
if (pn->node_type &&
strcmp(pn->node_type, "InternalStateEvent") == 0)
continue;
double mv = geo->members[m].membership;
if (mv < 0.0) mv = 0.0; else if (mv > 1.0) mv = 1.0;
double pact = mv * pscl;
if (pact < 0.01) continue; /* too cold to matter */
seeds[seed_count].idx = ri;
seeds[seed_count].act = pact;
seeds[seed_count].created_at = pn->created_at;
seed_count++;
best_bg[ri] = pact;
best_hops[ri] = 0;
reached[ri] = 1;
primed++;
}
_eg_act_geo_primed += primed;
free(geo_m);
mids = tmid; mw = tmw; mn = geo->n_members;
}
engram_geo_free(geo);
} else if (geo) {
engram_geo_free(geo);
}
}
free(seed_ids); free(vids);
if (mn > 0 && mids && mw) {
const double lo = eg_geo_seed_lo();
const double pscl = eg_geo_prime_scale();
const int pmax = eg_geo_prime_max();
/* membership per resident idx (-1 = not in the neighborhood). */
double* geo_m = malloc((size_t)g->node_count * sizeof(double));
if (geo_m) {
for (int64_t i = 0; i < g->node_count; i++) geo_m[i] = -1.0;
for (int m = 0; m < mn; m++) {
int64_t ri = engram_find_node_index(mids[m]);
if (ri >= 0 && ri < g->node_count) {
double mv = mw[m];
if (mv < 0.0) mv = 0.0; else if (mv > 1.0) mv = 1.0;
geo_m[ri] = mv;
}
}
/* (a) DAMP-ONLY seed reweight: factor = lo+(1-lo)*memb ∈ [lo,1].
* Off-domain members lose weight; anchor (memb→1) unchanged;
* seeds outside the neighborhood are left untouched. Never
* amplifies. Updates frontier act + best_bg (WM weight). */
for (int64_t s = 0; s < seed_count; s++) {
int64_t si = seeds[s].idx;
if (si < 0 || si >= g->node_count) continue;
double mv = geo_m[si];
if (mv < 0.0) continue; /* not in neighborhood */
double factor = lo + (1.0 - lo) * mv;
seeds[s].act *= factor;
best_bg[si] *= factor;
}
/* (b) PRIME sub-threshold: neighborhood members not already
* reached get a warm floor act=memb*pscl (pscl < WM gate ⇒
* cannot self-promote) and enter the frontier so a warm
* gradient spreads one hop then dies at the 0.02 BFS cutoff.
* Capped at pmax; ISE skipped. Safe: BFS keeps max, so this
* only RAISES a floor, never caps a legit activation. */
int primed = 0;
for (int m = 0; m < mn && primed < pmax; m++) {
int64_t ri = engram_find_node_index(mids[m]);
if (ri < 0 || ri >= g->node_count) continue;
if (reached[ri]) continue; /* already a seed */
EngramNode* pn = &g->nodes[ri];
if (pn->node_type &&
strcmp(pn->node_type, "InternalStateEvent") == 0)
continue;
double mv = mw[m];
if (mv < 0.0) mv = 0.0; else if (mv > 1.0) mv = 1.0;
double pact = mv * pscl;
if (pact < 0.01) continue; /* too cold to matter */
seeds[seed_count].idx = ri;
seeds[seed_count].act = pact;
seeds[seed_count].created_at = pn->created_at;
seed_count++;
best_bg[ri] = pact;
best_hops[ri] = 0;
reached[ri] = 1;
primed++;
}
_eg_act_geo_primed += primed;
free(geo_m);
}
}
free(tmid); free(tmw);
if (geo) engram_geo_free(geo);
free(seed_ids);
}
}
}