Files
el/engram/test/test_geometry.c
T
bigmerge bacaf3d39c
El SDK CI - dev / build-and-test (pull_request) Failing after 4m49s
engram: reconcile M8 HNSW vindex (#109) onto current dev, restore 3 fixes the branch predated
Lands feat/reframe-region-setop (PR #109: native set-based reframe_region,
decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency
work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated
seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's
actual current HEAD, plus engram-tiered-storage's still-unique test suite.

RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/
verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/
select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two
branches; #109's copy is a strict superset (adds vindex_harvest_from_store,
used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and
engram_verify.c are also byte-identical. #109's own branch point already
carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing
left to merge into #109 for those files. The one thing engram-tiered-storage
had that #109's tree dropped: its full test suite (test_vindex.c,
test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the
interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh
harnesses) — ported over here unchanged.

WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch
forked from dev on 2026-08-14 15:40 (before restructure-adjacent history
diverged the file's merge-base for `git merge` — it presented as an add/add
conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but
it silently reverted THREE dev fixes landed on 2026-08-14/15, after the
branch point, that the PR's diff had no way to know about:

  1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of
     the query-aware propagation gate dropped the shift-and-floor rescale
     about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate
     0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped
     around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi]
     is computed) is orthogonal to WHAT it gates on and both are kept.
  2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes
     wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is
     diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global).
     PR's tree predates this and dropped all three counters + their JSON
     stats fields. Restored declarations, all 4 direct increment sites, the
     eg_wm_carry_over bll increment, and the act-stats JSON fields --
     alongside (not instead of) the PR's own P4 afferent / API-reshape
     counters already in that same struct/JSON.
  3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev
     selects the STRONGEST qualifying candidate for consolidation each call;
     PR's tree predates this and reverted to hash-slot order (arbitrary wrt
     association strength) for edge formation -- the one path that writes
     PERMANENT structure. Restored the strongest-candidate while-loop,
     keeping the PR's own genuine improvement at that site
     (engram_adj_on_edge_added incremental-index append instead of a bare
     adj_dirty=1 full-rebuild flag).

engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env
flags, /api/nodes connected-count in responses) were pure additive: dev's
side was empty, PR's side added the feature. Took PR's side whole.

VERIFIED (nsbx sandbox only, live :8742/:7770 never touched):
  - cc -std=c11 -O2, clean link against the real engram/src/server.el via
    elc, zero errors.
  - vindex_bench (built standalone, read-only harvest) against the real
    production store clone (13,671 embedded nodes, 768-dim nomic-embed-text):
    recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs
    2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s
    for the full 13,671-node set -- see the flagged risk below.
  - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned
    snapshot of the live store, 13,424 nodes / 37,656 edges) and called
    /api/activate for real: first call after boot 41.5s (pays the one-time
    HNSW build inline -- matches the standalone bench), second/third calls
    356ms/605ms, no crash, correct results, act-stats JSON (including the
    restored evict_floor/cap/bll fields) reads correctly.

KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for
this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync
builds the HNSW index synchronously, inline, on the first engram_activate()
call after every process start (or index invalidation). On the real node
count that is a ~46s blocking stall on a single-threaded server -- the first
request after every restart (or its concurrent siblings) waits the full
build. Recommend a background/incremental build (or a bounded per-call build
budget) before this ever reaches the live daemon. See PR description / final
report for the fuller writeup.
2026-08-15 16:46:44 -05:00

160 lines
8.1 KiB
C

/* test_geometry.c — build + RUN gate for the M9 FOUNDATION geometry descriptor
* (engram_geometry.{c,h}). Self-contained: synthesizes a store with two KNOWN
* embedding clusters + intra-cluster hebb edges, then verifies the descriptor
* recovers the shape — centroid near the seeded cluster, skeleton = the strong
* intra-cluster edges, membership gradient, radius, positive co-registration.
*
* Pure C11; links engram_geometry.c + engram_store.c + engram_vindex.c; -lm.
* ASan/UBSan clean. Needs no live data.
*/
#include "engram_geometry.h"
#include "engram_store.h"
#include "engram_vindex.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <math.h>
#include <stdint.h>
#include <unistd.h>
#define DIM 64
static int g_fail=0;
#define CHECK(c,m) do{ if(!(c)){printf(" FAIL: %s\n",m); g_fail=1;} else printf(" ok: %s\n",m);}while(0)
static uint64_t rs=0x1234abcdULL;
static uint64_t xr(void){ uint64_t z=(rs+=0x9E3779B97F4A7C15ULL);
z=(z^(z>>30))*0xBF58476D1CE4E5B9ULL; z=(z^(z>>27))*0x94D049BB133111EBULL; return z^(z>>31); }
static float jitter(void){ return (float)(((double)(xr()>>11)*(1.0/9007199254740992.0))-0.5)*0.15f; }
/* two clusters: A centered on axis 0, B centered on axis 1. NA+NB nodes. */
#define NA 40
#define NB 40
int main(void){
printf("=== engram_geometry (M9 foundation) test suite ===\n");
char path[256]; snprintf(path,sizeof path,"/tmp/geo_test_store_%d.egm",(int)getpid());
unlink(path);
EngramPagedStore* st=store_create(path);
if(!st){ printf("FAIL: store_create\n"); return 1; }
char aids[NA][16], bids[NB][16];
/* cluster A: near +e0 ; cluster B: near +e1 */
for(int i=0;i<NA;i++){
StoreNode n; memset(&n,0,sizeof n);
snprintf(aids[i],16,"A%d",i); n.id=aids[i]; n.node_type="Concept"; n.tier="Semantic";
n.content="cluster-A"; n.salience=0.5+0.01*i;
float v[DIM]; for(int d=0;d<DIM;d++) v[d]=jitter(); v[0]=1.0f+jitter();
n.emb=v; n.emb_dim=DIM; store_put_node(st,&n);
}
for(int i=0;i<NB;i++){
StoreNode n; memset(&n,0,sizeof n);
snprintf(bids[i],16,"B%d",i); n.id=bids[i]; n.node_type="Concept"; n.tier="Semantic";
n.content="cluster-B"; n.salience=0.3;
float v[DIM]; for(int d=0;d<DIM;d++) v[d]=jitter(); v[1]=1.0f+jitter();
n.emb=v; n.emb_dim=DIM; store_put_node(st,&n);
}
/* strong intra-A hebb edges (a chain + hub), weaker cross edges A0<->B0 */
int ei=0;
for(int i=1;i<NA;i++){
StoreEdge e; memset(&e,0,sizeof e); char id[24]; snprintf(id,24,"eA%d",ei++);
e.id=id; e.from_id=aids[0]; e.to_id=aids[i]; e.relation="assoc"; e.weight=0.9; e.hebb=0.4;
store_put_edge(st,&e);
}
for(int i=1;i<NB;i++){
StoreEdge e; memset(&e,0,sizeof e); char id[24]; snprintf(id,24,"eB%d",ei++);
e.id=id; e.from_id=bids[0]; e.to_id=bids[i]; e.relation="assoc"; e.weight=0.9; e.hebb=0.4;
store_put_edge(st,&e);
}
{ StoreEdge e; memset(&e,0,sizeof e); e.id=(char*)"eX"; e.from_id=aids[0]; e.to_id=bids[0];
e.relation="assoc"; e.weight=0.5; e.hebb=0.0; store_put_edge(st,&e); }
store_close(st);
VIndex* ix=vindex_create(DIM,0,0);
char** ids=NULL; int nids=0;
int ins=vindex_build_from_store(ix, path, &ids, &nids);
CHECK(ins==NA+NB, "vindex built over all embedded nodes");
GeoParams P; engram_geo_default_params(&P); P.ann_k=20; P.max_members=0;
/* seed inside cluster A -> expect an A-dominated neighborhood */
st=store_open(path);
/* global-mean cache over the embedded set: the centering offset */
GeoMeanCache* mc=engram_geo_mean_build(st);
const float* gm=engram_geo_mean_vec(mc);
CHECK(mc!=NULL && engram_geo_mean_dim(mc)==DIM, "global-mean cache built over embedded set");
CHECK(engram_geo_mean_count(mc)==(uint64_t)(NA+NB), "global mean averaged all embedded nodes");
const char* seeds[1]={aids[0]};
/* CENTERED descriptor: pass the global mean so geometry runs in isotropic space */
GeoDescriptor* g=engram_geometry_descriptor(st, ix, ids, nids, seeds, 1, &P, gm);
CHECK(g!=NULL, "descriptor computed");
if(g){
printf(" members=%d embedded=%d edges=%d k_core=%d radius=%.4f co_reg=%.3f n_axes=%d\n",
g->n_members,g->n_embedded,g->n_edges,g->k_core,g->radius,g->co_registration,g->n_axes);
/* geometry ran in CENTERED space: g->centroid is the centered centroid,
* g->global_mean the applied offset. Reconstruct the raw prototype
* (centroid + global_mean) and check it sits on cluster-A's axis. */
CHECK(g->global_mean!=NULL, "descriptor recorded the centering offset (centered mode)");
int argmax=0; float best=-1.f;
for(int d=0;d<g->dim;d++){ float raw=g->centroid[d]+(g->global_mean?g->global_mean[d]:0.f);
if(fabsf(raw)>best){ best=fabsf(raw); argmax=d; } }
printf(" raw-prototype dominant axis = %d (expect 0); centered c[0]=%.3f c[1]=%.3f\n",
argmax, g->centroid[0], g->centroid[1]);
CHECK(argmax==0, "raw prototype sits on cluster-A's axis (near members)");
/* centering pushes A off cluster-B's axis: centered c[0] > c[1] */
CHECK(g->centroid[0] > g->centroid[1], "centered centroid leans off B's axis (isotropy)");
/* hub should be A0 (the intra-A hub with NA-1 strong edges) */
CHECK(g->hub_id && strcmp(g->hub_id,"A0")==0, "hub = the relational center A0");
/* membership: seed A0 == 1.0; A-members strong, B-members (if any) weaker */
double seedw=-1, minA=2, maxB=-1; int na=0,nb=0;
for(int i=0;i<g->n_members;i++){
const char* id=g->members[i].id; double w=g->members[i].membership;
if(strcmp(id,"A0")==0) seedw=w;
if(id[0]=='A'){ na++; if(w<minA)minA=w; }
if(id[0]=='B'){ nb++; if(w>maxB)maxB=w; }
}
printf(" A-members=%d B-members=%d seedw=%.3f\n", na,nb,seedw);
CHECK(fabs(seedw-1.0)<1e-9, "seed membership == 1.0");
CHECK(na>=NA-1, "neighborhood recovers cluster A");
/* skeleton = the strong intra-A edges: every edge eff_weight>=threshold,
* and edges connect A-nodes (co-registration should be positive: wired
* pairs are semantically near). */
int allstrong=1, allA=1;
for(int e=0;e<g->n_edges;e++){
if(g->edges[e].eff_weight < P.edge_min_weight) allstrong=0;
const char* a=g->members[g->edges[e].a].id, *b=g->members[g->edges[e].b].id;
if(!(a[0]=='A'&&b[0]=='A')) { /* the lone eX cross edge is allowed */
if(!((strcmp(a,"A0")==0&&strcmp(b,"B0")==0)||(strcmp(a,"B0")==0&&strcmp(b,"A0")==0))) allA=0; }
}
CHECK(allstrong, "skeleton holds only above-threshold (strong) edges");
CHECK(allA, "skeleton backbone is the intra-cluster wiring");
CHECK(g->co_registration>0.0, "co-registration positive (wired pairs are semantically near)");
/* principal axes: extents strictly non-increasing */
int mono=1; for(int i=1;i<g->n_axes;i++) if(g->axes[i].extent>g->axes[i-1].extent+1e-9) mono=0;
CHECK(g->n_axes>0 && mono, "principal axes sorted by descending extent");
CHECK(g->radius>0, "radius positive");
}
engram_geo_free(g);
/* edge cases: NULL store, no seeds, relational-only (NULL vindex) */
CHECK(engram_geometry_descriptor(NULL,ix,ids,nids,seeds,1,&P,gm)==NULL, "NULL store -> NULL");
CHECK(engram_geometry_descriptor(st,ix,ids,nids,seeds,0,&P,gm)==NULL, "zero seeds -> NULL");
GeoDescriptor* g2=engram_geometry_descriptor(st, NULL, NULL, 0, seeds, 1, &P, gm);
CHECK(g2!=NULL && g2->n_members>=NA-1, "relational-only path (no vindex) works");
engram_geo_free(g2);
/* raw (uncentered) mode still supported: global_mean=NULL -> no offset recorded */
GeoDescriptor* g3=engram_geometry_descriptor(st, ix, ids, nids, seeds, 1, &P, NULL);
CHECK(g3!=NULL && g3->global_mean==NULL, "raw mode (global_mean=NULL) leaves offset unset");
engram_geo_free(g3);
engram_geo_mean_free(mc);
for(int i=0;i<nids;i++) free(ids[i]); free(ids);
vindex_free(ix); store_close(st); unlink(path);
printf("\n=== %s ===\n", g_fail?"FAILURES PRESENT":"ALL TESTS PASSED");
return g_fail;
}