engram: reconcile M8 HNSW vindex (#109) onto current dev, restore 3 fixes the branch predated
El SDK CI - dev / build-and-test (pull_request) Failing after 4m49s
El SDK CI - dev / build-and-test (pull_request) Failing after 4m49s
Lands feat/reframe-region-setop (PR #109: native set-based reframe_region, decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's actual current HEAD, plus engram-tiered-storage's still-unique test suite. RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/ verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/ select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two branches; #109's copy is a strict superset (adds vindex_harvest_from_store, used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and engram_verify.c are also byte-identical. #109's own branch point already carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing left to merge into #109 for those files. The one thing engram-tiered-storage had that #109's tree dropped: its full test suite (test_vindex.c, test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh harnesses) — ported over here unchanged. WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch forked from dev on 2026-08-14 15:40 (before restructure-adjacent history diverged the file's merge-base for `git merge` — it presented as an add/add conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but it silently reverted THREE dev fixes landed on 2026-08-14/15, after the branch point, that the PR's diff had no way to know about: 1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of the query-aware propagation gate dropped the shift-and-floor rescale about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate 0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi] is computed) is orthogonal to WHAT it gates on and both are kept. 2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global). PR's tree predates this and dropped all three counters + their JSON stats fields. Restored declarations, all 4 direct increment sites, the eg_wm_carry_over bll increment, and the act-stats JSON fields -- alongside (not instead of) the PR's own P4 afferent / API-reshape counters already in that same struct/JSON. 3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev selects the STRONGEST qualifying candidate for consolidation each call; PR's tree predates this and reverted to hash-slot order (arbitrary wrt association strength) for edge formation -- the one path that writes PERMANENT structure. Restored the strongest-candidate while-loop, keeping the PR's own genuine improvement at that site (engram_adj_on_edge_added incremental-index append instead of a bare adj_dirty=1 full-rebuild flag). engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env flags, /api/nodes connected-count in responses) were pure additive: dev's side was empty, PR's side added the feature. Took PR's side whole. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): - cc -std=c11 -O2, clean link against the real engram/src/server.el via elc, zero errors. - vindex_bench (built standalone, read-only harvest) against the real production store clone (13,671 embedded nodes, 768-dim nomic-embed-text): recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs 2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s for the full 13,671-node set -- see the flagged risk below. - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned snapshot of the live store, 13,424 nodes / 37,656 edges) and called /api/activate for real: first call after boot 41.5s (pays the one-time HNSW build inline -- matches the standalone bench), second/third calls 356ms/605ms, no crash, correct results, act-stats JSON (including the restored evict_floor/cap/bll fields) reads correctly. KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync builds the HNSW index synchronously, inline, on the first engram_activate() call after every process start (or index invalidation). On the real node count that is a ~46s blocking stall on a single-threaded server -- the first request after every restart (or its concurrent siblings) waits the full build. Recommend a background/incremental build (or a bounded per-call build budget) before this ever reaches the live daemon. See PR description / final report for the fuller writeup.
This commit is contained in:
bigmerge
+972
-2
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,149 @@
|
||||
/* bench_discrimination.c — M9 REFINEMENT bench: measures whether mean-centering
|
||||
* the anisotropic nomic-embed-text space sharpens the §5 geometry operators on
|
||||
* REAL data. Read-only over a COPY of the live store (never the live file).
|
||||
*
|
||||
* usage: bench_discrimination [store.egm]
|
||||
* (or set ENGRAM_BENCH_STORE). If no store is given/openable it prints
|
||||
* SKIP and exits 0 — so it is safe in CI without live data.
|
||||
*
|
||||
* It picks two semantically distinct cohorts by keyword (domain A vs domain B),
|
||||
* computes the global mean over the embed-eligible set (via engram_geo_mean_build
|
||||
* — the same offset the descriptor uses), then reports BEFORE (raw unit space)
|
||||
* vs AFTER (mean-centered space):
|
||||
* - cross-centroid cosine (lower = better separated)
|
||||
* - cross-centroid Euclid dist (translation-invariant: a control)
|
||||
* - intra-cohesion per domain (member cos to own centroid)
|
||||
* - overlap operator (cross_cos / sqrt(intraA*intraB): ~1 = domains
|
||||
* indistinguishable, ~0 = cleanly separated)
|
||||
* - angular separation ratio z (centroid angle / summed angular spread)
|
||||
* - mean pairwise cosine sample (the anisotropy headline; ~0.55 raw -> ~0 ctr)
|
||||
*
|
||||
* Pure C11; links engram_store.c + engram_geometry.c; -lm.
|
||||
*/
|
||||
#include "engram_store.h"
|
||||
#include "engram_geometry.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <strings.h>
|
||||
#include <math.h>
|
||||
|
||||
#define CAP_DOMAIN 400
|
||||
#define CAP_SAMPLE 800
|
||||
|
||||
typedef struct { float** v; int n, cap, dim; } VecSet;
|
||||
static void vs_init(VecSet* s){ s->v=NULL; s->n=0; s->cap=0; s->dim=0; }
|
||||
static void vs_push(VecSet* s, const float* e, int dim, int cap){
|
||||
if(s->n>=cap) return;
|
||||
if(s->dim==0) s->dim=dim;
|
||||
if(s->n==s->cap){ int nc=s->cap?s->cap*2:64; s->v=realloc(s->v,(size_t)nc*sizeof*s->v); s->cap=nc; }
|
||||
float* c=malloc((size_t)dim*sizeof(float));
|
||||
double nn=0; for(int d=0;d<dim;d++) nn+=(double)e[d]*e[d]; nn=sqrt(nn);
|
||||
if(nn<1e-12){ free(c); return; }
|
||||
for(int d=0;d<dim;d++) c[d]=(float)(e[d]/nn); /* L2-normalized copy */
|
||||
s->v[s->n++]=c;
|
||||
}
|
||||
static void vs_free(VecSet* s){ for(int i=0;i<s->n;i++) free(s->v[i]); free(s->v); }
|
||||
|
||||
typedef struct { VecSet A, B, S; long idx; } Coh;
|
||||
static int has(const char* h, const char* n){ return h && strcasestr(h,n)!=NULL; }
|
||||
static void cb(const StoreNode* n, void* ctx){
|
||||
Coh* c=ctx;
|
||||
if(!(n->emb && n->emb_dim>0)) return;
|
||||
/* every 5th embedded node -> isotropy sample */
|
||||
if((c->idx++ % 5)==0) vs_push(&c->S, n->emb, n->emb_dim, CAP_SAMPLE);
|
||||
const char* t=n->content; const char* g=n->tags;
|
||||
int A = has(t,"quantiz")||has(g,"quantiz")||has(t,"lorablation")||has(t,"70B")||has(t,"LoRA merge");
|
||||
int B = has(t,"kubernetes")||has(t,"terraform")||has(t,"argo")||has(g,"infrastructure")||has(t,"vault")||has(t,"cloudflare");
|
||||
if(A && !B) vs_push(&c->A, n->emb, n->emb_dim, CAP_DOMAIN);
|
||||
else if(B && !A) vs_push(&c->B, n->emb, n->emb_dim, CAP_DOMAIN);
|
||||
}
|
||||
|
||||
/* mean of a VecSet into out (dim doubles). */
|
||||
static void mean_of(const VecSet* s, const float* gm, double* out){
|
||||
int dim=s->dim; for(int d=0;d<dim;d++) out[d]=0;
|
||||
for(int i=0;i<s->n;i++) for(int d=0;d<dim;d++) out[d]+=(double)s->v[i][d]-(gm?gm[d]:0.0);
|
||||
if(s->n) for(int d=0;d<dim;d++) out[d]/=s->n;
|
||||
}
|
||||
static double dnorm(const double* a, int dim){ double s=0; for(int d=0;d<dim;d++) s+=a[d]*a[d]; return sqrt(s); }
|
||||
static double dcos(const double* a, const double* b, int dim){
|
||||
double na=dnorm(a,dim), nb=dnorm(b,dim); if(na<1e-12||nb<1e-12) return 0;
|
||||
double s=0; for(int d=0;d<dim;d++) s+=a[d]*b[d]; double c=s/(na*nb);
|
||||
if(c>1)c=1; if(c<-1)c=-1; return c;
|
||||
}
|
||||
static double deuclid(const double* a, const double* b, int dim){
|
||||
double s=0; for(int d=0;d<dim;d++){ double x=a[d]-b[d]; s+=x*x; } return sqrt(s);
|
||||
}
|
||||
/* mean cosine of members (minus gm) to centroid c (already gm-subtracted). */
|
||||
static double cohesion(const VecSet* s, const float* gm, const double* c){
|
||||
int dim=s->dim; double nc=dnorm(c,dim); if(nc<1e-12||s->n==0) return 0;
|
||||
double acc=0; for(int i=0;i<s->n;i++){
|
||||
double dot=0, nv=0;
|
||||
for(int d=0;d<dim;d++){ double v=(double)s->v[i][d]-(gm?gm[d]:0.0); dot+=v*c[d]; nv+=v*v; }
|
||||
nv=sqrt(nv); if(nv<1e-12) continue; double cc=dot/(nv*nc);
|
||||
if(cc>1)cc=1; if(cc<-1)cc=-1; acc+=cc;
|
||||
}
|
||||
return acc/s->n;
|
||||
}
|
||||
/* mean pairwise cosine over a sample (isotropy metric). */
|
||||
static double mean_pairwise_cos(const VecSet* s, const float* gm){
|
||||
int dim=s->dim; if(s->n<2) return 0; double acc=0; long np=0;
|
||||
for(int i=0;i<s->n;i++) for(int j=i+1;j<s->n;j++){
|
||||
double dot=0, na=0, nb=0;
|
||||
for(int d=0;d<dim;d++){ double a=(double)s->v[i][d]-(gm?gm[d]:0.0), b=(double)s->v[j][d]-(gm?gm[d]:0.0);
|
||||
dot+=a*b; na+=a*a; nb+=b*b; }
|
||||
na=sqrt(na); nb=sqrt(nb); if(na<1e-12||nb<1e-12) continue;
|
||||
double c=dot/(na*nb); if(c>1)c=1; if(c<-1)c=-1; acc+=c; np++;
|
||||
}
|
||||
return np? acc/np : 0;
|
||||
}
|
||||
|
||||
static void report(const char* label, Coh* c, const float* gm){
|
||||
int dim=c->A.dim; double* ca=malloc((size_t)dim*sizeof(double)); double* cb=malloc((size_t)dim*sizeof(double));
|
||||
mean_of(&c->A, gm, ca); mean_of(&c->B, gm, cb);
|
||||
double xcos=dcos(ca,cb,dim), xeuc=deuclid(ca,cb,dim);
|
||||
double cohA=cohesion(&c->A,gm,ca), cohB=cohesion(&c->B,gm,cb);
|
||||
double overlap = (cohA>0&&cohB>0)? xcos/sqrt(cohA*cohB) : xcos;
|
||||
double theta = acos(xcos<-1?-1:(xcos>1?1:xcos));
|
||||
double sig = acos(cohA<-1?-1:(cohA>1?1:cohA)) + acos(cohB<-1?-1:(cohB>1?1:cohB));
|
||||
double z = (sig>1e-9)? theta/sig : 0;
|
||||
double mpc = mean_pairwise_cos(&c->S, gm);
|
||||
printf(" [%s]\n", label);
|
||||
printf(" cross-centroid cosine = %+.4f (lower = better separated)\n", xcos);
|
||||
printf(" cross-centroid Euclid = %.4f (translation-invariant control)\n", xeuc);
|
||||
printf(" intra-cohesion A / B = %.4f / %.4f\n", cohA, cohB);
|
||||
printf(" OVERLAP operator = %.4f (~1 = indistinguishable, ~0 = clean)\n", overlap);
|
||||
printf(" angular separation z = %.3f (centroid-angle / summed spread; >1 = separated)\n", z);
|
||||
printf(" mean pairwise cosine = %+.4f (isotropy: ~0.55 anisotropic -> ~0 isotropic)\n", mpc);
|
||||
free(ca); free(cb);
|
||||
}
|
||||
|
||||
int main(int argc, char** argv){
|
||||
const char* path = (argc>1)? argv[1] : getenv("ENGRAM_BENCH_STORE");
|
||||
if(!path){ printf("SKIP: no store path (arg or ENGRAM_BENCH_STORE)\n"); return 0; }
|
||||
EngramPagedStore* st=store_open(path);
|
||||
if(!st){ printf("SKIP: could not open %s\n", path); return 0; }
|
||||
|
||||
Coh c; vs_init(&c.A); vs_init(&c.B); vs_init(&c.S); c.idx=0;
|
||||
store_scan_nodes(st, cb, &c);
|
||||
printf("=== two-domain discrimination bench (real store copy) ===\n");
|
||||
printf("domain A (quantization) n=%d ; domain B (infrastructure) n=%d ; sample n=%d ; dim=%d\n",
|
||||
c.A.n, c.B.n, c.S.n, c.A.dim);
|
||||
if(c.A.n<3 || c.B.n<3){ printf("SKIP: a cohort is too small to be meaningful\n");
|
||||
vs_free(&c.A); vs_free(&c.B); vs_free(&c.S); store_close(st); return 0; }
|
||||
|
||||
GeoMeanCache* mc=engram_geo_mean_build(st);
|
||||
const float* gm=engram_geo_mean_vec(mc);
|
||||
printf("global-mean cache: dim=%d over %llu embedded nodes\n\n",
|
||||
engram_geo_mean_dim(mc), (unsigned long long)engram_geo_mean_count(mc));
|
||||
|
||||
printf("BEFORE (raw anisotropic unit space):\n");
|
||||
report("RAW", &c, NULL);
|
||||
printf("\nAFTER (mean-centered isotropic space):\n");
|
||||
report("CENTERED", &c, gm);
|
||||
|
||||
engram_geo_mean_free(mc);
|
||||
vs_free(&c.A); vs_free(&c.B); vs_free(&c.S);
|
||||
store_close(st);
|
||||
return 0;
|
||||
}
|
||||
Executable
+22
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env bash
|
||||
# M4 demand-paging buffer-pool gate. Pure C (NOT elb/elc). Writes only under /tmp.
|
||||
# Runs the suite twice: an -O2 correctness build and an ASan+UBSan build.
|
||||
set -e
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
SRC="$HERE/../../lang/runtime/engram_store.c"
|
||||
TST="$HERE/test_bufpool.c"
|
||||
|
||||
echo "== compiling (gcc -O2): test_bufpool.c engram_store.c =="
|
||||
BIN="/tmp/test_bufpool.$$"
|
||||
gcc -O2 -Wall -Wextra -std=c11 "$TST" "$SRC" -o "$BIN"
|
||||
"$BIN"; rc=$?
|
||||
rm -f "$BIN"; rm -rf /tmp/engram-bufpool-test-*
|
||||
[ $rc -ne 0 ] && exit $rc
|
||||
|
||||
echo
|
||||
echo "== ASan+UBSan build (memory-error + UB checks; LSan unavailable on macOS) =="
|
||||
ABIN="/tmp/test_bufpool_asan.$$"
|
||||
gcc -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -std=c11 "$TST" "$SRC" -o "$ABIN"
|
||||
ASAN_OPTIONS=detect_leaks=0 UBSAN_OPTIONS=halt_on_error=1 "$ABIN"; rc=$?
|
||||
rm -f "$ABIN"; rm -rf /tmp/engram-bufpool-test-*
|
||||
exit $rc
|
||||
Executable
+22
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env bash
|
||||
# M5 online-compaction + background-checkpointer gate. Pure C (NOT elb/elc).
|
||||
# Writes only under /tmp. Runs an -O2 correctness build then an ASan+UBSan build.
|
||||
set -e
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
SRC="$HERE/../../lang/runtime/engram_store.c"
|
||||
TST="$HERE/test_compaction.c"
|
||||
|
||||
echo "== compiling (gcc -O2): test_compaction.c engram_store.c =="
|
||||
BIN="/tmp/test_compaction.$$"
|
||||
gcc -O2 -Wall -Wextra -std=c11 "$TST" "$SRC" -o "$BIN"
|
||||
"$BIN"; rc=$?
|
||||
rm -f "$BIN"; rm -rf /tmp/engram-compact-test-*
|
||||
[ $rc -ne 0 ] && exit $rc
|
||||
|
||||
echo
|
||||
echo "== ASan+UBSan build (memory-error + UB checks; LSan unavailable on macOS) =="
|
||||
ABIN="/tmp/test_compaction_asan.$$"
|
||||
gcc -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -std=c11 "$TST" "$SRC" -o "$ABIN"
|
||||
ASAN_OPTIONS=detect_leaks=0 UBSAN_OPTIONS=halt_on_error=1 "$ABIN"; rc=$?
|
||||
rm -f "$ABIN"; rm -rf /tmp/engram-compact-test-*
|
||||
exit $rc
|
||||
Executable
+30
@@ -0,0 +1,30 @@
|
||||
#!/bin/sh
|
||||
# Build + RUN the M9 FOUNDATION geometry-descriptor tests. Pure C11 (gcc/cc),
|
||||
# stdlib + libm only. Standalone module — NOT folded through elb/elc. Two passes:
|
||||
# 1. PERF — optimised (-O2, no sanitizer): the functional gate.
|
||||
# 2. SAFETY — ASan + UBSan on the same suite (memory-safety is size-independent).
|
||||
set -e
|
||||
HERE=$(cd "$(dirname "$0")" && pwd)
|
||||
RT="$HERE/../../lang/runtime"
|
||||
CC=${CC:-cc}
|
||||
SRC="$HERE/test_geometry.c $RT/engram_geometry.c $RT/engram_store.c $RT/engram_vindex.c"
|
||||
WARN="-std=c11 -Wall -Wextra"
|
||||
TMP=$(mktemp -d)
|
||||
|
||||
echo "### PASS 1: PERF (optimised, un-sanitised) — functional gate"
|
||||
$CC $WARN -O2 -I"$RT" $SRC -lm -o "$TMP/perf"
|
||||
"$TMP/perf"
|
||||
|
||||
echo
|
||||
echo "### PASS 2: SAFETY (ASan/UBSan)"
|
||||
$CC $WARN -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -I"$RT" $SRC -lm -o "$TMP/safe"
|
||||
ASAN_OPTIONS=${ASAN_OPTIONS:-detect_leaks=0} UBSAN_OPTIONS=halt_on_error=1 "$TMP/safe"
|
||||
|
||||
# PASS 3 (OPTIONAL): mean-centering discrimination bench on a COPY of a real
|
||||
# store. Skips cleanly unless ENGRAM_BENCH_STORE points at a store .egm — never
|
||||
# touches the live store. Read-only; not part of the pass/fail gate.
|
||||
echo
|
||||
echo "### PASS 3: DISCRIMINATION BENCH (optional; set ENGRAM_BENCH_STORE)"
|
||||
BSRC="$HERE/bench_discrimination.c $RT/engram_geometry.c $RT/engram_store.c $RT/engram_vindex.c"
|
||||
$CC $WARN -O2 -I"$RT" $BSRC -lm -o "$TMP/bench"
|
||||
"$TMP/bench" "${ENGRAM_BENCH_STORE:-}"
|
||||
Executable
+86
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env bash
|
||||
# M-INTEROCEPTION P0 gate: engram_scan_nodes_emb_json read-only builtin.
|
||||
# Throwaway HOME + /tmp only. Never touches ~/.neuron or :8742.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p0-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compile (plain) =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p0_emb.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p0" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
D="$WORK/d"; mkdir -p "$D"
|
||||
"$WORK/p0" "$D" || { echo "FAIL: run"; fail=1; }
|
||||
|
||||
echo
|
||||
echo "== assertions =="
|
||||
python3 - "$D" <<'PY'
|
||||
import json, sys, os
|
||||
d = sys.argv[1]
|
||||
def load(n):
|
||||
with open(os.path.join(d,n)) as f: return json.load(f)
|
||||
rc = 0
|
||||
def check(c,m):
|
||||
global rc
|
||||
print((" PASS: " if c else " FAIL: ")+m)
|
||||
if not c: rc=1
|
||||
|
||||
alln = load("emb_all.json")
|
||||
check(len(alln)==3, f"emb dump returns all 3 nodes (got {len(alln)})")
|
||||
# salience-sorted: high, mid, low
|
||||
labels=[n["label"] for n in alln]
|
||||
check(labels==["emb-high","emb-mid","noemb-low"], f"salience-sorted order {labels}")
|
||||
for n in alln:
|
||||
L=len(n["emb"])
|
||||
check(L==n["emb_dim"], f"{n['label']}: len(emb)={L} == emb_dim={n['emb_dim']}")
|
||||
check(alln[0]["emb_dim"]==16 and alln[1]["emb_dim"]==16, "embedded nodes report dim 16")
|
||||
check(alln[2]["emb_dim"]==0 and alln[2]["emb"]==[], "un-embedded node -> emb_dim 0, emb []")
|
||||
# first emb value round-trips ~0.10
|
||||
check(abs(alln[0]["emb"][0]-0.10)<1e-3, f"emb[0] round-trips (~0.10, got {alln[0]['emb'][0]})")
|
||||
|
||||
pg0=load("emb_pg0.json"); pg1=load("emb_pg1.json")
|
||||
check(len(pg0)==1 and len(pg1)==1, "pagination: one node per page")
|
||||
check(pg0[0]["id"]=="n-high" and pg1[0]["id"]=="n-mid", f"pages disjoint & ordered ({pg0[0]['id']},{pg1[0]['id']})")
|
||||
|
||||
plain=load("plain.json")
|
||||
check(len(plain)==3, "existing scan_nodes_json still returns 3")
|
||||
check(all("emb" not in n for n in plain), "existing scan_nodes_json carries NO emb (behavior-neutral)")
|
||||
sys.exit(rc)
|
||||
PY
|
||||
[ $? -ne 0 ] && fail=1
|
||||
|
||||
echo
|
||||
echo "== latency (one 256-page over the 3-node copy) =="
|
||||
python3 - "$D" <<'PY'
|
||||
import os
|
||||
# timing was measured inside C not here; report emb payload size as a proxy
|
||||
sz=os.path.getsize(os.path.join(os.sys.argv[1] if False else __import__('sys').argv[1],"emb_all.json"))
|
||||
print(f" emb_all.json payload = {sz} bytes for 3 nodes")
|
||||
PY
|
||||
|
||||
echo
|
||||
echo "== ASan+UBSan =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p0_emb.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p0.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -20 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p0.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
DS="$WORK/ds"; mkdir -p "$DS"
|
||||
"$WORK/p0.san" "$DS" >/dev/null 2>"$WORK/san_run.log"
|
||||
if grep -qiE 'runtime error|AddressSanitizer|Sanitizer|ERROR: ' "$WORK/san_run.log"; then
|
||||
echo " FAIL: sanitizer findings:"; grep -iE 'runtime error|Sanitizer|ERROR' "$WORK/san_run.log" | head; fail=1
|
||||
else echo " ok: ASan+UBSan clean"; fi
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$fail" -eq 0 ]; then echo "====== P0 EMB-ENDPOINT GATE: PASS ======"; else echo "====== P0 EMB-ENDPOINT GATE: FAIL ======"; fi
|
||||
rm -rf "$WORK"
|
||||
exit $fail
|
||||
Executable
+147
@@ -0,0 +1,147 @@
|
||||
#!/usr/bin/env bash
|
||||
# M-INTEROCEPTION P1 gate: two-threshold consolidation (ENGRAM_CONSOLIDATION).
|
||||
# Throwaway HOME + /tmp only. Never touches ~/.neuron or :8742.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p1-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
unset ENGRAM_STORE ENGRAM_CONSOLIDATION ENGRAM_CONSOL_CONN_MIN ENGRAM_CONSOL_PERM_MIN ENGRAM_CONSOL_WM_TOPK
|
||||
fail=0
|
||||
|
||||
echo "== compile =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p1_consol.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p1" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
echo
|
||||
echo "== (a) HEADLINE: hebb accrual curve over N co-activations (flag OFF, pure trunk) =="
|
||||
D="$WORK/a"; mkdir -p "$D"
|
||||
( unset ENGRAM_CONSOLIDATION; "$WORK/p1" accrual "$D" ) >"$WORK/accrual.txt" 2>&1 || { echo "FAIL accrual run"; fail=1; }
|
||||
python3 - "$WORK/accrual.txt" <<'PY'
|
||||
import json,sys,re
|
||||
rows=[]
|
||||
for line in open(sys.argv[1]):
|
||||
m=re.match(r'SAMPLE (\d+) (\{.*\})',line.strip())
|
||||
if not m: continue
|
||||
n=int(m.group(1)); j=json.loads(m.group(2))
|
||||
hm=j.get("hebb_max",0.0); hc=j.get("hebb_cand_max",0.0)
|
||||
rows.append((n,hm,hc))
|
||||
print(" N hebb_max 1-0.9999^N (predicted EWMA)")
|
||||
rc=0
|
||||
for n,hm,hc in rows:
|
||||
pred=1-0.9999**n
|
||||
print(f" {n:<7} {hm:<12.6g} {pred:.6g}")
|
||||
# assertions: monotonic rise, starts near ETA, tracks EWMA prediction
|
||||
first=rows[0]; last=rows[-1]
|
||||
def check(c,m):
|
||||
global rc; print((" PASS: " if c else " FAIL: ")+m);
|
||||
if not c: rc=1
|
||||
check(abs(first[1]-0.0001)<5e-5, f"first sample hebb ~= ETA 0.0001 (got {first[1]:.6g})")
|
||||
check(all(rows[i][1] <= rows[i+1][1]+1e-9 for i in range(len(rows)-1)), "hebb_max is monotonically non-decreasing over N")
|
||||
check(last[1] > first[1]*50, f"hebb accrues substantially by N={last[0]} (got {last[1]:.4g} vs {first[1]:.4g})")
|
||||
# EWMA fit: measured should be within 25% of 1-0.9999^N at the mid samples
|
||||
mid=[r for r in rows if 100<=r[0]<=2000]
|
||||
ok=all(abs(hm-(1-0.9999**n))/(1-0.9999**n) < 0.25 for n,hm,hc in mid)
|
||||
check(ok, "measured curve tracks the 1-0.9999^N EWMA prediction within 25% (co-activation P~1)")
|
||||
sys.exit(rc)
|
||||
PY
|
||||
[ $? -ne 0 ] && fail=1
|
||||
|
||||
echo
|
||||
echo "== (b) CONNECTION threshold: strong ISE wires to wm_top, weak ISE wires nothing (flag ON) =="
|
||||
D="$WORK/b"; mkdir -p "$D"
|
||||
( export ENGRAM_CONSOLIDATION=1; "$WORK/p1" connect "$D" ) >"$WORK/connect.txt" 2>&1 || { echo "FAIL connect run"; fail=1; }
|
||||
cat "$WORK/connect.txt" | sed 's/^/ /'
|
||||
python3 - "$WORK/connect.txt" "$D/connect.json" <<'PY'
|
||||
import json,sys,re
|
||||
txt=open(sys.argv[1]).read()
|
||||
g=json.load(open(sys.argv[2]))
|
||||
def field(k):
|
||||
m=re.search(rf'{k} (\S+)',txt); return m.group(1) if m else None
|
||||
sid=field("ISE_STRONG_ID"); wid=field("ISE_WEAK_ID")
|
||||
m=re.search(r'EDGES before=(\d+) after_strong=(\d+) after_weak=(\d+)',txt)
|
||||
before,aftS,aftW=int(m.group(1)),int(m.group(2)),int(m.group(3))
|
||||
rc=0
|
||||
def check(c,mm):
|
||||
global rc; print((" PASS: " if c else " FAIL: ")+mm)
|
||||
if not c: rc=1
|
||||
strong_edges=[e for e in g["edges"] if e["from_id"]==sid and e["relation"]=="hebbian-associate"]
|
||||
weak_edges=[e for e in g["edges"] if e["from_id"]==wid]
|
||||
check(aftS>before, f"strong ISE formed connection edges ({before} -> {aftS})")
|
||||
check(aftW==aftS, f"weak ISE formed NO edges ({aftS} -> {aftW})")
|
||||
check(len(strong_edges)>=1, f"strong ISE has {len(strong_edges)} hebbian-associate edge(s) to wm_top")
|
||||
check(all('consolidated-from-ISE' in (e.get('metadata') or '') for e in strong_edges),
|
||||
"connection edges are provenance-tagged consolidated-from-ISE (reversible)")
|
||||
check(len(weak_edges)==0, "weak ISE (below connection bar) has zero outgoing edges")
|
||||
# targets must be the WM-top nodes (hebb-a / hebb-b), not distractors
|
||||
tgt_labels=set()
|
||||
byid={n["id"]:n for n in g["nodes"]}
|
||||
for e in strong_edges:
|
||||
t=byid.get(e["to_id"]);
|
||||
if t: tgt_labels.add(t.get("label"))
|
||||
print(f" connection targets: {sorted(tgt_labels)}")
|
||||
check(tgt_labels.issubset({"hebb-a","hebb-b"}) and len(tgt_labels)>=1,
|
||||
f"connections point at the wm_top nodes {sorted(tgt_labels)}")
|
||||
sys.exit(rc)
|
||||
PY
|
||||
[ $? -ne 0 ] && fail=1
|
||||
|
||||
echo
|
||||
echo "== (c) PERMANENCE threshold: promoted node survives 48h prune, ephemeral is swept (flag ON) =="
|
||||
D="$WORK/c"; mkdir -p "$D"
|
||||
( export ENGRAM_CONSOLIDATION=1 ENGRAM_CONSOL_PERM_MIN=-1000; "$WORK/p1" perm "$D" ) >"$WORK/perm.txt" 2>&1 || { echo "FAIL perm run"; fail=1; }
|
||||
cat "$WORK/perm.txt" | sed 's/^/ /'
|
||||
python3 - "$WORK/perm.txt" <<'PY'
|
||||
import sys,re,json
|
||||
txt=open(sys.argv[1]).read()
|
||||
rc=0
|
||||
def check(c,m):
|
||||
global rc; print((" PASS: " if c else " FAIL: ")+m)
|
||||
if not c: rc=1
|
||||
prom=int(re.search(r'PROMOTED (\d+)',txt).group(1))
|
||||
m=re.search(r'NODES before=(\d+) after=(\d+) removed=(\d+)',txt)
|
||||
before,after,removed=int(m.group(1)),int(m.group(2)),int(m.group(3))
|
||||
dur=re.search(r'DURABLE_NODE (\{.*\})',txt).group(1)
|
||||
eph=re.search(r'EPHEMERAL_NODE (\{.*\})',txt).group(1)
|
||||
durj=json.loads(dur); ephj=json.loads(eph)
|
||||
check(prom==1, "engram_consolidate_permanence promoted the node (returned 1)")
|
||||
check(before==2 and after==1 and removed==1, f"exactly one node pruned ({before}->{after}, removed={removed})")
|
||||
check(durj.get("id")=="ise-durable", "durable node SURVIVED the 48h telemetry prune")
|
||||
check('consolidated-from-ISE' in (durj.get("metadata") or ''), "durable node carries reversible provenance marker")
|
||||
check(ephj=={} or not ephj.get("id"), "ephemeral (non-permanent) ISE was swept")
|
||||
sys.exit(rc)
|
||||
PY
|
||||
[ $? -ne 0 ] && fail=1
|
||||
|
||||
echo
|
||||
echo "== (d) OFF path byte-identical: ISE creation forms no edges, permanence is a no-op =="
|
||||
D="$WORK/d"; mkdir -p "$D"
|
||||
( unset ENGRAM_CONSOLIDATION; "$WORK/p1" offcheck "$D" ) >"$WORK/off.txt" 2>&1
|
||||
rcoff=$?
|
||||
cat "$WORK/off.txt" | sed 's/^/ /'
|
||||
[ $rcoff -eq 0 ] && echo " PASS: flag OFF — ISE creation added 0 edges and permanence returned 0" \
|
||||
|| { echo " FAIL: OFF path changed behavior"; fail=1; }
|
||||
|
||||
echo
|
||||
echo "== ASan+UBSan (connect + perm + accrual-short) =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p1_consol.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p1.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p1.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
DS="$WORK/san"; mkdir -p "$DS"
|
||||
( export ENGRAM_CONSOLIDATION=1 ENGRAM_CONSOL_PERM_MIN=-1000; "$WORK/p1.san" connect "$DS" ) >/dev/null 2>"$WORK/san_run.log"
|
||||
( export ENGRAM_CONSOLIDATION=1 ENGRAM_CONSOL_PERM_MIN=-1000; "$WORK/p1.san" perm "$DS" ) >/dev/null 2>>"$WORK/san_run.log"
|
||||
if grep -qiE 'runtime error|AddressSanitizer|Sanitizer|ERROR: ' "$WORK/san_run.log"; then
|
||||
echo " FAIL: sanitizer findings:"; grep -iE 'runtime error|Sanitizer|ERROR' "$WORK/san_run.log" | head; fail=1
|
||||
else echo " ok: ASan+UBSan clean"; fi
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$fail" -eq 0 ]; then echo "====== P1 CONSOLIDATION GATE: PASS ======"; else echo "====== P1 CONSOLIDATION GATE: FAIL ======"; fi
|
||||
rm -rf "$WORK"
|
||||
exit $fail
|
||||
Executable
+96
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env bash
|
||||
# M-INTEROCEPTION P2 gate: chronoception (ENGRAM_CHRONOCEPTION).
|
||||
# Throwaway HOME + /tmp only. TC defaults to 3600s; we pin it for the math.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p2-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
export ENGRAM_CHRONO_TC=3600 # pin cooling time-constant for the math
|
||||
unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compile =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p2_chrono.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p2" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
sum_wm(){ python3 -c "import json,sys; g=json.load(open('$1')); print(sum(n.get('working_memory_weight',0) for n in g['nodes']))"; }
|
||||
|
||||
echo
|
||||
echo "== (a) cooling scales with dt (flag ON) =="
|
||||
for DT in 600000 1800000 3600000 7200000; do # 600s,1800s,3600s,7200s at TC=3600
|
||||
D="$WORK/dt$DT"; mkdir -p "$D"
|
||||
( export ENGRAM_CHRONOCEPTION=1; "$WORK/p2" once "$D" "$DT" ) >"$D/out.txt" 2>&1
|
||||
MAG=$(grep MAGNITUDE "$D/out.txt" | awk '{print $2}')
|
||||
WM=$(sum_wm "$D/field.json")
|
||||
PRED=$(python3 -c "import math; print(round(1-math.exp(-$DT/1000/3600),6))")
|
||||
echo " dt=${DT}ms magnitude=$MAG predicted 1-exp(-dt/TC)=$PRED field_wm_sum=$WM"
|
||||
python3 -c "import sys; m=float('$MAG'); p=float('$PRED'); sys.exit(0 if abs(m-p)<1e-4 else 1)" \
|
||||
&& echo " PASS: magnitude matches exp cooling" || { echo " FAIL"; fail=1; }
|
||||
done
|
||||
|
||||
echo
|
||||
echo "== (b) SCALE-INVARIANCE: age(dt) once == age(dt/N) N times (field within float tol) =="
|
||||
DT=3600000
|
||||
for N in 2 10 100; do
|
||||
DA="$WORK/inv_once_$N"; DB="$WORK/inv_split_$N"; mkdir -p "$DA" "$DB"
|
||||
( export ENGRAM_CHRONOCEPTION=1; "$WORK/p2" once "$DA" "$DT" ) >/dev/null 2>&1
|
||||
( export ENGRAM_CHRONOCEPTION=1; "$WORK/p2" split "$DB" "$DT" "$N" ) >/dev/null 2>&1
|
||||
WA=$(sum_wm "$DA/field.json"); WB=$(sum_wm "$DB/field.json")
|
||||
echo " N=$N once_wm=$WA split_wm=$WB |delta|=$(python3 -c "print(abs($WA-$WB))")"
|
||||
python3 -c "import sys; sys.exit(0 if abs($WA-$WB)<1e-9 else 1)" \
|
||||
&& echo " PASS: scale-invariant within 1e-9" || { echo " FAIL: not scale-invariant"; fail=1; }
|
||||
done
|
||||
|
||||
echo
|
||||
echo "== (c) REBOOT catch-up: one-shot cooling from persisted last-tick, reports MAGNITUDE not seconds =="
|
||||
D="$WORK/catch"; mkdir -p "$D"
|
||||
GAP=3600000 # 1h unconscious
|
||||
( export ENGRAM_CHRONOCEPTION=1 ENGRAM_DATA_DIR="$D"; "$WORK/p2" catchup "$D" "$GAP" ) >"$D/out.txt" 2>&1
|
||||
CMAG=$(grep CATCHUP_MAGNITUDE "$D/out.txt" | awk '{print $2}')
|
||||
CWM=$(sum_wm "$D/field.json")
|
||||
PRED=$(python3 -c "import math; print(round(1-math.exp(-$GAP/1000/3600),4))")
|
||||
echo " gap=${GAP}ms catchup_magnitude=$CMAG predicted=$PRED field_wm_sum=$CWM (was 0.6)"
|
||||
python3 -c "import sys; sys.exit(0 if abs(float('$CMAG')-float('$PRED'))<1e-2 else 1)" \
|
||||
&& echo " PASS: one-shot catch-up cooled by the elapsed gap, surfaced as a magnitude" \
|
||||
|| { echo " FAIL"; fail=1; }
|
||||
# honesty rail: magnitude is bounded [0,1), NOT an elapsed-seconds number
|
||||
python3 -c "import sys; m=float('$CMAG'); sys.exit(0 if 0<=m<1 else 1)" \
|
||||
&& echo " PASS: magnitude is a bounded drift signal in [0,1), never elapsed seconds" \
|
||||
|| { echo " FAIL: magnitude out of [0,1)"; fail=1; }
|
||||
|
||||
echo
|
||||
echo "== (d) OFF path: flag unset -> age & catchup return 0, field untouched =="
|
||||
D="$WORK/off"; mkdir -p "$D"
|
||||
( unset ENGRAM_CHRONOCEPTION; export ENGRAM_DATA_DIR="$D"; "$WORK/p2" offcheck "$D" 3600000 ) >"$D/out.txt" 2>&1
|
||||
cat "$D/out.txt" | sed 's/^/ /'
|
||||
OFFWM=$(sum_wm "$D/field.json")
|
||||
# loaded field wm sum = (1.0+0.8+0.6)*0.5 halving = 1.2 ; must be UNCHANGED
|
||||
echo " field_wm_sum=$OFFWM (expected 1.2, unchanged)"
|
||||
python3 -c "import sys; sys.exit(0 if abs($OFFWM-1.2)<1e-9 else 1)" \
|
||||
&& echo " PASS: OFF path leaves the field byte-identical (no aging)" \
|
||||
|| { echo " FAIL: OFF path modified the field"; fail=1; }
|
||||
|
||||
echo
|
||||
echo "== ASan+UBSan =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p2_chrono.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p2.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p2.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
DS="$WORK/san"; mkdir -p "$DS"
|
||||
( export ENGRAM_CHRONOCEPTION=1 ENGRAM_DATA_DIR="$DS"; "$WORK/p2.san" once "$DS" 3600000 ) >/dev/null 2>"$WORK/san.log"
|
||||
( export ENGRAM_CHRONOCEPTION=1 ENGRAM_DATA_DIR="$DS"; "$WORK/p2.san" catchup "$DS" 3600000 ) >/dev/null 2>>"$WORK/san.log"
|
||||
if grep -qiE 'runtime error|AddressSanitizer|Sanitizer|ERROR: ' "$WORK/san.log"; then
|
||||
echo " FAIL: sanitizer findings:"; grep -iE 'runtime error|Sanitizer|ERROR' "$WORK/san.log" | head; fail=1
|
||||
else echo " ok: ASan+UBSan clean"; fi
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$fail" -eq 0 ]; then echo "====== P2 CHRONOCEPTION GATE: PASS ======"; else echo "====== P2 CHRONOCEPTION GATE: FAIL ======"; fi
|
||||
rm -rf "$WORK"
|
||||
exit $fail
|
||||
Executable
+68
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env bash
|
||||
# M-INTEROCEPTION P3 gate: drift-sensor primitive engram_geo_displacement.
|
||||
# Read-only pure primitive; no store, no flag. Throwaway /tmp only.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p3-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
fail=0
|
||||
|
||||
echo "== compile =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p3_drift.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p3" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
"$WORK/p3" > "$WORK/out.txt" 2>&1 || { echo "FAIL run"; cat "$WORK/out.txt"; fail=1; }
|
||||
cat "$WORK/out.txt" | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "== assertions =="
|
||||
python3 - "$WORK/out.txt" <<'PY'
|
||||
import sys,re
|
||||
rows={}
|
||||
for line in open(sys.argv[1]):
|
||||
m=re.match(r'(\w+) (.*)',line.strip())
|
||||
if not m: continue
|
||||
tag=m.group(1); kv=dict(re.findall(r'(\w+)=([-\d.]+)',m.group(2)))
|
||||
rows[tag]={k:float(v) for k,v in kv.items()}
|
||||
rc=0
|
||||
def check(c,msg):
|
||||
global rc; print((" PASS: " if c else " FAIL: ")+msg)
|
||||
if not c: rc=1
|
||||
g=rows["GROWTH"]; c=rows["CORRUPTION"]; i=rows["IDENTITY"]
|
||||
check(g["core_disp"]<0.05, f"GROWTH: core displacement ~0 (core fixed) = {g['core_disp']}")
|
||||
check(g["periph_disp"]>0.30, f"GROWTH: periphery extended = {g['periph_disp']}")
|
||||
check(g["centroid_sep"]<1e-6, f"GROWTH: centroid unmoved = {g['centroid_sep']}")
|
||||
check(abs(g["radius_delta"]-0.4)<1e-4, f"GROWTH: radius grew by ~0.4 = {g['radius_delta']}")
|
||||
check(c["core_disp"]>0.40, f"CORRUPTION: core displaced strongly = {c['core_disp']}")
|
||||
check(c["periph_disp"]<0.05, f"CORRUPTION: periphery fixed = {c['periph_disp']}")
|
||||
check(c["centroid_sep"]>0.1, f"CORRUPTION: centroid moved = {c['centroid_sep']}")
|
||||
check(c["core_disp"] > 8*g["core_disp"]+0.3,
|
||||
f"SENSOR DISCRIMINATES: corruption core_disp ({c['core_disp']}) >> growth core_disp ({g['core_disp']})")
|
||||
check(i["core_disp"]==0 and i["periph_disp"]==0 and i["centroid_sep"]<1e-6,
|
||||
"IDENTITY: A vs A -> zero drift")
|
||||
sys.exit(rc)
|
||||
PY
|
||||
[ $? -ne 0 ] && fail=1
|
||||
|
||||
echo
|
||||
echo "== ASan+UBSan =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p3_drift.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p3.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p3.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
"$WORK/p3.san" >/dev/null 2>"$WORK/san.log"
|
||||
if grep -qiE 'runtime error|AddressSanitizer|Sanitizer|ERROR: ' "$WORK/san.log"; then
|
||||
echo " FAIL: sanitizer findings:"; grep -iE 'runtime error|Sanitizer|ERROR' "$WORK/san.log" | head; fail=1
|
||||
else echo " ok: ASan+UBSan clean"; fi
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$fail" -eq 0 ]; then echo "====== P3 DRIFT-SENSOR GATE: PASS ======"; else echo "====== P3 DRIFT-SENSOR GATE: FAIL ======"; fi
|
||||
rm -rf "$WORK"
|
||||
exit $fail
|
||||
Executable
+69
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
# M-INTEROCEPTION P4 gate: afferent input counters in act-stats (additive).
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p4-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compile =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p4_afferent.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p4" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
"$WORK/p4" > "$WORK/out.txt" 2>&1 || { echo "FAIL run"; cat "$WORK/out.txt"; fail=1; }
|
||||
grep -oE 'aff_[a-z_]+":[0-9]+' "$WORK/out.txt" | sed 's/^/ /' | head -30
|
||||
|
||||
echo
|
||||
echo "== assertions =="
|
||||
python3 - "$WORK/out.txt" <<'PY'
|
||||
import sys,re,json
|
||||
S={}
|
||||
for line in open(sys.argv[1]):
|
||||
m=re.match(r'(STATS\d) (\{.*\})',line.strip())
|
||||
if m: S[m.group(1)]=json.loads(m.group(2))
|
||||
rc=0
|
||||
def check(c,msg):
|
||||
global rc; print((" PASS: " if c else " FAIL: ")+msg)
|
||||
if not c: rc=1
|
||||
s0,s1,s2=S["STATS0"],S["STATS1"],S["STATS2"]
|
||||
# after creation, before any query
|
||||
check(s0["aff_node_creates"]==5, f"node_creates==5 (got {s0['aff_node_creates']})")
|
||||
check(s0["aff_ise_ingests"]==2, f"ise_ingests==2 (got {s0['aff_ise_ingests']})")
|
||||
check(s0["aff_edge_creates"]==2, f"edge_creates==2 (got {s0['aff_edge_creates']})")
|
||||
check(s0["aff_queries"]==0 and s0["aff_activations"]==0, "queries/activations start at 0")
|
||||
# after 4 queries
|
||||
check(s1["aff_queries"]==4, f"queries==4 (got {s1['aff_queries']})")
|
||||
check(s1["aff_activations"]==4, f"activations==4 (got {s1['aff_activations']})")
|
||||
check(s1["aff_node_creates"]==5 and s1["aff_ise_ingests"]==2 and s1["aff_edge_creates"]==2,
|
||||
"create counters unchanged by queries")
|
||||
# after 3 more queries — monotonic
|
||||
check(s2["aff_queries"]==7, f"queries==7 monotonic (got {s2['aff_queries']})")
|
||||
check(s2["aff_activations"]==7, f"activations==7 monotonic (got {s2['aff_activations']})")
|
||||
check(s2["aff_queries"]>s1["aff_queries"]>s0["aff_queries"], "queries strictly monotonic across readings")
|
||||
sys.exit(rc)
|
||||
PY
|
||||
[ $? -ne 0 ] && fail=1
|
||||
|
||||
echo
|
||||
echo "== ASan+UBSan =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p4_afferent.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p4.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p4.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
"$WORK/p4.san" >/dev/null 2>"$WORK/san.log"
|
||||
if grep -qiE 'runtime error|AddressSanitizer|Sanitizer|ERROR: ' "$WORK/san.log"; then
|
||||
echo " FAIL: sanitizer findings:"; grep -iE 'runtime error|Sanitizer|ERROR' "$WORK/san.log" | head; fail=1
|
||||
else echo " ok: ASan+UBSan clean"; fi
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$fail" -eq 0 ]; then echo "====== P4 AFFERENT-COUNTERS GATE: PASS ======"; else echo "====== P4 AFFERENT-COUNTERS GATE: FAIL ======"; fi
|
||||
rm -rf "$WORK"
|
||||
exit $fail
|
||||
Executable
+74
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env bash
|
||||
# M-INTEROCEPTION P5 gate: dream-recall builtin engram_dreams_json (honesty rail).
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p5-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compile =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p5_dreams.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p5" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
D="$WORK/d"; mkdir -p "$D"
|
||||
"$WORK/p5" "$D" > "$WORK/out.txt" 2>&1 || { echo "FAIL run"; cat "$WORK/out.txt"; fail=1; }
|
||||
cat "$WORK/out.txt" | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "== assertions =="
|
||||
python3 - "$WORK/out.txt" <<'PY'
|
||||
import sys,re,json
|
||||
L={}
|
||||
for line in open(sys.argv[1]):
|
||||
line=line.strip()
|
||||
m=re.match(r'(BEFORE|AFTER) (\[.*\])',line)
|
||||
if m: L[m.group(1)]=json.loads(m.group(2)); continue
|
||||
m=re.match(r'PRUNED (\d+)',line)
|
||||
if m: L['PRUNED']=int(m.group(1)); continue
|
||||
m=re.match(r'SINCE (\d+) (\[.*\])',line)
|
||||
if m: L['SINCE']=json.loads(m.group(2))
|
||||
rc=0
|
||||
def check(c,msg):
|
||||
global rc; print((" PASS: " if c else " FAIL: ")+msg)
|
||||
if not c: rc=1
|
||||
before_ids={d["id"] for d in L["BEFORE"]}
|
||||
after_ids={d["id"] for d in L["AFTER"]}
|
||||
since_ids={d["id"] for d in L["SINCE"]}
|
||||
check(before_ids=={"cur_old","cur_mid","cur_recent"}, f"before prune: all 3 curiosity_scan, heartbeat excluded (got {sorted(before_ids)})")
|
||||
check("hb_recent" not in before_ids, "heartbeat ISE never appears (not a dream)")
|
||||
check(L["PRUNED"]==1, f"prune rotated out exactly the ancient ISE (pruned={L['PRUNED']})")
|
||||
check(after_ids=={"cur_mid","cur_recent"}, f"after prune: rotated-out cur_old is ABSENT, not confabulated (got {sorted(after_ids)})")
|
||||
check("cur_old" not in after_ids, "honesty rail: pruned dream is gone = 'I don't remember', never synthesized")
|
||||
check(since_ids=={"cur_recent"}, f"since filter returns only events after the cutoff (got {sorted(since_ids)})")
|
||||
# no fabrication: every returned id was one we seeded
|
||||
seeded={"cur_old","cur_mid","cur_recent","hb_recent"}
|
||||
allret=before_ids|after_ids|since_ids
|
||||
check(allret<=seeded, f"no fabricated entries — every returned id was seeded ({sorted(allret)})")
|
||||
sys.exit(rc)
|
||||
PY
|
||||
[ $? -ne 0 ] && fail=1
|
||||
|
||||
echo
|
||||
echo "== ASan+UBSan =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p5_dreams.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p5.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p5.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
DS="$WORK/ds"; mkdir -p "$DS"
|
||||
"$WORK/p5.san" "$DS" >/dev/null 2>"$WORK/san.log"
|
||||
if grep -qiE 'runtime error|AddressSanitizer|Sanitizer|ERROR: ' "$WORK/san.log"; then
|
||||
echo " FAIL: sanitizer findings:"; grep -iE 'runtime error|Sanitizer|ERROR' "$WORK/san.log" | head; fail=1
|
||||
else echo " ok: ASan+UBSan clean"; fi
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$fail" -eq 0 ]; then echo "====== P5 DREAM-RECALL GATE: PASS ======"; else echo "====== P5 DREAM-RECALL GATE: FAIL ======"; fi
|
||||
rm -rf "$WORK"
|
||||
exit $fail
|
||||
Executable
+137
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env bash
|
||||
# M7 index-driven-traversal gate. Pure C harness (NOT elb/elc): links the real
|
||||
# el_runtime.c engram builtins + engram_store.c and drives ENGRAM_STORE off vs on.
|
||||
# Proves (1) byte-identical activation parity flag-on == flag-off across a
|
||||
# mutating query sequence, and (2) the O(E)-rebuild cost is eliminated flag-on.
|
||||
# Writes ONLY under a throwaway /tmp dir with a throwaway HOME.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-m7-XXXXXX)"
|
||||
DATA="$WORK/data"; mkdir -p "$DATA"
|
||||
BIN="$WORK/m7"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME" # never touch real ~/.neuron
|
||||
# Hermetic: point the embedder at a guaranteed-refused endpoint so eg_embed_fetch
|
||||
# fails fast, the circuit breaker opens, and cosq is deterministically absent in
|
||||
# EVERY run (no dependence on whether a dev Ollama happens to be listening). This
|
||||
# makes the byte-identical parity comparison reproducible and non-flaky.
|
||||
export EL_EMBED_URL="http://127.0.0.1:1/api/embeddings"
|
||||
unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compiling harness (gcc: el_runtime.c + engram_store.c + test_m7_traversal.c) =="
|
||||
gcc -O2 -std=c11 -I "$INC" "$HERE/test_m7_traversal.c" "$RT" "$ST" -lcurl -lm -o "$BIN" 2>"$WORK/cc.log"
|
||||
if [ $? -ne 0 ]; then echo "COMPILE FAILED:"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; fi
|
||||
echo " ok: compiled"
|
||||
|
||||
echo
|
||||
echo "== 1) PARITY: index-driven (M7 incremental) activation must be IDENTICAL to the"
|
||||
echo " full-rebuild scan path — proven under one identical ENGRAM_STORE=1 state,"
|
||||
echo " so the ONLY variable is how per-node adjacency is maintained."
|
||||
echo " (compared on deterministic fields: node label + activation_strength +"
|
||||
echo " working_memory_weight + epistemic_confidence + hops + promoted, IN ORDER;"
|
||||
echo " node id/timestamps are per-run random and are intentionally excluded.)"
|
||||
( unset ENGRAM_STORE; "$BIN" parity-off "$DATA" ) || { echo "FAIL: parity-off run"; fail=1; }
|
||||
ENGRAM_STORE=1 "$BIN" parity-on-rebuild "$DATA" || { echo "FAIL: parity-on-rebuild run"; fail=1; }
|
||||
ENGRAM_STORE=1 "$BIN" parity-on-incr "$DATA" || { echo "FAIL: parity-on-incr run"; fail=1; }
|
||||
python3 - "$DATA" <<'PY' || fail=1
|
||||
import json, sys, os
|
||||
d = sys.argv[1]
|
||||
def proj(prefix, i):
|
||||
a = json.load(open(os.path.join(d, f"{prefix}_act{i}.json")))
|
||||
out = []
|
||||
for e in a:
|
||||
n = e.get("node", {})
|
||||
out.append([n.get("label",""),
|
||||
e.get("activation_strength"), e.get("working_memory_weight"),
|
||||
e.get("epistemic_confidence"), e.get("hops"), e.get("promoted")])
|
||||
return out
|
||||
def compare(label, pa, pb, gate):
|
||||
rc = 0
|
||||
for i in (1,2,3,4):
|
||||
a, b = proj(pa, i), proj(pb, i)
|
||||
if a == b:
|
||||
print(f" #{i} identical (entries={len(a)}, promoted={sum(1 for r in a if r[5])})")
|
||||
else:
|
||||
if gate: rc = 1
|
||||
print(f" #{i} DIFFERS ({'FAIL' if gate else 'note'})")
|
||||
for x,y in zip(a,b):
|
||||
if x != y:
|
||||
print(f" first diff:\n {pa}={x}\n {pb}={y}"); break
|
||||
if len(a) != len(b): print(f" length: {pa}={len(a)} {pb}={len(b)}")
|
||||
print(f" {'PASS' if rc==0 else 'FAIL'}: {label}")
|
||||
return rc
|
||||
|
||||
print(" [CORE M7 GATE] flag-on incremental index == flag-on forced full rebuild:")
|
||||
rc1 = compare("index-driven activation == full-rebuild scan (same flag state)",
|
||||
"onincr", "onrb", gate=True)
|
||||
print(" [context] flag-on incremental index vs flag-off scan path (today's behavior):")
|
||||
rc2 = compare("M7 (flag-on) == flag-off scan path", "onincr", "off", gate=False)
|
||||
print(" [context] flag-off scan vs flag-on forced rebuild (isolates any pre-existing")
|
||||
print(" flag-on/off float difference, INDEPENDENT of M7's incremental path):")
|
||||
rc3 = compare("flag-off == flag-on (both rebuild path)", "off", "onrb", gate=False)
|
||||
sys.exit(rc1) # only the core M7 equivalence gates the result
|
||||
PY
|
||||
|
||||
echo
|
||||
echo "== 2) PERF: ~13k nodes / 43k edges, 200 (add-edge + activate) iterations =="
|
||||
NODES=13000; EDGES=43000; ITERS=120
|
||||
( unset ENGRAM_STORE; "$BIN" perf off "$DATA" "$NODES" "$EDGES" "$ITERS" ) | tee "$WORK/perf_off.txt"
|
||||
[ ${PIPESTATUS[0]} -ne 0 ] && { echo "FAIL: perf off"; fail=1; }
|
||||
ENGRAM_STORE=1 "$BIN" perf on "$DATA" "$NODES" "$EDGES" "$ITERS" | tee "$WORK/perf_on.txt"
|
||||
[ ${PIPESTATUS[0]} -ne 0 ] && { echo "FAIL: perf on"; fail=1; }
|
||||
python3 - "$WORK/perf_off.txt" "$WORK/perf_on.txt" <<'PY'
|
||||
import re, sys
|
||||
def parse(f):
|
||||
t = open(f).read()
|
||||
def g(k):
|
||||
m = re.search(k+r'=([\d.]+)', t); return float(m.group(1)) if m else 0.0
|
||||
return {'rw': g('rebuild_edge_work'), 'rb': g('rebuilds'), 'ap': g('incr_appends'),
|
||||
'loop_s': g('loop='), 'maint': g('adj_maint'),
|
||||
'perq': g('per_query')}
|
||||
off, on = parse(sys.argv[1]), parse(sys.argv[2])
|
||||
def ratio(a,b): return (a/b) if b else float('inf')
|
||||
print()
|
||||
print(f" ADJACENCY TRAVERSAL COST (the metric M7 changes):")
|
||||
print(f" edge-touches in rebuilds: off={off['rw']:.0f} on={on['rw']:.0f} "
|
||||
f"({ratio(off['rw'],on['rw']):.0f}x fewer on)")
|
||||
print(f" full O(E) rebuilds: off={off['rb']:.0f} on={on['rb']:.0f}")
|
||||
print(f" incremental O(1) appends: off={off['ap']:.0f} on={on['ap']:.0f}")
|
||||
print(f" adjacency-maint wall-time: off={off['maint']:.4f}s on={on['maint']:.4f}s "
|
||||
f"({ratio(off['maint'],on['maint']):.1f}x faster on)")
|
||||
print(f" END-TO-END per-query time: off={off['perq']:.2f}ms on={on['perq']:.2f}ms")
|
||||
print(f" (per-query is dominated by activation's O(N) node scoring over 13k nodes,")
|
||||
print(f" which M7 does not touch; the delta is the eliminated rebuild time.)")
|
||||
ok = on['rw'] < off['rw'] and on['maint'] < off['maint'] and on['rb'] < off['rb']
|
||||
print(" PASS: flag-on eliminates the O(E) per-query rebuild (fewer edge-touches, less maint time)"
|
||||
if ok else " FAIL: expected fewer edge-touches AND less adjacency-maint time on flag-on")
|
||||
sys.exit(0 if ok else 1)
|
||||
PY
|
||||
[ $? -ne 0 ] && fail=1
|
||||
|
||||
echo
|
||||
echo "== 3) ASan+UBSan clean across parity + a small perf loop (leaks off — harness intentionally leaks el_strdup) =="
|
||||
SANBIN="$WORK/m7.san"
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_m7_traversal.c" "$RT" "$ST" -lcurl -lm -o "$SANBIN" 2>"$WORK/san_cc.log"
|
||||
if [ $? -ne 0 ]; then echo " SAN COMPILE FAILED:"; tail -20 "$WORK/san_cc.log"; fail=1; else
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
D2="$WORK/data2"; mkdir -p "$D2"
|
||||
( unset ENGRAM_STORE; "$SANBIN" parity-off "$D2" ) >/dev/null 2>"$WORK/san_run.log" && \
|
||||
ENGRAM_STORE=1 "$SANBIN" parity-on-rebuild "$D2" >/dev/null 2>>"$WORK/san_run.log" && \
|
||||
ENGRAM_STORE=1 "$SANBIN" parity-on-incr "$D2" >/dev/null 2>>"$WORK/san_run.log" && \
|
||||
( unset ENGRAM_STORE; "$SANBIN" perf off "$D2" 1500 5000 40 ) >/dev/null 2>>"$WORK/san_run.log" && \
|
||||
ENGRAM_STORE=1 "$SANBIN" perf on "$D2" 1500 5000 40 >/dev/null 2>>"$WORK/san_run.log"
|
||||
if grep -qiE 'runtime error|AddressSanitizer|UndefinedBehavior|ERROR: ' "$WORK/san_run.log"; then
|
||||
echo " FAIL: sanitizer findings:"; grep -iE 'runtime error|Sanitizer|ERROR' "$WORK/san_run.log" | head; fail=1
|
||||
else
|
||||
echo " ok: ASan+UBSan clean across parity + perf (rebuild + incremental append + BFS)"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$fail" -eq 0 ]; then echo "================ M7 TRAVERSAL GATE: PASS ================"; else echo "================ M7 TRAVERSAL GATE: FAIL ================"; fi
|
||||
rm -rf "$WORK"
|
||||
exit $fail
|
||||
Executable
+23
@@ -0,0 +1,23 @@
|
||||
#!/bin/sh
|
||||
# Build + RUN the REASONING-layer tests (engram_reason.c): closed-form constructed
|
||||
# cases for ANALOGY / INDUCTION / ABDUCTION / CAUSAL / PLANNING, each composing the
|
||||
# §5 geometry OPERATORS (engram_geometry.c). Pure C11 (stdlib + libm). Standalone —
|
||||
# NOT folded through elc. Two passes:
|
||||
# 1. PERF — optimised (-O2, no sanitizer): the functional gate.
|
||||
# 2. SAFETY — ASan + UBSan on the same suite (memory-safety is size-independent).
|
||||
set -e
|
||||
HERE=$(cd "$(dirname "$0")" && pwd)
|
||||
RT="$HERE/../../lang/runtime"
|
||||
CC=${CC:-cc}
|
||||
SRC="$HERE/test_reason.c $RT/engram_reason.c $RT/engram_geometry.c $RT/engram_store.c $RT/engram_vindex.c"
|
||||
WARN="-std=c11 -Wall -Wextra"
|
||||
TMP=$(mktemp -d)
|
||||
|
||||
echo "### PASS 1: PERF (optimised, un-sanitised) — functional gate"
|
||||
$CC $WARN -O2 -I"$RT" $SRC -lm -o "$TMP/perf"
|
||||
"$TMP/perf"
|
||||
|
||||
echo
|
||||
echo "### PASS 2: SAFETY (ASan/UBSan)"
|
||||
$CC $WARN -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -I"$RT" $SRC -lm -o "$TMP/safe"
|
||||
ASAN_OPTIONS=${ASAN_OPTIONS:-detect_leaks=0} UBSAN_OPTIONS=halt_on_error=1 "$TMP/safe"
|
||||
Executable
+24
@@ -0,0 +1,24 @@
|
||||
#!/bin/sh
|
||||
# Build + RUN the VERIFIER-layer tests (engram_verify.c): closed-form constructed
|
||||
# cases for GROUNDING (anti-hallucination) and CONSISTENCY (polarity/negation
|
||||
# inversion + geometric contradiction), each composing the reasoning point-fit
|
||||
# (engram_reason.c) and the §5 geometry OPERATORS (engram_geometry.c). Pure C11
|
||||
# (stdlib + libm). Standalone — NOT folded through elc. Two passes:
|
||||
# 1. PERF — optimised (-O2, no sanitizer): the functional gate.
|
||||
# 2. SAFETY — ASan + UBSan on the same suite (memory-safety is size-independent).
|
||||
set -e
|
||||
HERE=$(cd "$(dirname "$0")" && pwd)
|
||||
RT="$HERE/../../lang/runtime"
|
||||
CC=${CC:-cc}
|
||||
SRC="$HERE/test_verify.c $RT/engram_verify.c $RT/engram_reason.c $RT/engram_geometry.c $RT/engram_store.c $RT/engram_vindex.c"
|
||||
WARN="-std=c11 -Wall -Wextra"
|
||||
TMP=$(mktemp -d)
|
||||
|
||||
echo "### PASS 1: PERF (optimised, un-sanitised) — functional gate"
|
||||
$CC $WARN -O2 -I"$RT" $SRC -lm -o "$TMP/perf"
|
||||
"$TMP/perf"
|
||||
|
||||
echo
|
||||
echo "### PASS 2: SAFETY (ASan/UBSan)"
|
||||
$CC $WARN -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -I"$RT" $SRC -lm -o "$TMP/safe"
|
||||
ASAN_OPTIONS=${ASAN_OPTIONS:-detect_leaks=0} UBSAN_OPTIONS=halt_on_error=1 "$TMP/safe"
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/bin/sh
|
||||
# Build + RUN the M8 HNSW vector-index tests. Pure C11 (gcc/cc), stdlib + libm
|
||||
# only. This is a standalone C module — NOT folded through elb/elc.
|
||||
#
|
||||
# Two passes:
|
||||
# 1. PERF — optimised (-O2, no sanitizer): the real recall@10 gate + speedup
|
||||
# numbers at full size (N=5000 recall, N=5000/20000 speedup).
|
||||
# 2. SAFETY — ASan + UBSan on the same suite at reduced size (VINDEX_QUICK=1);
|
||||
# memory-safety is size-independent, so this stays fast.
|
||||
set -e
|
||||
HERE=$(cd "$(dirname "$0")" && pwd)
|
||||
RT="$HERE/../../lang/runtime"
|
||||
CC=${CC:-cc}
|
||||
SRC="$HERE/test_vindex.c $RT/engram_vindex.c $RT/engram_store.c"
|
||||
WARN="-std=c11 -Wall -Wextra"
|
||||
TMP=$(mktemp -d)
|
||||
|
||||
echo "### PASS 1: PERF (optimised, un-sanitised) — recall gate + speedup"
|
||||
$CC $WARN -O2 -I"$RT" $SRC -lm -o "$TMP/perf"
|
||||
"$TMP/perf"
|
||||
|
||||
echo
|
||||
echo "### PASS 2: SAFETY (ASan/UBSan, reduced size)"
|
||||
$CC $WARN -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -I"$RT" $SRC -lm -o "$TMP/safe"
|
||||
VINDEX_QUICK=1 ASAN_OPTIONS=${ASAN_OPTIONS:-detect_leaks=0} UBSAN_OPTIONS=halt_on_error=1 "$TMP/safe"
|
||||
@@ -0,0 +1,496 @@
|
||||
/* test_bufpool.c — M4 gate for the demand-paging BUFFER POOL (engram_store.{c,h}).
|
||||
*
|
||||
* Pure C. Build: gcc -O2 test_bufpool.c ../../lang/runtime/engram_store.c -o t
|
||||
* Writes ONLY under a throwaway /tmp dir. Never touches ~/.neuron or live ports.
|
||||
*
|
||||
* Proves the M4 pool preserves every M1/M2 invariant when the pool is SMALLER
|
||||
* than the store (pages evict + re-fault): small-pool round-trip correctness,
|
||||
* LRU eviction policy (hot resident / cold evicted / no dirty stolen), pinned
|
||||
* residency (superblocks, index roots, explicit page + hot-layer pins), bounded
|
||||
* read-ahead, and crash safety (WAL replay + checkpoint-crash) under paging.
|
||||
*/
|
||||
#include "../../lang/runtime/engram_store.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <stdint.h>
|
||||
#include <unistd.h>
|
||||
#include <fcntl.h>
|
||||
#include <sys/stat.h>
|
||||
|
||||
static int g_pass = 0, g_fail = 0;
|
||||
static void ok(const char* name, int cond){
|
||||
printf(" [%s] %s\n", cond ? "PASS" : "FAIL", name);
|
||||
if (cond) g_pass++; else g_fail++;
|
||||
}
|
||||
|
||||
static char g_dir[512];
|
||||
static void mk_dir(void){
|
||||
snprintf(g_dir, sizeof g_dir, "/tmp/engram-bufpool-test-%d", (int)getpid());
|
||||
mkdir(g_dir, 0700);
|
||||
}
|
||||
static void path_in(char* out, size_t cap, const char* name){
|
||||
snprintf(out, cap, "%s/%s", g_dir, name);
|
||||
}
|
||||
|
||||
/* ── deterministic generators (bit-exact regeneration for oracles) ─────────── */
|
||||
static uint64_t xs(uint64_t* s){ uint64_t x=*s; x^=x<<13; x^=x>>7; x^=x<<17; *s=x; return x; }
|
||||
static uint64_t node_seed(int i){ return 0x9E3779B97F4A7C15ULL ^ ((uint64_t)(i+1)*0xD1B54A32D192ED03ULL); }
|
||||
static uint64_t edge_seed(int i){ return 0xC2B2AE3D27D4EB4FULL ^ ((uint64_t)(i+1)*0x165667B19E3779F9ULL); }
|
||||
static char* rnd_str(uint64_t* st, size_t len){
|
||||
char* s = (char*)malloc(len + 1);
|
||||
for (size_t i=0;i<len;i++) s[i] = (char)(33 + (xs(st) % 94));
|
||||
s[len] = 0; return s;
|
||||
}
|
||||
|
||||
#define NODE_COUNT 5000
|
||||
#define EDGE_COUNT 20000
|
||||
#define EMB_DIM 768
|
||||
#define CK_NODES 300
|
||||
|
||||
static void noop_node_cb(const StoreNode* n, void* ctx){ (void)n; (void)ctx; }
|
||||
|
||||
static void gen_node(int i, StoreNode* n){
|
||||
memset(n, 0, sizeof *n);
|
||||
uint64_t st = node_seed(i);
|
||||
char id[32]; snprintf(id, sizeof id, "node-%d", i);
|
||||
n->id = strdup(id);
|
||||
size_t clen = (i % 500 == 0) ? (size_t)(17000 + (xs(&st) % 6000)) : (size_t)(xs(&st) % 300);
|
||||
n->content = rnd_str(&st, clen);
|
||||
n->node_type = rnd_str(&st, 4 + (xs(&st) % 8));
|
||||
n->label = (i % 2) ? rnd_str(&st, 3 + (xs(&st) % 10)) : NULL;
|
||||
n->tier = rnd_str(&st, 4 + (xs(&st) % 6));
|
||||
n->tags = rnd_str(&st, xs(&st) % 40);
|
||||
n->metadata = (i % 3) ? rnd_str(&st, xs(&st) % 60) : NULL;
|
||||
n->salience = (double)(xs(&st) % 1000000) / 997.0;
|
||||
n->importance = (double)(xs(&st) % 1000000) / 131.0;
|
||||
n->confidence = (double)(xs(&st) % 1000000) / 733.0;
|
||||
n->temporal_decay_rate = (double)(xs(&st) % 1000000) / 101.0;
|
||||
n->activation_count = (int64_t)(xs(&st) % 100000);
|
||||
n->last_activated = (int64_t)xs(&st);
|
||||
n->created_at = (int64_t)(1600000000000LL + i);
|
||||
n->updated_at = (int64_t)xs(&st);
|
||||
n->background_activation = (double)(xs(&st) % 1000000) / 17.0;
|
||||
n->working_memory_weight = (double)(xs(&st) % 1000000) / 29.0;
|
||||
n->suppression_count = (int32_t)(xs(&st) % 50);
|
||||
n->layer_id = (uint32_t)(xs(&st) % 5);
|
||||
for (int k=0;k<STORE_BLL_K;k++) n->access_ts[k] = (int64_t)xs(&st);
|
||||
n->access_head = (int32_t)(xs(&st) % STORE_BLL_K);
|
||||
n->access_filled = (int32_t)(xs(&st) % (STORE_BLL_K + 1));
|
||||
n->wm_anchor = (double)(xs(&st) % 1000000) / 3.0;
|
||||
n->emb = (float*)malloc(EMB_DIM * sizeof(float));
|
||||
for (int k=0;k<EMB_DIM;k++){ uint32_t u=(uint32_t)xs(&st); memcpy(&n->emb[k], &u, 4); }
|
||||
n->emb_dim = EMB_DIM;
|
||||
}
|
||||
static void gen_edge(int i, StoreEdge* e){
|
||||
memset(e, 0, sizeof *e);
|
||||
uint64_t st = edge_seed(i);
|
||||
char id[32], from[32], to[32];
|
||||
snprintf(id, sizeof id, "edge-%d", i);
|
||||
snprintf(from, sizeof from, "node-%d", (int)(xs(&st) % NODE_COUNT));
|
||||
snprintf(to, sizeof to, "node-%d", (int)(xs(&st) % NODE_COUNT));
|
||||
e->id = strdup(id); e->from_id = strdup(from); e->to_id = strdup(to);
|
||||
e->relation = rnd_str(&st, 3 + (xs(&st) % 12));
|
||||
e->metadata = (i % 4) ? rnd_str(&st, xs(&st) % 40) : NULL;
|
||||
e->weight = (double)(xs(&st) % 1000000) / 111.0;
|
||||
e->hebb = (double)(xs(&st) % 1000000) / 1000000.0;
|
||||
e->confidence = (double)(xs(&st) % 1000000) / 777.0;
|
||||
e->created_at = (int64_t)(1600000000000LL + i);
|
||||
e->updated_at = (int64_t)xs(&st);
|
||||
e->last_fired = (int64_t)xs(&st);
|
||||
e->inhibitory = (int32_t)(xs(&st) % 2);
|
||||
e->layer_id = (uint32_t)(xs(&st) % 5);
|
||||
}
|
||||
static int streq(const char* a, const char* b){
|
||||
if (!a && !b) return 1;
|
||||
if (!a || !b) return 0;
|
||||
return strcmp(a,b)==0;
|
||||
}
|
||||
static int cmp_node(const StoreNode* a, const StoreNode* b){
|
||||
if (!streq(a->id,b->id) || !streq(a->content,b->content) ||
|
||||
!streq(a->node_type,b->node_type) || !streq(a->label,b->label) ||
|
||||
!streq(a->tier,b->tier) || !streq(a->tags,b->tags) ||
|
||||
!streq(a->metadata,b->metadata)) return 0;
|
||||
if (a->salience!=b->salience || a->importance!=b->importance ||
|
||||
a->confidence!=b->confidence || a->temporal_decay_rate!=b->temporal_decay_rate ||
|
||||
a->activation_count!=b->activation_count || a->last_activated!=b->last_activated ||
|
||||
a->created_at!=b->created_at || a->updated_at!=b->updated_at ||
|
||||
a->background_activation!=b->background_activation ||
|
||||
a->working_memory_weight!=b->working_memory_weight ||
|
||||
a->suppression_count!=b->suppression_count || a->layer_id!=b->layer_id ||
|
||||
a->access_head!=b->access_head || a->access_filled!=b->access_filled ||
|
||||
a->wm_anchor!=b->wm_anchor || a->emb_dim!=b->emb_dim) return 0;
|
||||
for (int k=0;k<STORE_BLL_K;k++) if (a->access_ts[k]!=b->access_ts[k]) return 0;
|
||||
if ((a->emb==NULL) != (b->emb==NULL)) return 0;
|
||||
if (a->emb && memcmp(a->emb, b->emb, (size_t)a->emb_dim*4)!=0) return 0;
|
||||
return 1;
|
||||
}
|
||||
static int cmp_edge(const StoreEdge* a, const StoreEdge* b){
|
||||
if (!streq(a->id,b->id) || !streq(a->from_id,b->from_id) || !streq(a->to_id,b->to_id) ||
|
||||
!streq(a->relation,b->relation) || !streq(a->metadata,b->metadata)) return 0;
|
||||
if (a->weight!=b->weight || a->hebb!=b->hebb || a->confidence!=b->confidence ||
|
||||
a->created_at!=b->created_at || a->updated_at!=b->updated_at ||
|
||||
a->last_fired!=b->last_fired || a->inhibitory!=b->inhibitory ||
|
||||
a->layer_id!=b->layer_id) return 0;
|
||||
return 1;
|
||||
}
|
||||
static void free_node_fields(StoreNode* n){
|
||||
free(n->id); free(n->content); free(n->node_type); free(n->label);
|
||||
free(n->tier); free(n->tags); free(n->metadata); free(n->emb); free(n->unknown);
|
||||
}
|
||||
static void free_edge_fields(StoreEdge* e){
|
||||
free(e->id); free(e->from_id); free(e->to_id); free(e->relation); free(e->metadata); free(e->unknown);
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* TEST 1 — SMALL-POOL CORRECTNESS: full M1 workload (5k nodes / 20k edges) with
|
||||
* a frame budget FAR smaller than the store → constant eviction + re-fault, yet
|
||||
* every read is bit-exact and the pool stays bounded.
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
static void test_small_pool_roundtrip(void){
|
||||
printf("\n== 1) small-pool correctness: %d nodes + %d edges, cap=%d frames ==\n",
|
||||
NODE_COUNT, EDGE_COUNT, 32);
|
||||
char path[600]; path_in(path, sizeof path, "small.store");
|
||||
unlink(path);
|
||||
EngramPagedStore* s = store_create(path);
|
||||
ok("store_create", s != NULL);
|
||||
if (!s) return;
|
||||
store__set_pool_frames(s, 32); /* pool << store */
|
||||
|
||||
for (int i=0;i<NODE_COUNT;i++){
|
||||
StoreNode n; gen_node(i,&n);
|
||||
if (store_put_node(s,&n)!=0){ ok("put_node", 0); free_node_fields(&n); store_close(s); return; }
|
||||
free_node_fields(&n);
|
||||
if ((i%500)==499) store_sync(s); /* checkpoint: dirty→clean so frames evictable */
|
||||
}
|
||||
for (int i=0;i<EDGE_COUNT;i++){
|
||||
StoreEdge e; gen_edge(i,&e);
|
||||
if (store_put_edge(s,&e)!=0){ ok("put_edge", 0); free_edge_fields(&e); store_close(s); return; }
|
||||
free_edge_fields(&e);
|
||||
if ((i%1000)==999) store_sync(s);
|
||||
}
|
||||
store_sync(s);
|
||||
|
||||
StorePoolStats st; store_pool_stats(s, &st);
|
||||
printf(" pages=%llu pool: cap=%zu resident=%zu pinned=%zu dirty=%zu evictions=%llu\n",
|
||||
(unsigned long long)store_page_count(s), st.cap, st.resident, st.pinned,
|
||||
st.dirty, (unsigned long long)st.evictions);
|
||||
ok("eviction actually fired (store exceeded the pool)", st.evictions > 0);
|
||||
ok("pool stayed bounded (resident <= cap)", st.resident <= st.cap);
|
||||
ok("no dirty frames after checkpoint", st.dirty == 0);
|
||||
|
||||
/* read back EVERY node bit-exact despite constant eviction/re-fault */
|
||||
int bad = 0;
|
||||
for (int i=0;i<NODE_COUNT;i++){
|
||||
StoreNode want; gen_node(i,&want);
|
||||
StoreNode got; int hit = store_get_node(s, want.id, &got);
|
||||
if (hit!=1 || !cmp_node(&want,&got)) bad++;
|
||||
if (hit==1) store_node_free(&got);
|
||||
free_node_fields(&want);
|
||||
}
|
||||
ok("all 5000 nodes bit-exact under eviction", bad==0);
|
||||
|
||||
/* sample 4000 edges bit-exact */
|
||||
int ebad = 0;
|
||||
for (int i=0;i<EDGE_COUNT;i+=5){
|
||||
StoreEdge want; gen_edge(i,&want);
|
||||
StoreEdge got; int hit = store_get_edge(s, want.id, &got);
|
||||
if (hit!=1 || !cmp_edge(&want,&got)) ebad++;
|
||||
if (hit==1) store_edge_free(&got);
|
||||
free_edge_fields(&want);
|
||||
}
|
||||
ok("sampled 4000 edges bit-exact under eviction", ebad==0);
|
||||
ok("store_check crc clean under paging", store_check(s, STORE_CHECK_CRC)==0);
|
||||
|
||||
store_pool_stats(s, &st);
|
||||
printf(" after reads: resident=%zu (<= cap=%zu) hits=%llu misses=%llu evictions=%llu\n",
|
||||
st.resident, st.cap, (unsigned long long)st.hits,
|
||||
(unsigned long long)st.misses, (unsigned long long)st.evictions);
|
||||
ok("still bounded after full read-back", st.resident <= st.cap);
|
||||
store_close(s);
|
||||
unlink(path);
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* TEST 2 — EVICTION POLICY: a repeatedly-touched HOT set stays resident (0 extra
|
||||
* faults) while a streaming COLD set is evicted; and a dirty-heavy write burst
|
||||
* proves dirty pages are NEVER stolen before a checkpoint (no-steal).
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
static void test_eviction_policy(void){
|
||||
printf("\n== 2) eviction policy: hot resident, cold evicted, no dirty stolen ==\n");
|
||||
char path[600]; path_in(path, sizeof path, "evict.store");
|
||||
unlink(path);
|
||||
|
||||
/* ---- part A: hot vs cold ---- */
|
||||
EngramPagedStore* s = store_create(path);
|
||||
if (!s){ ok("store_create", 0); return; }
|
||||
const int N = 1500;
|
||||
for (int i=0;i<N;i++){ StoreNode n; gen_node(i,&n); store_put_node(s,&n); free_node_fields(&n);
|
||||
if ((i%400)==399) store_sync(s); }
|
||||
store_sync(s);
|
||||
store__set_pool_frames(s, 64);
|
||||
|
||||
const int HOT = 8;
|
||||
/* warm the hot set */
|
||||
for (int h=0;h<HOT;h++){ char id[32]; snprintf(id,sizeof id,"node-%d",h);
|
||||
StoreNode g; if (store_get_node(s,id,&g)==1) store_node_free(&g); }
|
||||
|
||||
StorePoolStats a,b;
|
||||
uint64_t hot_faults = 0, cold_faults = 0;
|
||||
int cold = 200; /* streaming cold ids well outside hot set */
|
||||
for (int r=0;r<150;r++){
|
||||
for (int h=0;h<HOT;h++){
|
||||
char id[32]; snprintf(id,sizeof id,"node-%d",h);
|
||||
store_pool_stats(s,&a);
|
||||
StoreNode g; if (store_get_node(s,id,&g)==1) store_node_free(&g);
|
||||
store_pool_stats(s,&b);
|
||||
hot_faults += (b.misses - a.misses);
|
||||
}
|
||||
for (int c=0;c<3;c++){
|
||||
char id[32]; snprintf(id,sizeof id,"node-%d",cold++);
|
||||
if (cold>=N) cold=200;
|
||||
store_pool_stats(s,&a);
|
||||
StoreNode g; if (store_get_node(s,id,&g)==1) store_node_free(&g);
|
||||
store_pool_stats(s,&b);
|
||||
cold_faults += (b.misses - a.misses);
|
||||
}
|
||||
}
|
||||
printf(" hot re-get faults (post-warm)=%llu cold stream faults=%llu\n",
|
||||
(unsigned long long)hot_faults, (unsigned long long)cold_faults);
|
||||
ok("HOT pages stay resident (0 faults on re-access)", hot_faults == 0);
|
||||
ok("COLD pages get evicted + re-faulted", cold_faults > 0);
|
||||
store_pool_stats(s,&b);
|
||||
double hr = (double)b.hits / (double)(b.hits + b.misses);
|
||||
printf(" overall hit-rate = %.3f (hits=%llu misses=%llu)\n",
|
||||
hr, (unsigned long long)b.hits, (unsigned long long)b.misses);
|
||||
ok("hit-rate is sane (> 0.5)", hr > 0.5);
|
||||
store_close(s);
|
||||
unlink(path);
|
||||
|
||||
/* ---- part B: no-steal (dirty pages never evicted before checkpoint) ---- */
|
||||
EngramPagedStore* s2 = store_create(path);
|
||||
if (!s2){ ok("store_create(2)", 0); return; }
|
||||
store__set_pool_frames(s2, 8); /* tiny budget */
|
||||
for (int i=0;i<1200;i++){ StoreNode n; gen_node(i,&n); store_put_node(s2,&n); free_node_fields(&n); }
|
||||
/* NO sync: every mutated page is dirty and, by no-steal, unevictable */
|
||||
StorePoolStats d; store_pool_stats(s2,&d);
|
||||
printf(" tiny cap=%zu, unsynced burst: resident=%zu dirty=%zu evictions=%llu\n",
|
||||
d.cap, d.resident, d.dirty, (unsigned long long)d.evictions);
|
||||
ok("dirty pages pinned in RAM beyond budget (no-steal)", d.dirty > d.cap && d.resident > d.cap);
|
||||
/* a just-written node is served correctly from its dirty in-RAM page */
|
||||
{ StoreNode want; gen_node(777,&want); StoreNode got; int hit=store_get_node(s2,want.id,&got);
|
||||
ok("read served correctly from dirty (un-flushed) page", hit==1 && cmp_node(&want,&got));
|
||||
if (hit==1) store_node_free(&got); free_node_fields(&want); }
|
||||
store_sync(s2); /* checkpoint → dirty become clean/evictable */
|
||||
store_pool_stats(s2,&d);
|
||||
ok("checkpoint cleared all dirty frames", d.dirty == 0);
|
||||
/* durability across reopen after the no-steal burst */
|
||||
store_close(s2);
|
||||
EngramPagedStore* s3 = store_open(path);
|
||||
store__set_pool_frames(s3, 8);
|
||||
int miss=0; for (int i=0;i<1200;i++){ StoreNode want; gen_node(i,&want);
|
||||
StoreNode got; int hit=store_get_node(s3,want.id,&got);
|
||||
if (hit!=1 || !cmp_node(&want,&got)) miss++;
|
||||
if (hit==1) store_node_free(&got); free_node_fields(&want); }
|
||||
ok("all 1200 survive reopen, bit-exact, tiny pool", miss==0);
|
||||
store_close(s3);
|
||||
unlink(path);
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* TEST 3 — PINNED RESIDENCY: superblocks + index roots never evicted under heavy
|
||||
* thrash; an explicitly pinned page stays until unpinned; a pinned hot layer's
|
||||
* pages stay resident and are released on unpin.
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
static void test_pinning(void){
|
||||
printf("\n== 3) pinned residency: superblocks / index roots / page / layer ==\n");
|
||||
char path[600]; path_in(path, sizeof path, "pin.store");
|
||||
unlink(path);
|
||||
EngramPagedStore* s = store_create(path);
|
||||
if (!s){ ok("store_create", 0); return; }
|
||||
const int N = 1500;
|
||||
for (int i=0;i<N;i++){ StoreNode n; gen_node(i,&n); store_put_node(s,&n); free_node_fields(&n);
|
||||
if ((i%400)==399) store_sync(s); }
|
||||
store_sync(s);
|
||||
store_close(s);
|
||||
|
||||
s = store_open(path); /* reopen: SBs + roots auto-pinned */
|
||||
store__set_pool_frames(s, 24);
|
||||
|
||||
uint64_t P = store_page_count(s) / 2; /* an arbitrary interior page to pin */
|
||||
store_pin_page(s, P);
|
||||
|
||||
/* thrash: stream a large cold working set to force heavy eviction */
|
||||
for (int pass=0; pass<3; pass++)
|
||||
for (int i=0;i<N;i++){ char id[32]; snprintf(id,sizeof id,"node-%d",i);
|
||||
StoreNode g; if (store_get_node(s,id,&g)==1) store_node_free(&g); }
|
||||
|
||||
ok("superblock page 0 never evicted", store_pool_resident(s,0)==1);
|
||||
ok("superblock mirror page 1 never evicted", store_pool_resident(s,1)==1);
|
||||
ok("explicitly pinned page stayed resident under thrash", store_pool_resident(s,P)==1);
|
||||
|
||||
StorePoolStats st; store_pool_stats(s,&st);
|
||||
printf(" after thrash: resident=%zu pinned=%zu evictions=%llu\n",
|
||||
st.resident, st.pinned, (unsigned long long)st.evictions);
|
||||
ok("structural + explicit pins counted (>=4: 2 SB + 2 roots)", st.pinned >= 4);
|
||||
|
||||
/* unpin the page → it becomes evictable and is dropped under further thrash */
|
||||
store_unpin_page(s, P);
|
||||
for (int i=0;i<N;i++){ char id[32]; snprintf(id,sizeof id,"node-%d",i);
|
||||
StoreNode g; if (store_get_node(s,id,&g)==1) store_node_free(&g); }
|
||||
ok("unpinned page becomes evictable (dropped)", store_pool_resident(s,P)==0);
|
||||
|
||||
/* hot-layer pin: layer 3 is used by ~1/5 of the nodes */
|
||||
int npin = store_pin_layer(s, 3);
|
||||
printf(" store_pin_layer(3) pinned %d page(s)\n", npin);
|
||||
ok("pin_layer pinned a non-empty page set", npin > 0);
|
||||
store_pool_stats(s,&st);
|
||||
size_t pinned_with_layer = st.pinned;
|
||||
for (int pass=0; pass<3; pass++)
|
||||
for (int i=0;i<N;i++){ char id[32]; snprintf(id,sizeof id,"node-%d",i);
|
||||
StoreNode g; if (store_get_node(s,id,&g)==1) store_node_free(&g); }
|
||||
store_pool_stats(s,&st);
|
||||
ok("hot-layer pages stay resident under thrash", st.pinned >= pinned_with_layer);
|
||||
ok("layer pin holds >= npin extra frames", st.pinned >= (size_t)npin + 4);
|
||||
|
||||
store_unpin_layer(s, 3);
|
||||
store_pool_stats(s,&st);
|
||||
size_t after_unpin_max = st.pinned;
|
||||
for (int i=0;i<N;i++){ char id[32]; snprintf(id,sizeof id,"node-%d",i);
|
||||
StoreNode g; if (store_get_node(s,id,&g)==1) store_node_free(&g); }
|
||||
store_pool_stats(s,&st);
|
||||
printf(" pinned frames: with-layer=%zu after-unpin=%zu\n", pinned_with_layer, st.pinned);
|
||||
ok("unpin_layer released the layer's pins", st.pinned < pinned_with_layer && after_unpin_max <= pinned_with_layer);
|
||||
|
||||
store_close(s);
|
||||
unlink(path);
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* TEST 4 — PREFETCH: a sequential scan faults far fewer times with read-ahead on
|
||||
* than off (each cold cache; identical store).
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
static void test_prefetch(void){
|
||||
printf("\n== 4) prefetch: sequential scan faults fewer with read-ahead ==\n");
|
||||
char path[600]; path_in(path, sizeof path, "prefetch.store");
|
||||
unlink(path);
|
||||
EngramPagedStore* s = store_create(path);
|
||||
if (!s){ ok("store_create", 0); return; }
|
||||
for (int i=0;i<2000;i++){ StoreNode n; gen_node(i,&n); store_put_node(s,&n); free_node_fields(&n);
|
||||
if ((i%400)==399) store_sync(s); }
|
||||
store_sync(s);
|
||||
store_close(s);
|
||||
|
||||
/* prefetch OFF — cold cache */
|
||||
EngramPagedStore* a = store_open(path);
|
||||
store__set_pool_frames(a, 0); /* unlimited: isolate prefetch, no eviction */
|
||||
store__set_prefetch(a, 0);
|
||||
StorePoolStats o0, o1; store_pool_stats(a,&o0);
|
||||
int na = store_scan_nodes(a, noop_node_cb, NULL); /* walk + fault every page */
|
||||
(void)na;
|
||||
store_pool_stats(a,&o1);
|
||||
uint64_t faults_off = o1.misses - o0.misses;
|
||||
store_close(a);
|
||||
|
||||
/* prefetch ON — cold cache (fresh open) */
|
||||
EngramPagedStore* b = store_open(path);
|
||||
store__set_pool_frames(b, 0);
|
||||
store__set_prefetch(b, 16);
|
||||
StorePoolStats p0, p1; store_pool_stats(b,&p0);
|
||||
int nb = store_scan_nodes(b, noop_node_cb, NULL);
|
||||
(void)nb;
|
||||
store_pool_stats(b,&p1);
|
||||
uint64_t faults_on = p1.misses - p0.misses;
|
||||
uint64_t pref_reads = p1.prefetch_reads - p0.prefetch_reads;
|
||||
store_close(b);
|
||||
|
||||
printf(" scan demand-faults: prefetch OFF=%llu ON=%llu (read-ahead brought in %llu pages)\n",
|
||||
(unsigned long long)faults_off, (unsigned long long)faults_on,
|
||||
(unsigned long long)pref_reads);
|
||||
ok("prefetch reduced demand faults", faults_on < faults_off);
|
||||
ok("read-ahead actually ran", pref_reads > 0);
|
||||
unlink(path);
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* TEST 5 — CRASH SAFETY UNDER PAGING: WAL replay and checkpoint-crash recovery
|
||||
* with a tiny pool (pages evict + re-fault during replay).
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
static void test_crash_under_paging(void){
|
||||
printf("\n== 5) crash safety under a tiny pool (ENGRAM_POOL_FRAMES=16) ==\n");
|
||||
setenv("ENGRAM_POOL_FRAMES", "16", 1); /* every engram_open() below is paged */
|
||||
setenv("ENGRAM_WAL_SYNC", "always", 1);
|
||||
|
||||
/* ---- 5a: power-loss → WAL replay ---- */
|
||||
char dir[600]; path_in(dir, sizeof dir, "crash_wal"); mkdir(dir, 0700);
|
||||
EngramPagedStore* s = engram_open(dir);
|
||||
if (!s){ ok("engram_open", 0); return; }
|
||||
const int M = 400;
|
||||
for (int i=0;i<M;i++){ StoreNode n; gen_node(i,&n); store_put_node(s,&n); free_node_fields(&n); }
|
||||
store__crash(s); /* abandon RAM (dirty pages lost); WAL fsync'd */
|
||||
s = engram_open(dir); /* replay WAL under 16-frame pool */
|
||||
ok("reopened after crash (WAL replay, tiny pool)", s!=NULL);
|
||||
int bad=0; for (int i=0;i<M;i++){ StoreNode want; gen_node(i,&want);
|
||||
StoreNode got; int hit=store_get_node(s,want.id,&got);
|
||||
if (hit!=1 || !cmp_node(&want,&got)) bad++;
|
||||
if (hit==1) store_node_free(&got); free_node_fields(&want); }
|
||||
ok("all 400 nodes recovered bit-exact via WAL replay under paging", bad==0);
|
||||
ok("store_check crc clean post-recovery", store_check(s, STORE_CHECK_CRC)==0);
|
||||
engram_close(s);
|
||||
|
||||
/* ---- 5b: checkpoint-crash at each phase ---- */
|
||||
for (int phase=0; phase<=4; phase++){
|
||||
char cdir[620]; snprintf(cdir, sizeof cdir, "%s/ck%d", g_dir, phase); mkdir(cdir,0700);
|
||||
EngramPagedStore* c = engram_open(cdir);
|
||||
for (int i=0;i<CK_NODES;i++){ StoreNode n; gen_node(i,&n); store_put_node(c,&n); free_node_fields(&n); }
|
||||
store__checkpoint_crashat(c, phase); /* crash mid-checkpoint (frees c) */
|
||||
EngramPagedStore* r = engram_open(cdir); /* heal + replay under tiny pool */
|
||||
int miss=0; for (int i=0;i<CK_NODES;i++){ StoreNode want; gen_node(i,&want);
|
||||
StoreNode got; int hit=store_get_node(r,want.id,&got);
|
||||
if (hit!=1 || !cmp_node(&want,&got)) miss++;
|
||||
if (hit==1) store_node_free(&got); free_node_fields(&want); }
|
||||
char nm[64]; snprintf(nm,sizeof nm,"checkpoint-crash phase %d: all recovered (paged)", phase);
|
||||
ok(nm, miss==0);
|
||||
engram_close(r);
|
||||
}
|
||||
unsetenv("ENGRAM_POOL_FRAMES");
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* TEST 6 — DEFAULT POOL == PHASE 1: with the default (large) budget, no eviction
|
||||
* ever fires; the whole store is resident, exactly the pre-M4 behaviour.
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
static void test_default_is_phase1(void){
|
||||
printf("\n== 6) default (large) pool == Phase-1 resident (no eviction) ==\n");
|
||||
char path[600]; path_in(path, sizeof path, "default.store");
|
||||
unlink(path);
|
||||
EngramPagedStore* s = store_create(path); /* default cap, no override */
|
||||
if (!s){ ok("store_create", 0); return; }
|
||||
for (int i=0;i<1500;i++){ StoreNode n; gen_node(i,&n); store_put_node(s,&n); free_node_fields(&n); }
|
||||
store_sync(s);
|
||||
for (int i=0;i<1500;i++){ char id[32]; snprintf(id,sizeof id,"node-%d",i);
|
||||
StoreNode g; if (store_get_node(s,id,&g)==1) store_node_free(&g); }
|
||||
StorePoolStats st; store_pool_stats(s,&st);
|
||||
printf(" cap=%zu resident=%zu evictions=%llu (pages=%llu)\n",
|
||||
st.cap, st.resident, (unsigned long long)st.evictions,
|
||||
(unsigned long long)store_page_count(s));
|
||||
ok("default budget is large", st.cap >= (size_t)(1u<<20));
|
||||
ok("no eviction ever fired at default budget", st.evictions == 0);
|
||||
ok("whole store resident (every page cached)", st.resident == store_page_count(s));
|
||||
store_close(s);
|
||||
unlink(path);
|
||||
}
|
||||
|
||||
int main(void){
|
||||
mk_dir();
|
||||
printf("engram M4 buffer-pool gate — dir=%s\n", g_dir);
|
||||
test_small_pool_roundtrip();
|
||||
test_eviction_policy();
|
||||
test_pinning();
|
||||
test_prefetch();
|
||||
test_crash_under_paging();
|
||||
test_default_is_phase1();
|
||||
printf("\n================ %d passed, %d failed ================\n", g_pass, g_fail);
|
||||
return g_fail ? 1 : 0;
|
||||
}
|
||||
@@ -0,0 +1,421 @@
|
||||
/* test_compaction.c — M5 gate: ONLINE COMPACTION + background checkpointer.
|
||||
*
|
||||
* Pure C. Build: gcc -O2 test_compaction.c ../../lang/runtime/engram_store.c -o t
|
||||
* Writes ONLY under a throwaway /tmp dir. Never touches ~/.neuron or live ports.
|
||||
*
|
||||
* Proves:
|
||||
* 1) RECLAIM — tombstone/forget a large fraction of nodes + re-put many edges
|
||||
* (dead versions) + orphan large-record overflow chains, then compact:
|
||||
* page count AND file size drop, yet EVERY live record survives bit-exact and
|
||||
* the id + adjacency indexes resolve correctly at the relocated positions.
|
||||
* 2) CRASH-DURING-COMPACTION — kill at phases 0/1/2; recovery is always a
|
||||
* consistent store (crc clean, every live record intact), never corrupt.
|
||||
* 3) BACKGROUND CHECKPOINTER — a low ops / WAL-bytes threshold fires a checkpoint
|
||||
* automatically on the write path; the WAL prefix is reclaimed; recovery works.
|
||||
* 4) POOL COOPERATION — compaction under a tiny ENGRAM_POOL_FRAMES stays correct
|
||||
* with no stale frame surviving for a relocated page.
|
||||
*/
|
||||
#include "../../lang/runtime/engram_store.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <stdint.h>
|
||||
#include <unistd.h>
|
||||
#include <fcntl.h>
|
||||
#include <sys/stat.h>
|
||||
|
||||
static int g_pass = 0, g_fail = 0;
|
||||
static void ok(const char* name, int cond){
|
||||
printf(" [%s] %s\n", cond ? "PASS" : "FAIL", name);
|
||||
if (cond) g_pass++; else g_fail++;
|
||||
}
|
||||
|
||||
static char g_dir[512];
|
||||
static int g_dseq = 0;
|
||||
static void mk_dir(void){
|
||||
snprintf(g_dir, sizeof g_dir, "/tmp/engram-compact-test-%d-%d", (int)getpid(), g_dseq++);
|
||||
mkdir(g_dir, 0700);
|
||||
}
|
||||
static void egm_path(char* out, size_t cap){ snprintf(out, cap, "%s/neuron.egm", g_dir); }
|
||||
static void wal_path(char* out, size_t cap){ snprintf(out, cap, "%s/neuron.wal", g_dir); }
|
||||
static long file_size(const char* p){ struct stat st; return stat(p,&st)==0 ? (long)st.st_size : -1; }
|
||||
|
||||
/* ── deterministic generators (bit-exact regeneration for oracles) ─────────── */
|
||||
static uint64_t xs(uint64_t* s){ uint64_t x=*s; x^=x<<13; x^=x>>7; x^=x<<17; *s=x; return x; }
|
||||
static uint64_t node_seed(int i){ return 0x9E3779B97F4A7C15ULL ^ ((uint64_t)(i+1)*0xD1B54A32D192ED03ULL); }
|
||||
static uint64_t edge_seed(int i){ return 0xC2B2AE3D27D4EB4FULL ^ ((uint64_t)(i+1)*0x165667B19E3779F9ULL); }
|
||||
static char* rnd_str(uint64_t* st, size_t len){
|
||||
char* s = (char*)malloc(len + 1);
|
||||
for (size_t i=0;i<len;i++) s[i] = (char)(33 + (xs(st) % 94));
|
||||
s[len] = 0; return s;
|
||||
}
|
||||
|
||||
#define N_NODES 1500
|
||||
#define N_DEAD 1200 /* forget node-0 .. node-1199 (1200 dead / 300 live) */
|
||||
#define N_EDGES 3000
|
||||
#define EDGE_REPUT 2000 /* re-put edge-0 .. edge-1999 to version 3 */
|
||||
#define EMB_DIM 96
|
||||
|
||||
static int node_is_live(int i){ return i >= N_DEAD; }
|
||||
static int edge_live_version(int i){ return (i < EDGE_REPUT) ? 3 : 0; }
|
||||
|
||||
static void gen_node(int i, StoreNode* n){
|
||||
memset(n, 0, sizeof *n);
|
||||
uint64_t st = node_seed(i);
|
||||
char id[32]; snprintf(id, sizeof id, "node-%d", i);
|
||||
n->id = strdup(id);
|
||||
/* every 7th record is large → its own overflow chain (orphaned when it dies) */
|
||||
size_t clen = (i % 7 == 0) ? (size_t)(18000 + (xs(&st) % 4000)) : (size_t)(xs(&st) % 200);
|
||||
n->content = rnd_str(&st, clen);
|
||||
n->node_type = rnd_str(&st, 4 + (xs(&st) % 8));
|
||||
n->label = (i % 2) ? rnd_str(&st, 3 + (xs(&st) % 10)) : NULL;
|
||||
n->tier = rnd_str(&st, 4 + (xs(&st) % 6));
|
||||
n->tags = rnd_str(&st, xs(&st) % 40);
|
||||
n->metadata = (i % 3) ? rnd_str(&st, xs(&st) % 60) : NULL;
|
||||
n->salience = (double)(xs(&st) % 1000000) / 997.0;
|
||||
n->importance = (double)(xs(&st) % 1000000) / 131.0;
|
||||
n->confidence = (double)(xs(&st) % 1000000) / 733.0;
|
||||
n->temporal_decay_rate = (double)(xs(&st) % 1000000) / 101.0;
|
||||
n->activation_count = (int64_t)(xs(&st) % 100000);
|
||||
n->last_activated = (int64_t)xs(&st);
|
||||
n->created_at = (int64_t)(1600000000000LL + i);
|
||||
n->updated_at = (int64_t)xs(&st);
|
||||
n->background_activation = (double)(xs(&st) % 1000000) / 17.0;
|
||||
n->working_memory_weight = (double)(xs(&st) % 1000000) / 29.0;
|
||||
n->suppression_count = (int32_t)(xs(&st) % 50);
|
||||
n->layer_id = (uint32_t)(xs(&st) % 5);
|
||||
for (int k=0;k<STORE_BLL_K;k++) n->access_ts[k] = (int64_t)xs(&st);
|
||||
n->access_head = (int32_t)(xs(&st) % STORE_BLL_K);
|
||||
n->access_filled = (int32_t)(xs(&st) % (STORE_BLL_K + 1));
|
||||
n->wm_anchor = (double)(xs(&st) % 1000000) / 3.0;
|
||||
n->emb = (float*)malloc(EMB_DIM * sizeof(float));
|
||||
for (int k=0;k<EMB_DIM;k++){ uint32_t u=(uint32_t)xs(&st); memcpy(&n->emb[k], &u, 4); }
|
||||
n->emb_dim = EMB_DIM;
|
||||
}
|
||||
/* version alters weight/hebb/last_fired so a re-put is a distinct payload. */
|
||||
static void gen_edge(int i, int version, StoreEdge* e){
|
||||
memset(e, 0, sizeof *e);
|
||||
uint64_t st = edge_seed(i);
|
||||
char id[32], from[32], to[32];
|
||||
snprintf(id, sizeof id, "edge-%d", i);
|
||||
/* connect live nodes so adjacency queries on live nodes are meaningful */
|
||||
snprintf(from, sizeof from, "node-%d", N_DEAD + (int)(xs(&st) % (N_NODES - N_DEAD)));
|
||||
snprintf(to, sizeof to, "node-%d", N_DEAD + (int)(xs(&st) % (N_NODES - N_DEAD)));
|
||||
e->id = strdup(id); e->from_id = strdup(from); e->to_id = strdup(to);
|
||||
e->relation = rnd_str(&st, 3 + (xs(&st) % 12));
|
||||
e->metadata = (i % 4) ? rnd_str(&st, xs(&st) % 40) : NULL;
|
||||
e->weight = (double)(xs(&st) % 1000000) / 7.0 + version * 100.0;
|
||||
e->hebb = (double)(xs(&st) % 1000000) / 13.0 + version * 3.0;
|
||||
e->confidence = (double)(xs(&st) % 1000000) / 5.0;
|
||||
e->created_at = (int64_t)(1600000000000LL + i);
|
||||
e->updated_at = (int64_t)xs(&st) + version;
|
||||
e->last_fired = (int64_t)xs(&st) + version * 1000;
|
||||
e->inhibitory = (int32_t)(xs(&st) % 2);
|
||||
e->layer_id = (uint32_t)(xs(&st) % 5);
|
||||
}
|
||||
|
||||
static int streq(const char* a, const char* b){
|
||||
if (!a && !b) return 1; if (!a || !b) return 0; return strcmp(a,b)==0;
|
||||
}
|
||||
static int cmp_node(const StoreNode* a, const StoreNode* b){
|
||||
if (!streq(a->id,b->id) || !streq(a->content,b->content) ||
|
||||
!streq(a->node_type,b->node_type) || !streq(a->label,b->label) ||
|
||||
!streq(a->tier,b->tier) || !streq(a->tags,b->tags) ||
|
||||
!streq(a->metadata,b->metadata)) return 0;
|
||||
if (a->salience!=b->salience || a->importance!=b->importance ||
|
||||
a->confidence!=b->confidence || a->temporal_decay_rate!=b->temporal_decay_rate ||
|
||||
a->activation_count!=b->activation_count || a->last_activated!=b->last_activated ||
|
||||
a->created_at!=b->created_at || a->updated_at!=b->updated_at ||
|
||||
a->background_activation!=b->background_activation ||
|
||||
a->working_memory_weight!=b->working_memory_weight ||
|
||||
a->suppression_count!=b->suppression_count || a->layer_id!=b->layer_id ||
|
||||
a->access_head!=b->access_head || a->access_filled!=b->access_filled ||
|
||||
a->wm_anchor!=b->wm_anchor || a->emb_dim!=b->emb_dim) return 0;
|
||||
for (int k=0;k<STORE_BLL_K;k++) if (a->access_ts[k]!=b->access_ts[k]) return 0;
|
||||
if ((a->emb==NULL) != (b->emb==NULL)) return 0;
|
||||
if (a->emb && memcmp(a->emb, b->emb, (size_t)a->emb_dim*4)!=0) return 0;
|
||||
return 1;
|
||||
}
|
||||
static int cmp_edge(const StoreEdge* a, const StoreEdge* b){
|
||||
if (!streq(a->id,b->id) || !streq(a->from_id,b->from_id) || !streq(a->to_id,b->to_id) ||
|
||||
!streq(a->relation,b->relation) || !streq(a->metadata,b->metadata)) return 0;
|
||||
if (a->weight!=b->weight || a->hebb!=b->hebb || a->confidence!=b->confidence ||
|
||||
a->created_at!=b->created_at || a->updated_at!=b->updated_at ||
|
||||
a->last_fired!=b->last_fired || a->inhibitory!=b->inhibitory ||
|
||||
a->layer_id!=b->layer_id) return 0;
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* Populate a durable store with dead space: all nodes/edges, then forget the first
|
||||
* N_DEAD nodes and re-put the first EDGE_REPUT edges three times. */
|
||||
static void populate_with_dead_space(EngramPagedStore* s){
|
||||
for (int i=0;i<N_NODES;i++){ StoreNode n; gen_node(i,&n); store_put_node(s,&n); store_node_free(&n); }
|
||||
for (int i=0;i<N_EDGES;i++){ StoreEdge e; gen_edge(i,0,&e); store_put_edge(s,&e); store_edge_free(&e); }
|
||||
/* re-put (in-place field mutation) → prior versions become dead records */
|
||||
for (int v=1; v<=3; v++)
|
||||
for (int i=0;i<EDGE_REPUT;i++){ StoreEdge e; gen_edge(i,v,&e); store_put_edge(s,&e); store_edge_free(&e); }
|
||||
/* forget the cold nodes (tombstone; their large overflow chains orphan) */
|
||||
for (int i=0;i<N_DEAD;i++){ char id[32]; snprintf(id,sizeof id,"node-%d",i); store_forget(s,id); }
|
||||
}
|
||||
|
||||
/* Assert every live node/edge is present + bit-exact via point reads. */
|
||||
static int verify_live_set(EngramPagedStore* s){
|
||||
int bad = 0;
|
||||
for (int i=0;i<N_NODES;i++){
|
||||
char id[32]; snprintf(id,sizeof id,"node-%d",i);
|
||||
StoreNode got; int hit = store_get_node(s, id, &got);
|
||||
if (node_is_live(i)){
|
||||
StoreNode want; gen_node(i,&want);
|
||||
if (hit!=1 || !cmp_node(&want,&got)) bad++;
|
||||
if (hit==1) store_node_free(&got);
|
||||
store_node_free(&want);
|
||||
} else {
|
||||
if (hit!=0) bad++; /* forgotten → must be absent */
|
||||
if (hit==1) store_node_free(&got);
|
||||
}
|
||||
}
|
||||
for (int i=0;i<N_EDGES;i++){
|
||||
char id[32]; snprintf(id,sizeof id,"edge-%d",i);
|
||||
StoreEdge got; int hit = store_get_edge(s, id, &got);
|
||||
StoreEdge want; gen_edge(i, edge_live_version(i), &want);
|
||||
if (hit!=1 || !cmp_edge(&want,&got)) bad++;
|
||||
if (hit==1) store_edge_free(&got);
|
||||
store_edge_free(&want);
|
||||
}
|
||||
return bad;
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* TEST 1 — RECLAIM: dead space is reclaimed; live records + indexes survive.
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
static void test_reclaim(void){
|
||||
printf("\n== 1) reclaim: forget %d nodes + re-put %d edges x3, then compact ==\n",
|
||||
N_DEAD, EDGE_REPUT);
|
||||
mk_dir();
|
||||
char egm[600]; egm_path(egm, sizeof egm);
|
||||
EngramPagedStore* s = engram_open(g_dir);
|
||||
ok("engram_open", s != NULL);
|
||||
if (!s) return;
|
||||
|
||||
populate_with_dead_space(s);
|
||||
engram_checkpoint(s); /* flush so file size reflects state */
|
||||
|
||||
uint64_t pc_before = store_page_count(s);
|
||||
uint64_t free_before = store_free_page_count(s);
|
||||
long sz_before = file_size(egm);
|
||||
printf(" BEFORE: page_count=%llu free_pages=%llu file=%ld bytes (live records intact?)\n",
|
||||
(unsigned long long)pc_before, (unsigned long long)free_before, sz_before);
|
||||
ok("pre-compaction live set intact", verify_live_set(s)==0);
|
||||
|
||||
/* capture adjacency for a sample of live from-ids to compare post-compaction */
|
||||
#define NSAMP 12
|
||||
char samp[NSAMP][32]; size_t pre_cnt[NSAMP];
|
||||
for (int k=0;k<NSAMP;k++){
|
||||
snprintf(samp[k], sizeof samp[k], "node-%d", N_DEAD + k*20);
|
||||
StoreEdge* arr=NULL; size_t cnt=0;
|
||||
store_get_edges_from(s, samp[k], &arr, &cnt);
|
||||
pre_cnt[k]=cnt; store_edges_free(arr,cnt);
|
||||
}
|
||||
|
||||
int rc = store_compact(s);
|
||||
ok("store_compact returns 0", rc==0);
|
||||
|
||||
uint64_t pc_after = store_page_count(s);
|
||||
uint64_t free_after = store_free_page_count(s);
|
||||
long sz_after = file_size(egm);
|
||||
printf(" AFTER : page_count=%llu free_pages=%llu file=%ld bytes\n",
|
||||
(unsigned long long)pc_after, (unsigned long long)free_after, sz_after);
|
||||
printf(" RECLAIMED: %llu pages, %ld bytes (%.1f%% of file)\n",
|
||||
(unsigned long long)(pc_before - pc_after), sz_before - sz_after,
|
||||
sz_before ? 100.0*(sz_before-sz_after)/sz_before : 0.0);
|
||||
|
||||
ok("page count dropped (dead pages reclaimed)", pc_after < pc_before);
|
||||
ok("file size dropped (store physically shrank)", sz_after < sz_before);
|
||||
ok("store_check crc clean after compaction", store_check(s, STORE_CHECK_CRC)==0);
|
||||
ok("every LIVE record present + bit-exact at new locations", verify_live_set(s)==0);
|
||||
|
||||
/* adjacency index correct at relocated positions */
|
||||
int adj_bad = 0;
|
||||
for (int k=0;k<NSAMP;k++){
|
||||
StoreEdge* arr=NULL; size_t cnt=0;
|
||||
store_get_edges_from(s, samp[k], &arr, &cnt);
|
||||
if (cnt != pre_cnt[k]) adj_bad++;
|
||||
for (size_t j=0;j<cnt;j++){
|
||||
if (!streq(arr[j].from_id, samp[k])) { adj_bad++; break; }
|
||||
/* the returned edge must be the canonical latest live edge, bit-exact */
|
||||
int idx = atoi(arr[j].id + 5);
|
||||
StoreEdge want; gen_edge(idx, edge_live_version(idx), &want);
|
||||
if (!cmp_edge(&want,&arr[j])) adj_bad++;
|
||||
store_edge_free(&want);
|
||||
}
|
||||
store_edges_free(arr,cnt);
|
||||
}
|
||||
ok("adjacency (get_edges_from) correct + bit-exact post-compaction", adj_bad==0);
|
||||
|
||||
/* second compaction is a near no-op (no new dead space) and stays correct */
|
||||
uint64_t pc2_before = store_page_count(s);
|
||||
ok("compact again returns 0", store_compact(s)==0);
|
||||
ok("idempotent-ish: no growth on re-compact", store_page_count(s) <= pc2_before);
|
||||
ok("live set still intact after 2nd compaction", verify_live_set(s)==0);
|
||||
|
||||
engram_close(s);
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* TEST 2 — CRASH DURING COMPACTION: kill at phases 0/1/2 → consistent recovery.
|
||||
* Live set is identical whether we recover pre- or post-compaction, so the same
|
||||
* oracle must hold, and crc must always be clean (never corrupt).
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
static void test_crash_during_compaction(void){
|
||||
printf("\n== 2) crash during compaction at phases 0,1,2 → consistent store ==\n");
|
||||
for (int phase=0; phase<=2; phase++){
|
||||
mk_dir();
|
||||
char egm[600]; egm_path(egm, sizeof egm);
|
||||
EngramPagedStore* s = engram_open(g_dir);
|
||||
if (!s){ ok("engram_open", 0); continue; }
|
||||
populate_with_dead_space(s);
|
||||
engram_close(s); /* durable baseline on disk */
|
||||
|
||||
uint64_t pc_pre = 0;
|
||||
{ EngramPagedStore* p = engram_open(g_dir); pc_pre = store_page_count(p); engram_close(p); }
|
||||
|
||||
EngramPagedStore* c = engram_open(g_dir);
|
||||
store__compact_crashat(c, phase); /* crashes mid-compaction (frees c) */
|
||||
|
||||
EngramPagedStore* r = engram_open(g_dir); /* recover */
|
||||
char nm[80];
|
||||
snprintf(nm, sizeof nm, "phase %d: recovers, crc clean", phase);
|
||||
ok(nm, r && store_check(r, STORE_CHECK_CRC)==0);
|
||||
snprintf(nm, sizeof nm, "phase %d: every live record intact (not corrupt)", phase);
|
||||
ok(nm, r && verify_live_set(r)==0);
|
||||
if (r){
|
||||
uint64_t pc_now = store_page_count(r);
|
||||
if (phase < 2){
|
||||
snprintf(nm, sizeof nm, "phase %d: recovered PRE-compaction image", phase);
|
||||
ok(nm, pc_now == pc_pre);
|
||||
} else {
|
||||
snprintf(nm, sizeof nm, "phase %d: recovered POST-compaction (shrunk)", phase);
|
||||
ok(nm, pc_now < pc_pre);
|
||||
}
|
||||
/* store stays writable + durable after recovery */
|
||||
StoreNode n; gen_node(N_NODES+phase, &n); free(n.id);
|
||||
n.id = strdup("post-recovery-node");
|
||||
store_put_node(r, &n); store_node_free(&n);
|
||||
StoreNode g; int hit = store_get_node(r, "post-recovery-node", &g);
|
||||
snprintf(nm, sizeof nm, "phase %d: store writable after recovery", phase);
|
||||
ok(nm, hit==1);
|
||||
if (hit==1) store_node_free(&g);
|
||||
engram_close(r);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* TEST 3 — BACKGROUND CHECKPOINTER: a low threshold fires checkpoints on the
|
||||
* write path, reclaiming the WAL prefix automatically; recovery still correct.
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
static void test_background_checkpointer(void){
|
||||
printf("\n== 3) background checkpointer: auto-checkpoint on threshold ==\n");
|
||||
|
||||
/* (a) ops trigger */
|
||||
{
|
||||
mk_dir();
|
||||
char wal[600]; wal_path(wal, sizeof wal);
|
||||
EngramPagedStore* s = engram_open(g_dir);
|
||||
if (!s){ ok("engram_open", 0); return; }
|
||||
store_set_checkpoint_policy(s, /*ops*/50, /*dirty*/0, /*wal_bytes*/0, /*ms*/0);
|
||||
uint64_t ckpt0 = engram_last_checkpoint_lsn(s);
|
||||
for (int i=0;i<600;i++){ StoreNode n; gen_node(i,&n); store_put_node(s,&n); store_node_free(&n); }
|
||||
uint64_t ckpt1 = engram_last_checkpoint_lsn(s);
|
||||
long wsz = file_size(wal);
|
||||
printf(" ops-trigger: ckpt_lsn %llu -> %llu, WAL=%ld bytes after 600 puts\n",
|
||||
(unsigned long long)ckpt0, (unsigned long long)ckpt1, wsz);
|
||||
ok("ops trigger fired an automatic checkpoint", ckpt1 > ckpt0);
|
||||
ok("WAL prefix reclaimed (WAL stays small)", wsz >= 0 && wsz < 200000);
|
||||
/* crash (abandon RAM) then recover — everything durable via WAL+checkpoint */
|
||||
store__crash(s);
|
||||
EngramPagedStore* r = engram_open(g_dir);
|
||||
int bad=0;
|
||||
for (int i=0;i<600;i++){ char id[32]; snprintf(id,sizeof id,"node-%d",i);
|
||||
StoreNode w; gen_node(i,&w); StoreNode g; int hit=store_get_node(r,id,&g);
|
||||
if (hit!=1 || !cmp_node(&w,&g)) bad++; if(hit==1) store_node_free(&g); store_node_free(&w); }
|
||||
ok("recovery correct after auto-checkpoints (ops)", r && bad==0);
|
||||
ok("crc clean after recovery (ops)", r && store_check(r,STORE_CHECK_CRC)==0);
|
||||
if (r) engram_close(r);
|
||||
}
|
||||
/* (b) WAL-bytes trigger */
|
||||
{
|
||||
mk_dir();
|
||||
char wal[600]; wal_path(wal, sizeof wal);
|
||||
EngramPagedStore* s = engram_open(g_dir);
|
||||
if (!s){ ok("engram_open", 0); return; }
|
||||
store_set_checkpoint_policy(s, /*ops*/0, /*dirty*/0, /*wal_bytes*/64*1024, /*ms*/0);
|
||||
uint64_t ckpt0 = engram_last_checkpoint_lsn(s);
|
||||
for (int i=0;i<600;i++){ StoreNode n; gen_node(i,&n); store_put_node(s,&n); store_node_free(&n); }
|
||||
uint64_t ckpt1 = engram_last_checkpoint_lsn(s);
|
||||
long wsz = file_size(wal);
|
||||
printf(" wal-bytes-trigger: ckpt_lsn %llu -> %llu, WAL=%ld bytes\n",
|
||||
(unsigned long long)ckpt0, (unsigned long long)ckpt1, wsz);
|
||||
ok("wal-bytes trigger fired an automatic checkpoint", ckpt1 > ckpt0);
|
||||
ok("WAL kept bounded by byte threshold", wsz >= 0 && wsz < 2*1024*1024);
|
||||
engram_close(s);
|
||||
}
|
||||
/* (c) dirty-frames trigger (under a bounded pool) */
|
||||
{
|
||||
mk_dir();
|
||||
EngramPagedStore* s = engram_open(g_dir);
|
||||
if (!s){ ok("engram_open", 0); return; }
|
||||
store_set_checkpoint_policy(s, /*ops*/0, /*dirty*/16, /*wal_bytes*/0, /*ms*/0);
|
||||
uint64_t ckpt0 = engram_last_checkpoint_lsn(s);
|
||||
for (int i=0;i<400;i++){ StoreNode n; gen_node(i,&n); store_put_node(s,&n); store_node_free(&n); }
|
||||
uint64_t ckpt1 = engram_last_checkpoint_lsn(s);
|
||||
ok("dirty-frames trigger fired an automatic checkpoint", ckpt1 > ckpt0);
|
||||
engram_close(s);
|
||||
}
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* TEST 4 — POOL COOPERATION: compact under a tiny frame budget (constant eviction
|
||||
* + re-fault); correctness holds and no stale frame survives a relocated page.
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
static void test_pool_cooperation(void){
|
||||
printf("\n== 4) compaction under a small buffer pool (forced eviction) ==\n");
|
||||
setenv("ENGRAM_POOL_FRAMES", "24", 1); /* pool << store, and the temp build too */
|
||||
mk_dir();
|
||||
char egm[600]; egm_path(egm, sizeof egm);
|
||||
EngramPagedStore* s = engram_open(g_dir);
|
||||
ok("engram_open (24-frame pool)", s != NULL);
|
||||
if (!s){ unsetenv("ENGRAM_POOL_FRAMES"); return; }
|
||||
store__set_pool_frames(s, 24);
|
||||
|
||||
populate_with_dead_space(s);
|
||||
engram_checkpoint(s);
|
||||
uint64_t pc_before = store_page_count(s);
|
||||
|
||||
int rc = store_compact(s);
|
||||
ok("store_compact under tiny pool returns 0", rc==0);
|
||||
|
||||
StorePoolStats st; store_pool_stats(s, &st);
|
||||
printf(" post-compaction pool: cap=%zu resident=%zu pinned=%zu dirty=%zu\n",
|
||||
st.cap, st.resident, st.pinned, st.dirty);
|
||||
ok("pool respected budget after compaction (resident<=cap)", st.resident <= st.cap);
|
||||
ok("page count dropped under small pool", store_page_count(s) < pc_before);
|
||||
ok("crc clean under small pool", store_check(s, STORE_CHECK_CRC)==0);
|
||||
/* If any relocated page had a stale frame, a read would return wrong bytes. */
|
||||
ok("every live record bit-exact under small pool (no stale frames)", verify_live_set(s)==0);
|
||||
|
||||
engram_close(s);
|
||||
unsetenv("ENGRAM_POOL_FRAMES");
|
||||
}
|
||||
|
||||
int main(void){
|
||||
printf("=== M5 COMPACTION + BACKGROUND CHECKPOINTER GATE ===\n");
|
||||
test_reclaim();
|
||||
test_crash_during_compaction();
|
||||
test_background_checkpointer();
|
||||
test_pool_cooperation();
|
||||
printf("\n=== RESULT: %d passed, %d failed ===\n", g_pass, g_fail);
|
||||
/* cleanup */
|
||||
return g_fail ? 1 : 0;
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
/* test_geometry.c — build + RUN gate for the M9 FOUNDATION geometry descriptor
|
||||
* (engram_geometry.{c,h}). Self-contained: synthesizes a store with two KNOWN
|
||||
* embedding clusters + intra-cluster hebb edges, then verifies the descriptor
|
||||
* recovers the shape — centroid near the seeded cluster, skeleton = the strong
|
||||
* intra-cluster edges, membership gradient, radius, positive co-registration.
|
||||
*
|
||||
* Pure C11; links engram_geometry.c + engram_store.c + engram_vindex.c; -lm.
|
||||
* ASan/UBSan clean. Needs no live data.
|
||||
*/
|
||||
#include "engram_geometry.h"
|
||||
#include "engram_store.h"
|
||||
#include "engram_vindex.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <math.h>
|
||||
#include <stdint.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#define DIM 64
|
||||
static int g_fail=0;
|
||||
#define CHECK(c,m) do{ if(!(c)){printf(" FAIL: %s\n",m); g_fail=1;} else printf(" ok: %s\n",m);}while(0)
|
||||
|
||||
static uint64_t rs=0x1234abcdULL;
|
||||
static uint64_t xr(void){ uint64_t z=(rs+=0x9E3779B97F4A7C15ULL);
|
||||
z=(z^(z>>30))*0xBF58476D1CE4E5B9ULL; z=(z^(z>>27))*0x94D049BB133111EBULL; return z^(z>>31); }
|
||||
static float jitter(void){ return (float)(((double)(xr()>>11)*(1.0/9007199254740992.0))-0.5)*0.15f; }
|
||||
|
||||
/* two clusters: A centered on axis 0, B centered on axis 1. NA+NB nodes. */
|
||||
#define NA 40
|
||||
#define NB 40
|
||||
|
||||
int main(void){
|
||||
printf("=== engram_geometry (M9 foundation) test suite ===\n");
|
||||
char path[256]; snprintf(path,sizeof path,"/tmp/geo_test_store_%d.egm",(int)getpid());
|
||||
unlink(path);
|
||||
EngramPagedStore* st=store_create(path);
|
||||
if(!st){ printf("FAIL: store_create\n"); return 1; }
|
||||
|
||||
char aids[NA][16], bids[NB][16];
|
||||
/* cluster A: near +e0 ; cluster B: near +e1 */
|
||||
for(int i=0;i<NA;i++){
|
||||
StoreNode n; memset(&n,0,sizeof n);
|
||||
snprintf(aids[i],16,"A%d",i); n.id=aids[i]; n.node_type="Concept"; n.tier="Semantic";
|
||||
n.content="cluster-A"; n.salience=0.5+0.01*i;
|
||||
float v[DIM]; for(int d=0;d<DIM;d++) v[d]=jitter(); v[0]=1.0f+jitter();
|
||||
n.emb=v; n.emb_dim=DIM; store_put_node(st,&n);
|
||||
}
|
||||
for(int i=0;i<NB;i++){
|
||||
StoreNode n; memset(&n,0,sizeof n);
|
||||
snprintf(bids[i],16,"B%d",i); n.id=bids[i]; n.node_type="Concept"; n.tier="Semantic";
|
||||
n.content="cluster-B"; n.salience=0.3;
|
||||
float v[DIM]; for(int d=0;d<DIM;d++) v[d]=jitter(); v[1]=1.0f+jitter();
|
||||
n.emb=v; n.emb_dim=DIM; store_put_node(st,&n);
|
||||
}
|
||||
/* strong intra-A hebb edges (a chain + hub), weaker cross edges A0<->B0 */
|
||||
int ei=0;
|
||||
for(int i=1;i<NA;i++){
|
||||
StoreEdge e; memset(&e,0,sizeof e); char id[24]; snprintf(id,24,"eA%d",ei++);
|
||||
e.id=id; e.from_id=aids[0]; e.to_id=aids[i]; e.relation="assoc"; e.weight=0.9; e.hebb=0.4;
|
||||
store_put_edge(st,&e);
|
||||
}
|
||||
for(int i=1;i<NB;i++){
|
||||
StoreEdge e; memset(&e,0,sizeof e); char id[24]; snprintf(id,24,"eB%d",ei++);
|
||||
e.id=id; e.from_id=bids[0]; e.to_id=bids[i]; e.relation="assoc"; e.weight=0.9; e.hebb=0.4;
|
||||
store_put_edge(st,&e);
|
||||
}
|
||||
{ StoreEdge e; memset(&e,0,sizeof e); e.id=(char*)"eX"; e.from_id=aids[0]; e.to_id=bids[0];
|
||||
e.relation="assoc"; e.weight=0.5; e.hebb=0.0; store_put_edge(st,&e); }
|
||||
store_close(st);
|
||||
|
||||
VIndex* ix=vindex_create(DIM,0,0);
|
||||
char** ids=NULL; int nids=0;
|
||||
int ins=vindex_build_from_store(ix, path, &ids, &nids);
|
||||
CHECK(ins==NA+NB, "vindex built over all embedded nodes");
|
||||
|
||||
GeoParams P; engram_geo_default_params(&P); P.ann_k=20; P.max_members=0;
|
||||
|
||||
/* seed inside cluster A -> expect an A-dominated neighborhood */
|
||||
st=store_open(path);
|
||||
/* global-mean cache over the embedded set: the centering offset */
|
||||
GeoMeanCache* mc=engram_geo_mean_build(st);
|
||||
const float* gm=engram_geo_mean_vec(mc);
|
||||
CHECK(mc!=NULL && engram_geo_mean_dim(mc)==DIM, "global-mean cache built over embedded set");
|
||||
CHECK(engram_geo_mean_count(mc)==(uint64_t)(NA+NB), "global mean averaged all embedded nodes");
|
||||
const char* seeds[1]={aids[0]};
|
||||
/* CENTERED descriptor: pass the global mean so geometry runs in isotropic space */
|
||||
GeoDescriptor* g=engram_geometry_descriptor(st, ix, ids, nids, seeds, 1, &P, gm);
|
||||
CHECK(g!=NULL, "descriptor computed");
|
||||
if(g){
|
||||
printf(" members=%d embedded=%d edges=%d k_core=%d radius=%.4f co_reg=%.3f n_axes=%d\n",
|
||||
g->n_members,g->n_embedded,g->n_edges,g->k_core,g->radius,g->co_registration,g->n_axes);
|
||||
|
||||
/* geometry ran in CENTERED space: g->centroid is the centered centroid,
|
||||
* g->global_mean the applied offset. Reconstruct the raw prototype
|
||||
* (centroid + global_mean) and check it sits on cluster-A's axis. */
|
||||
CHECK(g->global_mean!=NULL, "descriptor recorded the centering offset (centered mode)");
|
||||
int argmax=0; float best=-1.f;
|
||||
for(int d=0;d<g->dim;d++){ float raw=g->centroid[d]+(g->global_mean?g->global_mean[d]:0.f);
|
||||
if(fabsf(raw)>best){ best=fabsf(raw); argmax=d; } }
|
||||
printf(" raw-prototype dominant axis = %d (expect 0); centered c[0]=%.3f c[1]=%.3f\n",
|
||||
argmax, g->centroid[0], g->centroid[1]);
|
||||
CHECK(argmax==0, "raw prototype sits on cluster-A's axis (near members)");
|
||||
/* centering pushes A off cluster-B's axis: centered c[0] > c[1] */
|
||||
CHECK(g->centroid[0] > g->centroid[1], "centered centroid leans off B's axis (isotropy)");
|
||||
|
||||
/* hub should be A0 (the intra-A hub with NA-1 strong edges) */
|
||||
CHECK(g->hub_id && strcmp(g->hub_id,"A0")==0, "hub = the relational center A0");
|
||||
|
||||
/* membership: seed A0 == 1.0; A-members strong, B-members (if any) weaker */
|
||||
double seedw=-1, minA=2, maxB=-1; int na=0,nb=0;
|
||||
for(int i=0;i<g->n_members;i++){
|
||||
const char* id=g->members[i].id; double w=g->members[i].membership;
|
||||
if(strcmp(id,"A0")==0) seedw=w;
|
||||
if(id[0]=='A'){ na++; if(w<minA)minA=w; }
|
||||
if(id[0]=='B'){ nb++; if(w>maxB)maxB=w; }
|
||||
}
|
||||
printf(" A-members=%d B-members=%d seedw=%.3f\n", na,nb,seedw);
|
||||
CHECK(fabs(seedw-1.0)<1e-9, "seed membership == 1.0");
|
||||
CHECK(na>=NA-1, "neighborhood recovers cluster A");
|
||||
|
||||
/* skeleton = the strong intra-A edges: every edge eff_weight>=threshold,
|
||||
* and edges connect A-nodes (co-registration should be positive: wired
|
||||
* pairs are semantically near). */
|
||||
int allstrong=1, allA=1;
|
||||
for(int e=0;e<g->n_edges;e++){
|
||||
if(g->edges[e].eff_weight < P.edge_min_weight) allstrong=0;
|
||||
const char* a=g->members[g->edges[e].a].id, *b=g->members[g->edges[e].b].id;
|
||||
if(!(a[0]=='A'&&b[0]=='A')) { /* the lone eX cross edge is allowed */
|
||||
if(!((strcmp(a,"A0")==0&&strcmp(b,"B0")==0)||(strcmp(a,"B0")==0&&strcmp(b,"A0")==0))) allA=0; }
|
||||
}
|
||||
CHECK(allstrong, "skeleton holds only above-threshold (strong) edges");
|
||||
CHECK(allA, "skeleton backbone is the intra-cluster wiring");
|
||||
CHECK(g->co_registration>0.0, "co-registration positive (wired pairs are semantically near)");
|
||||
|
||||
/* principal axes: extents strictly non-increasing */
|
||||
int mono=1; for(int i=1;i<g->n_axes;i++) if(g->axes[i].extent>g->axes[i-1].extent+1e-9) mono=0;
|
||||
CHECK(g->n_axes>0 && mono, "principal axes sorted by descending extent");
|
||||
CHECK(g->radius>0, "radius positive");
|
||||
}
|
||||
engram_geo_free(g);
|
||||
|
||||
/* edge cases: NULL store, no seeds, relational-only (NULL vindex) */
|
||||
CHECK(engram_geometry_descriptor(NULL,ix,ids,nids,seeds,1,&P,gm)==NULL, "NULL store -> NULL");
|
||||
CHECK(engram_geometry_descriptor(st,ix,ids,nids,seeds,0,&P,gm)==NULL, "zero seeds -> NULL");
|
||||
GeoDescriptor* g2=engram_geometry_descriptor(st, NULL, NULL, 0, seeds, 1, &P, gm);
|
||||
CHECK(g2!=NULL && g2->n_members>=NA-1, "relational-only path (no vindex) works");
|
||||
engram_geo_free(g2);
|
||||
/* raw (uncentered) mode still supported: global_mean=NULL -> no offset recorded */
|
||||
GeoDescriptor* g3=engram_geometry_descriptor(st, ix, ids, nids, seeds, 1, &P, NULL);
|
||||
CHECK(g3!=NULL && g3->global_mean==NULL, "raw mode (global_mean=NULL) leaves offset unset");
|
||||
engram_geo_free(g3);
|
||||
|
||||
engram_geo_mean_free(mc);
|
||||
for(int i=0;i<nids;i++) free(ids[i]); free(ids);
|
||||
vindex_free(ix); store_close(st); unlink(path);
|
||||
printf("\n=== %s ===\n", g_fail?"FAILURES PRESENT":"ALL TESTS PASSED");
|
||||
return g_fail;
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
/* test_interoception_p0_emb.c — M-INTEROCEPTION Priority 0.
|
||||
*
|
||||
* Verifies the new READ-ONLY builtin engram_scan_nodes_emb_json(limit,offset):
|
||||
* - every emitted node carries emb_dim and an emb JSON array of that length,
|
||||
* - nodes without an embedding emit emb_dim:0 / emb:[],
|
||||
* - pagination (limit/offset) is honoured,
|
||||
* - the count matches engram_node_count,
|
||||
* - the EXISTING engram_scan_nodes_json path is byte-unchanged (no emb field),
|
||||
* i.e. the addition is purely additive / behavior-neutral.
|
||||
*
|
||||
* Pure-C harness (no elc). We craft a snapshot with real emb vectors, load it
|
||||
* (engram_load parses "emb" comma-lists into node->emb via eg_parse_emb), then
|
||||
* dump via both scan paths. Assertions live in run_interoception_p0.sh.
|
||||
*/
|
||||
#include "el_runtime.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
static el_val_t S(const char* s){ return EL_STR(s); }
|
||||
|
||||
/* 16-d embedding as a comma list (>=8 required by eg_parse_emb). */
|
||||
static void emb_list(char* out, size_t cap, int dim, double base){
|
||||
size_t o = 0;
|
||||
for (int i = 0; i < dim; i++){
|
||||
o += snprintf(out+o, cap-o, "%s%.4f", i?",":"", base + 0.01*i);
|
||||
}
|
||||
}
|
||||
|
||||
int main(int argc, char** argv){
|
||||
if (argc < 2){ fprintf(stderr, "usage: %s <dir>\n", argv[0]); return 2; }
|
||||
const char* dir = argv[1];
|
||||
char snap[1024]; snprintf(snap, sizeof snap, "%s/seed.json", dir);
|
||||
|
||||
char e1[512], e2[512];
|
||||
emb_list(e1, sizeof e1, 16, 0.10);
|
||||
emb_list(e2, sizeof e2, 16, 0.50);
|
||||
|
||||
/* Two embedded nodes (distinct salience → deterministic sort order) and one
|
||||
* un-embedded node. */
|
||||
FILE* f = fopen(snap, "w");
|
||||
if (!f){ perror("fopen"); return 2; }
|
||||
fprintf(f,
|
||||
"{\"nodes\":["
|
||||
"{\"id\":\"n-high\",\"content\":\"high salience embedded\",\"node_type\":\"Concept\","
|
||||
"\"label\":\"emb-high\",\"tier\":\"Semantic\",\"salience\":0.9,\"importance\":0.8,"
|
||||
"\"confidence\":1.0,\"created_at\":1000,\"emb\":\"%s\"},"
|
||||
"{\"id\":\"n-mid\",\"content\":\"mid salience embedded\",\"node_type\":\"Concept\","
|
||||
"\"label\":\"emb-mid\",\"tier\":\"Semantic\",\"salience\":0.5,\"importance\":0.5,"
|
||||
"\"confidence\":1.0,\"created_at\":2000,\"emb\":\"%s\"},"
|
||||
"{\"id\":\"n-low\",\"content\":\"low salience no embedding\",\"node_type\":\"Fact\","
|
||||
"\"label\":\"noemb-low\",\"tier\":\"Semantic\",\"salience\":0.1,\"importance\":0.2,"
|
||||
"\"confidence\":1.0,\"created_at\":3000}"
|
||||
"],\"edges\":[]}", e1, e2);
|
||||
fclose(f);
|
||||
|
||||
if (!engram_load(S(snap))){ fprintf(stderr, "load failed\n"); return 2; }
|
||||
long long nc = (long long)(int64_t)engram_node_count();
|
||||
printf("node_count=%lld\n", nc);
|
||||
|
||||
/* full page */
|
||||
el_val_t all = engram_scan_nodes_emb_json((el_val_t)256, (el_val_t)0);
|
||||
char p[1024];
|
||||
snprintf(p, sizeof p, "%s/emb_all.json", dir);
|
||||
f = fopen(p, "w"); fputs(EL_CSTR(all), f); fclose(f);
|
||||
|
||||
/* pagination: one node at offset 0 and one at offset 1 */
|
||||
el_val_t pg0 = engram_scan_nodes_emb_json((el_val_t)1, (el_val_t)0);
|
||||
el_val_t pg1 = engram_scan_nodes_emb_json((el_val_t)1, (el_val_t)1);
|
||||
snprintf(p, sizeof p, "%s/emb_pg0.json", dir); f = fopen(p, "w"); fputs(EL_CSTR(pg0), f); fclose(f);
|
||||
snprintf(p, sizeof p, "%s/emb_pg1.json", dir); f = fopen(p, "w"); fputs(EL_CSTR(pg1), f); fclose(f);
|
||||
|
||||
/* existing path — must be unchanged / carry NO emb */
|
||||
el_val_t plain = engram_scan_nodes_json((el_val_t)256, (el_val_t)0);
|
||||
snprintf(p, sizeof p, "%s/plain.json", dir); f = fopen(p, "w"); fputs(EL_CSTR(plain), f); fclose(f);
|
||||
|
||||
printf("wrote dumps to %s\n", dir);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
/* test_interoception_p1_consol.c — M-INTEROCEPTION Priority 1.
|
||||
* Two-threshold consolidation (ENGRAM_CONSOLIDATION, default OFF).
|
||||
*
|
||||
* Modes:
|
||||
* accrual — flag OFF (pure trunk). Drive N co-activations of a WIRED pair and
|
||||
* print act-stats at sampled N so the run script can plot the
|
||||
* hebb accrual curve (headline measurement). No consolidation code
|
||||
* runs; this measures the EXISTING EWMA accrual.
|
||||
* connect — flag ON. Seed, activate to populate WM, then create a STRONG ISE
|
||||
* (connects to wm_top) and a WEAK ISE (below the bar → nothing).
|
||||
* Exports the graph so edges from each ISE can be counted.
|
||||
* perm — flag ON. Load two OLD InternalStateEvent nodes; promote one to
|
||||
* permanence; prune telemetry; export so the durable one is shown
|
||||
* to survive while the ephemeral one is swept.
|
||||
* offcheck — flag OFF. Prove creating an ISE forms NO edges and
|
||||
* engram_consolidate_permanence is a no-op (byte-identical OFF path).
|
||||
*/
|
||||
#include "el_runtime.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
static el_val_t S(const char* s){ return EL_STR(s); }
|
||||
static el_val_t F(double d){ return el_from_float(d); }
|
||||
|
||||
static void build_seed(void){
|
||||
el_val_t a = engram_node_full(S("hebbian potentiation strengthens co-active memory links"),
|
||||
S("Concept"), S("hebb-a"), F(0.9), F(0.85), F(1.0), S("Semantic"),
|
||||
S("hebbian,memory,activation"));
|
||||
el_val_t b = engram_node_full(S("co-active memory links accrue hebbian associative weight"),
|
||||
S("Concept"), S("hebb-b"), F(0.9), F(0.85), F(1.0), S("Semantic"),
|
||||
S("hebbian,memory,weight"));
|
||||
el_val_t c = engram_node_full(S("unrelated culinary recipe for sourdough bread"),
|
||||
S("Fact"), S("distractor-1"), F(0.4), F(0.4), F(1.0), S("Semantic"), S("food"));
|
||||
el_val_t d = engram_node_full(S("the weather forecast predicts rain tomorrow afternoon"),
|
||||
S("Fact"), S("distractor-2"), F(0.4), F(0.4), F(1.0), S("Semantic"), S("weather"));
|
||||
engram_connect(a, b, F(0.8), S("associate"));
|
||||
engram_connect(a, c, F(0.3), S("associate"));
|
||||
engram_connect(b, d, F(0.3), S("associate"));
|
||||
}
|
||||
static const char* QUERY =
|
||||
"hebbian potentiation co-active memory links associative weight";
|
||||
|
||||
int main(int argc, char** argv){
|
||||
if (argc < 3){ fprintf(stderr,"usage: %s <accrual|connect|perm|offcheck> <dir>\n",argv[0]); return 2; }
|
||||
const char* mode = argv[1];
|
||||
const char* dir = argv[2];
|
||||
char p[1024];
|
||||
|
||||
if (!strcmp(mode,"accrual")){
|
||||
build_seed();
|
||||
int samples[] = {1,10,50,100,250,500,1000,1625,2000,2500,3000};
|
||||
int ns = (int)(sizeof samples/sizeof samples[0]);
|
||||
int NMAX = samples[ns-1];
|
||||
int si = 0;
|
||||
for (int n=1; n<=NMAX; n++){
|
||||
engram_activate_json(S(QUERY), (el_val_t)3);
|
||||
if (si<ns && n==samples[si]){
|
||||
printf("SAMPLE %d %s\n", n, EL_CSTR(engram_act_stats_json()));
|
||||
si++;
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!strcmp(mode,"connect")){
|
||||
build_seed();
|
||||
engram_activate_json(S(QUERY), (el_val_t)3);
|
||||
long long e_before = (long long)(int64_t)engram_edge_count();
|
||||
/* STRONG ISE — should connect to wm_top */
|
||||
el_val_t ise_strong = engram_node_full(S("strong internal state: focused on hebbian consolidation"),
|
||||
S("InternalStateEvent"), S("ise-strong"), F(0.9), F(0.8), F(1.0), S("Working"), S("ise"));
|
||||
long long e_after_strong = (long long)(int64_t)engram_edge_count();
|
||||
/* WEAK ISE — below the connection bar (salience 0.3 < default 0.6) */
|
||||
el_val_t ise_weak = engram_node_full(S("weak internal state: idle drift"),
|
||||
S("InternalStateEvent"), S("ise-weak"), F(0.3), F(0.3), F(1.0), S("Working"), S("ise"));
|
||||
long long e_after_weak = (long long)(int64_t)engram_edge_count();
|
||||
printf("ISE_STRONG_ID %s\n", EL_CSTR(ise_strong));
|
||||
printf("ISE_WEAK_ID %s\n", EL_CSTR(ise_weak));
|
||||
printf("EDGES before=%lld after_strong=%lld after_weak=%lld\n",
|
||||
e_before, e_after_strong, e_after_weak);
|
||||
snprintf(p,sizeof p,"%s/connect.json",dir);
|
||||
el_val_t g = engram_save(S(p)); (void)g;
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!strcmp(mode,"perm")){
|
||||
/* Two OLD ISE nodes (created_at far in the past → prunable at 48h). */
|
||||
snprintf(p,sizeof p,"%s/seed.json",dir);
|
||||
FILE* f=fopen(p,"w");
|
||||
fprintf(f,"{\"nodes\":["
|
||||
"{\"id\":\"ise-durable\",\"content\":\"promoted internal state\",\"node_type\":\"InternalStateEvent\","
|
||||
"\"label\":\"ise-durable\",\"salience\":0.5,\"confidence\":1.0,\"created_at\":1000},"
|
||||
"{\"id\":\"ise-ephemeral\",\"content\":\"transient internal state\",\"node_type\":\"InternalStateEvent\","
|
||||
"\"label\":\"ise-ephemeral\",\"salience\":0.5,\"confidence\":1.0,\"created_at\":1000}"
|
||||
"],\"edges\":[]}");
|
||||
fclose(f);
|
||||
if(!engram_load(S(p))){ fprintf(stderr,"load failed\n"); return 2; }
|
||||
long long n_before = (long long)(int64_t)engram_node_count();
|
||||
el_val_t promoted = engram_consolidate_permanence(S("ise-durable"));
|
||||
long long removed = (long long)(int64_t)engram_prune_telemetry((el_val_t)0); /* default 48h */
|
||||
long long n_after = (long long)(int64_t)engram_node_count();
|
||||
printf("PROMOTED %lld\n", (long long)(int64_t)promoted);
|
||||
printf("NODES before=%lld after=%lld removed=%lld\n", n_before, n_after, removed);
|
||||
printf("DURABLE_NODE %s\n", EL_CSTR(engram_get_node_json(S("ise-durable"))));
|
||||
printf("EPHEMERAL_NODE %s\n", EL_CSTR(engram_get_node_json(S("ise-ephemeral"))));
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!strcmp(mode,"offcheck")){
|
||||
build_seed();
|
||||
engram_activate_json(S(QUERY), (el_val_t)3);
|
||||
long long e_before = (long long)(int64_t)engram_edge_count();
|
||||
engram_node_full(S("strong internal state with flag OFF"),
|
||||
S("InternalStateEvent"), S("ise-off"), F(0.9), F(0.8), F(1.0), S("Working"), S("ise"));
|
||||
long long e_after = (long long)(int64_t)engram_edge_count();
|
||||
el_val_t perm = engram_consolidate_permanence(S("ise-off"));
|
||||
printf("OFF edges before=%lld after=%lld perm_ret=%lld\n",
|
||||
e_before, e_after, (long long)(int64_t)perm);
|
||||
return (e_before==e_after && (int64_t)perm==0) ? 0 : 1;
|
||||
}
|
||||
|
||||
fprintf(stderr,"unknown mode %s\n",mode); return 2;
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
/* test_interoception_p2_chrono.c — M-INTEROCEPTION Priority 2.
|
||||
* Chronoception: engram_age_field(delta_ms) + reboot catch-up
|
||||
* (ENGRAM_CHRONOCEPTION, default OFF).
|
||||
*
|
||||
* Uses loaded snapshots with KNOWN working_memory_weight / background_activation
|
||||
* so the field is deterministic without depending on activation. (engram_load
|
||||
* halves WM on boot — the laundering step — so snapshot wm 1.0 -> 0.5 resident.)
|
||||
*
|
||||
* Modes:
|
||||
* once <dir> <dt_ms> — age the field once by dt; save field.json.
|
||||
* split <dir> <dt_ms> <N> — age by dt/N, N times; save field.json.
|
||||
* (once vs split must match: scale-invariance.)
|
||||
* catchup <dir> <gap_ms> — write a last-tick gap_ms in the past, then
|
||||
* engram_age_field_catchup(); print MAGNITUDE.
|
||||
* offcheck <dir> <dt_ms> — flag OFF: age returns 0 and field is untouched.
|
||||
*/
|
||||
#include "el_runtime.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/time.h>
|
||||
|
||||
static el_val_t S(const char* s){ return EL_STR(s); }
|
||||
|
||||
static void write_seed(const char* dir){
|
||||
char p[1024]; snprintf(p,sizeof p,"%s/seed.json",dir);
|
||||
FILE* f=fopen(p,"w");
|
||||
fprintf(f,"{\"nodes\":["
|
||||
"{\"id\":\"f1\",\"content\":\"field node 1\",\"node_type\":\"Concept\",\"label\":\"f1\","
|
||||
"\"salience\":0.9,\"confidence\":1.0,\"working_memory_weight\":1.0,\"background_activation\":0.5},"
|
||||
"{\"id\":\"f2\",\"content\":\"field node 2\",\"node_type\":\"Concept\",\"label\":\"f2\","
|
||||
"\"salience\":0.8,\"confidence\":1.0,\"working_memory_weight\":0.8,\"background_activation\":0.4},"
|
||||
"{\"id\":\"f3\",\"content\":\"field node 3\",\"node_type\":\"Concept\",\"label\":\"f3\","
|
||||
"\"salience\":0.7,\"confidence\":1.0,\"working_memory_weight\":0.6,\"background_activation\":0.3}"
|
||||
"],\"edges\":[]}");
|
||||
fclose(f);
|
||||
}
|
||||
static void load_seed(const char* dir){
|
||||
char p[1024]; snprintf(p,sizeof p,"%s/seed.json",dir);
|
||||
write_seed(dir);
|
||||
if(!engram_load(S(p))){ fprintf(stderr,"load failed\n"); exit(2); }
|
||||
}
|
||||
static void save_field(const char* dir){
|
||||
char p[1024]; snprintf(p,sizeof p,"%s/field.json",dir);
|
||||
engram_save(S(p));
|
||||
}
|
||||
|
||||
int main(int argc,char** argv){
|
||||
if(argc<3){ fprintf(stderr,"usage: %s <once|split|catchup|offcheck> <dir> ...\n",argv[0]); return 2; }
|
||||
const char* mode=argv[1];
|
||||
const char* dir =argv[2];
|
||||
|
||||
if(!strcmp(mode,"once")){
|
||||
double dt=atof(argv[3]);
|
||||
load_seed(dir);
|
||||
el_val_t mag=engram_age_field((el_val_t)(int64_t)dt);
|
||||
printf("MAGNITUDE %.10f\n", el_to_float(mag));
|
||||
save_field(dir);
|
||||
return 0;
|
||||
}
|
||||
if(!strcmp(mode,"split")){
|
||||
double dt=atof(argv[3]); int N=atoi(argv[4]); if(N<1)N=1;
|
||||
load_seed(dir);
|
||||
double sub=dt/(double)N;
|
||||
for(int i=0;i<N;i++) engram_age_field((el_val_t)(int64_t)sub);
|
||||
save_field(dir);
|
||||
printf("SPLIT dt=%.0f N=%d sub=%.4f\n", dt, N, sub);
|
||||
return 0;
|
||||
}
|
||||
if(!strcmp(mode,"catchup")){
|
||||
double gap=atof(argv[3]);
|
||||
load_seed(dir);
|
||||
/* Write a last-tick gap_ms in the past. ENGRAM_DATA_DIR is set == dir by
|
||||
* the runner, so the sidecar the runtime reads is <dir>/chrono_last_tick. */
|
||||
struct timeval tv; gettimeofday(&tv,NULL);
|
||||
long long now_ms=(long long)tv.tv_sec*1000+tv.tv_usec/1000;
|
||||
long long last=now_ms-(long long)gap;
|
||||
char p[1200]; snprintf(p,sizeof p,"%s/chrono_last_tick",dir);
|
||||
FILE* f=fopen(p,"w"); fprintf(f,"%lld\n",last); fclose(f);
|
||||
el_val_t mag=engram_age_field_catchup();
|
||||
printf("CATCHUP_MAGNITUDE %.10f\n", el_to_float(mag));
|
||||
save_field(dir);
|
||||
return 0;
|
||||
}
|
||||
if(!strcmp(mode,"offcheck")){
|
||||
double dt=atof(argv[3]);
|
||||
load_seed(dir);
|
||||
el_val_t mag=engram_age_field((el_val_t)(int64_t)dt);
|
||||
el_val_t magc=engram_age_field_catchup();
|
||||
printf("OFF age_mag=%.10f catchup_mag=%.10f\n", el_to_float(mag), el_to_float(magc));
|
||||
save_field(dir);
|
||||
return 0;
|
||||
}
|
||||
fprintf(stderr,"unknown mode %s\n",mode); return 2;
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
/* test_interoception_p3_drift.c — M-INTEROCEPTION Priority 3 (PARTIAL).
|
||||
* Drift-sensor primitive engram_geo_displacement: GROWTH vs CORRUPTION split.
|
||||
*
|
||||
* Constructs synthetic GeoDescriptors (the struct is public) — a baseline and
|
||||
* two perturbations — and checks the sensor reports LOW core-displacement for a
|
||||
* periphery-only change (growth) and HIGH core-displacement for a core change
|
||||
* (corruption). No store / embeddings needed: this exercises the primitive in
|
||||
* isolation, which is the honest scope given there is no persisted SelfAnchor
|
||||
* yet (see engram_geometry.c). */
|
||||
#include "engram_geometry.h"
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <stdio.h>
|
||||
|
||||
static GeoMember MK(const char* id, double centrality, double dist){
|
||||
GeoMember m; memset(&m,0,sizeof m);
|
||||
m.id=strdup(id); m.centrality=centrality; m.dist_centroid=dist;
|
||||
m.membership=1.0; m.embedded=1; return m;
|
||||
}
|
||||
/* 4 members: 2 core (high centrality), 2 periphery (low). */
|
||||
static GeoDescriptor* mkdesc(float cx,float cy,double radius,
|
||||
double c1,double c2,double p1,double p2){
|
||||
GeoDescriptor* g=calloc(1,sizeof *g);
|
||||
g->dim=4;
|
||||
g->centroid=calloc(4,sizeof(float));
|
||||
g->centroid[0]=cx; g->centroid[1]=cy;
|
||||
g->radius=radius;
|
||||
g->n_members=4;
|
||||
g->members=calloc(4,sizeof(GeoMember));
|
||||
g->members[0]=MK("core1",10.0,c1);
|
||||
g->members[1]=MK("core2", 9.0,c2);
|
||||
g->members[2]=MK("per1", 1.0,p1);
|
||||
g->members[3]=MK("per2", 0.9,p2);
|
||||
return g;
|
||||
}
|
||||
|
||||
int main(void){
|
||||
GeoDisplacement d;
|
||||
/* baseline: core at 0.10, periphery at 0.50, centroid [1,0], radius 1.0 */
|
||||
GeoDescriptor* A = mkdesc(1.0f,0.0f,1.0, 0.10,0.10, 0.50,0.50);
|
||||
|
||||
/* (i) GROWTH: periphery extends 0.50->0.90; core fixed; radius grows. */
|
||||
GeoDescriptor* G = mkdesc(1.0f,0.0f,1.4, 0.10,0.10, 0.90,0.90);
|
||||
engram_geo_displacement(A,G,0.5,&d);
|
||||
printf("GROWTH centroid_sep=%.4f centroid_cos=%.4f radius_delta=%.4f core_disp=%.4f periph_disp=%.4f core_n=%d periph_n=%d\n",
|
||||
d.centroid_sep,d.centroid_cos,d.radius_delta,d.core_disp,d.periph_disp,d.core_matched,d.periph_matched);
|
||||
|
||||
/* (ii) CORRUPTION: core displaces 0.10->0.60; periphery fixed; centroid shifts. */
|
||||
GeoDescriptor* C = mkdesc(0.6f,0.4f,1.0, 0.60,0.60, 0.50,0.50);
|
||||
engram_geo_displacement(A,C,0.5,&d);
|
||||
printf("CORRUPTION centroid_sep=%.4f centroid_cos=%.4f radius_delta=%.4f core_disp=%.4f periph_disp=%.4f core_n=%d periph_n=%d\n",
|
||||
d.centroid_sep,d.centroid_cos,d.radius_delta,d.core_disp,d.periph_disp,d.core_matched,d.periph_matched);
|
||||
|
||||
/* identity: A vs A -> zero drift */
|
||||
engram_geo_displacement(A,A,0.5,&d);
|
||||
printf("IDENTITY centroid_sep=%.4f core_disp=%.4f periph_disp=%.4f\n",
|
||||
d.centroid_sep,d.core_disp,d.periph_disp);
|
||||
|
||||
engram_geo_free(A); engram_geo_free(G); engram_geo_free(C);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
/* test_interoception_p4_afferent.c — M-INTEROCEPTION Priority 4.
|
||||
* Afferent input counters in engram_act_stats_json: additive observability.
|
||||
* Drives KNOWN counts and asserts the emitted counters match and are monotonic.
|
||||
*/
|
||||
#include "el_runtime.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
static el_val_t S(const char* s){ return EL_STR(s); }
|
||||
static el_val_t F(double d){ return el_from_float(d); }
|
||||
|
||||
int main(void){
|
||||
/* 3 plain node creates + 2 ISE creates = 5 node_creates, 2 ise_ingests */
|
||||
el_val_t a=engram_node_full(S("alpha concept about memory and time"),S("Concept"),S("a"),F(0.9),F(0.8),F(1.0),S("Semantic"),S("x"));
|
||||
el_val_t b=engram_node_full(S("beta concept about memory and links"),S("Concept"),S("b"),F(0.9),F(0.8),F(1.0),S("Semantic"),S("x"));
|
||||
engram_node_full(S("gamma distractor"),S("Fact"),S("c"),F(0.4),F(0.4),F(1.0),S("Semantic"),S("y"));
|
||||
engram_node_full(S("heartbeat internal state one"),S("InternalStateEvent"),S("i1"),F(0.5),F(0.5),F(1.0),S("Working"),S("ise"));
|
||||
engram_node_full(S("curiosity internal state two"),S("InternalStateEvent"),S("i2"),F(0.5),F(0.5),F(1.0),S("Working"),S("ise"));
|
||||
/* 2 edge creates */
|
||||
engram_connect(a,b,F(0.8),S("associate"));
|
||||
engram_connect(b,a,F(0.3),S("associate"));
|
||||
|
||||
/* first reading (0 queries so far) */
|
||||
printf("STATS0 %s\n", EL_CSTR(engram_act_stats_json()));
|
||||
|
||||
/* 4 queries -> 4 activations */
|
||||
for(int i=0;i<4;i++) engram_activate_json(S("memory and time and links"), (el_val_t)2);
|
||||
printf("STATS1 %s\n", EL_CSTR(engram_act_stats_json()));
|
||||
|
||||
/* 3 more queries -> monotonic increase */
|
||||
for(int i=0;i<3;i++) engram_activate_json(S("memory and time and links"), (el_val_t)2);
|
||||
printf("STATS2 %s\n", EL_CSTR(engram_act_stats_json()));
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
/* test_interoception_p5_dreams.c — M-INTEROCEPTION Priority 5.
|
||||
* Dream-recall-on-wake: engram_dreams_json(since_ms). Honesty rail — only
|
||||
* curiosity_scan ISEs still resident are returned; pruned (rotated-out) ones are
|
||||
* ABSENT (never confabulated); heartbeat ISEs are excluded.
|
||||
*/
|
||||
#include "el_runtime.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/time.h>
|
||||
|
||||
static el_val_t S(const char* s){ return EL_STR(s); }
|
||||
|
||||
int main(int argc,char** argv){
|
||||
if(argc<2){ fprintf(stderr,"usage: %s <dir>\n",argv[0]); return 2; }
|
||||
const char* dir=argv[1];
|
||||
struct timeval tv; gettimeofday(&tv,NULL);
|
||||
long long now=(long long)tv.tv_sec*1000+tv.tv_usec/1000;
|
||||
long long mid=now-3600000; /* 1h ago */
|
||||
long long ancient=1000; /* pruned by 48h retention */
|
||||
|
||||
char p[1024]; snprintf(p,sizeof p,"%s/seed.json",dir);
|
||||
FILE* f=fopen(p,"w");
|
||||
fprintf(f,"{\"nodes\":["
|
||||
"{\"id\":\"cur_old\",\"content\":\"{\\\"kind\\\":\\\"curiosity_scan\\\",\\\"q\\\":\\\"old wondering\\\"}\","
|
||||
"\"node_type\":\"InternalStateEvent\",\"label\":\"state-event\",\"created_at\":%lld},"
|
||||
"{\"id\":\"cur_mid\",\"content\":\"{\\\"kind\\\":\\\"curiosity_scan\\\",\\\"q\\\":\\\"mid wondering\\\"}\","
|
||||
"\"node_type\":\"InternalStateEvent\",\"label\":\"state-event\",\"created_at\":%lld},"
|
||||
"{\"id\":\"cur_recent\",\"content\":\"{\\\"kind\\\":\\\"curiosity_scan\\\",\\\"q\\\":\\\"recent wondering\\\"}\","
|
||||
"\"node_type\":\"InternalStateEvent\",\"label\":\"state-event\",\"created_at\":%lld},"
|
||||
"{\"id\":\"hb_recent\",\"content\":\"{\\\"kind\\\":\\\"heartbeat\\\",\\\"wm\\\":3}\","
|
||||
"\"node_type\":\"InternalStateEvent\",\"label\":\"state-event\",\"created_at\":%lld}"
|
||||
"],\"edges\":[]}", ancient, mid, now, now);
|
||||
fclose(f);
|
||||
if(!engram_load(S(p))){ fprintf(stderr,"load failed\n"); return 2; }
|
||||
|
||||
/* before prune: all resident curiosity_scan after since=0 */
|
||||
printf("BEFORE %s\n", EL_CSTR(engram_dreams_json((el_val_t)0)));
|
||||
/* prune 48h — cur_old (ancient) rotates out */
|
||||
long long pruned=(long long)(int64_t)engram_prune_telemetry((el_val_t)0);
|
||||
printf("PRUNED %lld\n", pruned);
|
||||
printf("AFTER %s\n", EL_CSTR(engram_dreams_json((el_val_t)0)));
|
||||
/* since filter: only events created after 30 min ago -> cur_recent only */
|
||||
long long since=now-1800000;
|
||||
printf("SINCE %lld %s\n", since, EL_CSTR(engram_dreams_json((el_val_t)(int64_t)since)));
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,208 @@
|
||||
/* test_m7_traversal.c — M7 index-driven activation traversal.
|
||||
*
|
||||
* Milestone 7 replaces the O(E) full adjacency rebuild that spreading activation
|
||||
* paid before every BFS with an incrementally-maintained per-node index, behind
|
||||
* the ENGRAM_STORE flag (flag-off = unchanged behavior). This harness links the
|
||||
* REAL el_runtime.c engram builtins (+ engram_store.c) and drives activation
|
||||
* directly — no EL interpreter, no store boot (the index optimization is a pure
|
||||
* in-RAM concern; the flag is read from the environment).
|
||||
*
|
||||
* Modes (argv[1]):
|
||||
* parity-off <dir> — ENGRAM_STORE unset: build a fixed graph, run a scripted
|
||||
* sequence of activations WITH mid-sequence edge/node
|
||||
* inserts, dump each activation's JSON to <dir>/off_actN.json.
|
||||
* parity-on <dir> — ENGRAM_STORE=1: identical graph + identical sequence,
|
||||
* dump to <dir>/on_actN.json. The runner asserts the off/on
|
||||
* files are BYTE-IDENTICAL (same activated set, weights,
|
||||
* ordering, hops, WM promotion).
|
||||
* perf <off|on> <dir> <nodes> <edges> <iters>
|
||||
* — build a large graph, then loop `iters` times doing
|
||||
* (add 1 edge + activate). Prints wall-time and the M7
|
||||
* instrumentation counters (rebuild calls / rebuild
|
||||
* edge-work / incremental appends).
|
||||
*
|
||||
* Writes ONLY under the caller-provided throwaway dir.
|
||||
*/
|
||||
#include "el_runtime.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <time.h>
|
||||
|
||||
/* M7 instrumentation getters (test-only; defined in el_runtime.c). */
|
||||
extern int64_t engram_adj_rebuild_calls(void);
|
||||
extern int64_t engram_adj_rebuild_edge_work(void);
|
||||
extern int64_t engram_adj_incr_appends(void);
|
||||
extern double engram_adj_maint_seconds(void);
|
||||
extern void engram_adj_test_force_dirty(void);
|
||||
extern int engram_store_enabled(void);
|
||||
|
||||
static el_val_t S(const char* s){ return EL_STR(s); }
|
||||
static el_val_t F(double d){ return el_from_float(d); }
|
||||
|
||||
/* Deterministic LCG so off/on processes build byte-identical graphs. */
|
||||
static uint64_t g_rng = 0x9E3779B97F4A7C15ULL;
|
||||
static void rng_seed(uint64_t s){ g_rng = s ? s : 1; }
|
||||
static uint64_t rng_next(void){ g_rng = g_rng * 6364136223846793005ULL + 1442695040888963407ULL; return g_rng >> 17; }
|
||||
|
||||
static el_val_t* g_handles = NULL; /* node id handles from engram_node_full */
|
||||
static int64_t g_nnodes = 0;
|
||||
|
||||
static void write_file(const char* path, const char* content){
|
||||
FILE* f = fopen(path, "wb");
|
||||
if (!f){ fprintf(stderr, "cannot open %s\n", path); exit(2); }
|
||||
if (content) fwrite(content, 1, strlen(content), f);
|
||||
fclose(f);
|
||||
}
|
||||
|
||||
/* Build `n` nodes whose content carries query-matchable tokens, then `m`
|
||||
* deterministic edges among them. Handles are retained for later connect. */
|
||||
static void build_graph(int64_t n, int64_t m){
|
||||
g_handles = malloc((size_t)n * sizeof(el_val_t));
|
||||
g_nnodes = n;
|
||||
static const char* topics[] = {
|
||||
"storage engine durable log", "spreading activation graph traversal",
|
||||
"hebbian potentiation memory", "buffer pool paging checkpoint",
|
||||
"adjacency index edge lookup", "working memory promotion",
|
||||
"b-tree primary index", "embeddings nearest neighbour" };
|
||||
for (int64_t i = 0; i < n; i++){
|
||||
char content[256];
|
||||
snprintf(content, sizeof content,
|
||||
"node %lld about %s and storage engine activation index",
|
||||
(long long)i, topics[(size_t)(i % 8)]);
|
||||
char label[32]; snprintf(label, sizeof label, "n%lld", (long long)i);
|
||||
g_handles[i] = engram_node_full(S(content), S("Concept"), S(label),
|
||||
F(0.7), F(0.6), F(1.0), S("Semantic"), S("storage,graph,index"));
|
||||
}
|
||||
for (int64_t k = 0; k < m; k++){
|
||||
int64_t a = (int64_t)(rng_next() % (uint64_t)n);
|
||||
int64_t b = (int64_t)(rng_next() % (uint64_t)n);
|
||||
if (a == b) b = (b + 1) % n;
|
||||
engram_connect(g_handles[a], g_handles[b], F(0.6), S("associate"));
|
||||
}
|
||||
}
|
||||
|
||||
static const char* Q1 = "storage engine activation and the durable log";
|
||||
static const char* Q2 = "adjacency index graph traversal";
|
||||
|
||||
/* One scripted activation with an optional forced full-rebuild first. */
|
||||
static el_val_t act(const char* q, int depth, int force_rebuild){
|
||||
if (force_rebuild) engram_adj_test_force_dirty();
|
||||
return engram_activate_json(S(q), (el_val_t)depth);
|
||||
}
|
||||
|
||||
/* Run the scripted parity sequence and dump each activation JSON. `tag` names
|
||||
* the output set. When force_rebuild is set, every activation first forces the
|
||||
* O(E) full-rebuild path (the pre-M7 "scan" behavior); otherwise the M7
|
||||
* incremental index is used. The graph build + query sequence are byte-for-byte
|
||||
* deterministic, so any difference between two runs is attributable solely to
|
||||
* the difference in adjacency maintenance (and/or the ENGRAM_STORE flag). */
|
||||
static int run_parity(const char* dir, const char* tag, int force_rebuild){
|
||||
char p[1024];
|
||||
rng_seed(0xC0FFEE123ULL);
|
||||
build_graph(60, 140);
|
||||
|
||||
el_val_t a1 = act(Q1, 3, force_rebuild);
|
||||
snprintf(p, sizeof p, "%s/%s_act1.json", dir, tag); write_file(p, EL_CSTR(a1));
|
||||
|
||||
/* Mutate the graph BETWEEN activations: this is exactly where the M7 path
|
||||
* appends incrementally while the rebuild path marks dirty + fully rebuilds.
|
||||
* Parity must hold across this divergence in HOW the index is maintained. */
|
||||
engram_connect(g_handles[0], g_handles[7], F(0.8), S("depends-on"));
|
||||
engram_connect(g_handles[7], g_handles[23], F(0.7), S("enables"));
|
||||
engram_connect(g_handles[23], g_handles[41],F(0.5), S("uses"));
|
||||
el_val_t hnew = engram_node_full(S("freshly minted storage index node about activation"),
|
||||
S("Concept"), S("nnew"), F(0.8), F(0.7), F(1.0), S("Semantic"), S("storage,index"));
|
||||
engram_connect(g_handles[0], hnew, F(0.9), S("about"));
|
||||
|
||||
el_val_t a2 = act(Q1, 3, force_rebuild);
|
||||
snprintf(p, sizeof p, "%s/%s_act2.json", dir, tag); write_file(p, EL_CSTR(a2));
|
||||
el_val_t a3 = act(Q2, 2, force_rebuild);
|
||||
snprintf(p, sizeof p, "%s/%s_act3.json", dir, tag); write_file(p, EL_CSTR(a3));
|
||||
el_val_t a4 = act(Q1, 3, force_rebuild);
|
||||
snprintf(p, sizeof p, "%s/%s_act4.json", dir, tag); write_file(p, EL_CSTR(a4));
|
||||
|
||||
printf("[parity-%s] enabled=%d force_rebuild=%d nodes=%lld edges=%lld "
|
||||
"rebuilds=%lld rebuild_edge_work=%lld incr_appends=%lld\n",
|
||||
tag, engram_store_enabled(), force_rebuild,
|
||||
(long long)(int64_t)engram_node_count(), (long long)(int64_t)engram_edge_count(),
|
||||
(long long)engram_adj_rebuild_calls(), (long long)engram_adj_rebuild_edge_work(),
|
||||
(long long)engram_adj_incr_appends());
|
||||
return 0;
|
||||
}
|
||||
|
||||
static double now_sec(void){
|
||||
struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts);
|
||||
return (double)ts.tv_sec + (double)ts.tv_nsec * 1e-9;
|
||||
}
|
||||
|
||||
static int run_perf(const char* dir, const char* tag, int64_t n, int64_t m, int64_t iters){
|
||||
(void)dir;
|
||||
rng_seed(0xBEEF7777ULL);
|
||||
double t_build0 = now_sec();
|
||||
build_graph(n, m);
|
||||
double t_build = now_sec() - t_build0;
|
||||
|
||||
int64_t rb0 = engram_adj_rebuild_calls();
|
||||
int64_t rw0 = engram_adj_rebuild_edge_work();
|
||||
int64_t ap0 = engram_adj_incr_appends();
|
||||
double mt0 = engram_adj_maint_seconds();
|
||||
|
||||
double t0 = now_sec();
|
||||
for (int64_t it = 0; it < iters; it++){
|
||||
/* One structural mutation per query — the curiosity-loop cadence that
|
||||
* makes the OLD path rebuild the whole adjacency before every BFS. */
|
||||
int64_t a = (int64_t)(rng_next() % (uint64_t)n);
|
||||
int64_t b = (int64_t)(rng_next() % (uint64_t)n);
|
||||
if (a == b) b = (b + 1) % n;
|
||||
engram_connect(g_handles[a], g_handles[b], F(0.6), S("associate"));
|
||||
el_val_t r = engram_activate_json(S(Q1), (el_val_t)2);
|
||||
(void)r;
|
||||
}
|
||||
double elapsed = now_sec() - t0;
|
||||
|
||||
double maint = engram_adj_maint_seconds() - mt0;
|
||||
printf("[perf-%s] flag=%d nodes=%lld edges=%lld iters=%lld build=%.3fs "
|
||||
"loop=%.3fs per_query=%.3fms adj_maint=%.4fs adj_maint_per_query=%.4fms | "
|
||||
"rebuilds=%lld rebuild_edge_work=%lld incr_appends=%lld\n",
|
||||
tag, engram_store_enabled(),
|
||||
(long long)(int64_t)engram_node_count(), (long long)(int64_t)engram_edge_count(),
|
||||
(long long)iters, t_build, elapsed, (elapsed / (double)iters) * 1e3,
|
||||
maint, (maint / (double)iters) * 1e3,
|
||||
(long long)(engram_adj_rebuild_calls() - rb0),
|
||||
(long long)(engram_adj_rebuild_edge_work() - rw0),
|
||||
(long long)(engram_adj_incr_appends() - ap0));
|
||||
return 0;
|
||||
}
|
||||
|
||||
int main(int argc, char** argv){
|
||||
if (argc < 3){ fprintf(stderr, "usage: %s <parity-off|parity-on|perf> ...\n", argv[0]); return 2; }
|
||||
const char* mode = argv[1];
|
||||
|
||||
if (!strcmp(mode, "parity-off")){
|
||||
/* flag-off, rebuild path = today's scan behavior (the baseline). */
|
||||
if (engram_store_enabled()){ fprintf(stderr, "parity-off requires ENGRAM_STORE unset\n"); return 2; }
|
||||
return run_parity(argv[2], "off", 0);
|
||||
}
|
||||
if (!strcmp(mode, "parity-on-rebuild")){
|
||||
/* flag-on, but force the O(E) rebuild before each activation. */
|
||||
if (!engram_store_enabled()){ fprintf(stderr, "parity-on-rebuild requires ENGRAM_STORE=1\n"); return 2; }
|
||||
return run_parity(argv[2], "onrb", 1);
|
||||
}
|
||||
if (!strcmp(mode, "parity-on-incr")){
|
||||
/* flag-on, M7 incremental index (the code path under test). */
|
||||
if (!engram_store_enabled()){ fprintf(stderr, "parity-on-incr requires ENGRAM_STORE=1\n"); return 2; }
|
||||
return run_parity(argv[2], "onincr", 0);
|
||||
}
|
||||
if (!strcmp(mode, "perf")){
|
||||
/* perf <off|on> <dir> <nodes> <edges> <iters> */
|
||||
if (argc < 7){ fprintf(stderr, "usage: %s perf <off|on> <dir> <nodes> <edges> <iters>\n", argv[0]); return 2; }
|
||||
const char* tag = argv[2];
|
||||
int64_t n = strtoll(argv[4], NULL, 10);
|
||||
int64_t m = strtoll(argv[5], NULL, 10);
|
||||
int64_t iters = strtoll(argv[6], NULL, 10);
|
||||
return run_perf(argv[3], tag, n, m, iters);
|
||||
}
|
||||
fprintf(stderr, "unknown mode %s\n", mode);
|
||||
return 2;
|
||||
}
|
||||
@@ -0,0 +1,255 @@
|
||||
/* Closed-form unit tests for the REASONING layer (engram_reason.c). All inputs are
|
||||
* hand-built synthetic descriptors whose answers are known in closed form. Every
|
||||
* reasoning MODE is proven, not declared. ASan/UBSan target. */
|
||||
#include "engram_reason.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <math.h>
|
||||
|
||||
static int failures = 0, checks = 0;
|
||||
static void ok(const char* what, int cond) {
|
||||
checks++;
|
||||
if (!cond) { failures++; printf(" FAIL: %s\n", what); }
|
||||
else printf(" ok: %s\n", what);
|
||||
}
|
||||
static void approx(const char* what, double got, double exp, double tol) {
|
||||
ok(what, fabs(got - exp) <= tol);
|
||||
if (fabs(got - exp) > tol) printf(" got=%.9g exp=%.9g\n", got, exp);
|
||||
}
|
||||
|
||||
/* ── descriptor builders (mirror scratchpad/test_geo_ops.c) ─────────────────── */
|
||||
static float* vec(const double* v, int dim) {
|
||||
float* f = malloc((size_t)dim * sizeof(float));
|
||||
for (int i = 0; i < dim; i++) f[i] = (float)v[i];
|
||||
return f;
|
||||
}
|
||||
static GeoDescriptor* mk(int dim, const double* centroid,
|
||||
int n_axes, const double* axis_flat, const double* extents,
|
||||
int n_members, const char** ids, double total_var) {
|
||||
GeoDescriptor* g = calloc(1, sizeof(GeoDescriptor));
|
||||
g->dim = dim;
|
||||
g->centroid = centroid ? vec(centroid, dim) : NULL;
|
||||
g->global_mean = NULL;
|
||||
g->n_axes = n_axes;
|
||||
g->axes = n_axes ? calloc((size_t)n_axes, sizeof(GeoAxis)) : NULL;
|
||||
double tr = 0;
|
||||
for (int k = 0; k < n_axes; k++) {
|
||||
g->axes[k].axis = vec(&axis_flat[(size_t)k * dim], dim);
|
||||
g->axes[k].extent = extents[k];
|
||||
tr += extents[k] * extents[k];
|
||||
}
|
||||
g->total_variance = (total_var >= 0) ? total_var : tr;
|
||||
g->radius = sqrt(g->total_variance > 0 ? g->total_variance : 0);
|
||||
g->n_members = n_members; g->n_embedded = n_members;
|
||||
g->members = n_members ? calloc((size_t)n_members, sizeof(GeoMember)) : NULL;
|
||||
for (int i = 0; i < n_members; i++) {
|
||||
g->members[i].id = strdup(ids[i]);
|
||||
g->members[i].membership = 1.0;
|
||||
g->members[i].centrality = (double)(n_members - i);
|
||||
g->members[i].embedded = 1;
|
||||
}
|
||||
g->hub_id = n_members ? strdup(ids[0]) : strdup("");
|
||||
g->k_core = 1; g->co_registration = 0.0; g->n_edges = 0; g->edges = NULL;
|
||||
return g;
|
||||
}
|
||||
|
||||
int main(void) {
|
||||
printf("== REASONING layer unit tests ==\n");
|
||||
|
||||
/* ══════════════════ ANALOGY — recover an affine A→B, apply to C ══════════ */
|
||||
/* A→B is a +90° rotation in the e0-e1 plane ((x,y)→(-y,x)) plus a +5 shift in e2.
|
||||
* A frame = (e0,e1); B frame = rotated (e1,-e0); cB = R·cA + t. Predict D from C. */
|
||||
{
|
||||
int dim = 4;
|
||||
double cA[4] = {1,0,0,0};
|
||||
double cB[4] = {0,1,5,0}; /* R·(1,0,0,0)=(0,1,0,0) + (0,0,5,0) */
|
||||
double cC[4] = {2,0,0,0};
|
||||
double axA[8] = {1,0,0,0, 0,1,0,0}; double exA[2] = {1,1};
|
||||
double axB[8] = {0,1,0,0, -1,0,0,0}; double exB[2] = {1,1}; /* R·e0, R·e1 */
|
||||
double axC[8] = {1,0,0,0, 0,1,0,0}; double exC[2] = {1,1};
|
||||
const char* idA[1] = {"A"}, *idB[1] = {"B"}, *idC[1] = {"C"};
|
||||
GeoDescriptor* A = mk(dim, cA, 2, axA, exA, 1, idA, -1);
|
||||
GeoDescriptor* B = mk(dim, cB, 2, axB, exB, 1, idB, -1);
|
||||
GeoDescriptor* C = mk(dim, cC, 2, axC, exC, 1, idC, -1);
|
||||
/* candidates: the true D + two distractors. true D = R·cC + t = (0,2,5,0). */
|
||||
double d_true[4] = {0,2,5,0}, d_far1[4] = {9,9,9,9}, d_far2[4] = {0,0,0,0};
|
||||
const char* idD[1] = {"Dt"}, *idF1[1] = {"F1"}, *idF2[1] = {"F2"};
|
||||
GeoDescriptor* Dt = mk(dim, d_true, 0, NULL, NULL, 1, idD, 0.0);
|
||||
GeoDescriptor* F1 = mk(dim, d_far1, 0, NULL, NULL, 1, idF1, 0.0);
|
||||
GeoDescriptor* F2 = mk(dim, d_far2, 0, NULL, NULL, 1, idF2, 0.0);
|
||||
const GeoDescriptor* cand[3] = {F1, Dt, F2}; /* true one at index 1 */
|
||||
GeoAnalogyResult res;
|
||||
int rc = engram_reason_analogy(A, B, C, cand, 3, &res);
|
||||
ok("analogy returns 0", rc == 0);
|
||||
printf("[analogy] residual=%.6f mapped=(%.4f,%.4f,%.4f,%.4f) best=%d bd=%.5f\n",
|
||||
res.analogy_residual, res.mapped_point[0], res.mapped_point[1],
|
||||
res.mapped_point[2], res.mapped_point[3], res.best, res.best_distance);
|
||||
approx("procrustes residual ~0", res.analogy_residual, 0.0, 1e-4);
|
||||
approx("mapped.x=0", res.mapped_point[0], 0.0, 1e-4);
|
||||
approx("mapped.y=2", res.mapped_point[1], 2.0, 1e-4);
|
||||
approx("mapped.z(e2)=5", res.mapped_point[2], 5.0, 1e-4);
|
||||
ok("nearest candidate = true D (idx 1)", res.best == 1);
|
||||
approx("best distance ~0", res.best_distance, 0.0, 1e-3);
|
||||
engram_reason_analogy_free(&res);
|
||||
engram_geo_free(A); engram_geo_free(B); engram_geo_free(C);
|
||||
engram_geo_free(Dt); engram_geo_free(F1); engram_geo_free(F2);
|
||||
}
|
||||
|
||||
/* ══════════════════ INDUCTION — recover a shared subspace + membership ═══ */
|
||||
/* 3 examples all spread over span(e0,e1) (ext 1 & 0.8), each with a small
|
||||
* idiosyncratic axis (e2 or e3, ext 0.2). Centroids all 0. The induced rule's
|
||||
* top-2 axes must lie in span(e0,e1); a held-out in-plane point fits, an
|
||||
* off-subspace point does not. */
|
||||
{
|
||||
int dim = 4;
|
||||
double c0[4] = {0,0,0,0};
|
||||
double axsh[8] = {1,0,0,0, 0,1,0,0}; double exsh[2] = {1.0, 0.8};
|
||||
double ax1[12] = {1,0,0,0, 0,1,0,0, 0,0,1,0}; double ex1[3] = {1.0,0.8,0.2}; /* +e2 */
|
||||
double ax2[12] = {1,0,0,0, 0,1,0,0, 0,0,0,1}; double ex2[3] = {1.0,0.8,0.2}; /* +e3 */
|
||||
const char* i1[2] = {"e1a","e1b"}, *i2[2] = {"e2a","e2b"}, *i3[2] = {"e3a","e3b"};
|
||||
GeoDescriptor* E1 = mk(dim, c0, 3, ax1, ex1, 2, i1, -1);
|
||||
GeoDescriptor* E2 = mk(dim, c0, 3, ax2, ex2, 2, i2, -1);
|
||||
GeoDescriptor* E3 = mk(dim, c0, 2, axsh, exsh, 2, i3, -1);
|
||||
const GeoDescriptor* ex[3] = {E1, E2, E3};
|
||||
GeoInduction ind;
|
||||
int rc = engram_reason_induce(ex, 3, 8, 1.0, &ind);
|
||||
ok("induce returns 0", rc == 0);
|
||||
printf("[induction] rule n_axes=%d ext0=%.4f ext1=%.4f\n",
|
||||
ind.rule->n_axes, ind.rule->n_axes > 0 ? ind.rule->axes[0].extent : 0,
|
||||
ind.rule->n_axes > 1 ? ind.rule->axes[1].extent : 0);
|
||||
/* top-2 axes lie in span(e0,e1): their e2,e3 components ~0. */
|
||||
int inplane = 1;
|
||||
for (int k = 0; k < 2 && k < ind.rule->n_axes; k++) {
|
||||
const float* a = ind.rule->axes[k].axis;
|
||||
printf(" axis%d=(%.3f,%.3f,%.3f,%.3f) ext=%.4f\n", k, a[0],a[1],a[2],a[3], ind.rule->axes[k].extent);
|
||||
if (fabs(a[2]) > 0.06 || fabs(a[3]) > 0.06) inplane = 0;
|
||||
}
|
||||
ok("induced top-2 axes lie in shared span(e0,e1)", inplane);
|
||||
approx("dominant extent ~1.0", ind.rule->axes[0].extent, 1.0, 0.06);
|
||||
approx("second extent ~0.8", ind.rule->axes[1].extent, 0.8, 0.06);
|
||||
/* membership: in-plane near-centroid positive fits; off-subspace negative doesn't. */
|
||||
float xpos[4] = {0.3f, -0.2f, 0, 0};
|
||||
float xneg[4] = {0, 0, 3.0f, 0}; /* large along e2 — outside the rule */
|
||||
float xfar[4] = {5.0f, 0, 0, 0}; /* in-plane but far — Mahalanobis blows up */
|
||||
double mp = engram_reason_membership(&ind, xpos);
|
||||
double mn = engram_reason_membership(&ind, xneg);
|
||||
double mf = engram_reason_membership(&ind, xfar);
|
||||
printf("[induction] membership pos=%.4f neg=%.4f far=%.4f\n", mp, mn, mf);
|
||||
ok("held-out positive fits (>0.5)", mp > 0.5);
|
||||
ok("off-subspace negative rejected (<0.3)", mn < 0.3);
|
||||
ok("in-plane-but-far rejected (<0.3)", mf < 0.3);
|
||||
ok("positive fits far better than negative", mp > mn + 0.4);
|
||||
engram_reason_induction_free(&ind);
|
||||
engram_geo_free(E1); engram_geo_free(E2); engram_geo_free(E3);
|
||||
}
|
||||
|
||||
/* ══════════════════ ABDUCTION — pick the best-explaining structure ═══════ */
|
||||
/* obs planted near H1's centroid among 3 candidate structures. */
|
||||
{
|
||||
int dim = 4;
|
||||
double h0[4] = {0,0,0,0}, h1[4] = {5,0,0,0}, h2[4] = {0,5,0,0};
|
||||
double ax[8] = {1,0,0,0, 0,1,0,0}; double ex[2] = {1,1};
|
||||
const char* n0[1] = {"H0"}, *n1[1] = {"H1"}, *n2[1] = {"H2"};
|
||||
GeoDescriptor* H0 = mk(dim, h0, 2, ax, ex, 1, n0, -1);
|
||||
GeoDescriptor* H1 = mk(dim, h1, 2, ax, ex, 1, n1, -1);
|
||||
GeoDescriptor* H2 = mk(dim, h2, 2, ax, ex, 1, n2, -1);
|
||||
const GeoDescriptor* H[3] = {H0, H1, H2};
|
||||
float obs[4] = {5.2f, 0.1f, 0, 0}; /* sits inside H1 */
|
||||
GeoAbduction ab;
|
||||
int rc = engram_reason_abduce(obs, dim, H, 3, 1.0, &ab);
|
||||
ok("abduce returns 0", rc == 0);
|
||||
printf("[abduction] best=%d best_score=%.4f rank=[%d,%d,%d] d=[%.3f,%.3f,%.3f]\n",
|
||||
ab.best, ab.best_score, ab.rank[0], ab.rank[1], ab.rank[2],
|
||||
ab.distances[0], ab.distances[1], ab.distances[2]);
|
||||
ok("best explanation = H1", ab.best == 1);
|
||||
ok("rank[0] = H1", ab.rank[0] == 1);
|
||||
ok("H1 has smallest distance", ab.distances[1] < ab.distances[0] && ab.distances[1] < ab.distances[2]);
|
||||
engram_reason_abduction_free(&ab);
|
||||
engram_geo_free(H0); engram_geo_free(H1); engram_geo_free(H2);
|
||||
}
|
||||
|
||||
/* ══════════════════ CAUSAL — direction + confounder flag ═════════════════ */
|
||||
/* Chain A→B→C along e0 (temporal 1<2<3). Confounder Z (e1) injects into A and
|
||||
* drives D (t=4). A–D correlate only via Z ⇒ must be flagged CONFOUNDED. */
|
||||
{
|
||||
int dim = 4;
|
||||
double cA[4] = {1,1,0,0}; /* e0 (chain) + e1 (confounder leak) */
|
||||
double cB[4] = {1,0,0,0}; /* e0 */
|
||||
double cC[4] = {2,0,0,0}; /* e0 */
|
||||
double cD[4] = {0,1,0,0}; /* e1 only — driven by Z */
|
||||
double cZ[4] = {0,1,0,0}; /* confounder centroid */
|
||||
double axZ[4] = {0,1,0,0}; double exZ[1] = {1}; /* Z's subspace = e1 */
|
||||
const char* idA[1]={"A"},*idB[1]={"B"},*idC[1]={"C"},*idD[1]={"D"},*idZ[1]={"Z"};
|
||||
GeoDescriptor* A = mk(dim, cA, 0, NULL, NULL, 1, idA, 0.0);
|
||||
GeoDescriptor* B = mk(dim, cB, 0, NULL, NULL, 1, idB, 0.0);
|
||||
GeoDescriptor* C = mk(dim, cC, 0, NULL, NULL, 1, idC, 0.0);
|
||||
GeoDescriptor* D = mk(dim, cD, 0, NULL, NULL, 1, idD, 0.0);
|
||||
GeoDescriptor* Z = mk(dim, cZ, 1, axZ, exZ, 1, idZ, -1);
|
||||
const GeoDescriptor* conf[1] = {Z};
|
||||
|
||||
GeoCausal ab, bc, ad, bd;
|
||||
engram_reason_causal(A, B, conf, 1, /*t*/1, 2, 0.5, &ab);
|
||||
engram_reason_causal(B, C, conf, 1, 2, 3, 0.5, &bc);
|
||||
engram_reason_causal(A, D, conf, 1, 1, 4, 0.5, &ad);
|
||||
engram_reason_causal(B, D, conf, 1, 2, 4, 0.5, &bd);
|
||||
printf("[causal] A->B: raw=%.3f ctrl=%.3f dir=%d verdict=%d strength=%.3f\n",
|
||||
ab.assoc_raw, ab.assoc_controlled, ab.temporal_dir, ab.verdict, ab.strength);
|
||||
printf("[causal] B->C: raw=%.3f ctrl=%.3f dir=%d verdict=%d\n", bc.assoc_raw, bc.assoc_controlled, bc.temporal_dir, bc.verdict);
|
||||
printf("[causal] A--D: raw=%.3f ctrl=%.3f dir=%d verdict=%d confounded=%d\n",
|
||||
ad.assoc_raw, ad.assoc_controlled, ad.temporal_dir, ad.verdict, ad.confounded);
|
||||
printf("[causal] B--D: raw=%.3f verdict=%d\n", bd.assoc_raw, bd.verdict);
|
||||
ok("A->B DIRECTED", ab.verdict == GEO_CAUSAL_DIRECTED);
|
||||
ok("A->B direction A precedes B", ab.temporal_dir == 1);
|
||||
ok("A->B association survives control (ctrl high)", ab.assoc_controlled > 0.6);
|
||||
ok("B->C DIRECTED", bc.verdict == GEO_CAUSAL_DIRECTED);
|
||||
ok("A--D CONFOUNDED (flagged)", ad.verdict == GEO_CAUSAL_CONFOUNDED && ad.confounded == 1);
|
||||
ok("A--D raw correlated but control kills it", ad.assoc_raw > 0.6 && ad.assoc_controlled < 0.2);
|
||||
ok("B--D NONE (no association at all)", bd.verdict == GEO_CAUSAL_NONE);
|
||||
engram_geo_free(A); engram_geo_free(B); engram_geo_free(C); engram_geo_free(D); engram_geo_free(Z);
|
||||
}
|
||||
|
||||
/* ══════════════════ PLANNING — geodesic path along a curved manifold ═════ */
|
||||
/* 6 neighborhoods on a semicircle (radius 10). Consecutive chord ~6.18,
|
||||
* skip-one ~11.76, endpoints ~20. neighbor_radius=7 admits only consecutive
|
||||
* hops ⇒ the plan must traverse the whole arc 0→1→2→3→4→5. */
|
||||
{
|
||||
int dim = 4; int N = 6; double R = 10.0;
|
||||
GeoDescriptor* nodes[6];
|
||||
char nm[6][8];
|
||||
for (int k = 0; k < N; k++) {
|
||||
double th = M_PI * (double)k / (double)(N - 1);
|
||||
double c[4] = { R * cos(th), R * sin(th), 0, 0 };
|
||||
snprintf(nm[k], sizeof nm[k], "n%d", k);
|
||||
const char* id[1] = { nm[k] };
|
||||
nodes[k] = mk(dim, c, 0, NULL, NULL, 1, id, 0.0);
|
||||
}
|
||||
const GeoDescriptor* cn[6];
|
||||
for (int k = 0; k < N; k++) cn[k] = nodes[k];
|
||||
GeoPlan plan;
|
||||
int rc = engram_reason_plan(cn, N, 0, 5, 7.0, 0, &plan);
|
||||
ok("plan returns 0", rc == 0);
|
||||
printf("[planning] reached=%d len=%d cost=%.4f path=[", plan.reached, plan.path_len, plan.total_cost);
|
||||
for (int i = 0; i < plan.path_len; i++) printf("%s%d", i ? "," : "", plan.path[i]);
|
||||
printf("]\n");
|
||||
ok("goal reached", plan.reached == 1);
|
||||
ok("path length = 6 (full arc)", plan.path_len == 6);
|
||||
int monotone = (plan.path_len == 6);
|
||||
for (int i = 0; i < plan.path_len; i++) if (plan.path[i] != i) monotone = 0;
|
||||
ok("path = 0,1,2,3,4,5 (the geodesic)", monotone);
|
||||
/* arc cost ~ 5 * 6.18 = 30.9, and strictly longer than the 20-unit chord. */
|
||||
approx("arc cost ~30.9", plan.total_cost, 30.9, 0.6);
|
||||
ok("arc longer than straight chord (20)", plan.total_cost > 20.0);
|
||||
engram_reason_plan_free(&plan);
|
||||
|
||||
/* negative control: radius too small to connect anything ⇒ unreachable. */
|
||||
GeoPlan p2;
|
||||
engram_reason_plan(cn, N, 0, 5, 1.0, 0, &p2);
|
||||
ok("unreachable when radius < min edge", p2.reached == 0);
|
||||
engram_reason_plan_free(&p2);
|
||||
for (int k = 0; k < N; k++) engram_geo_free(nodes[k]);
|
||||
}
|
||||
|
||||
printf("\n== %d checks, %d failures ==\n", checks, failures);
|
||||
return failures ? 1 : 0;
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
/* test_scan_collision.c — regression gate for the "saved but not findable" bug.
|
||||
*
|
||||
* ROOT CAUSE UNDER TEST: store_scan_nodes / store_scan_edges (the boot-load
|
||||
* path that populates the resident in-RAM graph — engram_store_boot ->
|
||||
* eg_load_node_cb) deduplicated emitted records by their 64-bit id_hash
|
||||
* (FNV-1a-64), NOT by the full id string. Two DISTINCT ids that collide under
|
||||
* id_hash therefore emitted only the FIRST: the second node/edge was durably
|
||||
* present in neuron.egm (store_get_node finds it), physically on a live page,
|
||||
* yet was SILENTLY DROPPED from the resident load. After any store reopen it
|
||||
* was unretrievable by id, absent from lexical search, and missing from the
|
||||
* recent list — exactly the reported symptom.
|
||||
*
|
||||
* The two ids below are real FNV-1a-64 collisions (found offline via Brent's
|
||||
* cycle detection over fnv1a(hex16(x))); both hash to 0x15141fdadfa24abe.
|
||||
*
|
||||
* Pure C. Writes ONLY under a throwaway /tmp dir. Never touches ~/.neuron.
|
||||
*/
|
||||
#include "../../lang/runtime/engram_store.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <stdint.h>
|
||||
#include <unistd.h>
|
||||
#include <sys/stat.h>
|
||||
|
||||
static int g_pass = 0, g_fail = 0;
|
||||
static void ok(const char* name, int cond){
|
||||
printf(" [%s] %s\n", cond ? "PASS" : "FAIL", name);
|
||||
if (cond) g_pass++; else g_fail++;
|
||||
}
|
||||
|
||||
/* Confirmed FNV-1a-64 collision (distinct strings, equal id_hash). */
|
||||
#define ID_A "d2c61ec7d015dc98"
|
||||
#define ID_B "bf85e965a2aefbdd"
|
||||
|
||||
static uint64_t fnv1a(const char* s){
|
||||
uint64_t h = 1469598103934665603ULL;
|
||||
for (; *s; ++s){ h ^= (uint8_t)*s; h *= 1099511628211ULL; }
|
||||
return h;
|
||||
}
|
||||
|
||||
static char g_dir[512];
|
||||
static void mk_dir(void){
|
||||
snprintf(g_dir, sizeof g_dir, "/tmp/engram-scancol-%d", (int)getpid());
|
||||
mkdir(g_dir, 0700);
|
||||
}
|
||||
|
||||
/* ── scan collectors: record which ids the boot-load scan actually emits ── */
|
||||
typedef struct { const char* want[8]; int seen[8]; int nwant; int total; } Collect;
|
||||
static void node_cb(const StoreNode* n, void* ctx){
|
||||
Collect* c = ctx; c->total++;
|
||||
for (int i=0;i<c->nwant;i++) if (n->id && strcmp(n->id, c->want[i])==0) c->seen[i]=1;
|
||||
}
|
||||
static void edge_cb(const StoreEdge* e, void* ctx){
|
||||
Collect* c = ctx; c->total++;
|
||||
for (int i=0;i<c->nwant;i++) if (e->id && strcmp(e->id, c->want[i])==0) c->seen[i]=1;
|
||||
}
|
||||
|
||||
static void mk_node(StoreNode* n, const char* id, const char* content){
|
||||
memset(n, 0, sizeof *n);
|
||||
n->id = strdup(id);
|
||||
n->content = strdup(content);
|
||||
n->node_type = strdup("Memory");
|
||||
n->label = strdup(content);
|
||||
n->tier = strdup("Working");
|
||||
n->tags = strdup("");
|
||||
n->metadata = strdup("{}");
|
||||
n->salience = 0.5; n->importance = 0.5; n->confidence = 1.0;
|
||||
n->created_at = 1700000000000LL; n->updated_at = 1700000000000LL;
|
||||
n->last_activated = 1700000000000LL;
|
||||
}
|
||||
static void mk_edge(StoreEdge* e, const char* id, const char* from, const char* to){
|
||||
memset(e, 0, sizeof *e);
|
||||
e->id = strdup(id); e->from_id = strdup(from); e->to_id = strdup(to);
|
||||
e->relation = strdup("assoc"); e->metadata = strdup("{}");
|
||||
e->weight = 1.0; e->confidence = 1.0;
|
||||
e->created_at = 1700000000000LL; e->updated_at = 1700000000000LL;
|
||||
}
|
||||
|
||||
int main(void){
|
||||
mk_dir();
|
||||
printf("== scan-collision regression (saved-but-not-findable) ==\n");
|
||||
printf(" id_hash(%s) = %016llx\n", ID_A, (unsigned long long)fnv1a(ID_A));
|
||||
printf(" id_hash(%s) = %016llx\n", ID_B, (unsigned long long)fnv1a(ID_B));
|
||||
ok("precondition: the two ids genuinely collide under id_hash",
|
||||
fnv1a(ID_A) == fnv1a(ID_B) && strcmp(ID_A, ID_B) != 0);
|
||||
|
||||
/* ---- Control: a single node survives a full store round-trip. ---- */
|
||||
{
|
||||
EngramPagedStore* s = engram_open(g_dir);
|
||||
StoreNode n; mk_node(&n, ID_A, "alpha distinctiveword");
|
||||
store_put_node(s, &n);
|
||||
engram_close(s); /* checkpoint + close */
|
||||
|
||||
EngramPagedStore* r = engram_open(g_dir);
|
||||
StoreNode got;
|
||||
ok("control: single node found by id after reopen", store_get_node(r, ID_A, &got)==1);
|
||||
if (0) {} else store_node_free(&got);
|
||||
Collect c = {{ID_A}, {0}, 1, 0};
|
||||
store_scan_nodes(r, node_cb, &c);
|
||||
ok("control: single node emitted by boot-load scan", c.seen[0]==1);
|
||||
engram_close(r);
|
||||
store_node_free(&n);
|
||||
}
|
||||
|
||||
/* ---- Bug: two id-hash-colliding NODES, both durable, both must load. ---- */
|
||||
{
|
||||
char dir2[600]; snprintf(dir2, sizeof dir2, "%s/nodes", g_dir); mkdir(dir2, 0700);
|
||||
EngramPagedStore* s = engram_open(dir2);
|
||||
StoreNode a, b;
|
||||
mk_node(&a, ID_A, "alpha distinctiveword-A");
|
||||
mk_node(&b, ID_B, "beta distinctiveword-B");
|
||||
store_put_node(s, &a);
|
||||
store_put_node(s, &b);
|
||||
engram_close(s);
|
||||
store_node_free(&a); store_node_free(&b);
|
||||
|
||||
EngramPagedStore* r = engram_open(dir2);
|
||||
/* Both are individually durable (store_get_node disambiguates by strcmp). */
|
||||
StoreNode ga, gb;
|
||||
int hit_a = store_get_node(r, ID_A, &ga); if (hit_a==1) store_node_free(&ga);
|
||||
int hit_b = store_get_node(r, ID_B, &gb); if (hit_b==1) store_node_free(&gb);
|
||||
ok("both colliding nodes are durably present (store_get_node)", hit_a==1 && hit_b==1);
|
||||
|
||||
/* THE REGRESSION: the boot-load scan must emit BOTH, not silently drop one. */
|
||||
Collect c = {{ID_A, ID_B}, {0,0}, 2, 0};
|
||||
store_scan_nodes(r, node_cb, &c);
|
||||
printf(" scan emitted A=%d B=%d (total=%d)\n", c.seen[0], c.seen[1], c.total);
|
||||
ok("boot-load scan emits node A (would be resident)", c.seen[0]==1);
|
||||
ok("boot-load scan emits node B (the dropped/unretrievable one)", c.seen[1]==1);
|
||||
engram_close(r);
|
||||
}
|
||||
|
||||
/* ---- Bug: two id-hash-colliding EDGES, both must load. ---- */
|
||||
{
|
||||
char dir3[600]; snprintf(dir3, sizeof dir3, "%s/edges", g_dir); mkdir(dir3, 0700);
|
||||
EngramPagedStore* s = engram_open(dir3);
|
||||
StoreNode na, nb; mk_node(&na, "src", "s"); mk_node(&nb, "dst", "d");
|
||||
store_put_node(s, &na); store_put_node(s, &nb);
|
||||
StoreEdge ea, eb;
|
||||
mk_edge(&ea, ID_A, "src", "dst");
|
||||
mk_edge(&eb, ID_B, "src", "dst");
|
||||
store_put_edge(s, &ea);
|
||||
store_put_edge(s, &eb);
|
||||
engram_close(s);
|
||||
store_node_free(&na); store_node_free(&nb);
|
||||
store_edge_free(&ea); store_edge_free(&eb);
|
||||
|
||||
EngramPagedStore* r = engram_open(dir3);
|
||||
Collect c = {{ID_A, ID_B}, {0,0}, 2, 0};
|
||||
store_scan_edges(r, edge_cb, &c);
|
||||
printf(" scan emitted edgeA=%d edgeB=%d\n", c.seen[0], c.seen[1]);
|
||||
ok("boot-load scan emits edge A", c.seen[0]==1);
|
||||
ok("boot-load scan emits edge B (the dropped one)", c.seen[1]==1);
|
||||
engram_close(r);
|
||||
}
|
||||
|
||||
printf("\n %d passed, %d failed\n", g_pass, g_fail);
|
||||
/* cleanup */
|
||||
char cmd[600]; snprintf(cmd, sizeof cmd, "rm -rf %s", g_dir); if (system(cmd)){}
|
||||
return g_fail ? 1 : 0;
|
||||
}
|
||||
@@ -0,0 +1,244 @@
|
||||
/* Closed-form unit tests for the VERIFIER layer (engram_verify.c). Every case is a
|
||||
* hand-built synthetic descriptor / claim point whose verdict is known in closed
|
||||
* form — the checks are PROVEN, not declared. ASan/UBSan target.
|
||||
*
|
||||
* The headline case is CONSISTENCY's polarity check: the reassurance→accusation
|
||||
* inversion ("you never fought" → "you argued") that no grammar check catches. */
|
||||
#include "engram_verify.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <math.h>
|
||||
|
||||
static int failures = 0, checks = 0;
|
||||
static void ok(const char* what, int cond) {
|
||||
checks++;
|
||||
if (!cond) { failures++; printf(" FAIL: %s\n", what); }
|
||||
else printf(" ok: %s\n", what);
|
||||
}
|
||||
static void approx(const char* what, double got, double exp, double tol) {
|
||||
ok(what, fabs(got - exp) <= tol);
|
||||
if (fabs(got - exp) > tol) printf(" got=%.9g exp=%.9g\n", got, exp);
|
||||
}
|
||||
|
||||
/* ── descriptor builder (mirrors test_reason.c) ─────────────────────────────── */
|
||||
static float* vec(const double* v, int dim) {
|
||||
float* f = malloc((size_t)dim * sizeof(float));
|
||||
for (int i = 0; i < dim; i++) f[i] = (float)v[i];
|
||||
return f;
|
||||
}
|
||||
static GeoDescriptor* mk(int dim, const double* centroid,
|
||||
int n_axes, const double* axis_flat, const double* extents,
|
||||
int n_members, const char** ids, double total_var) {
|
||||
GeoDescriptor* g = calloc(1, sizeof(GeoDescriptor));
|
||||
g->dim = dim;
|
||||
g->centroid = centroid ? vec(centroid, dim) : NULL;
|
||||
g->global_mean = NULL;
|
||||
g->n_axes = n_axes;
|
||||
g->axes = n_axes ? calloc((size_t)n_axes, sizeof(GeoAxis)) : NULL;
|
||||
double tr = 0;
|
||||
for (int k = 0; k < n_axes; k++) {
|
||||
g->axes[k].axis = vec(&axis_flat[(size_t)k * dim], dim);
|
||||
g->axes[k].extent = extents[k];
|
||||
tr += extents[k] * extents[k];
|
||||
}
|
||||
g->total_variance = (total_var >= 0) ? total_var : tr;
|
||||
g->radius = sqrt(g->total_variance > 0 ? g->total_variance : 0);
|
||||
g->n_members = n_members; g->n_embedded = n_members;
|
||||
g->members = n_members ? calloc((size_t)n_members, sizeof(GeoMember)) : NULL;
|
||||
for (int i = 0; i < n_members; i++) {
|
||||
g->members[i].id = strdup(ids[i]);
|
||||
g->members[i].membership = 1.0;
|
||||
g->members[i].centrality = (double)(n_members - i);
|
||||
g->members[i].embedded = 1;
|
||||
}
|
||||
g->hub_id = n_members ? strdup(ids[0]) : strdup("");
|
||||
g->k_core = 1; g->co_registration = 0.0; g->n_edges = 0; g->edges = NULL;
|
||||
return g;
|
||||
}
|
||||
|
||||
int main(void) {
|
||||
printf("== VERIFIER layer unit tests ==\n");
|
||||
|
||||
/* ══════════════════ GROUNDING — supported vs floating (hallucination) ════ */
|
||||
/* Two real neighborhoods: E0 at origin, E1 far along e0. A claim planted inside
|
||||
* E0 is grounded; a claim floating far off-manifold (along an unmodeled axis) is
|
||||
* flagged UNGROUNDED; a claim near E1 grounds to E1, not E0. */
|
||||
{
|
||||
int dim = 4;
|
||||
double c0[4] = {0,0,0,0}, c1[4] = {10,0,0,0};
|
||||
double ax[8] = {1,0,0,0, 0,1,0,0}; double ex[2] = {1,1};
|
||||
const char* i0[1] = {"E0"}, *i1[1] = {"E1"};
|
||||
GeoDescriptor* E0 = mk(dim, c0, 2, ax, ex, 1, i0, -1);
|
||||
GeoDescriptor* E1 = mk(dim, c1, 2, ax, ex, 1, i1, -1);
|
||||
const GeoDescriptor* ev[2] = {E0, E1};
|
||||
|
||||
/* (1) grounded claim — sits inside E0. */
|
||||
float in[4] = {0.3f, -0.2f, 0, 0};
|
||||
GeoGrounding g1;
|
||||
int rc = engram_verify_grounding(in, dim, ev, 2, 1.0, 0.5, &g1);
|
||||
ok("grounding returns 0", rc == 0);
|
||||
printf("[grounding] IN score=%.4f grounded=%d best=%d dist=%.3f ortho=%.3f nearL2=%.3f\n",
|
||||
g1.grounding, g1.grounded, g1.best, g1.best_distance, g1.best_ortho, g1.nearest_centroid_l2);
|
||||
ok("planted-inside claim is GROUNDED", g1.grounded == 1);
|
||||
ok("grounds to the nearest structure E0", g1.best == 0);
|
||||
ok("grounded score high (>0.7)", g1.grounding > 0.7);
|
||||
approx("off-model residual ~0 for in-distribution claim", g1.best_ortho, 0.0, 1e-4);
|
||||
engram_verify_grounding_free(&g1);
|
||||
|
||||
/* (2) hallucinated claim — floats far along the unmodeled e2 axis. */
|
||||
float out[4] = {0, 0, 50.0f, 0};
|
||||
GeoGrounding g2;
|
||||
engram_verify_grounding(out, dim, ev, 2, 1.0, 0.5, &g2);
|
||||
printf("[grounding] OUT score=%.6f grounded=%d best=%d dist=%.3f ortho=%.3f nearL2=%.3f\n",
|
||||
g2.grounding, g2.grounded, g2.best, g2.best_distance, g2.best_ortho, g2.nearest_centroid_l2);
|
||||
ok("floating claim is FLAGGED (ungrounded)", g2.grounded == 0);
|
||||
ok("floating claim scores near zero (<0.01)", g2.grounding < 0.01);
|
||||
ok("off-model residual is large (the hallucination signal)", g2.best_ortho > 40.0);
|
||||
ok("nearest real structure is far (L2>40)", g2.nearest_centroid_l2 > 40.0);
|
||||
engram_verify_grounding_free(&g2);
|
||||
|
||||
/* (3) selection — a claim near E1 grounds to E1. */
|
||||
float nearE1[4] = {9.8f, 0.1f, 0, 0};
|
||||
GeoGrounding g3;
|
||||
engram_verify_grounding(nearE1, dim, ev, 2, 1.0, 0.5, &g3);
|
||||
printf("[grounding] E1 score=%.4f grounded=%d best=%d\n", g3.grounding, g3.grounded, g3.best);
|
||||
ok("claim near E1 grounds to E1 (best=1)", g3.best == 1 && g3.grounded == 1);
|
||||
engram_verify_grounding_free(&g3);
|
||||
|
||||
engram_geo_free(E0); engram_geo_free(E1);
|
||||
}
|
||||
|
||||
/* ══════════════════ CONSISTENCY (a) — THE NEGATION-INVERSION CATCH ═══════ */
|
||||
/* The motivating failure, geometrically. Polarity axis along e0:
|
||||
* pole_pos = the AFFIRM region ("argued / fought") centroid (+5, …)
|
||||
* pole_neg = the NEGATE region ("never fought / at peace") centroid (−5, …)
|
||||
* The grounded TRUTH (context) is the reassurance "you never fought" → sits on
|
||||
* the NEGATE side (−5). The bad translation CLAIM "you argued" lands on the
|
||||
* AFFIRM side (+4). Opposite sides of the negation axis ⇒ INVERSION flagged —
|
||||
* even though "you argued" is perfectly grammatical. This is the catch. */
|
||||
{
|
||||
int dim = 4;
|
||||
double c_pos[4] = { 5, 0, 0, 0}; /* "argued / fought" */
|
||||
double c_neg[4] = {-5, 0, 0, 0}; /* "never fought / at peace"*/
|
||||
double c_truth[4] = {-5, 0, 0, 0}; /* context: the reassurance */
|
||||
double ax[4] = {1,0,0,0}; double ex[1] = {1};
|
||||
const char* ip[1]={"pos"},*in[1]={"neg"},*it[1]={"truth"};
|
||||
GeoDescriptor* POS = mk(dim, c_pos, 1, ax, ex, 1, ip, -1);
|
||||
GeoDescriptor* NEG = mk(dim, c_neg, 1, ax, ex, 1, in, -1);
|
||||
GeoDescriptor* CTX = mk(dim, c_truth, 1, ax, ex, 1, it, -1);
|
||||
|
||||
/* the plausible LIE: "you argued" — grammatical, fluent, and INVERTED. */
|
||||
float lie[4] = { 4, 0, 0, 0};
|
||||
GeoConsistency cl;
|
||||
int rc = engram_verify_consistency(lie, dim, CTX, POS, NEG, NULL,
|
||||
1.0, 0.10, 0.5, 0.0, &cl);
|
||||
ok("consistency returns 0", rc == 0);
|
||||
printf("[consistency] LIE verdict=%d inverted=%d claim_side=%.3f ref_side=%.3f sep=%.3f consist=%.3f\n",
|
||||
cl.verdict, cl.inverted, cl.polarity_claim, cl.polarity_reference, cl.polarity_separation, cl.consistency);
|
||||
ok("NEGATION INVERSION caught (inverted=1)", cl.inverted == 1);
|
||||
ok("verdict = POLARITY", cl.verdict == GEO_CONSIST_POLARITY);
|
||||
ok("claim sits on the AFFIRM pole (+)", cl.polarity_claim > 0);
|
||||
ok("truth sits on the NEGATE pole (−)", cl.polarity_reference < 0);
|
||||
ok("consistency collapses to 0 on inversion", cl.consistency < 1e-9);
|
||||
|
||||
/* the FAITHFUL translation: "you were at peace" — same pole as the truth. */
|
||||
float ok_claim[4] = {-4, 0, 0, 0};
|
||||
GeoConsistency cok;
|
||||
engram_verify_consistency(ok_claim, dim, CTX, POS, NEG, NULL,
|
||||
1.0, 0.10, 0.5, 0.0, &cok);
|
||||
printf("[consistency] TRUE verdict=%d inverted=%d claim_side=%.3f consist=%.3f\n",
|
||||
cok.verdict, cok.inverted, cok.polarity_claim, cok.consistency);
|
||||
ok("faithful claim NOT flagged (inverted=0)", cok.inverted == 0);
|
||||
ok("faithful claim verdict OK", cok.verdict == GEO_CONSIST_OK);
|
||||
ok("faithful claim consistency = 1", cok.consistency > 0.999);
|
||||
|
||||
/* a NEUTRAL claim near the midpoint must NOT false-trigger. */
|
||||
float neutral[4] = {0.1f, 0, 0, 0}; /* |side|=0.1 < deadzone 0.5 */
|
||||
GeoConsistency cn;
|
||||
engram_verify_consistency(neutral, dim, CTX, POS, NEG, NULL,
|
||||
1.0, 0.10, 0.5, 0.0, &cn);
|
||||
printf("[consistency] NEUT verdict=%d inverted=%d claim_side=%.3f consist=%.3f\n",
|
||||
cn.verdict, cn.inverted, cn.polarity_claim, cn.consistency);
|
||||
ok("neutral claim inside deadzone does NOT trigger inversion", cn.inverted == 0);
|
||||
|
||||
engram_geo_free(POS); engram_geo_free(NEG); engram_geo_free(CTX);
|
||||
}
|
||||
|
||||
/* ══════════════════ CONSISTENCY (b) — GEOMETRIC contradiction ════════════ */
|
||||
/* A claim that sits INSIDE a forbidden region it should be far from, and a claim
|
||||
* that violates a max-distance constraint to its context, are both flagged. */
|
||||
{
|
||||
int dim = 4;
|
||||
double c_ctx[4] = {0,0,0,0};
|
||||
double c_forb[4] = {0,10,0,0}; /* forbidden region, offset along e1 */
|
||||
double ax[8] = {0,1,0,0, 1,0,0,0}; double ex[2] = {1,1};
|
||||
const char* ic[1]={"ctx"},*ifb[1]={"forb"};
|
||||
GeoDescriptor* CTX = mk(dim, c_ctx, 2, ax, ex, 1, ic, -1);
|
||||
GeoDescriptor* FORB = mk(dim, c_forb, 2, ax, ex, 1, ifb, -1);
|
||||
|
||||
/* claim sitting inside the forbidden region → geometric contradiction. */
|
||||
float inside[4] = {0, 10.1f, 0, 0};
|
||||
GeoConsistency cf;
|
||||
engram_verify_consistency(inside, dim, CTX, NULL, NULL, FORB,
|
||||
1.0, 0.10, 0.5, 0.0, &cf);
|
||||
printf("[consistency] FORB verdict=%d geo_viol=%d forb_fit=%.4f consist=%.3f\n",
|
||||
cf.verdict, cf.geo_violation, cf.forbidden_fit, cf.consistency);
|
||||
ok("claim inside forbidden region FLAGGED", cf.geo_violation == 1);
|
||||
ok("verdict = GEOMETRIC", cf.verdict == GEO_CONSIST_GEOMETRIC);
|
||||
ok("forbidden fit is high (claim really is inside)", cf.forbidden_fit > 0.5);
|
||||
|
||||
/* claim well clear of the forbidden region → not flagged. */
|
||||
float clear[4] = {0.2f, 0.1f, 0, 0};
|
||||
GeoConsistency cc;
|
||||
engram_verify_consistency(clear, dim, CTX, NULL, NULL, FORB,
|
||||
1.0, 0.10, 0.5, 0.0, &cc);
|
||||
printf("[consistency] CLR verdict=%d geo_viol=%d forb_fit=%.4f\n",
|
||||
cc.verdict, cc.geo_violation, cc.forbidden_fit);
|
||||
ok("claim clear of forbidden NOT flagged", cc.geo_violation == 0 && cc.verdict == GEO_CONSIST_OK);
|
||||
|
||||
/* max-distance constraint: claim too far from context (off-axis, no poles). */
|
||||
float far[4] = {0, 8.0f, 0, 0};
|
||||
GeoConsistency cd;
|
||||
engram_verify_consistency(far, dim, CTX, NULL, NULL, NULL,
|
||||
1.0, 0.10, 0.5, /*max_distance*/3.0, &cd);
|
||||
printf("[consistency] DIST verdict=%d geo_viol=%d ctx_dist=%.3f\n",
|
||||
cd.verdict, cd.geo_violation, cd.context_distance);
|
||||
ok("claim beyond max_distance FLAGGED", cd.geo_violation == 1 && cd.verdict == GEO_CONSIST_GEOMETRIC);
|
||||
approx("context distance measured correctly", cd.context_distance, 8.0, 1e-4);
|
||||
|
||||
engram_geo_free(CTX); engram_geo_free(FORB);
|
||||
}
|
||||
|
||||
/* ══════════════════ COMBINED — grounded but INVERTED (the full plausible lie) */
|
||||
/* The most dangerous output: fluent, GROUNDED in real vocabulary, yet polarity-
|
||||
* inverted. Grounding alone passes it; only consistency catches the lie. This is
|
||||
* exactly why the verifier needs BOTH checks. */
|
||||
{
|
||||
int dim = 4;
|
||||
double c_pos[4] = { 5, 0, 0, 0}, c_neg[4] = {-5, 0, 0, 0};
|
||||
double ax[4] = {1,0,0,0}; double ex[1] = {2};
|
||||
const char* ip[1]={"pos"},*in[1]={"neg"};
|
||||
GeoDescriptor* POS = mk(dim, c_pos, 1, ax, ex, 1, ip, -1);
|
||||
GeoDescriptor* NEG = mk(dim, c_neg, 1, ax, ex, 1, in, -1);
|
||||
const GeoDescriptor* ev[2] = {POS, NEG};
|
||||
|
||||
float lie[4] = {5, 0, 0, 0}; /* "argued" — sits dead-center in the affirm region */
|
||||
GeoGrounding g;
|
||||
engram_verify_grounding(lie, dim, ev, 2, 1.0, 0.5, &g);
|
||||
GeoConsistency c;
|
||||
engram_verify_consistency(lie, dim, NEG /*truth=never fought*/, POS, NEG, NULL,
|
||||
1.0, 0.10, 0.5, 0.0, &c);
|
||||
printf("[combined] grounded=%d (score=%.3f) inverted=%d verdict=%d\n",
|
||||
g.grounded, g.grounding, c.inverted, c.verdict);
|
||||
ok("plausible lie PASSES grounding (it is real vocabulary)", g.grounded == 1);
|
||||
ok("plausible lie is CAUGHT by consistency (inverted)", c.inverted == 1);
|
||||
ok("=> grounding alone is insufficient; consistency is the catch",
|
||||
g.grounded == 1 && c.verdict == GEO_CONSIST_POLARITY);
|
||||
engram_verify_grounding_free(&g);
|
||||
engram_geo_free(POS); engram_geo_free(NEG);
|
||||
}
|
||||
|
||||
printf("\n== %d checks, %d failures ==\n", checks, failures);
|
||||
return failures ? 1 : 0;
|
||||
}
|
||||
@@ -0,0 +1,312 @@
|
||||
/* test_vindex.c — build + RUN gate for the M8 HNSW vector index.
|
||||
*
|
||||
* Covers: recall@10 vs brute-force oracle, brute-force-vs-index speedup,
|
||||
* correctness edge cases (k>N, identical vectors, self-query, zero vector),
|
||||
* determinism (seeded PRNG → identical graphs), and vindex_build_from_store
|
||||
* over a real engram_store on-disk file.
|
||||
*
|
||||
* Pure C11; links engram_vindex.c + engram_store.c; -lm. ASan/UBSan clean.
|
||||
*/
|
||||
#include "engram_vindex.h"
|
||||
#include "engram_store.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <math.h>
|
||||
#include <stdint.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#define DIM 768
|
||||
|
||||
static int g_fail = 0;
|
||||
/* VINDEX_QUICK=1 shrinks the two large builds so the ASan/UBSan pass (which runs
|
||||
* ~5-10x slower) stays fast — memory-safety is size-independent. The perf numbers
|
||||
* (recall gate + speedup) come from the un-sanitized, full-size pass. */
|
||||
static int g_quick = 0;
|
||||
static int envint(const char* k, int dflt){ const char* s=getenv(k); return s?atoi(s):dflt; }
|
||||
#define CHECK(cond, msg) do{ if(!(cond)){ printf(" FAIL: %s\n", msg); g_fail=1; } else { printf(" ok: %s\n", msg); } }while(0)
|
||||
|
||||
/* deterministic test PRNG (splitmix64) */
|
||||
static uint64_t rng_state = 0xABCDEF0123456789ULL;
|
||||
static uint64_t xrng(uint64_t* s){
|
||||
uint64_t z=(*s+=0x9E3779B97F4A7C15ULL);
|
||||
z=(z^(z>>30))*0xBF58476D1CE4E5B9ULL; z=(z^(z>>27))*0x94D049BB133111EBULL;
|
||||
return z^(z>>31);
|
||||
}
|
||||
static float frand(uint64_t* s){ return (float)((xrng(s)>>11)*(1.0/9007199254740992.0)) - 0.5f; }
|
||||
|
||||
static double now_s(void){
|
||||
struct timespec t; clock_gettime(CLOCK_MONOTONIC,&t);
|
||||
return t.tv_sec + t.tv_nsec*1e-9;
|
||||
}
|
||||
|
||||
/* fill vec[N*DIM]: mostly random, some clustered groups (center + small noise). */
|
||||
static void gen_vectors(float* v, int N, uint64_t seed){
|
||||
uint64_t s = seed;
|
||||
int clustered = N/5; /* last fifth is clustered */
|
||||
int ncenters = 20;
|
||||
float* centers = (float*)malloc((size_t)ncenters*DIM*sizeof(float));
|
||||
for (int c=0;c<ncenters;c++) for(int d=0;d<DIM;d++) centers[c*DIM+d]=frand(&s);
|
||||
for (int i=0;i<N;i++){
|
||||
if (i < N-clustered){
|
||||
for (int d=0;d<DIM;d++) v[i*DIM+d]=frand(&s);
|
||||
} else {
|
||||
int c = (int)(xrng(&s)%ncenters);
|
||||
for (int d=0;d<DIM;d++) v[i*DIM+d]=centers[c*DIM+d] + 0.05f*frand(&s);
|
||||
}
|
||||
}
|
||||
free(centers);
|
||||
}
|
||||
|
||||
static float cosdist(const float* a, const float* b){
|
||||
double da=0,db=0,dot=0;
|
||||
for(int i=0;i<DIM;i++){ da+=(double)a[i]*a[i]; db+=(double)b[i]*b[i]; dot+=(double)a[i]*b[i]; }
|
||||
if (da<=0||db<=0) return 1.0f;
|
||||
return (float)(1.0 - dot/(sqrt(da)*sqrt(db)));
|
||||
}
|
||||
|
||||
/* brute-force top-k node ids into ids[k] (ascending distance). */
|
||||
static void brute_topk(const float* v, int N, const float* q, int k, int* ids){
|
||||
float* bd = (float*)malloc((size_t)k*sizeof(float));
|
||||
for (int i=0;i<k;i++){ ids[i]=-1; bd[i]=1e30f; }
|
||||
for (int i=0;i<N;i++){
|
||||
float d = cosdist(q, v+(size_t)i*DIM);
|
||||
if (d < bd[k-1]){
|
||||
int p=k-1;
|
||||
while (p>0 && bd[p-1]>d){ bd[p]=bd[p-1]; ids[p]=ids[p-1]; p--; }
|
||||
bd[p]=d; ids[p]=i;
|
||||
}
|
||||
}
|
||||
free(bd);
|
||||
}
|
||||
|
||||
/* ── Test 1: recall@10 vs brute force + latency/recall tradeoff ────────────── */
|
||||
static void test_recall(void){
|
||||
int N=envint("VINDEX_N_RECALL", g_quick?1500:5000), Q=200, K=10;
|
||||
printf("\n== Test 1: recall@10 vs brute force (N=%d, DIM=768) ==\n", N);
|
||||
float* v = (float*)malloc((size_t)N*DIM*sizeof(float));
|
||||
gen_vectors(v, N, 111);
|
||||
|
||||
double t0=now_s();
|
||||
VIndex* ix = vindex_create(DIM, VINDEX_DEFAULT_M, VINDEX_DEFAULT_EF_CONSTRUCTION);
|
||||
for (int i=0;i<N;i++) vindex_insert(ix, (uint64_t)i, v+(size_t)i*DIM);
|
||||
double build_s = now_s()-t0;
|
||||
printf(" build: %d vectors in %.2fs (M=%d, ef_construction=%d)\n",
|
||||
N, build_s, VINDEX_DEFAULT_M, VINDEX_DEFAULT_EF_CONSTRUCTION);
|
||||
|
||||
/* queries: half random, half near a real vector (perturbed). */
|
||||
float* qs = (float*)malloc((size_t)Q*DIM*sizeof(float));
|
||||
uint64_t s=999;
|
||||
for (int i=0;i<Q;i++){
|
||||
if (i<Q/2) for(int d=0;d<DIM;d++) qs[i*DIM+d]=frand(&s);
|
||||
else { int base=(int)(xrng(&s)%N); for(int d=0;d<DIM;d++) qs[i*DIM+d]=v[base*DIM+d]+0.03f*frand(&s); }
|
||||
}
|
||||
|
||||
/* oracle */
|
||||
int* oracle = (int*)malloc((size_t)Q*K*sizeof(int));
|
||||
for (int i=0;i<Q;i++) brute_topk(v, N, qs+(size_t)i*DIM, K, oracle+(size_t)i*K);
|
||||
|
||||
int efs[] = { 10, 32, 64, 128 };
|
||||
for (int e=0;e<4;e++){
|
||||
int ef=efs[e];
|
||||
uint64_t ids[64]; float dd[64];
|
||||
int hits=0;
|
||||
double qt0=now_s();
|
||||
for (int i=0;i<Q;i++){
|
||||
int n=vindex_search(ix, qs+(size_t)i*DIM, K, ef, ids, dd);
|
||||
for (int a=0;a<n;a++) for(int b=0;b<K;b++) if((int)ids[a]==oracle[i*K+b]){ hits++; break; }
|
||||
}
|
||||
double qs_ms = (now_s()-qt0)*1000.0/Q;
|
||||
double recall = (double)hits/(Q*K);
|
||||
printf(" ef_search=%-4d recall@10=%.4f latency=%.3f ms/query\n", ef, recall, qs_ms);
|
||||
if (ef==VINDEX_DEFAULT_EF_SEARCH && !g_quick)
|
||||
CHECK(recall >= 0.90, "recall@10 >= 0.90 at default ef_search=128");
|
||||
}
|
||||
free(oracle); free(qs); free(v); vindex_free(ix);
|
||||
}
|
||||
|
||||
/* ── Test 2: speedup vs brute force ───────────────────────────────────────── */
|
||||
static void speedup_at(int N){
|
||||
int Q=100, K=10;
|
||||
float* v=(float*)malloc((size_t)N*DIM*sizeof(float));
|
||||
gen_vectors(v,N,222);
|
||||
VIndex* ix=vindex_create(DIM,16,200);
|
||||
double bt0=now_s();
|
||||
for(int i=0;i<N;i++) vindex_insert(ix,(uint64_t)i,v+(size_t)i*DIM);
|
||||
printf(" N=%d build=%.2fs\n", N, now_s()-bt0);
|
||||
|
||||
float* qs=(float*)malloc((size_t)Q*DIM*sizeof(float));
|
||||
uint64_t s=333; for(int i=0;i<Q*DIM;i++) qs[i]=frand(&s);
|
||||
|
||||
/* brute force */
|
||||
int scratch[16];
|
||||
double b0=now_s();
|
||||
for(int i=0;i<Q;i++) brute_topk(v,N,qs+(size_t)i*DIM,K,scratch);
|
||||
double bf=(now_s()-b0)/Q;
|
||||
|
||||
/* index */
|
||||
uint64_t ids[16]; float dd[16];
|
||||
double i0=now_s();
|
||||
for(int i=0;i<Q;i++) vindex_search(ix,qs+(size_t)i*DIM,K,64,ids,dd);
|
||||
double iq=(now_s()-i0)/Q;
|
||||
|
||||
printf(" N=%d brute=%.4f ms/q index=%.4f ms/q speedup=%.1fx\n",
|
||||
N, bf*1000, iq*1000, bf/iq);
|
||||
CHECK(iq < bf, "index query faster than brute force");
|
||||
free(qs); free(v); vindex_free(ix);
|
||||
}
|
||||
static void test_speedup(void){
|
||||
printf("\n== Test 2: brute-force vs index speedup ==\n");
|
||||
speedup_at(g_quick?2000:5000);
|
||||
speedup_at(envint("VINDEX_N_BIG", g_quick?3000:20000));
|
||||
}
|
||||
|
||||
/* ── Test 3: edge cases ───────────────────────────────────────────────────── */
|
||||
static void test_edges(void){
|
||||
printf("\n== Test 3: correctness edge cases ==\n");
|
||||
/* k larger than node count */
|
||||
{
|
||||
VIndex* ix=vindex_create(DIM,16,200);
|
||||
float vec[DIM]; uint64_t s=1;
|
||||
for(int i=0;i<3;i++){ for(int d=0;d<DIM;d++) vec[d]=frand(&s); vindex_insert(ix,(uint64_t)i,vec); }
|
||||
uint64_t ids[50]; float dd[50];
|
||||
int n=vindex_search(ix, vec, 50, 64, ids, dd);
|
||||
CHECK(n==3, "k > node count returns exactly node-count results");
|
||||
vindex_free(ix);
|
||||
}
|
||||
/* duplicate / identical vectors */
|
||||
{
|
||||
VIndex* ix=vindex_create(DIM,16,200);
|
||||
float a[DIM]; uint64_t s=2; for(int d=0;d<DIM;d++) a[d]=frand(&s);
|
||||
for(int i=0;i<10;i++) vindex_insert(ix,(uint64_t)i,a); /* all identical */
|
||||
float b[DIM]; for(int d=0;d<DIM;d++) b[d]=frand(&s);
|
||||
vindex_insert(ix,100,b);
|
||||
uint64_t ids[5]; float dd[5];
|
||||
int n=vindex_search(ix,a,5,64,ids,dd);
|
||||
CHECK(n==5, "identical-vector index returns k results");
|
||||
CHECK(dd[0] < 1e-4f, "top-1 distance ~0 for a duplicated vector");
|
||||
vindex_free(ix);
|
||||
}
|
||||
/* query equal to an indexed vector returns itself as top-1, dist ~0 */
|
||||
{
|
||||
VIndex* ix=vindex_create(DIM,16,200);
|
||||
int N=500; float* v=(float*)malloc((size_t)N*DIM*sizeof(float)); gen_vectors(v,N,7);
|
||||
for(int i=0;i<N;i++) vindex_insert(ix,(uint64_t)(1000+i),v+(size_t)i*DIM);
|
||||
int probe=137;
|
||||
uint64_t ids[3]; float dd[3];
|
||||
int n=vindex_search(ix, v+(size_t)probe*DIM, 3, 64, ids, dd);
|
||||
CHECK(n>=1 && ids[0]==(uint64_t)(1000+probe), "self-query returns itself as top-1");
|
||||
CHECK(dd[0] < 1e-4f, "self-query top-1 distance ~0");
|
||||
free(v); vindex_free(ix);
|
||||
}
|
||||
/* zero vector: no NaN, handled */
|
||||
{
|
||||
VIndex* ix=vindex_create(DIM,16,200);
|
||||
float z[DIM]; memset(z,0,sizeof z);
|
||||
float a[DIM]; uint64_t s=3; for(int d=0;d<DIM;d++) a[d]=frand(&s);
|
||||
vindex_insert(ix,0,z); vindex_insert(ix,1,a);
|
||||
uint64_t ids[2]; float dd[2];
|
||||
int n=vindex_search(ix, z, 2, 64, ids, dd); /* zero query */
|
||||
int nan=0; for(int i=0;i<n;i++) if(isnan(dd[i])||isinf(dd[i])) nan=1;
|
||||
CHECK(n>=1 && !nan, "zero vector query produces no NaN/Inf");
|
||||
n=vindex_search(ix, a, 2, 64, ids, dd); /* zero indexed */
|
||||
nan=0; for(int i=0;i<n;i++) if(isnan(dd[i])||isinf(dd[i])) nan=1;
|
||||
CHECK(!nan, "indexed zero vector produces no NaN/Inf");
|
||||
vindex_free(ix);
|
||||
}
|
||||
}
|
||||
|
||||
/* ── Test 4: determinism ──────────────────────────────────────────────────── */
|
||||
static void test_determinism(void){
|
||||
printf("\n== Test 4: determinism (seeded PRNG → identical results) ==\n");
|
||||
int N=1500;
|
||||
float* v=(float*)malloc((size_t)N*DIM*sizeof(float)); gen_vectors(v,N,55);
|
||||
uint64_t ids1[10],ids2[10]; float d1[10],d2[10];
|
||||
int identical=1;
|
||||
for (int build=0; build<2; build++){
|
||||
VIndex* ix=vindex_create(DIM,16,200);
|
||||
for(int i=0;i<N;i++) vindex_insert(ix,(uint64_t)i,v+(size_t)i*DIM);
|
||||
/* probe several queries */
|
||||
for (int q=0;q<20;q++){
|
||||
uint64_t* ida = build? ids2 : ids1; float* da = build? d2 : d1;
|
||||
vindex_search(ix, v+(size_t)(q*37%N)*DIM, 10, 64, ida, da);
|
||||
if (build==1){
|
||||
/* re-run build-0 query stored? simpler: compare within-run below */
|
||||
}
|
||||
}
|
||||
vindex_free(ix);
|
||||
}
|
||||
/* Proper comparison: run two fresh builds, same single query. */
|
||||
identical=1;
|
||||
for (int q=0;q<25;q++){
|
||||
int qi=(q*61)%N;
|
||||
VIndex* a=vindex_create(DIM,16,200); for(int i=0;i<N;i++) vindex_insert(a,(uint64_t)i,v+(size_t)i*DIM);
|
||||
VIndex* b=vindex_create(DIM,16,200); for(int i=0;i<N;i++) vindex_insert(b,(uint64_t)i,v+(size_t)i*DIM);
|
||||
int na=vindex_search(a, v+(size_t)qi*DIM,10,64,ids1,d1);
|
||||
int nb=vindex_search(b, v+(size_t)qi*DIM,10,64,ids2,d2);
|
||||
if (na!=nb) identical=0;
|
||||
for(int i=0;i<na;i++) if(ids1[i]!=ids2[i] || d1[i]!=d2[i]) identical=0;
|
||||
vindex_free(a); vindex_free(b);
|
||||
}
|
||||
CHECK(identical, "two independent builds give byte-identical query results");
|
||||
free(v);
|
||||
}
|
||||
|
||||
/* ── Test 5: build_from_store ─────────────────────────────────────────────── */
|
||||
static void test_build_from_store(void){
|
||||
printf("\n== Test 5: vindex_build_from_store over a real engram_store ==\n");
|
||||
char path[256];
|
||||
snprintf(path,sizeof path,"/tmp/vindex_test_store_%d.engram",(int)getpid());
|
||||
unlink(path);
|
||||
EngramPagedStore* st = store_create(path);
|
||||
if (!st){ printf(" FAIL: store_create\n"); g_fail=1; return; }
|
||||
|
||||
int N=300;
|
||||
float* v=(float*)malloc((size_t)N*DIM*sizeof(float)); gen_vectors(v,N,88);
|
||||
for (int i=0;i<N;i++){
|
||||
StoreNode n; memset(&n,0,sizeof n);
|
||||
char id[32]; snprintf(id,sizeof id,"node-%d",i);
|
||||
n.id=id; n.content="x"; n.node_type="concept"; n.tier="Semantic";
|
||||
n.emb = v+(size_t)i*DIM; n.emb_dim=DIM;
|
||||
if (store_put_node(st,&n)!=0){ printf(" FAIL: put_node %d\n",i); g_fail=1; }
|
||||
}
|
||||
/* a node WITHOUT an emb — must be skipped by build_from_store. */
|
||||
{ StoreNode n; memset(&n,0,sizeof n); n.id=(char*)"no-emb"; n.content="y"; n.node_type="concept"; n.tier="Semantic";
|
||||
store_put_node(st,&n); }
|
||||
store_close(st);
|
||||
|
||||
VIndex* ix = vindex_create(DIM,16,200);
|
||||
char** ids=NULL; int nids=0;
|
||||
int ins = vindex_build_from_store(ix, path, &ids, &nids);
|
||||
printf(" build_from_store inserted %d vectors (expected %d; 1 emb-less skipped)\n", ins, N);
|
||||
CHECK(ins==N, "build_from_store inserts exactly the emb'd nodes");
|
||||
CHECK((size_t)ins==vindex_size(ix), "index size matches insert count");
|
||||
|
||||
/* query with a known vector → must return its own node id as top-1. */
|
||||
int probe=42;
|
||||
uint64_t rids[5]; float dd[5];
|
||||
int n=vindex_search(ix, v+(size_t)probe*DIM, 5, 64, rids, dd);
|
||||
int correct = (n>=1 && rids[0]<(uint64_t)nids && strcmp(ids[rids[0]], "node-42")==0);
|
||||
printf(" query for node-42's vector → top-1 id=%s dist=%.5f\n",
|
||||
(n>=1 && rids[0]<(uint64_t)nids)? ids[rids[0]] : "?", n?dd[0]:-1);
|
||||
CHECK(correct, "build_from_store query resolves to the right node id");
|
||||
CHECK(n>=1 && dd[0]<1e-4f, "top-1 distance ~0 for exact stored vector");
|
||||
|
||||
for (int i=0;i<nids;i++) free(ids[i]);
|
||||
free(ids); free(v); vindex_free(ix); unlink(path);
|
||||
}
|
||||
|
||||
int main(void){
|
||||
(void)rng_state;
|
||||
g_quick = envint("VINDEX_QUICK", 0);
|
||||
printf("=== engram_vindex (HNSW) test suite ===%s\n", g_quick?" [QUICK]":"");
|
||||
test_recall();
|
||||
test_speedup();
|
||||
test_edges();
|
||||
test_determinism();
|
||||
test_build_from_store();
|
||||
printf("\n=== %s ===\n", g_fail? "FAILURES PRESENT" : "ALL TESTS PASSED");
|
||||
return g_fail;
|
||||
}
|
||||
Reference in New Issue
Block a user