will.anderson
8cae0f94eb
The nomic-embed-text space over the corpus is strongly anisotropic (mean pairwise cosine ~0.55), which compresses cosine-based domain separation almost to nothing so the design-doc s5 operators (distance/overlap/Wasserstein) cannot discriminate. Subtracting the global mean of the normalized embeddings restores isotropy (mean pairwise cosine ~0) and sharpens the operators. - add GeoMeanCache (engram_geo_mean_build / _maybe_refresh / _vec / _free): a store-derived centering offset over the embed-eligible set, cached and refreshed on significant drift; lives in geometry.c, not the store. - engram_geometry_descriptor gains an optional global_mean: when supplied the centroid, per-member cosine distance, and co-registration run in centered space (GM=zeros reproduces the legacy raw path exactly). - co-registration choice (b): the ANN query stays in raw unit space (index unchanged) since centering is a rigid translation that ~preserves neighborhood membership; only the descriptor statistics move to the centered frame. Covariance/axes/radius are translation-invariant and therefore unchanged. - test: synthetic ground-truth suite stays green (PERF + ASan/UBSan), plus new centered/raw/mean-cache assertions. - add bench_discrimination.c (env-gated, read-only, skips in CI): on a copy of the real store the two-domain overlap operator drops 1.13 -> 0.008 and cross-centroid cosine 0.899 -> 0.003 after centering, Euclid distance unchanged (translation-invariant control). No change to activation/retrieval behavior; wiring geometry into retrieval is a separate, behavior-changing cutover.
31 lines
1.4 KiB
Bash
Executable File
31 lines
1.4 KiB
Bash
Executable File
#!/bin/sh
|
|
# Build + RUN the M9 FOUNDATION geometry-descriptor tests. Pure C11 (gcc/cc),
|
|
# stdlib + libm only. Standalone module — NOT folded through elb/elc. Two passes:
|
|
# 1. PERF — optimised (-O2, no sanitizer): the functional gate.
|
|
# 2. SAFETY — ASan + UBSan on the same suite (memory-safety is size-independent).
|
|
set -e
|
|
HERE=$(cd "$(dirname "$0")" && pwd)
|
|
RT="$HERE/../../lang/runtime"
|
|
CC=${CC:-cc}
|
|
SRC="$HERE/test_geometry.c $RT/engram_geometry.c $RT/engram_store.c $RT/engram_vindex.c"
|
|
WARN="-std=c11 -Wall -Wextra"
|
|
TMP=$(mktemp -d)
|
|
|
|
echo "### PASS 1: PERF (optimised, un-sanitised) — functional gate"
|
|
$CC $WARN -O2 -I"$RT" $SRC -lm -o "$TMP/perf"
|
|
"$TMP/perf"
|
|
|
|
echo
|
|
echo "### PASS 2: SAFETY (ASan/UBSan)"
|
|
$CC $WARN -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -I"$RT" $SRC -lm -o "$TMP/safe"
|
|
ASAN_OPTIONS=${ASAN_OPTIONS:-detect_leaks=0} UBSAN_OPTIONS=halt_on_error=1 "$TMP/safe"
|
|
|
|
# PASS 3 (OPTIONAL): mean-centering discrimination bench on a COPY of a real
|
|
# store. Skips cleanly unless ENGRAM_BENCH_STORE points at a store .egm — never
|
|
# touches the live store. Read-only; not part of the pass/fail gate.
|
|
echo
|
|
echo "### PASS 3: DISCRIMINATION BENCH (optional; set ENGRAM_BENCH_STORE)"
|
|
BSRC="$HERE/bench_discrimination.c $RT/engram_geometry.c $RT/engram_store.c $RT/engram_vindex.c"
|
|
$CC $WARN -O2 -I"$RT" $BSRC -lm -o "$TMP/bench"
|
|
"$TMP/bench" "${ENGRAM_BENCH_STORE:-}"
|