bacaf3d39c
El SDK CI - dev / build-and-test (pull_request) Failing after 4m49s
Lands feat/reframe-region-setop (PR #109: native set-based reframe_region, decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's actual current HEAD, plus engram-tiered-storage's still-unique test suite. RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/ verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/ select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two branches; #109's copy is a strict superset (adds vindex_harvest_from_store, used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and engram_verify.c are also byte-identical. #109's own branch point already carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing left to merge into #109 for those files. The one thing engram-tiered-storage had that #109's tree dropped: its full test suite (test_vindex.c, test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh harnesses) — ported over here unchanged. WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch forked from dev on 2026-08-14 15:40 (before restructure-adjacent history diverged the file's merge-base for `git merge` — it presented as an add/add conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but it silently reverted THREE dev fixes landed on 2026-08-14/15, after the branch point, that the PR's diff had no way to know about: 1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of the query-aware propagation gate dropped the shift-and-floor rescale about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate 0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi] is computed) is orthogonal to WHAT it gates on and both are kept. 2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global). PR's tree predates this and dropped all three counters + their JSON stats fields. Restored declarations, all 4 direct increment sites, the eg_wm_carry_over bll increment, and the act-stats JSON fields -- alongside (not instead of) the PR's own P4 afferent / API-reshape counters already in that same struct/JSON. 3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev selects the STRONGEST qualifying candidate for consolidation each call; PR's tree predates this and reverted to hash-slot order (arbitrary wrt association strength) for edge formation -- the one path that writes PERMANENT structure. Restored the strongest-candidate while-loop, keeping the PR's own genuine improvement at that site (engram_adj_on_edge_added incremental-index append instead of a bare adj_dirty=1 full-rebuild flag). engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env flags, /api/nodes connected-count in responses) were pure additive: dev's side was empty, PR's side added the feature. Took PR's side whole. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): - cc -std=c11 -O2, clean link against the real engram/src/server.el via elc, zero errors. - vindex_bench (built standalone, read-only harvest) against the real production store clone (13,671 embedded nodes, 768-dim nomic-embed-text): recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs 2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s for the full 13,671-node set -- see the flagged risk below. - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned snapshot of the live store, 13,424 nodes / 37,656 edges) and called /api/activate for real: first call after boot 41.5s (pays the one-time HNSW build inline -- matches the standalone bench), second/third calls 356ms/605ms, no crash, correct results, act-stats JSON (including the restored evict_floor/cap/bll fields) reads correctly. KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync builds the HNSW index synchronously, inline, on the first engram_activate() call after every process start (or index invalidation). On the real node count that is a ~46s blocking stall on a single-threaded server -- the first request after every restart (or its concurrent siblings) waits the full build. Recommend a background/incremental build (or a bounded per-call build budget) before this ever reaches the live daemon. See PR description / final report for the fuller writeup.
148 lines
7.1 KiB
Bash
Executable File
148 lines
7.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# M-INTEROCEPTION P1 gate: two-threshold consolidation (ENGRAM_CONSOLIDATION).
|
|
# Throwaway HOME + /tmp only. Never touches ~/.neuron or :8742.
|
|
set -u
|
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
RT="$HERE/../../lang/runtime/el_runtime.c"
|
|
ST="$HERE/../../lang/runtime/engram_store.c"
|
|
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
|
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
|
INC="$HERE/../../lang/runtime"
|
|
WORK="$(mktemp -d /tmp/engram-p1-XXXXXX)"
|
|
export HOME="$WORK/home"; mkdir -p "$HOME"
|
|
unset ENGRAM_STORE ENGRAM_CONSOLIDATION ENGRAM_CONSOL_CONN_MIN ENGRAM_CONSOL_PERM_MIN ENGRAM_CONSOL_WM_TOPK
|
|
fail=0
|
|
|
|
echo "== compile =="
|
|
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p1_consol.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
|
-lcurl -lm -o "$WORK/p1" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
|
|
|
echo
|
|
echo "== (a) HEADLINE: hebb accrual curve over N co-activations (flag OFF, pure trunk) =="
|
|
D="$WORK/a"; mkdir -p "$D"
|
|
( unset ENGRAM_CONSOLIDATION; "$WORK/p1" accrual "$D" ) >"$WORK/accrual.txt" 2>&1 || { echo "FAIL accrual run"; fail=1; }
|
|
python3 - "$WORK/accrual.txt" <<'PY'
|
|
import json,sys,re
|
|
rows=[]
|
|
for line in open(sys.argv[1]):
|
|
m=re.match(r'SAMPLE (\d+) (\{.*\})',line.strip())
|
|
if not m: continue
|
|
n=int(m.group(1)); j=json.loads(m.group(2))
|
|
hm=j.get("hebb_max",0.0); hc=j.get("hebb_cand_max",0.0)
|
|
rows.append((n,hm,hc))
|
|
print(" N hebb_max 1-0.9999^N (predicted EWMA)")
|
|
rc=0
|
|
for n,hm,hc in rows:
|
|
pred=1-0.9999**n
|
|
print(f" {n:<7} {hm:<12.6g} {pred:.6g}")
|
|
# assertions: monotonic rise, starts near ETA, tracks EWMA prediction
|
|
first=rows[0]; last=rows[-1]
|
|
def check(c,m):
|
|
global rc; print((" PASS: " if c else " FAIL: ")+m);
|
|
if not c: rc=1
|
|
check(abs(first[1]-0.0001)<5e-5, f"first sample hebb ~= ETA 0.0001 (got {first[1]:.6g})")
|
|
check(all(rows[i][1] <= rows[i+1][1]+1e-9 for i in range(len(rows)-1)), "hebb_max is monotonically non-decreasing over N")
|
|
check(last[1] > first[1]*50, f"hebb accrues substantially by N={last[0]} (got {last[1]:.4g} vs {first[1]:.4g})")
|
|
# EWMA fit: measured should be within 25% of 1-0.9999^N at the mid samples
|
|
mid=[r for r in rows if 100<=r[0]<=2000]
|
|
ok=all(abs(hm-(1-0.9999**n))/(1-0.9999**n) < 0.25 for n,hm,hc in mid)
|
|
check(ok, "measured curve tracks the 1-0.9999^N EWMA prediction within 25% (co-activation P~1)")
|
|
sys.exit(rc)
|
|
PY
|
|
[ $? -ne 0 ] && fail=1
|
|
|
|
echo
|
|
echo "== (b) CONNECTION threshold: strong ISE wires to wm_top, weak ISE wires nothing (flag ON) =="
|
|
D="$WORK/b"; mkdir -p "$D"
|
|
( export ENGRAM_CONSOLIDATION=1; "$WORK/p1" connect "$D" ) >"$WORK/connect.txt" 2>&1 || { echo "FAIL connect run"; fail=1; }
|
|
cat "$WORK/connect.txt" | sed 's/^/ /'
|
|
python3 - "$WORK/connect.txt" "$D/connect.json" <<'PY'
|
|
import json,sys,re
|
|
txt=open(sys.argv[1]).read()
|
|
g=json.load(open(sys.argv[2]))
|
|
def field(k):
|
|
m=re.search(rf'{k} (\S+)',txt); return m.group(1) if m else None
|
|
sid=field("ISE_STRONG_ID"); wid=field("ISE_WEAK_ID")
|
|
m=re.search(r'EDGES before=(\d+) after_strong=(\d+) after_weak=(\d+)',txt)
|
|
before,aftS,aftW=int(m.group(1)),int(m.group(2)),int(m.group(3))
|
|
rc=0
|
|
def check(c,mm):
|
|
global rc; print((" PASS: " if c else " FAIL: ")+mm)
|
|
if not c: rc=1
|
|
strong_edges=[e for e in g["edges"] if e["from_id"]==sid and e["relation"]=="hebbian-associate"]
|
|
weak_edges=[e for e in g["edges"] if e["from_id"]==wid]
|
|
check(aftS>before, f"strong ISE formed connection edges ({before} -> {aftS})")
|
|
check(aftW==aftS, f"weak ISE formed NO edges ({aftS} -> {aftW})")
|
|
check(len(strong_edges)>=1, f"strong ISE has {len(strong_edges)} hebbian-associate edge(s) to wm_top")
|
|
check(all('consolidated-from-ISE' in (e.get('metadata') or '') for e in strong_edges),
|
|
"connection edges are provenance-tagged consolidated-from-ISE (reversible)")
|
|
check(len(weak_edges)==0, "weak ISE (below connection bar) has zero outgoing edges")
|
|
# targets must be the WM-top nodes (hebb-a / hebb-b), not distractors
|
|
tgt_labels=set()
|
|
byid={n["id"]:n for n in g["nodes"]}
|
|
for e in strong_edges:
|
|
t=byid.get(e["to_id"]);
|
|
if t: tgt_labels.add(t.get("label"))
|
|
print(f" connection targets: {sorted(tgt_labels)}")
|
|
check(tgt_labels.issubset({"hebb-a","hebb-b"}) and len(tgt_labels)>=1,
|
|
f"connections point at the wm_top nodes {sorted(tgt_labels)}")
|
|
sys.exit(rc)
|
|
PY
|
|
[ $? -ne 0 ] && fail=1
|
|
|
|
echo
|
|
echo "== (c) PERMANENCE threshold: promoted node survives 48h prune, ephemeral is swept (flag ON) =="
|
|
D="$WORK/c"; mkdir -p "$D"
|
|
( export ENGRAM_CONSOLIDATION=1 ENGRAM_CONSOL_PERM_MIN=-1000; "$WORK/p1" perm "$D" ) >"$WORK/perm.txt" 2>&1 || { echo "FAIL perm run"; fail=1; }
|
|
cat "$WORK/perm.txt" | sed 's/^/ /'
|
|
python3 - "$WORK/perm.txt" <<'PY'
|
|
import sys,re,json
|
|
txt=open(sys.argv[1]).read()
|
|
rc=0
|
|
def check(c,m):
|
|
global rc; print((" PASS: " if c else " FAIL: ")+m)
|
|
if not c: rc=1
|
|
prom=int(re.search(r'PROMOTED (\d+)',txt).group(1))
|
|
m=re.search(r'NODES before=(\d+) after=(\d+) removed=(\d+)',txt)
|
|
before,after,removed=int(m.group(1)),int(m.group(2)),int(m.group(3))
|
|
dur=re.search(r'DURABLE_NODE (\{.*\})',txt).group(1)
|
|
eph=re.search(r'EPHEMERAL_NODE (\{.*\})',txt).group(1)
|
|
durj=json.loads(dur); ephj=json.loads(eph)
|
|
check(prom==1, "engram_consolidate_permanence promoted the node (returned 1)")
|
|
check(before==2 and after==1 and removed==1, f"exactly one node pruned ({before}->{after}, removed={removed})")
|
|
check(durj.get("id")=="ise-durable", "durable node SURVIVED the 48h telemetry prune")
|
|
check('consolidated-from-ISE' in (durj.get("metadata") or ''), "durable node carries reversible provenance marker")
|
|
check(ephj=={} or not ephj.get("id"), "ephemeral (non-permanent) ISE was swept")
|
|
sys.exit(rc)
|
|
PY
|
|
[ $? -ne 0 ] && fail=1
|
|
|
|
echo
|
|
echo "== (d) OFF path byte-identical: ISE creation forms no edges, permanence is a no-op =="
|
|
D="$WORK/d"; mkdir -p "$D"
|
|
( unset ENGRAM_CONSOLIDATION; "$WORK/p1" offcheck "$D" ) >"$WORK/off.txt" 2>&1
|
|
rcoff=$?
|
|
cat "$WORK/off.txt" | sed 's/^/ /'
|
|
[ $rcoff -eq 0 ] && echo " PASS: flag OFF — ISE creation added 0 edges and permanence returned 0" \
|
|
|| { echo " FAIL: OFF path changed behavior"; fail=1; }
|
|
|
|
echo
|
|
echo "== ASan+UBSan (connect + perm + accrual-short) =="
|
|
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
|
-I "$INC" "$HERE/test_interoception_p1_consol.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
|
-lcurl -lm -o "$WORK/p1.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
|
if [ -x "$WORK/p1.san" ]; then
|
|
export ASAN_OPTIONS=detect_leaks=0
|
|
DS="$WORK/san"; mkdir -p "$DS"
|
|
( export ENGRAM_CONSOLIDATION=1 ENGRAM_CONSOL_PERM_MIN=-1000; "$WORK/p1.san" connect "$DS" ) >/dev/null 2>"$WORK/san_run.log"
|
|
( export ENGRAM_CONSOLIDATION=1 ENGRAM_CONSOL_PERM_MIN=-1000; "$WORK/p1.san" perm "$DS" ) >/dev/null 2>>"$WORK/san_run.log"
|
|
if grep -qiE 'runtime error|AddressSanitizer|Sanitizer|ERROR: ' "$WORK/san_run.log"; then
|
|
echo " FAIL: sanitizer findings:"; grep -iE 'runtime error|Sanitizer|ERROR' "$WORK/san_run.log" | head; fail=1
|
|
else echo " ok: ASan+UBSan clean"; fi
|
|
fi
|
|
|
|
echo
|
|
if [ "$fail" -eq 0 ]; then echo "====== P1 CONSOLIDATION GATE: PASS ======"; else echo "====== P1 CONSOLIDATION GATE: FAIL ======"; fi
|
|
rm -rf "$WORK"
|
|
exit $fail
|