Neuron
8e9d88fc01
El SDK CI - dev / build-and-test (pull_request) Failing after 10m59s
The crash (SIGTRAP in engram_activate -> eg_vindex_sync -> vindex_insert -> _realloc) had three read paths mutating five process-global statics. engram_activate, eg_knn_for_node (whose own comment says "No writes.") and engram_geo_reify_run_json all called eg_vindex_sync, which frees the index, reallocs the seen-map and inserts — on a read. Three moves, in decreasing order of how much they dissolve: 1. Misfiled scratch is not shared state. visited/visit_epoch/visited_cap were never owned by the index; they are one traversal's local, hoisted into struct VIndex as an allocation optimisation. They want neither a lock nor a capability nor a pool — just to go back in the call frame. Two concurrent READS stomped each other purely because of this. 2. const IS the capability. Once the scratch leaves the struct, search reads and nothing else, so vindex_search takes a const VIndex*. That is exactly what a capability-pointer ABI would have bought — a read path physically cannot call vindex_insert, enforced by the compiler on every future caller — for one qualifier instead of an ABI swept across hundreds of builtins. 3. What survives is publication, not ownership. HNSW insert is NOT an append: it rewires the neighbour links of already-existing elements and reallocs elems[], so the store's append-only property does not transfer to the index derived from it. eg_vindex_sync therefore splits into eg_vindex_maintain (exclusive, sole mutator) and eg_vindex_view (shared, returns const VIndex*). A read path may demand that a current snapshot exist — a request to the owner, not a mutation by the reader. Write-side owner: eg_vindex_note_embedded hooks the embedding-ASSIGNMENT sites rather than the append sites, because a node with no embedding cannot be in a vector index — embedding assignment is the event that owns index membership. One O(log n) insert, no O(node_count) presence scan. This also retires the "STALENESS (honest tradeoff)" note where a lazily-embedded older node stayed invisible to route_nearest/autoconnect until a full rebuild (the embed-gap #20 shape). Evidence. The existing harness conflated two hazards, which is why fixing half of it read as failure. Split into four: single (3000 vec, ASan+UBSan) clean -> clean readers (4 readers, no writer, TSan) RACE -> clean unsynchronized (writer+reader, bare) race -> race, expected forever published (owner + 4 readers) n/a -> clean, 3000/3000 landed RESULT: PASS. recall@10 = 0.9365 at ef_search=128 (gate >= 0.90); determinism byte-identical across two independent builds. The unsynchronized half is now permanently expected to race, deliberately: it is the executable proof that the boundary must live above the data structure, not inside it. fb32d15's guard is KEPT, correcting this design's own section 5. Measured, it guards TWO structures and only one was converted here: g->nodes/g->edges are realloc'd in place (el_runtime.c:7618,7629) and engram_activate_inner's embed-backfill writes n->emb through exactly such a borrowed pointer. Deleting the guard reintroduces a measured 11171->9579 edge loss. Its comment is narrowed to the RAM graph and the deletion precondition named. That corrects the ordering claim too: the residual is not one ABI that dissolves everything at once, it is a PROPERTY applied per structure. Residues evaporate in the order the property is applied, and a residue whose structure has not been converted must be left standing.
108 lines
4.5 KiB
Bash
Executable File
108 lines
4.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# run_vindex_concurrency_tests.sh — regression harness for the 2026-08-16 soul crash.
|
|
#
|
|
# Four halves. The SET is the point: it separates two hazards the original two-half
|
|
# version conflated, and which have fixes in different files.
|
|
#
|
|
# 1. single ASan+UBSan, one thread. MUST be clean. Hard failure.
|
|
#
|
|
# 2. readers TSan, N readers, NO writer. Hazard (a): the visited set used
|
|
# to live on the index, so two pure READS stamped each other's
|
|
# epoch. Fixed in engram_vindex.c (frame-owned VVisit +
|
|
# `const VIndex*` search). MUST be clean. Hard failure.
|
|
#
|
|
# 3. unsynchronized TSan, writer + reader on a BARE index. Hazard (b): in-place
|
|
# HNSW insert rewires existing elements' neighbour lists and
|
|
# reallocs elems[]. EXPECTED TO RACE, PERMANENTLY. This is not
|
|
# a bug to fix inside engram_vindex.c — it is the executable
|
|
# proof that a publication boundary must exist above it.
|
|
# Not a failure. If it ever goes CLEAN, the test stopped
|
|
# interleaving and half 4 is no longer meaningful either.
|
|
#
|
|
# 4. published TSan, owner + N readers through a publication boundary
|
|
# (rwlock: readers shared, owner exclusive) mirroring
|
|
# eg_vindex_view / eg_vindex_maintain in lang/runtime/el_runtime.c.
|
|
# MUST be clean, and all inserts must land. Hard failure.
|
|
#
|
|
# See test_vindex_concurrency.c for the full story (SIGSEGV at ASCII address
|
|
# "gramNode", heap corruption in xzm_realloc, etc).
|
|
#
|
|
# usage: run_vindex_concurrency_tests.sh
|
|
set -uo pipefail
|
|
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
RUNTIME="$(cd "$HERE/../../lang/runtime" && pwd)"
|
|
WORK="$(mktemp -d)"
|
|
trap 'rm -rf "$WORK"' EXIT
|
|
|
|
SRC="$HERE/test_vindex_concurrency.c"
|
|
VINDEX="$RUNTIME/engram_vindex.c"
|
|
|
|
fail=0
|
|
|
|
echo "== [1/4] single-threaded control under AddressSanitizer =="
|
|
cc -std=c11 -g -O1 -fsanitize=address,undefined -fno-omit-frame-pointer \
|
|
-I"$RUNTIME" -o "$WORK/single" "$SRC" "$VINDEX" -lm || { echo "BUILD FAILED"; exit 2; }
|
|
if ASAN_OPTIONS=detect_leaks=0 "$WORK/single" single; then
|
|
echo " -> OK"
|
|
else
|
|
echo " -> FAIL: the single-threaded control must always be clean."
|
|
echo " If this fails the bug is NOT (only) concurrency — look for a real"
|
|
echo " out-of-bounds or lifetime error in engram_vindex.c."
|
|
fail=1
|
|
fi
|
|
|
|
cc -std=c11 -g -O1 -fsanitize=thread -fno-omit-frame-pointer \
|
|
-I"$RUNTIME" -o "$WORK/conc" "$SRC" "$VINDEX" -lm || { echo "BUILD FAILED"; exit 2; }
|
|
|
|
# run_tsan <mode> <logfile>; echoes nothing, sets $tsan_raced
|
|
run_tsan() {
|
|
TSAN_OPTIONS="halt_on_error=0" "$WORK/conc" "$1" >"$2" 2>&1
|
|
tsan_rc=$?
|
|
if grep -q "ThreadSanitizer: data race" "$2"; then tsan_raced=1; else tsan_raced=0; fi
|
|
}
|
|
|
|
echo
|
|
echo "== [2/4] concurrent READERS, no writer (visited-set gate) =="
|
|
run_tsan readers "$WORK/readers.log"
|
|
if [ "$tsan_raced" = "1" ]; then
|
|
echo " -> REGRESSION: two concurrent reads still race."
|
|
grep -m1 -A6 "ThreadSanitizer: data race" "$WORK/readers.log" | sed 's/^/ /'
|
|
echo " The visited set was supposed to be owned by the call frame."
|
|
fail=1
|
|
else
|
|
echo " -> clean (concurrent reads are safe)"
|
|
fi
|
|
|
|
echo
|
|
echo "== [3/4] writer+reader on a BARE index (expected-race probe) =="
|
|
run_tsan unsynchronized "$WORK/unsync.log"
|
|
if [ "$tsan_raced" = "1" ]; then
|
|
echo " -> RACE DETECTED, as expected:"
|
|
grep -m1 -A4 "ThreadSanitizer: data race" "$WORK/unsync.log" | sed 's/^/ /'
|
|
echo " In-place HNSW insert mutates existing elements. Not fixable inside"
|
|
echo " engram_vindex.c — this is why the publication boundary exists."
|
|
else
|
|
echo " -> NOTE: no race reported. The probe did not interleave; half 4's"
|
|
echo " clean result proves less than it should. Investigate."
|
|
fi
|
|
|
|
echo
|
|
echo "== [4/4] owner+readers through the publication boundary (boundary gate) =="
|
|
run_tsan published "$WORK/pub.log"
|
|
if [ "$tsan_raced" = "1" ]; then
|
|
echo " -> REGRESSION: the publication boundary did not serialize the owner."
|
|
grep -m1 -A6 "ThreadSanitizer: data race" "$WORK/pub.log" | sed 's/^/ /'
|
|
fail=1
|
|
elif [ "$tsan_rc" != "0" ]; then
|
|
echo " -> FAIL: boundary clean under TSan but the run failed:"
|
|
tail -3 "$WORK/pub.log" | sed 's/^/ /'
|
|
fail=1
|
|
else
|
|
echo " -> clean (readers project concurrently; the owner's inserts all landed)"
|
|
fi
|
|
|
|
echo
|
|
[ "$fail" -eq 0 ] && echo "RESULT: PASS" || echo "RESULT: FAIL"
|
|
exit "$fail"
|