runtime: extract engram_text.c, and repair 10 harnesses that could not link
First concern moved out of el_runtime.c under the ratchet, and the move is
deliberately small: it exists to prove the mechanism end to end before anything
large depends on it.
engram_text.{c,h} — query tokenization, candidate-token hygiene, word-boundary
matching, and the text-damage signature. Four functions, moved verbatim; only
`static` was dropped and each doc comment travelled with the code. They touch no
EL value type and no engram store type: plain C over <ctype.h>/<string.h> over
char buffers. They were never el_runtime.c's business.
el_runtime.c 20,527 -> 20,427 lines (BUDGET max_lines ratcheted down)
engram fns 279 -> 275 (BUDGET max_engram_fns ratcheted down)
The Stage 1 extension point worked as designed: adding the file to
lang/runtime/SOURCES was one line, and every build path picked it up. The
Stage 2 drift guard then caught that I had NOT added it to install.sh's
standalone list — the exact class of drift it was written for, on its first
real change, before the commit rather than after a broken SDK shipped.
WHY ONLY 100 LINES, AND WHAT ACTUALLY BLOCKS THE REST
Measured, not estimated: of 273 engram-domain functions in el_runtime.c
(~9,700 lines), only 75 (~1,058 lines) can move today, and they are scattered
rather than clustered. The blocker is a single fact:
EngramNode, EngramEdge, EngramStore, EngramLayer, EngramWal and EngramIdSlot
are typedef'd INSIDE el_runtime.c. No sibling can see them. engram_store.h
defines a SEPARATE serializable "node view" struct and maps between the two.
So every engram function that takes an EngramNode* — which is most of them, 109
of 273 by direct type reference — cannot compile in engram_store.c until those
types move to a shared header. That extraction is the real Stage 3 enabler and
it deserves its own change: it touches the most load-bearing struct in the
system, and doing it in the same commit as a code move would make a regression
impossible to bisect.
REPAIRED: 10 engram harnesses that had silently stopped linking
Not new breakage from this move — verified against unmodified dev, where
el_runtime.c + engram_store.c alone already failed with undefined symbols.
They had been dead for as long as el_runtime.c has been calling into the
siblings, and nothing noticed because nothing ran them.
run_m3_parity, run_m7_traversal, run_m35_hebb_persist,
run_interoception_p0..p5 — now build from $(scripts/el-runtime-sources.sh)
run_wal_tests — its two TUs #include "el_runtime.c" directly, so
it links the SIBLINGS ONLY; adding el_runtime.c
to that link line would define every symbol twice
(That #include'd .c is worth recording: the runtime does have one, in
engram/test/test_wal.c and the generated test_failloud.c.)
Verified locally — every one of these was run, not assumed:
* m3_parity ............ PASS, incl. ASan+UBSan clean across seed/on/reboot
* m7_traversal ......... PASS
* m35_hebb_persist ..... PASS (the gate over the original prod hebb bug)
* interoception p0..p5 . PASS (all six)
* wal_tests ............ 66 passed, 0 failed, + fail-loud exit check
* self-host fixpoint ... byte-identical, AND the emitted C is byte-identical
to the pre-move compiler output — the move changes
nothing the compiler produces
* engram/src/server.el . compiles and links
* native suites ........ 8 of 13, unchanged from before the move; the same 5
pre-existing failures, no regression
* both runtime guards .. green at the new, lower budget
Also fixes a block comment left unterminated by the extraction (the deleted
range carried its closing */), restoring the compile to its single pre-existing
-Wcomment warning.
This commit is contained in:
bigmerge
@@ -3,10 +3,14 @@
|
||||
# Throwaway HOME + /tmp only. Never touches ~/.neuron or :8742.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
RTSRC="$("$HERE/../../scripts/el-runtime-sources.sh" "$HERE/../../lang/runtime")"
|
||||
# The runtime is MULTI-FILE (lang/runtime/SOURCES). This harness used to link
|
||||
# el_runtime.c + engram_store.c only, which stopped linking once el_runtime.c
|
||||
# began calling into the other engram siblings. Unquoted on purpose: a list.
|
||||
SSLFLAGS=""
|
||||
if command -v brew >/dev/null 2>&1 && O="$(brew --prefix openssl@3 2>/dev/null)"; then
|
||||
SSLFLAGS="-I$O/include -L$O/lib"
|
||||
fi
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p0-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
@@ -14,8 +18,8 @@ unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compile (plain) =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p0_emb.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p0" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p0_emb.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p0" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
D="$WORK/d"; mkdir -p "$D"
|
||||
"$WORK/p0" "$D" || { echo "FAIL: run"; fail=1; }
|
||||
@@ -69,8 +73,8 @@ PY
|
||||
echo
|
||||
echo "== ASan+UBSan =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p0_emb.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p0.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -20 "$WORK/san_cc.log"; fail=1; }
|
||||
-I "$INC" "$HERE/test_interoception_p0_emb.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p0.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -20 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p0.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
DS="$WORK/ds"; mkdir -p "$DS"
|
||||
|
||||
@@ -3,10 +3,14 @@
|
||||
# Throwaway HOME + /tmp only. Never touches ~/.neuron or :8742.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
RTSRC="$("$HERE/../../scripts/el-runtime-sources.sh" "$HERE/../../lang/runtime")"
|
||||
# The runtime is MULTI-FILE (lang/runtime/SOURCES). This harness used to link
|
||||
# el_runtime.c + engram_store.c only, which stopped linking once el_runtime.c
|
||||
# began calling into the other engram siblings. Unquoted on purpose: a list.
|
||||
SSLFLAGS=""
|
||||
if command -v brew >/dev/null 2>&1 && O="$(brew --prefix openssl@3 2>/dev/null)"; then
|
||||
SSLFLAGS="-I$O/include -L$O/lib"
|
||||
fi
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p1-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
@@ -14,8 +18,8 @@ unset ENGRAM_STORE ENGRAM_CONSOLIDATION ENGRAM_CONSOL_CONN_MIN ENGRAM_CONSOL_PER
|
||||
fail=0
|
||||
|
||||
echo "== compile =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p1_consol.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p1" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p1_consol.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p1" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
echo
|
||||
echo "== (a) HEADLINE: hebb accrual curve over N co-activations (flag OFF, pure trunk) =="
|
||||
@@ -129,8 +133,8 @@ cat "$WORK/off.txt" | sed 's/^/ /'
|
||||
echo
|
||||
echo "== ASan+UBSan (connect + perm + accrual-short) =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p1_consol.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p1.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
-I "$INC" "$HERE/test_interoception_p1_consol.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p1.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p1.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
DS="$WORK/san"; mkdir -p "$DS"
|
||||
|
||||
@@ -3,10 +3,14 @@
|
||||
# Throwaway HOME + /tmp only. TC defaults to 3600s; we pin it for the math.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
RTSRC="$("$HERE/../../scripts/el-runtime-sources.sh" "$HERE/../../lang/runtime")"
|
||||
# The runtime is MULTI-FILE (lang/runtime/SOURCES). This harness used to link
|
||||
# el_runtime.c + engram_store.c only, which stopped linking once el_runtime.c
|
||||
# began calling into the other engram siblings. Unquoted on purpose: a list.
|
||||
SSLFLAGS=""
|
||||
if command -v brew >/dev/null 2>&1 && O="$(brew --prefix openssl@3 2>/dev/null)"; then
|
||||
SSLFLAGS="-I$O/include -L$O/lib"
|
||||
fi
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p2-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
@@ -15,8 +19,8 @@ unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compile =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p2_chrono.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p2" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p2_chrono.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p2" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
sum_wm(){ python3 -c "import json,sys; g=json.load(open('$1')); print(sum(n.get('working_memory_weight',0) for n in g['nodes']))"; }
|
||||
|
||||
@@ -78,8 +82,8 @@ python3 -c "import sys; sys.exit(0 if abs($OFFWM-1.2)<1e-9 else 1)" \
|
||||
echo
|
||||
echo "== ASan+UBSan =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p2_chrono.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p2.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
-I "$INC" "$HERE/test_interoception_p2_chrono.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p2.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p2.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
DS="$WORK/san"; mkdir -p "$DS"
|
||||
|
||||
@@ -3,18 +3,22 @@
|
||||
# Read-only pure primitive; no store, no flag. Throwaway /tmp only.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
RTSRC="$("$HERE/../../scripts/el-runtime-sources.sh" "$HERE/../../lang/runtime")"
|
||||
# The runtime is MULTI-FILE (lang/runtime/SOURCES). This harness used to link
|
||||
# el_runtime.c + engram_store.c only, which stopped linking once el_runtime.c
|
||||
# began calling into the other engram siblings. Unquoted on purpose: a list.
|
||||
SSLFLAGS=""
|
||||
if command -v brew >/dev/null 2>&1 && O="$(brew --prefix openssl@3 2>/dev/null)"; then
|
||||
SSLFLAGS="-I$O/include -L$O/lib"
|
||||
fi
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p3-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
fail=0
|
||||
|
||||
echo "== compile =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p3_drift.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p3" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p3_drift.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p3" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
"$WORK/p3" > "$WORK/out.txt" 2>&1 || { echo "FAIL run"; cat "$WORK/out.txt"; fail=1; }
|
||||
cat "$WORK/out.txt" | sed 's/^/ /'
|
||||
@@ -52,8 +56,8 @@ PY
|
||||
echo
|
||||
echo "== ASan+UBSan =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p3_drift.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p3.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
-I "$INC" "$HERE/test_interoception_p3_drift.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p3.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p3.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
"$WORK/p3.san" >/dev/null 2>"$WORK/san.log"
|
||||
|
||||
@@ -2,10 +2,14 @@
|
||||
# M-INTEROCEPTION P4 gate: afferent input counters in act-stats (additive).
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
RTSRC="$("$HERE/../../scripts/el-runtime-sources.sh" "$HERE/../../lang/runtime")"
|
||||
# The runtime is MULTI-FILE (lang/runtime/SOURCES). This harness used to link
|
||||
# el_runtime.c + engram_store.c only, which stopped linking once el_runtime.c
|
||||
# began calling into the other engram siblings. Unquoted on purpose: a list.
|
||||
SSLFLAGS=""
|
||||
if command -v brew >/dev/null 2>&1 && O="$(brew --prefix openssl@3 2>/dev/null)"; then
|
||||
SSLFLAGS="-I$O/include -L$O/lib"
|
||||
fi
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p4-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
@@ -13,8 +17,8 @@ unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compile =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p4_afferent.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p4" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p4_afferent.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p4" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
"$WORK/p4" > "$WORK/out.txt" 2>&1 || { echo "FAIL run"; cat "$WORK/out.txt"; fail=1; }
|
||||
grep -oE 'aff_[a-z_]+":[0-9]+' "$WORK/out.txt" | sed 's/^/ /' | head -30
|
||||
@@ -53,8 +57,8 @@ PY
|
||||
echo
|
||||
echo "== ASan+UBSan =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p4_afferent.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p4.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
-I "$INC" "$HERE/test_interoception_p4_afferent.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p4.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p4.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
"$WORK/p4.san" >/dev/null 2>"$WORK/san.log"
|
||||
|
||||
@@ -2,10 +2,14 @@
|
||||
# M-INTEROCEPTION P5 gate: dream-recall builtin engram_dreams_json (honesty rail).
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
GEO="$HERE/../../lang/runtime/engram_geometry.c"
|
||||
VIDX="$HERE/../../lang/runtime/engram_vindex.c"
|
||||
RTSRC="$("$HERE/../../scripts/el-runtime-sources.sh" "$HERE/../../lang/runtime")"
|
||||
# The runtime is MULTI-FILE (lang/runtime/SOURCES). This harness used to link
|
||||
# el_runtime.c + engram_store.c only, which stopped linking once el_runtime.c
|
||||
# began calling into the other engram siblings. Unquoted on purpose: a list.
|
||||
SSLFLAGS=""
|
||||
if command -v brew >/dev/null 2>&1 && O="$(brew --prefix openssl@3 2>/dev/null)"; then
|
||||
SSLFLAGS="-I$O/include -L$O/lib"
|
||||
fi
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-p5-XXXXXX)"
|
||||
export HOME="$WORK/home"; mkdir -p "$HOME"
|
||||
@@ -13,8 +17,8 @@ unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compile =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p5_dreams.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p5" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_interoception_p5_dreams.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p5" 2>"$WORK/cc.log" || { echo "COMPILE FAILED"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; }
|
||||
|
||||
D="$WORK/d"; mkdir -p "$D"
|
||||
"$WORK/p5" "$D" > "$WORK/out.txt" 2>&1 || { echo "FAIL run"; cat "$WORK/out.txt"; fail=1; }
|
||||
@@ -57,8 +61,8 @@ PY
|
||||
echo
|
||||
echo "== ASan+UBSan =="
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_interoception_p5_dreams.c" "$RT" "$ST" "$GEO" "$VIDX" \
|
||||
-lcurl -lm -o "$WORK/p5.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
-I "$INC" "$HERE/test_interoception_p5_dreams.c" $RTSRC $SSLFLAGS \
|
||||
-lcurl -lssl -lcrypto -lpthread -lm -lm -o "$WORK/p5.san" 2>"$WORK/san_cc.log" || { echo "SAN COMPILE FAILED"; tail -25 "$WORK/san_cc.log"; fail=1; }
|
||||
if [ -x "$WORK/p5.san" ]; then
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
DS="$WORK/ds"; mkdir -p "$DS"
|
||||
|
||||
@@ -6,8 +6,14 @@
|
||||
# Writes ONLY under a throwaway /tmp dir with a throwaway HOME.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
RTSRC="$("$HERE/../../scripts/el-runtime-sources.sh" "$HERE/../../lang/runtime")"
|
||||
# The runtime is MULTI-FILE (lang/runtime/SOURCES). This harness used to link
|
||||
# el_runtime.c + engram_store.c only, which stopped linking once el_runtime.c
|
||||
# began calling into the other engram siblings. Unquoted on purpose: a list.
|
||||
SSLFLAGS=""
|
||||
if command -v brew >/dev/null 2>&1 && O="$(brew --prefix openssl@3 2>/dev/null)"; then
|
||||
SSLFLAGS="-I$O/include -L$O/lib"
|
||||
fi
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-m35-XXXXXX)"
|
||||
BIN="$WORK/m35"
|
||||
@@ -17,7 +23,7 @@ unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compiling harness (gcc: el_runtime.c + engram_store.c + test_m35_hebb_persist.c) =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_m35_hebb_persist.c" "$RT" "$ST" -lcurl -o "$BIN" 2>"$WORK/cc.log"
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_m35_hebb_persist.c" $RTSRC $SSLFLAGS -lcurl -lssl -lcrypto -lpthread -lm -o "$BIN" 2>"$WORK/cc.log"
|
||||
if [ $? -ne 0 ]; then echo "COMPILE FAILED:"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; fi
|
||||
|
||||
echo
|
||||
@@ -139,7 +145,7 @@ echo
|
||||
echo "== 5) ASan+UBSan build, exercise the full persist+reboot flow (leaks off — harness intentionally leaks el_strdup) =="
|
||||
SANBIN="$WORK/m35.san"
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_m35_hebb_persist.c" "$RT" "$ST" -lcurl -o "$SANBIN" 2>"$WORK/san_cc.log"
|
||||
-I "$INC" "$HERE/test_m35_hebb_persist.c" $RTSRC $SSLFLAGS -lcurl -lssl -lcrypto -lpthread -lm -o "$SANBIN" 2>"$WORK/san_cc.log"
|
||||
if [ $? -ne 0 ]; then echo " SAN COMPILE FAILED:"; tail -20 "$WORK/san_cc.log"; fail=1; else
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
DSAN="$WORK/san"; mkdir -p "$DSAN"
|
||||
|
||||
@@ -4,8 +4,14 @@
|
||||
# Writes ONLY under a throwaway /tmp dir with a throwaway HOME + ENGRAM_DATA_DIR.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
RTSRC="$("$HERE/../../scripts/el-runtime-sources.sh" "$HERE/../../lang/runtime")"
|
||||
# The runtime is MULTI-FILE (lang/runtime/SOURCES). This harness used to link
|
||||
# el_runtime.c + engram_store.c only, which stopped linking once el_runtime.c
|
||||
# began calling into the other engram siblings. Unquoted on purpose: a list.
|
||||
SSLFLAGS=""
|
||||
if command -v brew >/dev/null 2>&1 && O="$(brew --prefix openssl@3 2>/dev/null)"; then
|
||||
SSLFLAGS="-I$O/include -L$O/lib"
|
||||
fi
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-m3-XXXXXX)"
|
||||
DATA="$WORK/data"; mkdir -p "$DATA"
|
||||
@@ -17,7 +23,7 @@ unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compiling harness (gcc: el_runtime.c + engram_store.c + test_m3_parity.c) =="
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_m3_parity.c" "$RT" "$ST" -lcurl -o "$BIN" 2>"$WORK/cc.log"
|
||||
gcc -O1 -std=c11 -I "$INC" "$HERE/test_m3_parity.c" $RTSRC $SSLFLAGS -lcurl -lssl -lcrypto -lpthread -lm -o "$BIN" 2>"$WORK/cc.log"
|
||||
if [ $? -ne 0 ]; then echo "COMPILE FAILED:"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; fi
|
||||
grep -i warning "$WORK/cc.log" | grep -iE 'engram_store|eg_store|eg_load|scan_nodes|scan_edges' && echo "(warnings in M3 code above)" || true
|
||||
|
||||
@@ -106,7 +112,7 @@ echo
|
||||
echo "== 5) ASan+UBSan build, exercise M3 scan/boot/hooks (leaks off — harness intentionally leaks el_strdup) =="
|
||||
SANBIN="$WORK/m3.san"
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_m3_parity.c" "$RT" "$ST" -lcurl -o "$SANBIN" 2>"$WORK/san_cc.log"
|
||||
-I "$INC" "$HERE/test_m3_parity.c" $RTSRC $SSLFLAGS -lcurl -lssl -lcrypto -lpthread -lm -o "$SANBIN" 2>"$WORK/san_cc.log"
|
||||
if [ $? -ne 0 ]; then echo " SAN COMPILE FAILED:"; tail -20 "$WORK/san_cc.log"; fail=1; else
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
DATA2="$WORK/data2"; mkdir -p "$DATA2"
|
||||
|
||||
@@ -6,8 +6,14 @@
|
||||
# Writes ONLY under a throwaway /tmp dir with a throwaway HOME.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
RT="$HERE/../../lang/runtime/el_runtime.c"
|
||||
ST="$HERE/../../lang/runtime/engram_store.c"
|
||||
RTSRC="$("$HERE/../../scripts/el-runtime-sources.sh" "$HERE/../../lang/runtime")"
|
||||
# The runtime is MULTI-FILE (lang/runtime/SOURCES). This harness used to link
|
||||
# el_runtime.c + engram_store.c only, which stopped linking once el_runtime.c
|
||||
# began calling into the other engram siblings. Unquoted on purpose: a list.
|
||||
SSLFLAGS=""
|
||||
if command -v brew >/dev/null 2>&1 && O="$(brew --prefix openssl@3 2>/dev/null)"; then
|
||||
SSLFLAGS="-I$O/include -L$O/lib"
|
||||
fi
|
||||
INC="$HERE/../../lang/runtime"
|
||||
WORK="$(mktemp -d /tmp/engram-m7-XXXXXX)"
|
||||
DATA="$WORK/data"; mkdir -p "$DATA"
|
||||
@@ -22,7 +28,7 @@ unset ENGRAM_STORE
|
||||
fail=0
|
||||
|
||||
echo "== compiling harness (gcc: el_runtime.c + engram_store.c + test_m7_traversal.c) =="
|
||||
gcc -O2 -std=c11 -I "$INC" "$HERE/test_m7_traversal.c" "$RT" "$ST" -lcurl -lm -o "$BIN" 2>"$WORK/cc.log"
|
||||
gcc -O2 -std=c11 -I "$INC" "$HERE/test_m7_traversal.c" $RTSRC $SSLFLAGS -lcurl -lssl -lcrypto -lpthread -lm -o "$BIN" 2>"$WORK/cc.log"
|
||||
if [ $? -ne 0 ]; then echo "COMPILE FAILED:"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; fi
|
||||
echo " ok: compiled"
|
||||
|
||||
@@ -115,7 +121,7 @@ echo
|
||||
echo "== 3) ASan+UBSan clean across parity + a small perf loop (leaks off — harness intentionally leaks el_strdup) =="
|
||||
SANBIN="$WORK/m7.san"
|
||||
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
|
||||
-I "$INC" "$HERE/test_m7_traversal.c" "$RT" "$ST" -lcurl -lm -o "$SANBIN" 2>"$WORK/san_cc.log"
|
||||
-I "$INC" "$HERE/test_m7_traversal.c" $RTSRC $SSLFLAGS -lcurl -lssl -lcrypto -lpthread -lm -lm -o "$SANBIN" 2>"$WORK/san_cc.log"
|
||||
if [ $? -ne 0 ]; then echo " SAN COMPILE FAILED:"; tail -20 "$WORK/san_cc.log"; fail=1; else
|
||||
export ASAN_OPTIONS=detect_leaks=0
|
||||
D2="$WORK/data2"; mkdir -p "$D2"
|
||||
|
||||
@@ -3,8 +3,17 @@
|
||||
set -e
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
REL="$HERE/../../lang/runtime"
|
||||
# test_wal.c and test_failloud.c #include "el_runtime.c" directly, so el_runtime.c
|
||||
# is already IN the translation unit — link the SIBLINGS only, or every symbol in
|
||||
# it is defined twice. The siblings are still required: el_runtime.c calls into
|
||||
# all six engram TUs. (lang/runtime/SOURCES is the source of truth.)
|
||||
RTSIB="$("$HERE/../../scripts/el-runtime-sources.sh" "$REL" | grep -v '/el_runtime\.c$')"
|
||||
SSLFLAGS=""
|
||||
if command -v brew >/dev/null 2>&1 && O="$(brew --prefix openssl@3 2>/dev/null)"; then
|
||||
SSLFLAGS="-I$O/include -L$O/lib"
|
||||
fi
|
||||
cc -O2 -fbracket-depth=1024 -Wno-parentheses-equality -I"$REL" \
|
||||
"$HERE/test_wal.c" -lcurl -lpthread -o /tmp/test_wal
|
||||
"$HERE/test_wal.c" $RTSIB $SSLFLAGS -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/test_wal
|
||||
HOME=/tmp/engram-throwaway-home /tmp/test_wal
|
||||
# Fail-loud data-dir check (must exit 1 with a FATAL line):
|
||||
cat > /tmp/test_failloud.c <<'C'
|
||||
@@ -12,5 +21,5 @@ cat > /tmp/test_failloud.c <<'C'
|
||||
int main(void){ unsetenv("ENGRAM_DATA_DIR"); unsetenv("HOME");
|
||||
engram_resolve_data_dir(); printf("REACHED\n"); return 0; }
|
||||
C
|
||||
cc -O2 -fbracket-depth=1024 -Wno-parentheses-equality -I"$REL" /tmp/test_failloud.c -lcurl -lpthread -o /tmp/test_failloud
|
||||
cc -O2 -fbracket-depth=1024 -Wno-parentheses-equality -I"$REL" /tmp/test_failloud.c $RTSIB $SSLFLAGS -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/test_failloud
|
||||
if env -u HOME -u ENGRAM_DATA_DIR /tmp/test_failloud; then echo "FAIL: should have exited"; exit 1; else echo "[PASS] fail-loud exit on unresolvable HOME"; fi
|
||||
|
||||
@@ -60,12 +60,14 @@ RUNTIME_SOURCES=(
|
||||
el_runtime.c el_seed.c
|
||||
engram_store.c engram_vindex.c engram_geometry.c
|
||||
engram_reason.c engram_verify.c engram_cognition.c
|
||||
engram_text.c
|
||||
eg_cosine_batch.c eg_cosine_batch_strategy_cpu.c
|
||||
)
|
||||
RUNTIME_HEADERS=(
|
||||
el_runtime.h el_seed.h
|
||||
engram_store.h engram_vindex.h engram_geometry.h
|
||||
engram_reason.h engram_verify.h engram_cognition.h
|
||||
engram_text.h
|
||||
eg_cosine_batch.h eg_cosine_batch_strategy.h
|
||||
)
|
||||
|
||||
|
||||
+6
-4
@@ -28,12 +28,14 @@
|
||||
# FORMAT: <key> <value> — `#` comments and blank lines ignored.
|
||||
|
||||
# Maximum lines in lang/runtime/el_runtime.c.
|
||||
# 2026-08-16: 20,527 — the high-water mark, ratcheted from here.
|
||||
max_lines 20527
|
||||
# 2026-08-16: 20,527 — the high-water mark.
|
||||
# 2026-08-16: 20,427 — engram_text.c extracted (tokenize, token hygiene,
|
||||
# word-boundary match, damage signature). Ratcheted down.
|
||||
max_lines 20427
|
||||
|
||||
# Maximum top-level engram/eg_/cog_ function definitions in el_runtime.c.
|
||||
# ~47.5% of the file is engram code, and engram already owns six dedicated
|
||||
# sibling files (engram_{store,vindex,geometry,reason,verify,cognition}.c).
|
||||
# Every one of these belongs in one of them. This is the Stage 3 scoreboard.
|
||||
# 2026-08-16: 279.
|
||||
max_engram_fns 279
|
||||
# 2026-08-16: 279 -> 275 (4 moved to engram_text.c).
|
||||
max_engram_fns 275
|
||||
|
||||
@@ -44,6 +44,11 @@ engram_reason.c
|
||||
engram_verify.c
|
||||
engram_cognition.c
|
||||
|
||||
# --- Text: tokenization, token hygiene, damage signature ---------------------
|
||||
# Extracted from el_runtime.c 2026-08-16. Plain C over <ctype.h>/<string.h> —
|
||||
# touches no EL value type and no engram store type. New text helpers go HERE.
|
||||
engram_text.c
|
||||
|
||||
# --- Vector math: batch cosine + its CPU strategy ----------------------------
|
||||
# The ggml strategy (eg_cosine_batch_strategy_ggml.c) is an OPTIONAL swap-in and
|
||||
# is deliberately NOT in the default set — it needs ggml headers. Link it in
|
||||
|
||||
+5
-105
@@ -8638,6 +8638,7 @@ static char* engram_first_n_chars(const char* s, size_t n) {
|
||||
* mutation (node/edge create, forget) is mirrored through the store's
|
||||
* WAL-logged API so neuron.egm/neuron.wal stay authoritative.
|
||||
* ══════════════════════════════════════════════════════════════════════════ */
|
||||
#include "engram_text.h" /* text: tokenize, token hygiene, loss signature */
|
||||
#include "engram_store.h"
|
||||
#include "engram_vindex.h" /* M8: ANN (HNSW) index for activation seed selection */
|
||||
#include "engram_geometry.h" /* M9: centered relational-neighborhood geometry (priming) */
|
||||
@@ -9061,31 +9062,9 @@ el_val_t engram_node(el_val_t content, el_val_t node_type, el_val_t salience) {
|
||||
* RIGHT NOW" — which is the regression question, and the one that
|
||||
* would have caught this in a day instead of two months.
|
||||
*
|
||||
* SIGNATURE. Conservative on purpose — a false alarm that cries corruption
|
||||
* over ordinary punctuation is worse than useless. Two patterns, both of
|
||||
* which are essentially absent from well-formed English prose:
|
||||
* (a) alnum '?' alnum — "na?ve", "caf?s", "don?t". A real question mark
|
||||
* never sits between two word characters.
|
||||
* (b) ' ? ' followed by a lowercase letter — a lost em/en dash. A real
|
||||
* question mark is not preceded by a space, and
|
||||
* what follows one starts a new sentence.
|
||||
* Deliberately NOT flagged: a trailing '?' after a word, '? ' before a
|
||||
* capital, or '?' at end of string — all legitimate. This under-counts (it
|
||||
* cannot see a mangled 'café ' where the '?' landed before a space), so the
|
||||
* census is a floor on the damage, never an exaggeration of it. */
|
||||
static int eg_text_loss_signature(const char* s) {
|
||||
if (!s) return 0;
|
||||
for (const char* p = s; *p; p++) {
|
||||
if (*p != '?') continue;
|
||||
unsigned char prev = (p == s) ? 0 : (unsigned char)p[-1];
|
||||
unsigned char next = (unsigned char)p[1];
|
||||
/* (a) sandwiched between word characters. */
|
||||
if (isalnum(prev) && isalnum(next)) return 1;
|
||||
/* (b) spaced, with lowercase continuation — a lost dash. */
|
||||
if (prev == ' ' && next == ' ' && islower((unsigned char)p[2])) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
* The SIGNATURE itself (eg_text_loss_signature) moved to engram_text.c on
|
||||
* 2026-08-16 — it is plain C over <ctype.h> and touches nothing in here. The
|
||||
* stock/flow gauges below stay, because they touch store and EL value types. */
|
||||
|
||||
/* Damaged-node creations since process start. See the block comment above. */
|
||||
static int64_t _eg_txt_write_damaged = 0;
|
||||
@@ -9697,37 +9676,7 @@ static int istr_contains(const char* hay, const char* needle) {
|
||||
* fix landed but never reached this release runtime — the copy the engram
|
||||
* binary actually builds against.) */
|
||||
#define ENGRAM_MAX_QTOKENS 32
|
||||
#define ENGRAM_QTOK_LEN 256
|
||||
|
||||
/* Split q on whitespace into up to ENGRAM_MAX_QTOKENS distinct
|
||||
* (case-insensitive) tokens. Returns the token count. Over-long tokens are
|
||||
* truncated to ENGRAM_QTOK_LEN-1; over-count tokens are ignored. */
|
||||
static int engram_tokenize_query(const char* q,
|
||||
char toks[][ENGRAM_QTOK_LEN], int maxtok) {
|
||||
int n = 0;
|
||||
if (!q) return 0;
|
||||
const char* p = q;
|
||||
while (*p && n < maxtok) {
|
||||
while (*p && isspace((unsigned char)*p)) p++;
|
||||
if (!*p) break;
|
||||
char buf[ENGRAM_QTOK_LEN];
|
||||
size_t tl = 0;
|
||||
while (*p && !isspace((unsigned char)*p)) {
|
||||
if (tl < sizeof(buf) - 1) buf[tl++] = *p;
|
||||
p++;
|
||||
}
|
||||
buf[tl] = '\0';
|
||||
if (tl == 0) continue;
|
||||
int dup = 0;
|
||||
for (int s = 0; s < n; s++) {
|
||||
if (strcasecmp(toks[s], buf) == 0) { dup = 1; break; }
|
||||
}
|
||||
if (dup) continue;
|
||||
memcpy(toks[n], buf, tl + 1);
|
||||
n++;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
/* ENGRAM_QTOK_LEN and engram_tokenize_query moved to engram_text.h/.c. */
|
||||
|
||||
/* Count how many of the ntok distinct query tokens appear (case-insensitive)
|
||||
* in the node's content, label, or tags. 0 == no match. */
|
||||
@@ -16389,29 +16338,6 @@ el_val_t engram_label_df(el_val_t term) {
|
||||
#define ENGRAM_ST_TOKLEN 64
|
||||
#define ENGRAM_ST_SCANCHARS 400
|
||||
|
||||
/* Trim leading/trailing non-alphanumerics, then accept only tokens whose core
|
||||
* is alphanumeric plus '-' and '_' with at least 3 letters. This subsumes the
|
||||
* quoted-title guard (2026-07-25) and the "<!--" flood (2026-08-03)
|
||||
* structurally: markup and punctuation-bearing tokens never become
|
||||
* candidates, rather than being blocklisted after the fact. */
|
||||
static int eg_st_clean_token(const char* raw, size_t rawlen,
|
||||
char* out, size_t outcap) {
|
||||
size_t s = 0, e = rawlen;
|
||||
while (s < e && !isalnum((unsigned char)raw[s])) s++;
|
||||
while (e > s && !isalnum((unsigned char)raw[e - 1])) e--;
|
||||
size_t len = e - s;
|
||||
if (len < 4 || len >= outcap) return 0;
|
||||
int alpha = 0;
|
||||
for (size_t i = 0; i < len; i++) {
|
||||
unsigned char c = (unsigned char)raw[s + i];
|
||||
if (isalpha(c)) alpha++;
|
||||
else if (!isdigit(c) && c != '-' && c != '_') return 0;
|
||||
}
|
||||
if (alpha < 3) return 0;
|
||||
memcpy(out, raw + s, len);
|
||||
out[len] = '\0';
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* ENGRAM_ST_DEBUG=1 dumps the full scored candidate set to stderr. One
|
||||
* cached branch in production. This exists because the first live run of this
|
||||
@@ -16424,32 +16350,6 @@ static int _eg_st_debug(void) {
|
||||
return v;
|
||||
}
|
||||
|
||||
/* Word-boundary document frequency. engram_label_df uses istr_contains, i.e.
|
||||
* SUBSTRING matching, and that is the wrong estimator for term specificity on
|
||||
* short tokens: "them" hits inside "theme" and "anthem", "about" and "whole"
|
||||
* come back with df 2 and 1 rather than 0. That matters here specifically
|
||||
* because the min_df floor is what rejects English function words, and it can
|
||||
* only do that job if their df is honestly zero. Substring df quietly handed
|
||||
* them a survival ticket. Measured on the live store before this fix, "whole"
|
||||
* (df=1, idf=8.76) and "about" (df=2, idf=8.36) were outscoring real topical
|
||||
* terms and losing only on position — one node whose text happened to open
|
||||
* with a function word would have seeded on it.
|
||||
*
|
||||
* engram_label_df keeps substring semantics: it is a separate published
|
||||
* measure with existing callers, and changing it underneath them is not this
|
||||
* change's business. */
|
||||
static int eg_st_label_has_word(const char* hay, const char* word) {
|
||||
size_t wl = strlen(word);
|
||||
for (const char* p = hay; *p; p++) {
|
||||
if (strncasecmp(p, word, wl) != 0) continue;
|
||||
char before = (p == hay) ? '\0' : p[-1];
|
||||
char after = p[wl];
|
||||
if (before && (isalnum((unsigned char)before) || before == '_')) continue;
|
||||
if (after && (isalnum((unsigned char)after) || after == '_')) continue;
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* YAKE T_Case, adapted to this corpus. YAKE up-weights all-caps tokens
|
||||
* because in ordinary prose an acronym is rare and carries topic. That
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
/* engram_text.c — see engram_text.h.
|
||||
*
|
||||
* Moved verbatim out of el_runtime.c (2026-08-16). Bodies are unchanged; only
|
||||
* `static` was dropped so they link from this translation unit, and each
|
||||
* function's doc comment travelled with it.
|
||||
*/
|
||||
#include "engram_text.h"
|
||||
|
||||
#include <ctype.h>
|
||||
#include <string.h>
|
||||
|
||||
/* Split q on whitespace into up to ENGRAM_MAX_QTOKENS distinct
|
||||
* (case-insensitive) tokens. Returns the token count. Over-long tokens are
|
||||
* truncated to ENGRAM_QTOK_LEN-1; over-count tokens are ignored. */
|
||||
int engram_tokenize_query(const char* q,
|
||||
char toks[][ENGRAM_QTOK_LEN], int maxtok) {
|
||||
int n = 0;
|
||||
if (!q) return 0;
|
||||
const char* p = q;
|
||||
while (*p && n < maxtok) {
|
||||
while (*p && isspace((unsigned char)*p)) p++;
|
||||
if (!*p) break;
|
||||
char buf[ENGRAM_QTOK_LEN];
|
||||
size_t tl = 0;
|
||||
while (*p && !isspace((unsigned char)*p)) {
|
||||
if (tl < sizeof(buf) - 1) buf[tl++] = *p;
|
||||
p++;
|
||||
}
|
||||
buf[tl] = '\0';
|
||||
if (tl == 0) continue;
|
||||
int dup = 0;
|
||||
for (int s = 0; s < n; s++) {
|
||||
if (strcasecmp(toks[s], buf) == 0) { dup = 1; break; }
|
||||
}
|
||||
if (dup) continue;
|
||||
memcpy(toks[n], buf, tl + 1);
|
||||
n++;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
/* Trim leading/trailing non-alphanumerics, then accept only tokens whose core
|
||||
* is alphanumeric plus '-' and '_' with at least 3 letters. This subsumes the
|
||||
* quoted-title guard (2026-07-25) and the "<!--" flood (2026-08-03)
|
||||
* structurally: markup and punctuation-bearing tokens never become
|
||||
* candidates, rather than being blocklisted after the fact. */
|
||||
int eg_st_clean_token(const char* raw, size_t rawlen,
|
||||
char* out, size_t outcap) {
|
||||
size_t s = 0, e = rawlen;
|
||||
while (s < e && !isalnum((unsigned char)raw[s])) s++;
|
||||
while (e > s && !isalnum((unsigned char)raw[e - 1])) e--;
|
||||
size_t len = e - s;
|
||||
if (len < 4 || len >= outcap) return 0;
|
||||
int alpha = 0;
|
||||
for (size_t i = 0; i < len; i++) {
|
||||
unsigned char c = (unsigned char)raw[s + i];
|
||||
if (isalpha(c)) alpha++;
|
||||
else if (!isdigit(c) && c != '-' && c != '_') return 0;
|
||||
}
|
||||
if (alpha < 3) return 0;
|
||||
memcpy(out, raw + s, len);
|
||||
out[len] = '\0';
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* Word-boundary document frequency. engram_label_df uses istr_contains, i.e.
|
||||
* SUBSTRING matching, and that is the wrong estimator for term specificity on
|
||||
* short tokens: "them" hits inside "theme" and "anthem", "about" and "whole"
|
||||
* come back with df 2 and 1 rather than 0. That matters here specifically
|
||||
* because the min_df floor is what rejects English function words, and it can
|
||||
* only do that job if their df is honestly zero. Substring df quietly handed
|
||||
* them a survival ticket. Measured on the live store before this fix, "whole"
|
||||
* (df=1, idf=8.76) and "about" (df=2, idf=8.36) were outscoring real topical
|
||||
* terms and losing only on position — one node whose text happened to open
|
||||
* with a function word would have seeded on it.
|
||||
*
|
||||
* engram_label_df keeps substring semantics: it is a separate published
|
||||
* measure with existing callers, and changing it underneath them is not this
|
||||
* change's business. */
|
||||
int eg_st_label_has_word(const char* hay, const char* word) {
|
||||
size_t wl = strlen(word);
|
||||
for (const char* p = hay; *p; p++) {
|
||||
if (strncasecmp(p, word, wl) != 0) continue;
|
||||
char before = (p == hay) ? '\0' : p[-1];
|
||||
char after = p[wl];
|
||||
if (before && (isalnum((unsigned char)before) || before == '_')) continue;
|
||||
if (after && (isalnum((unsigned char)after) || after == '_')) continue;
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Text-damage signature. Extracted with the function from el_runtime.c's
|
||||
* "Text-integrity instrumentation" block; the stock/flow gauges that use it
|
||||
* (engram_text_health_json, _eg_txt_write_damaged) stay there because they
|
||||
* touch store and EL value types.
|
||||
*
|
||||
* SIGNATURE. Conservative on purpose — a false alarm that cries corruption
|
||||
* over ordinary punctuation is worse than useless. Two patterns, both of
|
||||
* which are essentially absent from well-formed English prose:
|
||||
* (a) alnum '?' alnum — "na?ve", "caf?s", "don?t". A real question mark
|
||||
* never sits between two word characters.
|
||||
* (b) ' ? ' followed by a lowercase letter — a lost em/en dash. A real
|
||||
* question mark is not preceded by a space, and
|
||||
* what follows one starts a new sentence.
|
||||
* Deliberately NOT flagged: a trailing '?' after a word, '? ' before a
|
||||
* capital, or '?' at end of string — all legitimate. This under-counts (it
|
||||
* cannot see a mangled 'café ' where the '?' landed before a space), so the
|
||||
* census is a floor on the damage, never an exaggeration of it. */
|
||||
int eg_text_loss_signature(const char* s) {
|
||||
if (!s) return 0;
|
||||
for (const char* p = s; *p; p++) {
|
||||
if (*p != '?') continue;
|
||||
unsigned char prev = (p == s) ? 0 : (unsigned char)p[-1];
|
||||
unsigned char next = (unsigned char)p[1];
|
||||
/* (a) sandwiched between word characters. */
|
||||
if (isalnum(prev) && isalnum(next)) return 1;
|
||||
/* (b) spaced, with lowercase continuation — a lost dash. */
|
||||
if (prev == ' ' && next == ' ' && islower((unsigned char)p[2])) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
/* engram_text.h — text handling for the engram: query tokenization, candidate
|
||||
* token hygiene, word-boundary matching, and the text-damage signature.
|
||||
*
|
||||
* WHY THIS FILE EXISTS
|
||||
* --------------------
|
||||
* These functions lived in el_runtime.c, which is a 2026-05-03 build shim that
|
||||
* was scheduled for deletion, never retired, and grew to 20,527 lines. They do
|
||||
* not belong there: they touch no EL value type and no engram store type. They
|
||||
* are plain C over <ctype.h>/<string.h> operating on char buffers, and they are
|
||||
* a concern of their own — so they get a translation unit of their own.
|
||||
*
|
||||
* Adding a new text helper? Add it HERE, not to el_runtime.c. A new .c costs
|
||||
* exactly one line in lang/runtime/SOURCES, and every build path picks it up.
|
||||
* Placement is a LINK-TIME concern: the compiler cannot tell which .c a symbol
|
||||
* came from (builtin_arity is an arity guard, not a dispatch table), so a
|
||||
* function defined here is exactly as linkable as one defined in el_runtime.c.
|
||||
*/
|
||||
#ifndef ENGRAM_TEXT_H
|
||||
#define ENGRAM_TEXT_H
|
||||
|
||||
#include <stddef.h>
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/* Max bytes per query token, including the NUL. */
|
||||
#define ENGRAM_QTOK_LEN 256
|
||||
|
||||
/* Split q on whitespace into up to ENGRAM_MAX_QTOKENS distinct
|
||||
* (case-insensitive) tokens. Returns the token count. Over-long tokens are
|
||||
* truncated to ENGRAM_QTOK_LEN-1; over-count tokens are ignored. */
|
||||
int engram_tokenize_query(const char* q, char toks[][ENGRAM_QTOK_LEN], int maxtok);
|
||||
|
||||
/* Trim leading/trailing non-alphanumerics, then accept only tokens whose core
|
||||
* is alphanumeric plus '-' and '_' with at least 3 letters. This subsumes the
|
||||
* quoted-title guard (2026-07-25) and the "<!--" flood (2026-08-03)
|
||||
* structurally: markup and punctuation-bearing tokens never become
|
||||
* candidates, rather than being blocklisted after the fact. */
|
||||
int eg_st_clean_token(const char* raw, size_t rawlen, char* out, size_t outcap);
|
||||
|
||||
/* Word-boundary document frequency. engram_label_df uses istr_contains, i.e.
|
||||
* SUBSTRING matching, and that is the wrong estimator for term specificity on
|
||||
* short tokens: "them" hits inside "theme" and "anthem", "about" and "whole"
|
||||
* come back with df 2 and 1 rather than 0. That matters here specifically
|
||||
* because the min_df floor is what rejects English function words, and it can
|
||||
* only do that job if their df is honestly zero. Substring df quietly handed
|
||||
* them a survival ticket. Measured on the live store before this fix, "whole"
|
||||
* (df=1, idf=8.76) and "about" (df=2, idf=8.36) were outscoring real topical
|
||||
* terms and losing only on position — one node whose text happened to open
|
||||
* with a function word would have seeded on it.
|
||||
*
|
||||
* engram_label_df keeps substring semantics: it is a separate published
|
||||
* measure with existing callers, and changing it underneath them is not this
|
||||
* change's business. */
|
||||
int eg_st_label_has_word(const char* hay, const char* word);
|
||||
|
||||
/* Whether s carries the text-loss signature left by the \uXXXX -> '?' parser
|
||||
* defect. Conservative by design; see engram_text.c for the full rationale. */
|
||||
int eg_text_loss_signature(const char* s);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* ENGRAM_TEXT_H */
|
||||
Reference in New Issue
Block a user