runtime: extract engram_text.c, and repair 10 harnesses that could not link
First concern moved out of el_runtime.c under the ratchet, and the move is
deliberately small: it exists to prove the mechanism end to end before anything
large depends on it.
engram_text.{c,h} — query tokenization, candidate-token hygiene, word-boundary
matching, and the text-damage signature. Four functions, moved verbatim; only
`static` was dropped and each doc comment travelled with the code. They touch no
EL value type and no engram store type: plain C over <ctype.h>/<string.h> over
char buffers. They were never el_runtime.c's business.
el_runtime.c 20,527 -> 20,427 lines (BUDGET max_lines ratcheted down)
engram fns 279 -> 275 (BUDGET max_engram_fns ratcheted down)
The Stage 1 extension point worked as designed: adding the file to
lang/runtime/SOURCES was one line, and every build path picked it up. The
Stage 2 drift guard then caught that I had NOT added it to install.sh's
standalone list — the exact class of drift it was written for, on its first
real change, before the commit rather than after a broken SDK shipped.
WHY ONLY 100 LINES, AND WHAT ACTUALLY BLOCKS THE REST
Measured, not estimated: of 273 engram-domain functions in el_runtime.c
(~9,700 lines), only 75 (~1,058 lines) can move today, and they are scattered
rather than clustered. The blocker is a single fact:
EngramNode, EngramEdge, EngramStore, EngramLayer, EngramWal and EngramIdSlot
are typedef'd INSIDE el_runtime.c. No sibling can see them. engram_store.h
defines a SEPARATE serializable "node view" struct and maps between the two.
So every engram function that takes an EngramNode* — which is most of them, 109
of 273 by direct type reference — cannot compile in engram_store.c until those
types move to a shared header. That extraction is the real Stage 3 enabler and
it deserves its own change: it touches the most load-bearing struct in the
system, and doing it in the same commit as a code move would make a regression
impossible to bisect.
REPAIRED: 10 engram harnesses that had silently stopped linking
Not new breakage from this move — verified against unmodified dev, where
el_runtime.c + engram_store.c alone already failed with undefined symbols.
They had been dead for as long as el_runtime.c has been calling into the
siblings, and nothing noticed because nothing ran them.
run_m3_parity, run_m7_traversal, run_m35_hebb_persist,
run_interoception_p0..p5 — now build from $(scripts/el-runtime-sources.sh)
run_wal_tests — its two TUs #include "el_runtime.c" directly, so
it links the SIBLINGS ONLY; adding el_runtime.c
to that link line would define every symbol twice
(That #include'd .c is worth recording: the runtime does have one, in
engram/test/test_wal.c and the generated test_failloud.c.)
Verified locally — every one of these was run, not assumed:
* m3_parity ............ PASS, incl. ASan+UBSan clean across seed/on/reboot
* m7_traversal ......... PASS
* m35_hebb_persist ..... PASS (the gate over the original prod hebb bug)
* interoception p0..p5 . PASS (all six)
* wal_tests ............ 66 passed, 0 failed, + fail-loud exit check
* self-host fixpoint ... byte-identical, AND the emitted C is byte-identical
to the pre-move compiler output — the move changes
nothing the compiler produces
* engram/src/server.el . compiles and links
* native suites ........ 8 of 13, unchanged from before the move; the same 5
pre-existing failures, no regression
* both runtime guards .. green at the new, lower budget
Also fixes a block comment left unterminated by the extraction (the deleted
range carried its closing */), restoring the compile to its single pre-existing
-Wcomment warning.
This commit is contained in:
bigmerge
+5
-105
@@ -8638,6 +8638,7 @@ static char* engram_first_n_chars(const char* s, size_t n) {
|
||||
* mutation (node/edge create, forget) is mirrored through the store's
|
||||
* WAL-logged API so neuron.egm/neuron.wal stay authoritative.
|
||||
* ══════════════════════════════════════════════════════════════════════════ */
|
||||
#include "engram_text.h" /* text: tokenize, token hygiene, loss signature */
|
||||
#include "engram_store.h"
|
||||
#include "engram_vindex.h" /* M8: ANN (HNSW) index for activation seed selection */
|
||||
#include "engram_geometry.h" /* M9: centered relational-neighborhood geometry (priming) */
|
||||
@@ -9061,31 +9062,9 @@ el_val_t engram_node(el_val_t content, el_val_t node_type, el_val_t salience) {
|
||||
* RIGHT NOW" — which is the regression question, and the one that
|
||||
* would have caught this in a day instead of two months.
|
||||
*
|
||||
* SIGNATURE. Conservative on purpose — a false alarm that cries corruption
|
||||
* over ordinary punctuation is worse than useless. Two patterns, both of
|
||||
* which are essentially absent from well-formed English prose:
|
||||
* (a) alnum '?' alnum — "na?ve", "caf?s", "don?t". A real question mark
|
||||
* never sits between two word characters.
|
||||
* (b) ' ? ' followed by a lowercase letter — a lost em/en dash. A real
|
||||
* question mark is not preceded by a space, and
|
||||
* what follows one starts a new sentence.
|
||||
* Deliberately NOT flagged: a trailing '?' after a word, '? ' before a
|
||||
* capital, or '?' at end of string — all legitimate. This under-counts (it
|
||||
* cannot see a mangled 'café ' where the '?' landed before a space), so the
|
||||
* census is a floor on the damage, never an exaggeration of it. */
|
||||
static int eg_text_loss_signature(const char* s) {
|
||||
if (!s) return 0;
|
||||
for (const char* p = s; *p; p++) {
|
||||
if (*p != '?') continue;
|
||||
unsigned char prev = (p == s) ? 0 : (unsigned char)p[-1];
|
||||
unsigned char next = (unsigned char)p[1];
|
||||
/* (a) sandwiched between word characters. */
|
||||
if (isalnum(prev) && isalnum(next)) return 1;
|
||||
/* (b) spaced, with lowercase continuation — a lost dash. */
|
||||
if (prev == ' ' && next == ' ' && islower((unsigned char)p[2])) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
* The SIGNATURE itself (eg_text_loss_signature) moved to engram_text.c on
|
||||
* 2026-08-16 — it is plain C over <ctype.h> and touches nothing in here. The
|
||||
* stock/flow gauges below stay, because they touch store and EL value types. */
|
||||
|
||||
/* Damaged-node creations since process start. See the block comment above. */
|
||||
static int64_t _eg_txt_write_damaged = 0;
|
||||
@@ -9697,37 +9676,7 @@ static int istr_contains(const char* hay, const char* needle) {
|
||||
* fix landed but never reached this release runtime — the copy the engram
|
||||
* binary actually builds against.) */
|
||||
#define ENGRAM_MAX_QTOKENS 32
|
||||
#define ENGRAM_QTOK_LEN 256
|
||||
|
||||
/* Split q on whitespace into up to ENGRAM_MAX_QTOKENS distinct
|
||||
* (case-insensitive) tokens. Returns the token count. Over-long tokens are
|
||||
* truncated to ENGRAM_QTOK_LEN-1; over-count tokens are ignored. */
|
||||
static int engram_tokenize_query(const char* q,
|
||||
char toks[][ENGRAM_QTOK_LEN], int maxtok) {
|
||||
int n = 0;
|
||||
if (!q) return 0;
|
||||
const char* p = q;
|
||||
while (*p && n < maxtok) {
|
||||
while (*p && isspace((unsigned char)*p)) p++;
|
||||
if (!*p) break;
|
||||
char buf[ENGRAM_QTOK_LEN];
|
||||
size_t tl = 0;
|
||||
while (*p && !isspace((unsigned char)*p)) {
|
||||
if (tl < sizeof(buf) - 1) buf[tl++] = *p;
|
||||
p++;
|
||||
}
|
||||
buf[tl] = '\0';
|
||||
if (tl == 0) continue;
|
||||
int dup = 0;
|
||||
for (int s = 0; s < n; s++) {
|
||||
if (strcasecmp(toks[s], buf) == 0) { dup = 1; break; }
|
||||
}
|
||||
if (dup) continue;
|
||||
memcpy(toks[n], buf, tl + 1);
|
||||
n++;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
/* ENGRAM_QTOK_LEN and engram_tokenize_query moved to engram_text.h/.c. */
|
||||
|
||||
/* Count how many of the ntok distinct query tokens appear (case-insensitive)
|
||||
* in the node's content, label, or tags. 0 == no match. */
|
||||
@@ -16389,29 +16338,6 @@ el_val_t engram_label_df(el_val_t term) {
|
||||
#define ENGRAM_ST_TOKLEN 64
|
||||
#define ENGRAM_ST_SCANCHARS 400
|
||||
|
||||
/* Trim leading/trailing non-alphanumerics, then accept only tokens whose core
|
||||
* is alphanumeric plus '-' and '_' with at least 3 letters. This subsumes the
|
||||
* quoted-title guard (2026-07-25) and the "<!--" flood (2026-08-03)
|
||||
* structurally: markup and punctuation-bearing tokens never become
|
||||
* candidates, rather than being blocklisted after the fact. */
|
||||
static int eg_st_clean_token(const char* raw, size_t rawlen,
|
||||
char* out, size_t outcap) {
|
||||
size_t s = 0, e = rawlen;
|
||||
while (s < e && !isalnum((unsigned char)raw[s])) s++;
|
||||
while (e > s && !isalnum((unsigned char)raw[e - 1])) e--;
|
||||
size_t len = e - s;
|
||||
if (len < 4 || len >= outcap) return 0;
|
||||
int alpha = 0;
|
||||
for (size_t i = 0; i < len; i++) {
|
||||
unsigned char c = (unsigned char)raw[s + i];
|
||||
if (isalpha(c)) alpha++;
|
||||
else if (!isdigit(c) && c != '-' && c != '_') return 0;
|
||||
}
|
||||
if (alpha < 3) return 0;
|
||||
memcpy(out, raw + s, len);
|
||||
out[len] = '\0';
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* ENGRAM_ST_DEBUG=1 dumps the full scored candidate set to stderr. One
|
||||
* cached branch in production. This exists because the first live run of this
|
||||
@@ -16424,32 +16350,6 @@ static int _eg_st_debug(void) {
|
||||
return v;
|
||||
}
|
||||
|
||||
/* Word-boundary document frequency. engram_label_df uses istr_contains, i.e.
|
||||
* SUBSTRING matching, and that is the wrong estimator for term specificity on
|
||||
* short tokens: "them" hits inside "theme" and "anthem", "about" and "whole"
|
||||
* come back with df 2 and 1 rather than 0. That matters here specifically
|
||||
* because the min_df floor is what rejects English function words, and it can
|
||||
* only do that job if their df is honestly zero. Substring df quietly handed
|
||||
* them a survival ticket. Measured on the live store before this fix, "whole"
|
||||
* (df=1, idf=8.76) and "about" (df=2, idf=8.36) were outscoring real topical
|
||||
* terms and losing only on position — one node whose text happened to open
|
||||
* with a function word would have seeded on it.
|
||||
*
|
||||
* engram_label_df keeps substring semantics: it is a separate published
|
||||
* measure with existing callers, and changing it underneath them is not this
|
||||
* change's business. */
|
||||
static int eg_st_label_has_word(const char* hay, const char* word) {
|
||||
size_t wl = strlen(word);
|
||||
for (const char* p = hay; *p; p++) {
|
||||
if (strncasecmp(p, word, wl) != 0) continue;
|
||||
char before = (p == hay) ? '\0' : p[-1];
|
||||
char after = p[wl];
|
||||
if (before && (isalnum((unsigned char)before) || before == '_')) continue;
|
||||
if (after && (isalnum((unsigned char)after) || after == '_')) continue;
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* YAKE T_Case, adapted to this corpus. YAKE up-weights all-caps tokens
|
||||
* because in ordinary prose an acronym is rare and carries topic. That
|
||||
|
||||
Reference in New Issue
Block a user