feat(engram): word-start match primitive + corpus-vocabulary gate on recall

The retrieval match test is a raw substring scan, so a query token matches
anywhere INSIDE a corpus word: "throom" matches "bathroom". Measured over the
38-query gold set on this corpus that is not a rare accident - q28's lexical
leg is 36,954 records of which only 13 contain a query token at a word start
(99.96% mid-word noise), six further queries carry ~20,500 mid-word-only
records each, and the nonsense control q35 returns 7 records ALL of which
match only mid-word.

istr_contains_wordstart() anchors a token to a word start (preceding char not
alphanumeric) while still matching suffixes, so "value" still hits "values".
That empties the lexical leg for gibberish, and the nhits==0 corpus-vocabulary
gate (iteration 6's mechanism, feat/claim24-unfloored-semantic) then makes the
whole query decline rather than let the semantic leg answer it.

Measured vs feat/bm25-lexical-leg on the embedded corpus, 2 runs each,
0 queries of run-to-run drift on both sides:
  net +1 (nonsense:q35), 0 losses, McNemar p=1.0 -> NOT-SHOWN (floor is 6)
  nonsense clean 2/3 -> 3/3; exact_rare 100%, phrase 100%, paraphrase 61.5%,
  associative 66.7%, superseded 2/3 all UNCHANGED
  latency p50 1184 -> 543 ms (0.46x)

Iteration 6 called q35 "a DEFECTIVE CONTROL ... cannot be cleaned without
breaking the lexical leg". It can: the defect was the match primitive, and
cleaning it cost nothing.

Also committed: results-wsclaim24.json + cmp-nogate.json, a measured negative
for bundling the claim-24 unfloored semantic leg on top (gains q14/q25, breaks
q15/q28/q33/q34, net -2) - it independently reproduces iteration 6's q15/q28
losses and shows unflooring REQUIRES the vocabulary gate.

Reproducers: legs.py (leg-level replica, reproduces baseline hit@5 exactly on
all 38 queries), policy2.py, ceiling.py, wb2.py.
This commit is contained in:
Tim Lingo
2026-08-07 16:58:34 -05:00
parent 55f9ee3cb0
commit 4eb4c9e287
10 changed files with 3618 additions and 7 deletions
+52 -7
View File
@@ -7327,6 +7327,39 @@ static int istr_contains(const char* hay, const char* needle) {
return 0;
}
/* Word-START-anchored variant of istr_contains.
*
* WHY. The retrieval match primitive is a raw substring test, so a query token
* matches ANYWHERE inside a corpus word: "throom" matches "bathroom", "cat"
* matches "concatenate". Measured on this corpus over the 38-query gold set,
* that is not a rare accident it is the bulk of some queries' candidate
* sets. q28's lexical leg is 36,954 records of which only 13 contain a query
* token at a word start (99.96% mid-word noise); six other queries carry
* ~20,500 mid-word-only records each; and the nonsense control q35
* ("xxqzzt vurblenacht throom") returns 7 records ALL of which match only
* mid-word, which is the entire reason that control has been dirty since main.
*
* WHAT CHANGES. A token must begin at a word boundary the preceding
* character is not alphanumeric. Suffixes are still matched ("value" still
* hits "values", "unjailbreakable" still hits "unjailbreakables"), so this is
* strictly a prefix anchor, not whole-word equality; whole-word equality would
* break the morphological matching the phrase category depends on.
*
* PROVENANCE, stated honestly: this restores no engram claim. Will's design
* has no lexical leg at all (05-detailed-description l.64 takes "one or more
* seed node UUIDs representing the current active context" as its input), so
* the lexical leg is the seed-finding step that feeds the designed mechanism.
* Cleaner seeds serve that mechanism; they do not replace it. */
static int istr_contains_wordstart(const char* hay, const char* needle) {
if (!hay || !needle || !*needle) return 0;
size_t nl = strlen(needle);
for (const char* p = hay; *p; p++) {
if (p != hay && isalnum((unsigned char)p[-1])) continue;
if (strncasecmp(p, needle, nl) == 0) return 1;
}
return 0;
}
/* ── Tokenized query matching ───────────────────────────────────────────
* The engram query surface (search / activate / goal-bias) historically
* matched the ENTIRE raw query string as a single case-insensitive
@@ -7379,9 +7412,9 @@ static int engram_node_match_score(const EngramNode* n,
char toks[][ENGRAM_QTOK_LEN], int ntok) {
int score = 0;
for (int t = 0; t < ntok; t++) {
if (istr_contains(n->content, toks[t]) ||
istr_contains(n->label, toks[t]) ||
istr_contains(n->tags, toks[t]))
if (istr_contains_wordstart(n->content, toks[t]) ||
istr_contains_wordstart(n->label, toks[t]) ||
istr_contains_wordstart(n->tags, toks[t]))
score++;
}
return score;
@@ -7396,9 +7429,9 @@ static uint32_t engram_node_match_mask(const EngramNode* n,
char toks[][ENGRAM_QTOK_LEN], int ntok) {
uint32_t m = 0;
for (int t = 0; t < ntok && t < 32; t++) {
if (istr_contains(n->content, toks[t]) ||
istr_contains(n->label, toks[t]) ||
istr_contains(n->tags, toks[t]))
if (istr_contains_wordstart(n->content, toks[t]) ||
istr_contains_wordstart(n->label, toks[t]) ||
istr_contains_wordstart(n->tags, toks[t]))
m |= (uint32_t)1u << t;
}
return m;
@@ -9712,7 +9745,19 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) {
? engram_assoc_leg(g, hits, nhits, semseed, nsemseed,
qv, qdim, assoc, ENGRAM_ASSOC_MAX)
: 0;
int64_t* order = malloc((size_t)lim * sizeof(int64_t));
/* Corpus-vocabulary gate. If no stored record contains ANY
* query token in its content, label or tags, the query is
* outside this graph's vocabulary: there are no seeds, and
* 05-detailed-description l.64 makes retrieval downstream of
* seeds ("the caller provides one or more seed node UUIDs
* representing the current active context"). No seeds, no
* retrieval the graph declines rather than confabulating a
* nearest neighbour for gibberish. The mechanism is iteration
* 6's (feat/claim24-unfloored-semantic); it is required here
* because word-start matching empties the lexical leg for
* q35-style queries whose only "hits" were mid-word, and the
* semantic leg would otherwise answer them anyway. */
int64_t* order = (nhits > 0) ? malloc((size_t)lim * sizeof(int64_t)) : NULL;
if (order) {
int64_t no = engram_interleave3(hits, nhits, sem, nsem,
assoc, nassoc, lim, order);