feat(engram): word-start match primitive + corpus-vocabulary gate on recall
The retrieval match test is a raw substring scan, so a query token matches anywhere INSIDE a corpus word: "throom" matches "bathroom". Measured over the 38-query gold set on this corpus that is not a rare accident - q28's lexical leg is 36,954 records of which only 13 contain a query token at a word start (99.96% mid-word noise), six further queries carry ~20,500 mid-word-only records each, and the nonsense control q35 returns 7 records ALL of which match only mid-word. istr_contains_wordstart() anchors a token to a word start (preceding char not alphanumeric) while still matching suffixes, so "value" still hits "values". That empties the lexical leg for gibberish, and the nhits==0 corpus-vocabulary gate (iteration 6's mechanism, feat/claim24-unfloored-semantic) then makes the whole query decline rather than let the semantic leg answer it. Measured vs feat/bm25-lexical-leg on the embedded corpus, 2 runs each, 0 queries of run-to-run drift on both sides: net +1 (nonsense:q35), 0 losses, McNemar p=1.0 -> NOT-SHOWN (floor is 6) nonsense clean 2/3 -> 3/3; exact_rare 100%, phrase 100%, paraphrase 61.5%, associative 66.7%, superseded 2/3 all UNCHANGED latency p50 1184 -> 543 ms (0.46x) Iteration 6 called q35 "a DEFECTIVE CONTROL ... cannot be cleaned without breaking the lexical leg". It can: the defect was the match primitive, and cleaning it cost nothing. Also committed: results-wsclaim24.json + cmp-nogate.json, a measured negative for bundling the claim-24 unfloored semantic leg on top (gains q14/q25, breaks q15/q28/q33/q34, net -2) - it independently reproduces iteration 6's q15/q28 losses and shows unflooring REQUIRES the vocabulary gate. Reproducers: legs.py (leg-level replica, reproduces baseline hit@5 exactly on all 38 queries), policy2.py, ceiling.py, wb2.py.
This commit is contained in:
+52
-7
@@ -7327,6 +7327,39 @@ static int istr_contains(const char* hay, const char* needle) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Word-START-anchored variant of istr_contains.
|
||||
*
|
||||
* WHY. The retrieval match primitive is a raw substring test, so a query token
|
||||
* matches ANYWHERE inside a corpus word: "throom" matches "bathroom", "cat"
|
||||
* matches "concatenate". Measured on this corpus over the 38-query gold set,
|
||||
* that is not a rare accident — it is the bulk of some queries' candidate
|
||||
* sets. q28's lexical leg is 36,954 records of which only 13 contain a query
|
||||
* token at a word start (99.96% mid-word noise); six other queries carry
|
||||
* ~20,500 mid-word-only records each; and the nonsense control q35
|
||||
* ("xxqzzt vurblenacht throom") returns 7 records ALL of which match only
|
||||
* mid-word, which is the entire reason that control has been dirty since main.
|
||||
*
|
||||
* WHAT CHANGES. A token must begin at a word boundary — the preceding
|
||||
* character is not alphanumeric. Suffixes are still matched ("value" still
|
||||
* hits "values", "unjailbreakable" still hits "unjailbreakables"), so this is
|
||||
* strictly a prefix anchor, not whole-word equality; whole-word equality would
|
||||
* break the morphological matching the phrase category depends on.
|
||||
*
|
||||
* PROVENANCE, stated honestly: this restores no engram claim. Will's design
|
||||
* has no lexical leg at all (05-detailed-description l.64 takes "one or more
|
||||
* seed node UUIDs representing the current active context" as its input), so
|
||||
* the lexical leg is the seed-finding step that feeds the designed mechanism.
|
||||
* Cleaner seeds serve that mechanism; they do not replace it. */
|
||||
static int istr_contains_wordstart(const char* hay, const char* needle) {
|
||||
if (!hay || !needle || !*needle) return 0;
|
||||
size_t nl = strlen(needle);
|
||||
for (const char* p = hay; *p; p++) {
|
||||
if (p != hay && isalnum((unsigned char)p[-1])) continue;
|
||||
if (strncasecmp(p, needle, nl) == 0) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* ── Tokenized query matching ───────────────────────────────────────────
|
||||
* The engram query surface (search / activate / goal-bias) historically
|
||||
* matched the ENTIRE raw query string as a single case-insensitive
|
||||
@@ -7379,9 +7412,9 @@ static int engram_node_match_score(const EngramNode* n,
|
||||
char toks[][ENGRAM_QTOK_LEN], int ntok) {
|
||||
int score = 0;
|
||||
for (int t = 0; t < ntok; t++) {
|
||||
if (istr_contains(n->content, toks[t]) ||
|
||||
istr_contains(n->label, toks[t]) ||
|
||||
istr_contains(n->tags, toks[t]))
|
||||
if (istr_contains_wordstart(n->content, toks[t]) ||
|
||||
istr_contains_wordstart(n->label, toks[t]) ||
|
||||
istr_contains_wordstart(n->tags, toks[t]))
|
||||
score++;
|
||||
}
|
||||
return score;
|
||||
@@ -7396,9 +7429,9 @@ static uint32_t engram_node_match_mask(const EngramNode* n,
|
||||
char toks[][ENGRAM_QTOK_LEN], int ntok) {
|
||||
uint32_t m = 0;
|
||||
for (int t = 0; t < ntok && t < 32; t++) {
|
||||
if (istr_contains(n->content, toks[t]) ||
|
||||
istr_contains(n->label, toks[t]) ||
|
||||
istr_contains(n->tags, toks[t]))
|
||||
if (istr_contains_wordstart(n->content, toks[t]) ||
|
||||
istr_contains_wordstart(n->label, toks[t]) ||
|
||||
istr_contains_wordstart(n->tags, toks[t]))
|
||||
m |= (uint32_t)1u << t;
|
||||
}
|
||||
return m;
|
||||
@@ -9712,7 +9745,19 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) {
|
||||
? engram_assoc_leg(g, hits, nhits, semseed, nsemseed,
|
||||
qv, qdim, assoc, ENGRAM_ASSOC_MAX)
|
||||
: 0;
|
||||
int64_t* order = malloc((size_t)lim * sizeof(int64_t));
|
||||
/* Corpus-vocabulary gate. If no stored record contains ANY
|
||||
* query token in its content, label or tags, the query is
|
||||
* outside this graph's vocabulary: there are no seeds, and
|
||||
* 05-detailed-description l.64 makes retrieval downstream of
|
||||
* seeds ("the caller provides one or more seed node UUIDs
|
||||
* representing the current active context"). No seeds, no
|
||||
* retrieval — the graph declines rather than confabulating a
|
||||
* nearest neighbour for gibberish. The mechanism is iteration
|
||||
* 6's (feat/claim24-unfloored-semantic); it is required here
|
||||
* because word-start matching empties the lexical leg for
|
||||
* q35-style queries whose only "hits" were mid-word, and the
|
||||
* semantic leg would otherwise answer them anyway. */
|
||||
int64_t* order = (nhits > 0) ? malloc((size_t)lim * sizeof(int64_t)) : NULL;
|
||||
if (order) {
|
||||
int64_t no = engram_interleave3(hits, nhits, sem, nsem,
|
||||
assoc, nassoc, lim, order);
|
||||
|
||||
Reference in New Issue
Block a user