bigmerge
678dac5efc
First concern moved out of el_runtime.c under the ratchet, and the move is
deliberately small: it exists to prove the mechanism end to end before anything
large depends on it.
engram_text.{c,h} — query tokenization, candidate-token hygiene, word-boundary
matching, and the text-damage signature. Four functions, moved verbatim; only
`static` was dropped and each doc comment travelled with the code. They touch no
EL value type and no engram store type: plain C over <ctype.h>/<string.h> over
char buffers. They were never el_runtime.c's business.
el_runtime.c 20,527 -> 20,427 lines (BUDGET max_lines ratcheted down)
engram fns 279 -> 275 (BUDGET max_engram_fns ratcheted down)
The Stage 1 extension point worked as designed: adding the file to
lang/runtime/SOURCES was one line, and every build path picked it up. The
Stage 2 drift guard then caught that I had NOT added it to install.sh's
standalone list — the exact class of drift it was written for, on its first
real change, before the commit rather than after a broken SDK shipped.
WHY ONLY 100 LINES, AND WHAT ACTUALLY BLOCKS THE REST
Measured, not estimated: of 273 engram-domain functions in el_runtime.c
(~9,700 lines), only 75 (~1,058 lines) can move today, and they are scattered
rather than clustered. The blocker is a single fact:
EngramNode, EngramEdge, EngramStore, EngramLayer, EngramWal and EngramIdSlot
are typedef'd INSIDE el_runtime.c. No sibling can see them. engram_store.h
defines a SEPARATE serializable "node view" struct and maps between the two.
So every engram function that takes an EngramNode* — which is most of them, 109
of 273 by direct type reference — cannot compile in engram_store.c until those
types move to a shared header. That extraction is the real Stage 3 enabler and
it deserves its own change: it touches the most load-bearing struct in the
system, and doing it in the same commit as a code move would make a regression
impossible to bisect.
REPAIRED: 10 engram harnesses that had silently stopped linking
Not new breakage from this move — verified against unmodified dev, where
el_runtime.c + engram_store.c alone already failed with undefined symbols.
They had been dead for as long as el_runtime.c has been calling into the
siblings, and nothing noticed because nothing ran them.
run_m3_parity, run_m7_traversal, run_m35_hebb_persist,
run_interoception_p0..p5 — now build from $(scripts/el-runtime-sources.sh)
run_wal_tests — its two TUs #include "el_runtime.c" directly, so
it links the SIBLINGS ONLY; adding el_runtime.c
to that link line would define every symbol twice
(That #include'd .c is worth recording: the runtime does have one, in
engram/test/test_wal.c and the generated test_failloud.c.)
Verified locally — every one of these was run, not assumed:
* m3_parity ............ PASS, incl. ASan+UBSan clean across seed/on/reboot
* m7_traversal ......... PASS
* m35_hebb_persist ..... PASS (the gate over the original prod hebb bug)
* interoception p0..p5 . PASS (all six)
* wal_tests ............ 66 passed, 0 failed, + fail-loud exit check
* self-host fixpoint ... byte-identical, AND the emitted C is byte-identical
to the pre-move compiler output — the move changes
nothing the compiler produces
* engram/src/server.el . compiles and links
* native suites ........ 8 of 13, unchanged from before the move; the same 5
pre-existing failures, no regression
* both runtime guards .. green at the new, lower budget
Also fixes a block comment left unterminated by the extraction (the deleted
range carried its closing */), restoring the compile to its single pre-existing
-Wcomment warning.
67 lines
3.1 KiB
C
67 lines
3.1 KiB
C
/* engram_text.h — text handling for the engram: query tokenization, candidate
|
|
* token hygiene, word-boundary matching, and the text-damage signature.
|
|
*
|
|
* WHY THIS FILE EXISTS
|
|
* --------------------
|
|
* These functions lived in el_runtime.c, which is a 2026-05-03 build shim that
|
|
* was scheduled for deletion, never retired, and grew to 20,527 lines. They do
|
|
* not belong there: they touch no EL value type and no engram store type. They
|
|
* are plain C over <ctype.h>/<string.h> operating on char buffers, and they are
|
|
* a concern of their own — so they get a translation unit of their own.
|
|
*
|
|
* Adding a new text helper? Add it HERE, not to el_runtime.c. A new .c costs
|
|
* exactly one line in lang/runtime/SOURCES, and every build path picks it up.
|
|
* Placement is a LINK-TIME concern: the compiler cannot tell which .c a symbol
|
|
* came from (builtin_arity is an arity guard, not a dispatch table), so a
|
|
* function defined here is exactly as linkable as one defined in el_runtime.c.
|
|
*/
|
|
#ifndef ENGRAM_TEXT_H
|
|
#define ENGRAM_TEXT_H
|
|
|
|
#include <stddef.h>
|
|
|
|
#ifdef __cplusplus
|
|
extern "C" {
|
|
#endif
|
|
|
|
/* Max bytes per query token, including the NUL. */
|
|
#define ENGRAM_QTOK_LEN 256
|
|
|
|
/* Split q on whitespace into up to ENGRAM_MAX_QTOKENS distinct
|
|
* (case-insensitive) tokens. Returns the token count. Over-long tokens are
|
|
* truncated to ENGRAM_QTOK_LEN-1; over-count tokens are ignored. */
|
|
int engram_tokenize_query(const char* q, char toks[][ENGRAM_QTOK_LEN], int maxtok);
|
|
|
|
/* Trim leading/trailing non-alphanumerics, then accept only tokens whose core
|
|
* is alphanumeric plus '-' and '_' with at least 3 letters. This subsumes the
|
|
* quoted-title guard (2026-07-25) and the "<!--" flood (2026-08-03)
|
|
* structurally: markup and punctuation-bearing tokens never become
|
|
* candidates, rather than being blocklisted after the fact. */
|
|
int eg_st_clean_token(const char* raw, size_t rawlen, char* out, size_t outcap);
|
|
|
|
/* Word-boundary document frequency. engram_label_df uses istr_contains, i.e.
|
|
* SUBSTRING matching, and that is the wrong estimator for term specificity on
|
|
* short tokens: "them" hits inside "theme" and "anthem", "about" and "whole"
|
|
* come back with df 2 and 1 rather than 0. That matters here specifically
|
|
* because the min_df floor is what rejects English function words, and it can
|
|
* only do that job if their df is honestly zero. Substring df quietly handed
|
|
* them a survival ticket. Measured on the live store before this fix, "whole"
|
|
* (df=1, idf=8.76) and "about" (df=2, idf=8.36) were outscoring real topical
|
|
* terms and losing only on position — one node whose text happened to open
|
|
* with a function word would have seeded on it.
|
|
*
|
|
* engram_label_df keeps substring semantics: it is a separate published
|
|
* measure with existing callers, and changing it underneath them is not this
|
|
* change's business. */
|
|
int eg_st_label_has_word(const char* hay, const char* word);
|
|
|
|
/* Whether s carries the text-loss signature left by the \uXXXX -> '?' parser
|
|
* defect. Conservative by design; see engram_text.c for the full rationale. */
|
|
int eg_text_loss_signature(const char* s);
|
|
|
|
#ifdef __cplusplus
|
|
}
|
|
#endif
|
|
|
|
#endif /* ENGRAM_TEXT_H */
|