engram: native set-based reframe_region + engine hardening (embed-gap, lazy cosine, vindex harvest) #109
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: neuron-technologies/el#109
Reference in New Issue
Block a user
Delete Branch "feat/reframe-region-setop"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Preserving already-committed and uncommitted work from an open worktree during a worktree audit. Adds native set-based reframe_region on the cognition engine, decorator-as-seam @route codegen port, teacher-summon (default-off, byte-inert when unset), plus geometry ops + primitive agentic tools over the one geometry (6 commits). On top: fixes lazy-embed index gap #20 (O(1) seen bitmap so lazily-embedded older nodes are picked up incrementally, not only on full rebuild), replaces engram_activate's O(N*D) cosine prescan with a lazy-memoized cosine cache (proven bit-identical), and extracts a vindex_harvest_from_store primitive with a new brute-force-vs-HNSW bench/oracle harness (vindex_bench.c). See git log on this branch for full commit-by-commit detail.
auto_term_empty_streak — the counter the 2026-08-06 review added to catch exactly this — read 50 and climbing. Fifty consecutive curiosity scans where the soul's dynamic seeding produced nothing and the loop fell back to four hardcoded phrases. The live WM top said why in one look: every slot was a Memory node labelled "memory:remembered". The extractor read the LABEL only, the sentinel guard correctly rejects sentinels, so there was never anything to extract. It was written against Knowledge nodes, which have real titles, and was structurally blind to the node type that dominates working memory. Rather than add a sixth guard to the five that accumulated across four reviews (genre words, quoted titles, stopwords, label-df), invert the algorithm. The old one was: take the first word, then check whether it is acceptable. That shape forces quality to be expressed as rejection, and rejection can only ever encode floods that already happened. engram_salient_term() scores EVERY candidate token and returns the argmax of idf · position · casing (YAKE, Campos et al. 2020, with real corpus IDF substituted for YAKE's corpus-free proxies), falling back from a sentinel label to the node's content. Term quality becomes the selection criterion instead of a veto: a bad token loses to a better token in the same text without needing to be on any list. Tabu is applied during the argmax, so inhibition-of-return costs seed quality rather than costing the whole scan. Two defects found by instrumenting rather than assuming, which is the lesson this codebase keeps relearning: - The first live run returned five ALL-CAPS terms in a row. Memory content conventionally opens with an all-caps header, so YAKE's acronym bonus was handing the seed to whatever word the heading started with. Restricted to tokens <= 5 chars, where all-caps is evidence of an acronym rather than evidence of a heading. Long headers now compete on specificity. - df via istr_contains is substring matching, so "them" hit inside "theme" and function words came back with nonzero df. Added word-boundary df locally; engram_label_df keeps substring semantics for its callers. An earlier draft claimed the min_df floor subsumed the 73 stopwords that 08-03 measured label-df as missing. Re-measured: about:2, whole:1, them:2 — they clear a floor of 1. The claim was false and the comment now records the correction. The floor buys lexical reachability; the argmax buys quality; the stopword list still earns its keep. Measured on 60 live Memory nodes before shipping: 0 empty, versus 60 of 60 under the old extractor. Terms are topical — HEBBIAN, CONSOLIDATION, TEMPORAL, crash-loop, PRIMING, NEIGHBORHOOD, DRIFT. Three of sixty are weak header words; left alone deliberately, because listing them is the move that produced four blocklists. ENGRAM_ST_DEBUG=1 dumps the scored candidate set. It exists because there was no way to see whether the all-caps run was the corpus or the casing weight without guessing.Two changes to the activation path, both grounded in measurement on the live store rather than on the spec. 1. Rescale cosine before the query gate. The propagation gate (arXiv:2606.30133, added in an earlier review) fed RAW cosine into FLOOR + (1-FLOOR)*c. Raw cosine from nomic-embed is compressed into a narrow high band, so that expression is close to a constant. Measured, 400 random UNRELATED node pairs on the live store: median 0.562, central 98% span [0.381, 0.743] So a node with no semantic relation to the query was propagating at 0.25 + 0.75*0.562 = 0.67. Two thirds strength. The gate was a small tax. Fixed by shifting and flooring about ENGRAM_EMBED_S0 -- which is already in this file, already 0.45, and already used exactly this way by the Pass-2 WM term. The propagation gate simply never used it. Same 400 pairs after: median unrelated pair falls to 0.40, top of range preserved (0.85 vs 0.92), gate spread widens 0.42 -> 0.60. Only 8.5% reach the floor, so dissimilar lexical/structural pathways are damped, never severed. Range is unchanged at [0.25, 1.0], and cosq == NULL still degrades to no gating at all. 2. Decompose the WM eviction counter by cause. _eg_act_wm_evicted was incremented from six sites with four distinct causes and collapsed all of them into one integer. Today's review measured 175,547 evictions over 13.5h (~216/min against 24 slots) and could not tell healthy rotation from cap thrashing from duplicate churn. That is this file's most-repeated defect: dup_wm and dup_wm_global exist only because the aggregate could not answer "why" during the 08-02 and 08-06 incidents. Each of those needed a NEW gauge before it was diagnosable. evict_floor / evict_cap / evict_bll complete the decomposition, so wm_evicted == floor + cap + bll + dup_wm + dup_wm_global holds as an identity and each term implies a different correction. Verified on an isolated instance: 30 nodes, 24 filled the cap, wm_evicted 6 == evict_cap 6, all other terms 0. Built and smoke-tested out of tree. The live daemon runs a pinned binary and was deliberately not restarted -- the store compaction workstream is in flight.The buildable form of the "one operation" theory (memory bdc8a488). Maps the theory onto what is already compiled: the five reasoning operators in engram_reason.c already collapse onto ONE primitive — engram_reason_point_fit — plus the geo-algebra (combine/subtract/analogy-rotate/distance), and engram_verify.c is built on the same fit. So the operator-collapse is already half-written; what is missing is not the primitive. What is missing, and what this doc specifies: - think(anchor, prior) -> gradient (a distribution/direction, not a point); each named faculty = {point_fit + a prior}, the operation frozen, the prior learned. - Prior as a first-class stored node (warp + calibration), superseding the intrinsic importance/salience scalar with a relational, grounded-for-whom edge. Confirmed against the runtime: importance is already a live activation computation (el_runtime.c:13013), never trusted as a static field. - vantage_read(anchor, aperture) — one op, three settings: self / foreign-field / veil. - The reflexive correspondence-loop as the learning engine: move the grounding check from offline Python into the geometry, reflexive, reusing the DORMANT verifier (engram_verify_grounding has no runtime caller and no El binding today) turned inward. grounding = learning = one loop. - hold/ground/assert kept distinct: the engram holds anything, grounding is an edge, the honesty floor is on assertion only; ungrounded content is first-class. - metastability: keystone core (read-mostly priors) + plastic everything else. Seven staged milestones, earliest is a real end-to-end slice (induction as {primitive + grounded prior} with the loop closing on it, reboot-proven on a snapshot). Build rails stated: offline/secondary, snapshot-first, reboot-prove, zero-loss, gated launchctl cutover. Design only; no code changed this pass.