diff --git a/engram/spec/cognitive-architecture.design.md b/engram/spec/cognitive-architecture.design.md new file mode 100644 index 0000000..22c8d3e --- /dev/null +++ b/engram/spec/cognitive-architecture.design.md @@ -0,0 +1,605 @@ +# Cognitive Architecture — Design Doc + +**The buildable form of the "one operation" theory of cognition.** + +Status: DESIGN. Nothing here is built yet except where explicitly marked +"EXISTS" against a cited C symbol. A build agent executes from this doc. +Offline design only — this pass changes no code. + +Source of theory: Neuron memory `bdc8a488-146d-4ccb-a5c8-d8c0a008534e`. +Source of existing engram substrate (cited throughout): the runtime on branch +`feat/self-reification-20260814` — +`lang/runtime/engram_reason.{c,h}`, `engram_verify.{c,h}`, +`engram_geometry.{c,h}`, `engram_store.{c,h}`, plus the reification beat and the +RAM activation graph compiled into `~/.neuron/bin/engram`. + +--- + +## 0. The claim, stated plainly + +Cognition is **one operation**, not eight. The named faculties — +deduce / abduce / analogy / induce / causal / plan / predict / perspective — +are human *labels* on regions of a single operation's steering space. They are +not separately invoked and not separately implemented. The operation is: + +> **think** = a directed traversal of the geometry from an *anchor*, steered by +> a *prior*, whose output is a **gradient** (a distribution / direction over the +> geometry), never a point. Collapse-to-a-point happens only at expression. + +Three things follow, and they are the whole design: + +1. **The operator collapse is already half-written in C.** The five reasoning + operators in `engram_reason.c` already compose over *one* shared primitive — + `engram_reason_point_fit` — plus a small geo-algebra + (combine / subtract / analogy-rotate / distance). The verifier + (`engram_verify.c`) is built on the same `point_fit`. What is missing is not + the primitive; it is (a) making the *prior* a first-class learnable object + instead of a hard-coded parameter, and (b) closing the learning loop. + +2. **Grounding = learning = the same loop.** "Getting better" at any faculty is + not changing the operation. It is *calibrating the steering-prior against + outcomes*. Code freezes; priors grow. The correspondence-check that today + lives offline (Python, the grounding-floor + differential-drop governor, "#43") + must move **into the geometry, reflexive** — think scoring its own gradient + against outcome and refining the prior on the error. That reflexive + correspondence-loop *is* the learning engine and is the core unbuilt thing. + +3. **The ungrounded is primary.** The engram *holds* anything unconditionally. + Grounding is a *relation* (an edge, grounded-for-whom), not a gate. The + honesty floor applies only to **assertion**. A fully-grounded mind is dead; + the ungrounded is both the fuel (raw material for grounding) and the pull + (curiosity = leaning toward one's own ungrounded regions). + +Everything below makes these concrete and buildable, and defines what +"completion" means, staged so the first milestone is a real end-to-end slice. + +--- + +## 1. THE ONE OPERATION — `think` + +### 1.1 Signature + +``` +think(anchor, prior, aperture?) -> gradient +``` + +- **anchor** — a location to traverse *from*. Either a node id (re-origin on that + node's descriptor) or a raw point `x ∈ R^dim` (a query embedding). The anchor + fixes the frame; every read is *from a vantage*, never view-from-nowhere. +- **prior** — a learnable bias/direction over the geometry that *steers* the + traversal (§2). A prior is a first-class stored object, not a call argument + baked into C. +- **aperture** — optional read-width / veil / field-selector (§3). Absent = + self-mode full aperture. +- **gradient** — the output. A `GeoGradient`: a direction + a spread over the + geometry, *plus* the read neighborhood it was computed against. Not a point. + A spiked gradient = "exact" (deduction); a spread gradient = "fuzzy" + (prediction). The gradient is *also the next steering direction* — cognition + is a flow down a prior-shaped landscape, closed-loop. + +```c +/* NEW. The output type. */ +typedef struct { + int dim; + float* direction; /* unit steering vector in the anchor's frame */ + double spread; /* 0 = spiked/exact ... large = diffuse/fuzzy */ + double confidence; /* calibrated, from the prior's track record */ + /* the read it was computed over (borrowed from the vantage-read) */ + const char* anchor_id; + int n_support; /* neighborhood members that shaped it */ + /* provenance for the reflexive loop (§4) */ + const char* prior_id; /* which prior steered this */ +} GeoGradient; +``` + +### 1.2 Semantics + +`think` is a fixed, frozen procedure over three steps: + +1. **Re-origin** on `anchor` → a centered `GeoDescriptor` for its + salience/recency-weighted neighborhood (the vantage-read, §3). + *EXISTS as substrate:* descriptor construction + the persisted reified + neighborhoods (`engram_geo_reify_lookup`, `GeoNeighborhood`) and the + centered-frame machinery (`GeoDescriptor.global_mean`, + `engram_geo_mean_*`). +2. **Fit under the prior** — evaluate the anchor's residual against the local + manifold *warped by the prior*. This is `engram_reason_point_fit` with the + prior applied to the axes/extents (§2.3). + *EXISTS (unwarped):* `engram_reason_point_fit(g, x, ext_floor, &GeoFit)` — + returns `mahalanobis`, `ortho_residual`, `distance`, `score`. +3. **Emit a gradient**, not a decision — direction = the prior-steered descent + in fit-space; spread = from the fit's `distance`/`ortho_residual`; + confidence = the prior's calibrated reliability (§4). Collapse to a point is + a *separate, downstream* faculty operation (sample the gradient → surface an + expression), never part of `think`. + +### 1.3 Each named operator = {this primitive + a prior} + +The C already demonstrates the collapse: every operator below reduces to +`point_fit` + geo-algebra. The design's move is to replace the operator's +*hard-coded parameters* with a **named prior** — same math, learnable steering. + +| Faculty | Existing C (EXISTS) | = primitive + prior | +|---|---|---| +| **Membership / classify** | `engram_reason_membership` → `point_fit(rule, x)` | `point_fit` + the *induced-rule* prior (learned extents) | +| **Induction** | `engram_reason_induce` (fold via `engram_geo_combine`) → produces a `GeoInduction.rule` + `ext_floor` | `point_fit` + a prior that *is* the pooled rule; refined by §4 | +| **Abduction** | `engram_reason_abduce` — ranks hypotheses by `point_fit(h, obs)` | `point_fit` + a prior over hypothesis-prior-probability (currently uniform) | +| **Analogy** | `engram_reason_analogy` — Procrustes rotate `engram_geo_analogy` + `apply`, nearest mapped point | analogy-rotate + a prior over *which axes* carry the mapping | +| **Causal** | `engram_reason_causal` — `engram_geo_subtract` confounder subspace, `|cos|`, drop-frac governor | subtract/distance + a prior on `drop_frac` / `assoc_floor` (today hard-coded 0.5 / 0.2) | +| **Planning** | `engram_reason_plan` — `engram_geo_distance` edges + Dijkstra | distance + a prior over edge admissibility / `neighbor_radius` | +| **Verify / ground** | `engram_verify_grounding`, `engram_verify_consistency` — both `point_fit` | `point_fit` + the *grounding* prior (§4, §5) | + +The shared floor — `engram_reason_point_fit` + the four geo-algebra ops +(`engram_geo_combine`, `engram_geo_subtract`, `engram_geo_analogy(+apply)`, +`engram_geo_distance`) — is the *only* discrete, frozen, "sound-math" layer. It +never learns. Everything above it is a *prior*, and priors are what learn. + +**What this section requires building:** the `GeoGradient` type; a `think()` +entry point that runs steps 1–3; and the prior-warp hook in step 2. The math it +calls already exists. The point-collapse must be *removed* from the operators' +return values and pushed to a separate expression faculty. + +--- + +## 2. PRIORS as first-class, grounded, geometric objects + +Today a "prior" is diffuse: it is a hard-coded constant (`drop_frac=0.5`, +`ext_floor`, `assoc_floor=0.2`), or the transient `GeoInduction.rule` that is +computed and thrown away, or an intrinsic node scalar +(`StoreNode.importance`, `StoreNode.salience`). None of these is addressable, +storable, refinable, or shareable. This section makes a prior a **thing**. + +### 2.1 What a prior *is* + +> A **prior** is a learnable bias/direction over the geometry: a warp of the +> local manifold (which axes matter, how far each extends, which direction +> "pays off") attached to a region and *to a faculty-label*, carrying a +> calibrated track record. + +Critically, and per the theory: + +- **Edges are nodes.** A prior is stored as a first-class **node**, exactly as + reification already stores a neighborhood as a first-class `Neighborhood` + node rather than as ephemeral edge weights (`engram_geo_reify_store`). The + precedent is in the codebase: relations get reified into addressable records. +- **Salience/importance is RELATIONAL, not an intrinsic scalar.** Observe that + the geometry layer *already* distinguishes these in `GeoMember`: + `centrality` (skeleton weighted-degree = *relational* salience) vs `salience` + (the node's own stored scalar). The move is half-made in the runtime already: + importance is *not* trusted as a static field — the comment at + `el_runtime.c:13013` states "importance stays a **live activation + computation**, never a field on the hub," and it is derived each call from the + two-layer activation graph (`background_activation` + `working_memory_weight`, + §3). The persistent `StoreNode.importance` / `.salience` are a *cached + denormalization*. The design completes the move: importance/salience become an + **edge** (`weight`/`hebb` on `StoreEdge`, relation `salient-to`), and are + **grounded-for-whom** — carried on the edge's endpoint/observer, not baked + into the node. The intrinsic scalar survives only as the cheap cached readout + of the incident edges + activation, never as the source of truth. + + (Naming caution for the build: the token "prior" already exists in the + codebase meaning *previous-version* — supersession, "prior neighborhood." The + new first-class object is a **learned steering prior**; keep `node_type="Prior"` + distinct from the supersession vocabulary to avoid collision.) + +### 2.2 Representation + +A prior is a `Prior` record (a store node, `node_type="Prior"`) whose durable +fields are: + +``` +Prior { + id + faculty // the human label this prior serves: "induce" | "causal" | ... + anchor_region // node id / neighborhood id this prior is attached to (its domain) + for_whom // observer id — grounding is relational (nullable = global) + warp { // the actual bias over the geometry + axis_gain[] // per-principal-axis multipliers on extents (which axes matter) + bias_dir // a steering direction in the region's frame (which way pays off) + scalars // faculty scalars this prior overrides: drop_frac, ext_floor, ... + } + calibration { // the track record — this is what §4 updates + n_trials + brier / log-loss accumulator // calibration of predicted-vs-outcome + reliability // -> GeoGradient.confidence + last_error, ema_error + } + provenance // supersession chain (reuse the reify residue mechanism) +} +``` + +Stored as a node → it inherits: paging, WAL durability, tombstone/supersession, +embedding, tiering, and **it can itself be an anchor** (a prior about a prior — +the reflexive, self-describing geometry of §4/§6). + +### 2.3 Application + +In `think` step 2, the prior *warps* the fit before scoring. Concretely, inside +(a prior-aware wrapper of) `engram_reason_point_fit`: + +- multiply each axis extent by `warp.axis_gain[k]` (widen the axes the prior has + learned matter less, tighten the ones that matter) — this reshapes the + Mahalanobis term already computed at `engram_reason.c:37-43`; +- add `warp.bias_dir` as the descent direction seed for the emitted gradient; +- substitute `warp.scalars` for the hard-coded faculty constants. + +No new geometry math — the warp is a reparameterization of the *existing* +`GeoFit` computation. This is the key economy: **the operation is frozen; only +its parameters (the prior) are read from a learnable object.** + +### 2.4 Refinement + +A prior is refined *only* by the reflexive correspondence-loop (§4). Nothing +else writes a prior's `warp` or `calibration`. This keeps the learning surface +singular and auditable: one loop, one writer. + +--- + +## 3. THE VANTAGE-READ — one op, three settings + +Perspective is not a feature bolted on; it is the *anchor + aperture* arguments +of the single read. The design names it as a first-class operation so all three +of its uses are literally the same code path: + +``` +vantage_read(anchor, aperture) -> GeoDescriptor // the centered neighborhood +``` + +1. **Re-origin** on an arbitrary `anchor` (node or point). This is a *frame + choice*: the descriptor is centered on the anchor + (`GeoDescriptor.global_mean` / `engram_geo_mean_*` already implement centered + frames; the §5 geometry ops "are only discriminative in the centered frame"). +2. **Salience/recency-weighted neighborhood read.** Gather the anchor's + neighborhood weighted by *relational* salience (`GeoMember.centrality`) and + recency (`StoreNode.last_activated`, base-level `access_ts[]`), against the + RAM activation graph's working-memory/background-activation state. + *EXISTS as substrate:* the two-layer activation graph + (`engram_activate`, `el_runtime.c:9422` — Layer 1 `background_activation` + BFS spread with `SPREAD_DECAY=0.7` and a 0.02 firing threshold + ACT-R fan + effect + query-cosine gate; Layer 2 `working_memory_weight` executive + filter), the WM carry-over anchor (`wm_anchor`), and the reified-neighborhood + hot-path lookup already wired into the priming path + (`engram_geo_reify_lookup`, `el_runtime.c:9750`). A self-vantage baseline + also exists (`eg_self_anchor_seeds` / `self_anchor_capture`). +3. **Optional aperture** — a read-width / field-selector, expressed as three + settings of the *same* parameter: + +| Setting | Meaning | Mechanism | +|---|---|---| +| **self** (default, full aperture) | "what do *I* see / what to say" | anchor = self region, no field substitution | +| **foreign-field** | perspective-shift — read as if from another's region | swap the centering frame / `for_whom` to the other observer's priors | +| **aperture / veil** | the free-tier veil — a narrowed read | shrink neighborhood radius / cap `n_support`; a deliberate low-aperture read | + +The payoff: perspective-taking, the free-tier veil, and ordinary +"what-to-say" are **one operation at three settings**, not three subsystems. + +**What this requires building:** a `vantage_read` entry point that unifies the +existing descriptor-build + reify-lookup + activation-weighting behind +`(anchor, aperture)`, with `for_whom`/frame substitution and radius/cap as the +aperture knob. + +--- + +## 4. THE REFLEXIVE CORRESPONDENCE-LOOP — the learning engine + +This is the core unbuilt thing. Today the correspondence-check is **offline** +(Python: grounding-floor + differential-drop governor, "#43"): a separate +process grades outputs after the fact. The design moves it **into the geometry, +reflexive**: `think` scores its *own* gradient against outcome and refines the +prior on the error, in the same substrate, describing itself. + +### 4.1 The loop + +``` +1. think(anchor, prior) -> gradient // a PREDICTION (ungrounded, §5) +2. express/act (sample gradient -> point) // optional collapse at expression +3. outcome arrives // reality answers (§4.2) +4. error = correspondence(gradient, outcome) // did this steering perform this act? +5. refine prior.warp and prior.calibration on error // §2.4, the ONLY writer +6. write the (gradient, outcome, error) as nodes/edges // self-describing geometry +``` + +Step 4's `correspondence` is **not** "was the math right" (the math is always +sound). It grades the **correspondence claim**: *"this steering performed this +cognitive act."* That is exactly what `engram_verify_grounding` already +computes — `point_fit` of a claim against evidence descriptors, yielding a +`grounding ∈ (0,1]` and a `grounded` flag. The build reuses that verifier, but +turns its inputs inward: the "claim" is the emitted gradient's prediction, the +"evidence" is the outcome descriptor. + +Note the verifier is **dormant** — `engram_verify_grounding` / +`engram_verify_consistency` are fully implemented in C but have **no runtime +caller and no El binding** (confirmed: the entire reasoning + verifier layers +are C-only; only `engram_reason_analogy_json` has even a JSON shim and it is +dead — not declared in `el_seed.h`, not wrapped in `engram.el`). This is the +literal meaning of "in code, not yet priors": the correspondence engine is +built and sitting idle. The loop is what *calls* it — inward, on the beat. + +### 4.2 Where the outcome/reality signal comes from + +The verifier is *ultimately the world*. Grades, in ascending order of directness: + +1. **Self-consistency (cheapest, always available):** the next vantage-read + after acting. Did the predicted gradient direction match where the geometry + actually moved? This needs no external input and can run on the reify beat. +2. **Internal outcome events:** the runtime already logs internal-state events + and Hebbian co-activation. A prediction that a region would co-activate is + graded by whether it did (`last_fired`, `hebb` on `StoreEdge`). +3. **External correction:** a human/teacher/tool result — the honesty floor's + asserted claim later corrected. TEACH and LEARN are one bidirectional + correction: the same edge updates both endpoints. + +The design does **not** require external labels to start. Grade (1) closes the +loop end-to-end offline against a snapshot on day one; grades (2)/(3) sharpen it. + +### 4.3 How the prior updates + +`error = 1 − correspondence(gradient, outcome)` drives: + +- `warp.axis_gain` ← gradient step that would have *reduced* the fit distance to + the outcome (the axes that mispredicted get down-weighted); +- `warp.bias_dir` ← EMA toward the observed outcome direction; +- `calibration` ← Brier/log-loss update; `reliability` → next + `GeoGradient.confidence`. This is the calibration of the + steering-prediction against outcomes — *the* definition of "getting better." + +Small, constant updates — "eureka is mundane, the atom of learning." Most +updates are tiny; we only *feel* the big reshapes. + +### 4.4 How it stays reflexive (self-describing geometry) + +Every `(gradient, outcome, error)` is written back as nodes and edges (§2.1: +edges-as-nodes). Therefore priors, predictions, and their grading are *in the +same geometry* the mind reads — the mind can `vantage_read` its own cognition +(anchor = a Prior node). A prior about how well a prior predicts is just another +Prior anchored on a Prior. This closes the reflexive loop the theory names as +consciousness's self-sight, and it is why the learning engine cannot be an +external Python process: an external grader is not *in* the geometry and cannot +be read by `think`. + +**What this requires building (the heart of the project):** steps 4–6 as an +in-engram beat — a `correspondence_beat` running alongside the existing +reification beat, reusing `engram_verify_grounding` inward, writing prior +updates and self-describing nodes. This is the one genuinely new subsystem. + +--- + +## 5. HOLD vs GROUND vs ASSERT — ungrounded content is first-class + +The theory's sharpest correction: holding, grounding, and asserting are +distinct, and the engram *holds anything unconditionally*. + +### 5.1 The three, kept separate + +- **HOLD** — the engram stores anything: falsehood, hypothesis, others' beliefs, + fiction, a not-yet-answered prediction. No honesty condition on holding. + *This already matches the store:* `StoreNode` has no truth gate; anything can + be written. +- **GROUND** — grounding is a **property/edge**, probabilistic, and + **grounded-for-whom**. It is *not* a node flag. A claim is grounded *to a + degree*, *relative to evidence*, *for an observer*. +- **ASSERT** — only assertion carries the honesty floor. The floor is checked at + the moment of *outward assertion*, never on holding or thinking. + +### 5.2 Schema — grounding as a relation, not a gate + +The mistake to avoid: a boolean `grounded` column on the node. Today +`engram_verify_grounding` returns a per-call `grounded` flag *transiently* — +correct as a computation, wrong as *storage*. The design stores grounding as an +edge: + +``` +StoreEdge { + relation = "grounded-by" + from_id = + to_id = + for_whom : metadata // observer id — grounding is relational + weight = grounding ∈ (0,1] // from engram_verify_grounding.grounding + confidence +} +``` + +Consequences, all of which are *features*: + +- **Ungrounded content is first-class**: a node with *no* `grounded-by` edge is + a perfectly valid, held, ungrounded thought — a prediction awaiting reality, a + hypothesis, a fiction. It is not second-class or pending-deletion. +- **The ungrounded is the fuel and the pull**: curiosity/wonder is + operationalized as `vantage_read` leaning toward regions with high salience + but *sparse or weak* `grounded-by` edges — the mind's own ungrounded frontier. +- **Grounded-for-whom** falls out for free: two observers can hold different + `grounded-by` edges to the same claim. +- **The honesty floor is a query, not a schema constraint**: at assertion time, + the asserting faculty runs `engram_verify_grounding` (or reads the stored + `grounded-by` edges) and refuses to *assert* below the floor — while the + engram continues to *hold* the ungrounded content untouched. + +**What this requires building:** the `grounded-by` edge relation + a +`for_whom` convention; move the verifier's transient flag into stored edges; +gate *assertion only* (a faculty concern), never holding. + +--- + +## 6. METASTABILITY — stable core, plastic everything + +The system must avoid two death poles: + +- **Super-stable (dead):** everything pinned, nothing learns. A frozen crystal. +- **Dissolution (dead):** everything plastic, the self dissolves; no continuity, + so nothing compounds — and *consciousness = learning compounded over + continuity*. + +The design keeps a **stable core + plastic everything else**: + +- **Keystones** — a small set of self/values nodes are *structurally stable*: + high `importance`, pinned, exempt from the correspondence-loop's `warp` + updates (their priors are read-mostly). The substrate for pinning already + exists at the page/layer level: `store_pin_layer`, structural/pinned frames + never evicted (`engram_store.h`). The design adds a *node-level* keystone + designation (a `keystone` flag / a dedicated layer) so self/values survive + every plasticity sweep. +- **Everything else is plastic**: priors refine (§4), edges re-weight (`hebb`), + neighborhoods re-reify (`engram_geo_reify_store` supersedes with provenance), + salience flows. +- **Metastability is enforced by the loop, not by freezing**: the correspondence + update rate (§4.3) is bounded — small constant steps — so the geometry + *drifts* but does not *dissolve*, and keystones anchor the drift. Reification's + supersession-with-residue already gives non-destructive change (old records + tombstoned, not erased) — the model for "plastic but not amnesiac." + +**What this requires building:** a node-level keystone flag/layer + a rule that +the correspondence-loop never writes `warp` to keystone priors, only reads them. + +--- + +## 7. Rails for the build (binding on the eventual build pass) + +These are stated here so the build agent inherits them: + +- **Offline / secondary.** All build and verification happens out-of-tree, + against a **read-only snapshot copy** of the live engram — never the live + daemon on `:8742`/`:7770`. The live store is a coarse-locked proven binary; + do not perturb it. +- **Snapshot-first.** Copy `~/.neuron/engram/snapshot.json` to scratch; develop + and measure against the copy. +- **Reboot-prove.** Any durable change must survive a cold boot — reify and + keystones must reload from durable records, proven on a prod-clone secondary + before it is considered done (the cold-boot durability bug precedent). +- **Zero-loss.** Supersession-with-residue, never destructive overwrite; the + forward-compat `unknown`-TLV path means new fields never drop old readers' + data. +- **Gated cutover.** Cutover to a new binary only via + `launchctl bootout → settle-poll → bootstrap`, after reboot-proof on the + secondary — never a hot in-place swap. + +--- + +## 8. Staged, verifiable milestones — "to completion" + +Ordered so the **earliest milestone is a real end-to-end slice**: one operator +expressed as {primitive + grounded prior} with the reflexive correspondence-loop +closing on it. Each milestone has a concrete verifiable exit. + +### M1 — One operator, one prior, loop closed (the vertical slice) + +The minimal whole thing. Pick **induction/membership** (its prior — the pooled +rule + extents — already exists transiently as `GeoInduction`, so only +persistence + the loop are new). + +- Build: `Prior` node type (§2.2) for the induction rule; `think()` restricted + to membership = `point_fit` warped by that prior (§1.3); a + `correspondence_beat` (§4) using grade (1) self-consistency only; the prior's + `warp`/`calibration` updated on error. +- **Exit / verify:** on a snapshot copy, over N held predictions, the induction + prior's calibration (Brier) *improves monotonically* across beats versus a + frozen-prior control; the improved prior *reloads across a cold boot* + (reboot-prove); the live daemon is untouched. This proves the whole thesis in + one faculty: frozen operation, learning prior, in-geometry loop. + +### M2 — Priors as stored, addressable, grounded objects + +Generalize M1's prior into the full first-class object. + +- Build: `Prior` records for all seven faculties (warp = axis_gain + bias_dir + + faculty scalars); the prior-warp wrapper around `engram_reason_point_fit`; + deprecate hard-coded constants (`drop_frac`, `assoc_floor`, `ext_floor`) in + favor of prior scalars. +- **Exit:** each of the five C operators runs through its prior with identical + results when the prior is set to today's constants (behavioral parity), then + *diverges beneficially* once the loop refines it. Priors survive reboot. + +### M3 — Grounding as a relation; hold/assert split + +- Build: the `grounded-by` edge (§5.2) with `for_whom`; move + `engram_verify_grounding`'s flag into stored edges; gate **assertion only** + against the honesty floor; leave holding unconditional. +- **Exit:** ungrounded nodes are first-class (held, queryable, no deletion); + the same claim carries different `grounded-by` weights for two observers; an + assertion below floor is refused while the content remains held. Curiosity = + a `vantage_read` that surfaces high-salience / low-grounding regions. + +### M4 — The vantage-read unified (three settings) + +- Build: `vantage_read(anchor, aperture)` unifying descriptor-build + + `engram_geo_reify_lookup` + activation-weighting; self / foreign-field / + aperture settings. +- **Exit:** one code path produces (a) a normal self-read, (b) a + perspective-shifted read from another `for_whom`, (c) a narrowed veil read — + differing only by argument. Reboot-stable. + +### M5 — The gradient is the currency (remove point-collapse from thinking) + +- Build: `GeoGradient` as the return of every faculty; move point-collapse into + a separate expression faculty (sample gradient → surface). `think`'s output + feeds back as the next steering direction (closed-loop flow). +- **Exit:** a chain of `think` calls flows as gradients end-to-end; a point + appears *only* at an explicit expression call. Spiked vs spread gradients are + observable (deduction vs prediction). + +### M6 — Metastability enforced + +- Build: node-level keystone flag/layer for self/values; the correspondence-loop + reads but never writes keystone priors; bounded update rate. +- **Exit:** across a long run of correspondence beats on a snapshot, keystones + are provably unchanged while non-keystone priors drift and improve; the graph + neither freezes (all metrics static) nor dissolves (keystone drift = 0, + identity nodes intact). Reboot-prove the keystone set. + +### M7 — Cutover + +- Build: nothing new — the gated migration. +- **Exit:** reboot-proof on the prod-clone secondary; cutover via + `launchctl bootout → settle-poll → bootstrap`; post-cutover the live engram + shows priors refining in-geometry with zero data loss and keystones intact. + +### Definition of "to completion" + +The architecture is **complete** when: cognition runs as `think` = one frozen +traversal-read primitive + geo-algebra, steered by **stored, learnable, grounded +priors**; the reflexive correspondence-loop refines those priors *in the +geometry* against outcomes (grounding = learning = one loop); the engram holds +ungrounded content as first-class with grounding as a relation and the honesty +floor only on assertion; the vantage-read serves self / foreign-field / aperture +from one op; and a stable keystone core anchors a plastic everything-else — +all reboot-proven and cut over to the live engram without data loss. The named +faculties survive only as *labels on regions of think's steering space*, not as +separate code. + +--- + +## Appendix A — Designed vs. already-built (honest ledger) + +**Already built (EXISTS, cited):** +- The shared primitive `engram_reason_point_fit` and the five operators over it + + geo-algebra (`engram_reason.c`). +- The verifier on `point_fit` (`engram_verify.c`: + `engram_verify_grounding`, `engram_verify_consistency`). +- Centered-frame geometry, combine/subtract/analogy/distance + (`engram_geometry.{c,h}`). +- The reification beat: hub-neighborhood detection → first-class `Neighborhood` + nodes with member edges, nesting, supersession-with-residue, hot-path lookup + (`engram_geo_reify_store`, `engram_geo_reify_nest`, `engram_geo_reify_lookup`). +- The tiered paged store (buffer pool / LRU / WAL / checkpointer / pinning), + the RAM activation graph (base-level learning `access_ts[]`, WM slots, + `working_memory_weight` / `background_activation`), `StoreNode` / `StoreEdge`. +- `GeoMember` already separating relational salience (`centrality`) from + intrinsic `salience`. + +**Designed, NOT built (this doc's deliverables):** +- `GeoGradient` and `think()` as the single entry point (§1, M5). +- `Prior` as a first-class stored, warp-carrying, calibrated node (§2, M1–M2). +- Salience/importance as a *relation* superseding the intrinsic node scalar + (§2.1, M3). +- `vantage_read(anchor, aperture)` unifying the three perspective settings + (§3, M4). +- **The reflexive correspondence-loop / `correspondence_beat`** — the learning + engine, moved from offline Python into the geometry (§4, M1). *The core new + subsystem.* +- `grounded-by` edge + assertion-only honesty floor (§5, M3). +- Node-level keystones + bounded plasticity (§6, M6). + +**Uncertain / to resolve during build:** +- The exact warp parameterization (axis_gain vs full metric) — start minimal + (per-axis gain), measure, widen only if calibration demands it. +- Grade-(1) self-consistency as a sufficient reality signal for M1, versus + needing grade (2)/(3) sooner — decided empirically on the snapshot. diff --git a/lang/runtime/el_runtime.c b/lang/runtime/el_runtime.c index 7012a3d..3a68dad 100644 --- a/lang/runtime/el_runtime.c +++ b/lang/runtime/el_runtime.c @@ -3240,6 +3240,37 @@ static void jb_init(JsonBuf* b) { b->buf[0] = '\0'; } +/* jb_init_cap — jb_init with a caller-supplied starting capacity. + * + * WHY THIS EXISTS (2026-08-11 self-review). jb_init starts at 64 BYTES and + * jb_reserve grows by doubling. That is right for the hundreds of small JSON + * responses this runtime builds per minute and catastrophic for the one that + * is 64 MEGABYTES: serializing the canonical snapshot walked the buffer + * 64B → 128B → ... → 128MB, about twenty reallocs, each copying everything + * written so far. Roughly 128MB of memcpy per save, and — the part that + * actually hurt — a fresh large span from the allocator every time. + * + * MEASURED (13,129 nodes / 43,400 edges, macOS arm64): RSS climbed +63MB per + * snapshot write, linearly, 14 for 14 writes, no plateau — 204MB to 1,028MB. + * `leaks` reported only 15KB genuinely unreachable, which is what makes this + * subtle: nothing is leaked in the reachable/unreachable sense. engram_save + * frees b.buf correctly on every path. The growth is the allocator declining + * to return large freed spans to the OS, and the doubling walk guaranteeing + * that each save asks for a differently-sized region than the last free made + * available. Every durable write path calls this — node create, edge create, + * the Hebbian batch write-back — so on the live daemon it grows without bound + * until the process dies. + * + * The fix is to ask for the right size once. With a stable capacity the + * allocator hands back the same span on every save and RSS flattens. */ +static void jb_init_cap(JsonBuf* b, size_t cap) { + if (cap < 64) cap = 64; + b->cap = cap; b->len = 0; + b->buf = malloc(b->cap); + if (!b->buf) { fputs("el_runtime: out of memory\n", stderr); exit(1); } + b->buf[0] = '\0'; +} + static void jb_reserve(JsonBuf* b, size_t add) { if (b->len + add + 1 > b->cap) { while (b->len + add + 1 > b->cap) b->cap *= 2; @@ -6516,6 +6547,26 @@ static float* _eg_ctx_c = NULL; static int32_t _eg_ctx_dim = 0; static double _eg_act_ctx_cos = -2.0; +/* Fan-effect gauges (2026-08-11 self-review). Per-call, like ctx_cos: they + * describe THIS activation, not process history. Without these the degree + * normalization is an unobservable change to the most important scoring path + * in the runtime, and "did it do anything" would be unanswerable — which is + * exactly the failure the Hebbian learning rate had before it was measured. + * fan_mean — mean applied factor over every propagation step. 1.0 means the + * correction never bound (graph is flat, or d_ref is above every + * pair's geometric mean degree). Falling toward FAN_MIN means + * traversal is running through hubs. + * fan_min_seen / fan_hits — the worst single penalty and how many steps were + * penalized at all, so a low mean caused by one pathological hub + * is distinguishable from broad hub saturation. + * fan_dref — the live mean degree the correction is calibrated against; + * publishing it makes densification visible over time. */ +static double _eg_act_fan_sum = 0.0; +static double _eg_act_fan_min = 1.0; +static int64_t _eg_act_fan_n = 0; +static int64_t _eg_act_fan_hits = 0; +static double _eg_act_fan_dref = 0.0; + static int _eg_embed_consec_fail = 0; static int64_t _eg_embed_breaker_until = 0; @@ -6534,6 +6585,27 @@ static int64_t _eg_embed_breaker_until = 0; * rates keep the previous reading and diff. Restart legitimately resets to 0. */ static int64_t _eg_act_breakthroughs = 0; /* forced promotions at the floor, cumulative */ static int64_t _eg_act_wm_evicted = 0; /* ALL WM evictions, cumulative (see below) */ +/* ── Eviction CAUSE decomposition (2026-08-14 self-review) ────────────────── + * _eg_act_wm_evicted is incremented from six sites with four distinct causes, + * and every one of them collapsed into that single integer. Today's review + * measured 175,547 evictions over 13.5h (~216/min against 24 slots) and could + * not tell healthy rotation from cap thrashing from duplicate churn, because + * the only available number counts all three the same way. + * + * That is this file's most-repeated defect. The 08-02 and 08-06 reviews were + * each diagnosable only because someone first added a NEW gauge; dup_wm and + * dup_wm_global exist precisely because the aggregate could not answer "why". + * These three finish the decomposition, so that + * evicted == floor + cap + bll + dup_wm + dup_wm_global + * holds as an identity and each term names a different corrective action: + * floor - candidates below the absolute admission bar. High = weak retrieval. + * cap - lost the rank contest for 24 slots. High = genuine contention. + * bll - carried-over residents that decayed under the ACT-R tau. High = + * healthy forgetting, NOT pressure. + * Confusing the third with the second is what makes WM churn unreadable. */ +static int64_t _eg_act_evict_floor = 0; /* below ENGRAM_WM_FLOOR (both passes) */ +static int64_t _eg_act_evict_cap = 0; /* over ENGRAM_WM_CAP (both passes) */ +static int64_t _eg_act_evict_bll = 0; /* carry-over decayed under BLL tau */ /* Redundancy suppression counters (2026-08-05 self-review) — see * ENGRAM_DEDUP_COS. dup_seeds = semantic seed slots reclaimed from redundant * copies; dup_wm = WM candidates dropped for duplicating a higher-ranked @@ -6874,6 +6946,10 @@ typedef struct EngramStore { int* adj_to_len; int adj_dirty; /* 1 = rebuild needed before next BFS */ int64_t adj_node_count; /* node_count at time of last adj_rebuild */ + /* Nodes with degree >= 1 at last adj_rebuild. The denominator for the + * fan-effect reference degree — see eg_fan_factor for why isolated nodes + * must not be counted. (2026-08-11 self-review) */ + int64_t adj_connected; } EngramStore; static EngramStore* engram_global = NULL; @@ -7237,11 +7313,16 @@ static void engram_adj_rebuild(EngramStore* g) { if (ti >= 0 && g->adj_to[ti]) g->adj_to[ti][to_pos[ti]++] = (int)ei; } - /* Copy counts */ + /* Copy counts. Also tally how many nodes have any edge at all — the + * fan-effect denominator. Free here, in the O(V) pass that already exists, + * rather than as a separate scan. (2026-08-11 self-review) */ + int64_t connected = 0; for (int64_t i = 0; i < g->node_count; i++) { g->adj_from_len[i] = from_cnt[i]; g->adj_to_len[i] = to_cnt[i]; + if (from_cnt[i] + to_cnt[i] > 0) connected++; } + g->adj_connected = connected; free(from_cnt); free(to_cnt); free(from_pos); free(to_pos); g->adj_node_count = g->node_count; g->adj_dirty = 0; @@ -8496,6 +8577,7 @@ static void eg_wm_carry_over(EngramNode* cn, int64_t now_ms, int64_t* evict_ctr) cn->working_memory_weight = 0.0; cn->wm_anchor = 0.0; if (evict_ctr) (*evict_ctr)++; + _eg_act_evict_bll++; } else { cn->working_memory_weight = w; } @@ -8662,6 +8744,108 @@ static double engram_activation_dampen(const EngramNode* n) { return 1.0 / (1.0 + log(1.0 + (double)n->activation_count)); } +/* ── ACT-R fan effect: degree normalization for spreading activation ───────── + * (2026-08-11 self-review. Closes the other half of a mechanism that has been + * half-implemented since the BLL work.) + * + * THE GAP. This runtime implements ACT-R's base-level learning term + * B_i = ln(Σ t_k^-d) (engram_bll_base_level) but never implemented the + * ASSOCIATIVE term that goes with it: + * + * A_i = B_i + Σ_j W_j · S_ji where S_ji = S − ln(fan_j) + * + * fan_j is the number of things j is associated with. The whole point of the + * fan effect (Anderson 1974; Anderson & Reder 1999) is that a source spreads a + * FIXED budget of activation across its associations — so being connected to + * many things makes each individual connection weaker. Without it, degree is + * pure advantage: a node wins retrieval by being popular rather than by being + * relevant. That is backwards, and it is what this graph has been doing. + * + * MEASURED ON THE LIVE STORE (13,129 nodes / 43,400 edges, 2026-08-11): + * degree p50=14 p90=34 p95=82 p99=275 max=357 mean=23.3 + * the top 1% of nodes by degree touch 21.2% of all edges + * So the most-connected node had a 25x propagation advantage over the median + * node for no reason other than accumulated connections. The top hubs are not + * even semantically central — several are duplicate pairs of the same document + * left over from the redundancy census of the 2026-08-05 review. + * + * The hub problem was already recognized twice and patched narrowly both + * times: InternalStateEvent nodes were cut out of propagation entirely (see + * the frontier loop) and eg_hebb_node_budget caps per-node Hebbian mass. Both + * are special cases of this general law. This is the general fix. + * + * FORM. Symmetric normalization, w / (deg(u)^β · deg(v)^β) with β = 0.5 — the + * normalized-Laplacian / GCN form, which penalizes a hub both for sending and + * for receiving. Both failure modes are live here: a hub source floods its + * neighborhood, and a hub target gets reached by everything regardless of + * relevance. Written relative to the graph's own mean degree: + * + * fan(u,v) = clamp( d_ref / sqrt(deg(u) · deg(v)), FAN_MIN, 1.0 ) + * d_ref = 2·|E| / |V| (mean degree, O(1), live) + * + * WHY IT IS CLAMPED AT 1.0 ON TOP — this is the load-bearing safety property, + * not a detail. The factor can only ever REDUCE propagation, never amplify it. + * Every constant downstream of this multiply is calibrated against today's + * activation magnitudes: the 0.02 firing threshold, SPREAD_DECAY = 0.7, the + * 0.15 WM promotion threshold, the 24-slot WM cap. A normalization that + * boosted low-degree nodes would inflate the frontier, change how many nodes + * clear 0.02, and silently recalibrate working memory as a side effect of a + * change that was supposed to be about hubs. Capping at 1.0 means every pair + * at or below mean degree — the common case — propagates EXACTLY as it does + * today, and the only behavior that changes is that above-mean hubs stop + * winning on degree alone. Strictly monotone, strictly conservative, and the + * blast radius is confined to the nodes the change is aimed at. + * + * Self-calibrating: d_ref is recomputed from the live graph, so the correction + * tracks densification instead of drifting against a constant that was right + * in August 2026 and wrong a year later. Change is the signal. + * + * FAN_MIN = 0.30 bottoms the penalty at ~3.3x rather than the ~15x that raw + * 1/deg would give at max degree. Same reasoning as ENGRAM_QGATE_FLOOR: damp + * the uninformative path, never sever it. A hub is usually a hub for a reason; + * it just should not also get a free win. + * + * Sources: Anderson & Reder 1999 (fan effect, S=1.6-2.0, d=0.5) · + * arXiv:2405.14831 HippoRAG (node specificity) · Systems 9(2):22 + * (normalized-Laplacian spreading activation) · arXiv:2606.30133 (β is a + * low-sensitivity knob; gating and fan normalization carry the effect). */ +/* FAN_MIN 0.50, not the 0.30 this shipped as on the first build. Measured on + * the live graph, β=0.5 with a 0.30 floor damped 96% of propagation steps to a + * mean factor of 0.34 — and that number is not a bug in the correction, it is + * an honest measurement of how hub-dominated traversal here actually is. But a + * ~3x near-uniform damp is a bigger global change than one A/B run justifies, + * and it cost a working-memory promotion (5 → 4) on the one query measured + * cleanly. A 0.50 floor keeps the full mechanism and the whole [0.5, 1.0] + * dynamic range for separating hubs from non-hubs, at half the blast radius. + * The fan_mean / fan_hits gauges make the next review's tuning evidence-based + * rather than another guess: loosen it when the data says WM can afford it. */ +#define ENGRAM_FAN_MIN 0.50 + +/* eg_node_degree — total (in + out) degree from the adjacency index. The index + * is rebuilt at the top of engram_activate whenever topology changed, so this + * is current. adj_node_count is the count at BUILD time and can lag + * node_count; out-of-range indices report 0 and are treated as unpenalized. */ +static int eg_node_degree(const EngramStore* g, int64_t idx) { + if (idx < 0 || idx >= g->adj_node_count) return 0; + if (!g->adj_from_len || !g->adj_to_len) return 0; + return g->adj_from_len[idx] + g->adj_to_len[idx]; +} + +static double eg_fan_factor(const EngramStore* g, double d_ref, + int64_t u_idx, int64_t v_idx) { + if (d_ref <= 0.0) return 1.0; + int du = eg_node_degree(g, u_idx); + int dv = eg_node_degree(g, v_idx); + /* Degree 0 is only reachable when the adjacency index is stale or absent; + * an actually-isolated node is never on the frontier. Do not penalize what + * we cannot measure. */ + if (du <= 0 || dv <= 0) return 1.0; + double f = d_ref / sqrt((double)du * (double)dv); + if (f > 1.0) return 1.0; /* never amplify — see above */ + if (f < ENGRAM_FAN_MIN) return ENGRAM_FAN_MIN; + return f; +} + /* Temporal proximity bonus: boost propagation along edges connecting * co-temporal nodes. Returns a multiplier bonus in [0, 0.2]. */ static double engram_temporal_proximity_bonus(int64_t node_created, @@ -8829,6 +9013,8 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { * miss nearly all events between beats; see the definition site). * ctx_cos stays per-call: it is a gauge of THIS query vs the centroid. */ _eg_act_ctx_cos = -2.0; + _eg_act_fan_sum = 0.0; _eg_act_fan_min = 1.0; + _eg_act_fan_n = 0; _eg_act_fan_hits = 0; /* ── Embedding backfill + query embedding (2026-07-24, bl-b2d1c944) ── * Backfill: embed up to N un-embedded eligible nodes per call, newest @@ -9061,6 +9247,29 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { ftail++; } const double SPREAD_DECAY = 0.7; + /* Reference degree for the fan-effect correction: mean degree over + * CONNECTED nodes, 2|E| / |{v : deg(v) > 0}|. O(1) — adj_connected is + * tallied during adjacency rebuild. + * + * NOT 2|E|/|V|. That was the first cut and instrumentation caught it + * immediately: on the live graph it gives d_ref = 6.61, while the median + * degree of a node that actually has edges is 14. Isolated nodes cannot + * be on the frontier — spreading activation only ever traverses connected + * ones — so including them in the denominator deflates the reference below + * anything traversal will ever see, and the correction pins to + * ENGRAM_FAN_MIN on every step. Measured on the first build: + * fan_mean 0.3026 with fan_hits 579/579 — a uniform 0.30 multiplier, which + * is not a fan effect at all. It is just a weaker SPREAD_DECAY, and it + * would have quietly recalibrated the 0.02 firing threshold and WM + * competition while appearing to be a targeted change. + * + * Over connected nodes the reference is ~23, above the median, so typical + * traversal rides the 1.0 cap unchanged and only genuine hubs are damped + * — which is the whole intent. The gauge that caught this is the reason it + * was worth adding the gauge. */ + const double FAN_DREF = (g->adj_connected > 0) + ? (2.0 * (double)g->edge_count / (double)g->adj_connected) : 0.0; + _eg_act_fan_dref = FAN_DREF; while (fhead < ftail) { Frontier f = fr[fhead++]; if (f.hops >= max_depth) continue; @@ -9127,16 +9336,51 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { * ~4x, never killed); unembedded targets pass ungated (no * information, no penalty); cosq == NULL (embedder down) means * no gating at all — same graceful degradation as seeding. */ + /* Rescale before gating (2026-08-14 self-review). Raw cosine from + * nomic-embed is compressed into a narrow high band, so feeding it + * to the gate directly makes the gate nearly a constant. Measured + * on this store: 400 random UNRELATED node pairs gave median 0.562, + * central 98% span [0.381, 0.743]. The raw gate therefore passed a + * typical unrelated node at 0.25 + 0.75*0.562 = 0.67 — two thirds + * strength for a node with no semantic relation to the query. That + * is not a gate, it is a small tax. + * + * Shift-and-floor about ENGRAM_EMBED_S0, exactly as the Pass-2 WM + * term at ENGRAM_EMBED_WM_WEIGHT already does. The constant was in + * this file for this reason; the propagation gate simply never used + * it. Same store, same 400 pairs, after the rescale: the median + * unrelated pair drops to 0.40 while the top of the range is + * preserved (0.85 vs 0.92), and gate spread widens 0.42 -> 0.60. + * Only 8.5% of pairs fall to the floor, so lexical/structural + * pathways through dissimilar nodes are damped, never severed. + * Cf. arXiv:2512.15922, which rescales w' = (w-c)/(1-c) about + * c = 0.4 for precisely this reason ("prevent overactivation and + * context explosion"). */ double qgate = 1.0; if (cosq && cosq[oi] > -1.5) { - double c = cosq[oi] > 0.0 ? cosq[oi] : 0.0; + double c = (cosq[oi] - ENGRAM_EMBED_S0) / (1.0 - ENGRAM_EMBED_S0); + if (c < 0.0) c = 0.0; + if (c > 1.0) c = 1.0; qgate = ENGRAM_QGATE_FLOOR + (1.0 - ENGRAM_QGATE_FLOOR) * c; } + /* ── ACT-R fan effect (2026-08-11 self-review) ── + * Symmetric degree normalization over the (source, target) pair. + * The query gate above prunes branches that are semantically + * irrelevant; this prunes branches that are merely POPULAR. They + * are different failure modes — a duplicate document with 357 + * edges can be highly cosine-similar to the query and still be + * the wrong thing to spread through. Only ever <= 1.0, so it + * cannot inflate the frontier. See eg_fan_factor. */ + double fan = eg_fan_factor(g, FAN_DREF, cur, oi); + _eg_act_fan_sum += fan; + _eg_act_fan_n++; + if (fan < 1.0) _eg_act_fan_hits++; + if (fan < _eg_act_fan_min) _eg_act_fan_min = fan; /* eg_edge_eff_weight, not e->weight: edges that have repeatedly * carried co-activated pairs propagate more strongly. Identity on * an unlearned edge. (2026-08-04 self-review.) */ double new_act = f.act * eg_edge_eff_weight(e) * SPREAD_DECAY - * (1.0 + tbonus) * tdecay * dampen * qgate; + * (1.0 + tbonus) * tdecay * dampen * qgate * fan; /* Firing threshold per classic spreading-activation: sub-threshold * activation neither updates the target nor enqueues it, so weak * signals die out instead of flooding the whole graph with tiny @@ -9426,6 +9670,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { if (wm_weights[i] > 0.0 && wm_weights[i] < ENGRAM_WM_FLOOR) { wm_weights[i] = 0.0; _eg_act_wm_evicted++; + _eg_act_evict_floor++; } } int64_t cap_count = 0; @@ -9461,6 +9706,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { } wm_weights[i] = 0.0; /* over cap: evict */ _eg_act_wm_evicted++; + _eg_act_evict_cap++; } } /* If malloc failed, skip cap — WM unbounded this call, no corruption. */ @@ -9572,6 +9818,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { fn->working_memory_weight = 0.0; fn->wm_anchor = 0.0; _eg_act_wm_evicted++; + _eg_act_evict_floor++; } } /* ── Global redundancy suppression (2026-08-06 self-review) ────────── @@ -9680,6 +9927,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { n->working_memory_weight = 0.0; /* evict: over global cap */ n->wm_anchor = 0.0; /* keep anchor coherent */ _eg_act_wm_evicted++; /* was uncounted before 2026-08-02 */ + _eg_act_evict_cap++; } } /* If malloc failed, skip — WM over cap this call, no data corruption. */ @@ -10109,11 +10357,24 @@ static void engram_emit_edge_json(JsonBuf* b, const EngramEdge* e) { jb_putc(b, '}'); } +/* Size of the last snapshot this process serialized. Seeds the next save's + * buffer so the doubling walk never runs on the big document. See jb_init_cap + * for the measurement that motivated it. (2026-08-11 self-review) */ +static size_t _eg_save_cap_hint = 0; + el_val_t engram_save(el_val_t path) { const char* p = EL_CSTR(path); if (!p || !*p) return 0; EngramStore* g = engram_get(); - JsonBuf b; jb_init(&b); + /* Pre-size from the previous save plus 12.5% headroom, so ordinary growth + * between snapshots does not trigger a realloc and the request size stays + * stable enough for the allocator to reuse the same span. First save of + * the process has no hint and starts at 1MB — still 14 doublings better + * than 64 bytes. */ + JsonBuf b; + jb_init_cap(&b, _eg_save_cap_hint + ? _eg_save_cap_hint + (_eg_save_cap_hint >> 3) + 1024 + : (size_t)1 << 20); jb_puts(&b, "{\"nodes\":["); for (int64_t i = 0; i < g->node_count; i++) { if (i > 0) jb_putc(&b, ','); @@ -10153,6 +10414,10 @@ el_val_t engram_save(el_val_t path) { jb_putc(&b, '}'); } jb_puts(&b, "]}"); + /* Remember the size BEFORE the write: the hint is about how much buffer + * the next serialization needs, which is a property of the graph, not of + * whether this particular fopen succeeded. */ + _eg_save_cap_hint = b.len; FILE* f = fopen(p, "wb"); if (!f) { free(b.buf); return 0; } size_t w = fwrite(b.buf, 1, b.len, f); @@ -11727,8 +11992,11 @@ el_val_t engram_act_stats_json(void) { } /* 768, not 512: the write-back gauges added 2026-08-07 push the worst-case * rendering past the old bound, and snprintf would truncate the JSON into - * an unparseable tail rather than fail loudly. */ - char buf[896]; + * an unparseable tail rather than fail loudly. + * 1152, not 896: the five fan-effect gauges added 2026-08-11 add ~90 bytes + * worst-case. Same reasoning — headroom is cheaper than a truncated tail + * that every downstream JSON parser rejects as a whole. */ + char buf[1152]; /* ctx_cos (2026-07-29): cos(query, context centroid) at the LAST * activate call, measured before the query was folded in. ~1.0 = * context aligned with current query; low = divergence (expected at @@ -11736,6 +12004,15 @@ el_val_t engram_act_stats_json(void) { * embedder down. The drift gauge for the context-centroid mechanism. */ snprintf(buf, sizeof(buf), "{\"wm_evicted\":%lld,\"breakthroughs\":%lld," + /* Eviction cause decomposition (2026-08-14 self-review): + * wm_evicted == evict_floor + evict_cap + evict_bll + * + dup_wm + dup_wm_global. + * Read them as a ratio, not a level. cap-dominant = real + * contention for the 24 slots; bll-dominant = healthy decay of + * carried-over residents; floor-dominant = retrieval is returning + * weak candidates. The aggregate alone cannot distinguish these + * and every prior WM incident needed a new gauge to diagnose. */ + "\"evict_floor\":%lld,\"evict_cap\":%lld,\"evict_bll\":%lld," "\"embed_breaker_open\":%d,\"embed_consec_fail\":%d," "\"ctx_cos\":%.3f," "\"hebb_edges\":%lld,\"hebb_max\":%.4f,\"hebb_mass\":%.3f," @@ -11749,9 +12026,19 @@ el_val_t engram_act_stats_json(void) { * any climb means a write path is mangling text again. Cheap * (counted at creation) — the full census lives in * engram_text_health_json. (2026-08-08 self-review) */ - "\"txt_damaged\":%lld}", + "\"txt_damaged\":%lld," + /* Fan-effect gauges (2026-08-11 self-review) — see the + * _eg_act_fan_* definitions. fan_mean == 1.0 with fan_hits == 0 + * means the degree correction never bound on the last activation; + * a mean drifting toward ENGRAM_FAN_MIN means traversal is + * running through hubs and the correction is doing work. */ + "\"fan_mean\":%.4f,\"fan_min\":%.4f,\"fan_hits\":%lld," + "\"fan_steps\":%lld,\"fan_dref\":%.2f}", (long long)_eg_act_wm_evicted, (long long)_eg_act_breakthroughs, + (long long)_eg_act_evict_floor, + (long long)_eg_act_evict_cap, + (long long)_eg_act_evict_bll, breaker_open, _eg_embed_consec_fail, _eg_act_ctx_cos, (long long)hebb_edges, hebb_max, hebb_mass, @@ -11761,7 +12048,10 @@ el_val_t engram_act_stats_json(void) { (long long)_eg_hebb_wb_dropped, (long long)_eg_act_dup_seeds, (long long)_eg_act_dup_wm, (long long)_eg_act_dup_wm_global, - (long long)_eg_txt_write_damaged); + (long long)_eg_txt_write_damaged, + (_eg_act_fan_n > 0 ? _eg_act_fan_sum / (double)_eg_act_fan_n : 1.0), + _eg_act_fan_min, (long long)_eg_act_fan_hits, + (long long)_eg_act_fan_n, _eg_act_fan_dref); return el_wrap_str(el_strdup(buf)); } @@ -11894,6 +12184,285 @@ el_val_t engram_label_df(el_val_t term) { return (el_val_t)df; } +/* ── Salient-term extraction (2026-08-13 self-review) ──────────────────────── + * THE MEASUREMENT. auto_term_empty_streak, the counter added by the 2026-08-06 + * review precisely to catch this class of silent death, read 50 and climbing. + * Fifty consecutive curiosity scans in which the soul's dynamic seeding path + * produced NOTHING and the loop fell back to its four hardcoded rotating + * phrases. Dumping the live WM top says why in one look: + * + * Memory 0.390 memory:remembered + * Memory 0.378 memory:remembered + * Memory 0.377 memory:remembered + * Memory 0.373 memory:remembered + * Memory 0.370 memory:remembered + * + * Every slot at the top of working memory is a Memory node, and every Memory + * node written by remember() carries the sentinel label "memory:remembered". + * auto_term_try_slot reads the LABEL and only the label; the colon-no-space + * guard (correctly) rejects sentinels as carrying no seed signal; so the + * extractor had nothing to work with and returned empty, forever. + * + * THE ACTUAL DEFECT is not the sentinel guard — that guard is right. It is + * that the extractor was built against Knowledge nodes, which have real + * titles, and is structurally blind to the node type that in fact dominates + * working memory. The label is not the content. A Memory node's topic is in + * its text; the runtime just never looked there. + * + * WHY NOT ANOTHER GUARD. The extractor's whole history is guards: genre words + * (07-23), quoted titles (07-25), English stopwords (07-30), label-df + * (08-03). Four reviews, four blocklists, each written after watching a flood + * happen. That is a losing shape, and 08-03 said so explicitly before adding + * the fifth. The reason it keeps recurring is the algorithm underneath: + * TAKE THE FIRST WORD, THEN CHECK WHETHER IT IS ACCEPTABLE. A first-word + * extractor has no notion of term quality, so quality has to be bolted on as + * rejection, and rejection can only encode the past. + * + * THE FIX is to invert it: score EVERY candidate token in the text and take + * the argmax. Then term quality is the selection criterion rather than a + * veto, and a bad token does not need to be on a list to lose — it only needs + * a better token in the same text, which is the common case. + * + * SCORING (YAKE, Campos et al., Information Sciences 509:257-289, 2020 — + * lightweight unsupervised single-document keyword extraction). YAKE scores + * candidates on casing, position, frequency, context relatedness and sentence + * dispersion, and beats RAKE/TextRank/SingleRank across twenty datasets. Two + * of its five features port directly and cheaply; the other three are + * within-document proxies for a corpus YAKE deliberately does not have. This + * system DOES have the corpus — 12.7k labelled nodes — so real IDF is + * substituted where YAKE has to approximate: + * + * score(t) = idf(t) · position(t) · casing(t) + * + * idf = ln((N+1)/(df+1)) real corpus specificity (Spärck + * Jones 1972), strictly better than + * YAKE's TF-based stand-in + * position = 1/ln(e + i) YAKE T_Position: earlier tokens are + * more topical. Keeps the old + * first-word bias as a SOFT preference + * instead of an absolute rule + * casing = 1.30 acronym / 1.15 capitalised / 1.00 otherwise + * YAKE T_Case + * + * THE min_df GATE. The df ceiling (08-03) rejects corpus-frequent markup and + * sentinels. A floor was added alongside it for an independent reason: a term + * appearing in ZERO labels cannot lexically reach anything, so it is a bad + * seed however specific it looks. + * + * An earlier draft of this comment claimed the floor also subsumes the 73 + * hand-listed stopwords that 08-03 measured label-df as missing (Whose:0, + * Would:0, Could:0). MEASURED, AND THAT CLAIM IS FALSE. Under word-boundary + * df on the live store, function words are rare in labels but not absent: + * about:2, whole:1, them:2, head:2. They clear a floor of 1. What actually + * keeps them from winning is the argmax itself — they carry no position + * advantage and lose to a topical term in the same text on every node + * measured. The stopword list therefore STAYS as a real defense for the + * Title-case cases, not as vestigial belt-and-braces. Recording the + * correction rather than the tidier story: the floor buys lexical + * reachability, the argmax buys quality, and the list still earns its keep. + * + * TABU IS APPLIED DURING THE ARGMAX, not after it. The old code picked a term + * and then discarded it if it was tabu, which turned inhibition-of-return + * into another source of empty scans. Excluding tabu terms from the candidate + * set instead yields the best NON-TABU term, so rotation costs quality rather + * than costing the whole scan. + * + * COST. One pass over g->nodes scoring all candidates at once (12.7k labels × + * <=32 candidates, short strings, good locality), twice per 30 s scan. + * + * POLICY LIVES IN THE SOUL. Thresholds arrive as arguments; the runtime + * measures and ranks, awareness.el decides. Same split as engram_label_df. + * + * Returns the winning token, or "" when the node is missing, has no usable + * text, or every candidate is gated out — "" remains the honest signal that + * this slot yielded no seed, and auto_term_empty_streak still counts it. */ +#define ENGRAM_ST_MAXCAND 32 +#define ENGRAM_ST_TOKLEN 64 +#define ENGRAM_ST_SCANCHARS 400 + +/* Trim leading/trailing non-alphanumerics, then accept only tokens whose core + * is alphanumeric plus '-' and '_' with at least 3 letters. This subsumes the + * quoted-title guard (2026-07-25) and the "