From 3a3d3e1611e018c7665c2915c399e351eb3a530c Mon Sep 17 00:00:00 2001 From: "will.anderson" Date: Fri, 14 Aug 2026 11:23:38 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20geometric=20retrieval=20+=20=C2=A74=20m?= =?UTF-8?q?anaged-memory=20cure=20(live)=20and=20autonomous=20superseding?= =?UTF-8?q?=20self-reification=20(design)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 06 §2.5: shipped 2026-08-14 substrate — structure-gated geometric retrieval (P@5 0.700, semantic-not-lexical) and the §4 write-barrier + generational GC cure (store 1.616GB→38.5MB, RSS→82MB, zero loss, reboot-proven) + LLM token telemetry. - 06 §4.1: autonomous, continuous, superseding self-reification on the heartbeat — reification as an operation OF the engram; explicit reify/rename/run-a-pass as the degenerate case; no gate/pause; flat + overlapping domains; contextual importance; supersession-as-residue; secondary-soul validation, flag-gated. - 06 §6.0: relating as the primitive (one capability; the rest is terrain); perspective calculated via geometric transformations. - 06 §6.4: reasoning as constructive self-argument governed by the verifier. - 03: supersession-as-residue note; live geometric retrieval on route_search. --- docs/architecture/03-data-and-memory.md | 16 +++- .../architecture/06-cognitive-architecture.md | 88 ++++++++++++++++++- 2 files changed, 101 insertions(+), 3 deletions(-) diff --git a/docs/architecture/03-data-and-memory.md b/docs/architecture/03-data-and-memory.md index b5c721b..b2ea7cb 100644 --- a/docs/architecture/03-data-and-memory.md +++ b/docs/architecture/03-data-and-memory.md @@ -165,6 +165,13 @@ Engram nodes are immutable (`memory.el:64-69`). The model is: returns both ids so the caller re-points. This is the `supersedes_id` pattern: new node linked, old preserved, full audit trail. +> **Supersession is residue, not garbage.** The superseded node is the *trail of +> how the current understanding was reached* — kept deliberately, because sometimes +> the truth was in the **old** idea even when the old idea was not itself the truth. +> This is what lets autonomous self-reification (`06` §4.1) run ungated: every +> rename/re-cluster supersedes into this residue chain, so nothing it does is ever +> destructive — the safety is *after* the act, not a gate before it. + > **The hole to know about.** The raw runtime `engram_forget` **does** hard-delete > (frees node + edges, `el_runtime.c:7647`), and the engram HTTP route > `DELETE /api/nodes/:id` calls it directly (`server.el:322-328`). Immutability @@ -228,6 +235,11 @@ Retrieval is **spreading activation, not query matching**: cosine(query, target)` — multiplicative, top-N, with the two-layer background → working-memory promotion (`README.md:27-36`; `el_runtime.c:5892+, 6094+`). `mem_recall` / `/api/activate` fire this and mutate WM; `mem_search` / -`/api/search` are passive lexical scans. The cognitive API's `begin_session` and -`compile_ctx` return a **bounded projection** of the activated set, never the raw +`/api/search` are passive lexical scans — **but as of 2026-08-14 the live +`route_search` runs structure-gated *geometric* retrieval** +(`engram_retrieve_geometric_json`; held-out **P@5 = 0.700**, semantic not lexical — +`skill` returns skill nodes and *rejects* the false-positive `rainfall`), with the +old lexical scan retained at `/api/search-lexical` (see `06` §2.5). The cognitive +API's `begin_session` and `compile_ctx` return a **bounded projection** of the +activated set, never the raw graph (doc 02, §2). diff --git a/docs/architecture/06-cognitive-architecture.md b/docs/architecture/06-cognitive-architecture.md index 7fd1acf..00f5bef 100644 --- a/docs/architecture/06-cognitive-architecture.md +++ b/docs/architecture/06-cognitive-architecture.md @@ -1,6 +1,6 @@ # Neuron — Cognitive Architecture -> **Status: living design document, grounded in source and probed against the live soul (2026-08-13).** +> **Status: living design document, grounded in source and probed against the live soul (2026-08-13; retrieval + §4 managed-memory cutovers and the self-reification design added 2026-08-14).** > This is the *middle layer* of the documentation: below the whitepaper's thesis > (`~/Writing/whitepapers/engram-cognitive-architecture-whitepaper.md`, **v1.5**) and above the > endpoint reference (`~/work/engram-api-reference.md`). It documents *how the mind is designed and why*, @@ -148,6 +148,28 @@ a **clean reseed**: rebuild the durable graph from a known-good snapshot/export, repopulate the `Neighborhood` nodes, and let the soul re-sync. The 28→187 neighborhood reseed (§4) is an instance of this: reification is a derivable pass, so the geometry can always be regrown from the substrate. +### 2.5 Bounded store — the §4 managed-memory cure + geometric retrieval (LIVE / reboot-proven, 2026-08-14) + +Two cutovers landed on the live soul on 2026-08-14, both reboot-proven, zero data loss: + +- **Geometric retrieval (LIVE).** `route_search` now runs structure-gated **geometric retrieval** + (`engram_retrieve_geometric_json`) in place of the old lexical scan; the lexical path is retained as + `/api/search-lexical`. On the held-out set, **P@5 = 0.700** — semantic, not lexical: the query `skill` + returns skill nodes and *rejects* the lexical false-positive `rainfall`. Keystones and edge counts intact. +- **The §4 managed-memory cure (LIVE, flag-gated).** The store bloat — records re-appended on every + checkpoint's full-walk, the CCR's missing managed-memory layer — is cured at the source. A **write-barrier** + (`ENGRAM_WRITE_BARRIER=1`) hashes a node's durable fields and *skips the whole put when unchanged* (no LSN, + no WAL record), flattening checkpoint growth (offline reproduction: 8× growth over 10 think-only checkpoints + → **zero growth** with the barrier on); **generational minor GC** (`ENGRAM_GC=1`) returns whole-dead + node/edge pages to the free list each checkpoint. Backlog reclaimed via the existing merge-safe + `store_compact`: **egm 1.616 GB → 38.5 MB (97.6%)**, pages 98,650 → 2,351, **RSS 1,077 MB → 82 MB**, + nodes/edges preserved exactly (zero loss), boot alive in ~4 s. Also folded in: **LLM token telemetry** + (`llm_last_usage()` now parses nested `usage.{input,output}_tokens`, previously dropped at the C→EL + boundary). Rollback armed at `~/.neuron/engram-rollback-s4-20260814-153754/REVERT.sh`. + +Together these **bound the store's size permanently** (growth flat, not merely swept) while the retrieval it +serves is now semantic — the substrate under everything in §§4–7. + --- ## 3. The data model @@ -232,6 +254,39 @@ per-query recomputation. (the 28 persisted, durable neighborhoods prove it ran) though the write itself was not exercised under the read-only rail. +### 4.1 Autonomous, superseding self-reification (DESIGNED / BUILDING — validating on a secondary soul, 2026-08-14) + +Reification today runs as an explicit pass (`POST /api/reify`). The designed end-state is that **reification is +an operation *of* the engram, not a call made *to* it** — a continuous, autonomous process on the heartbeat, +next to Hebbian edge-formation (§3.3) and consolidation (§6.3), that clusters, names, nests, and promotes its +own neighborhoods as the geometry grows and co-activates. The organizing insight: a mind does not tell itself +"file this under mathematics" — the substrate settles it there. So an explicit `reify` / `rename` / "run a +pass" is the **degenerate, manual-override case** of an operation whose core is always-on and unbidden. + +Design constraints (being validated on a snapshot-clone secondary soul before any prod flag-flip; flag-gated +default-off, so prod is byte-unchanged until enabled): + +- **It just runs — no gate, no pause, no "important call."** There is no privileged tier of reifications that + earns approval-before-commit. It is safe to run ungated *because* of immutability (§3.4): every name/grouping + is **superseded, never overwritten**, so there is no irreversible moment to gate on. Safety lives *after* the + act (supersede), not *before* it (approval). +- **Supersession is residue, not a tombstone.** A re-clustered or renamed neighborhood keeps its prior names as + an ordered chain — the trail of how the understanding matured, with the cause of each shift (autonomous drift + vs. explicit override) recorded. Kept deliberately, because *sometimes the truth was in the old idea even + when the old idea was not itself the truth*; nothing is deleted. +- **Domains are flat and overlapping.** No static importance hierarchy over domains — math is not privileged + over comedy over English. The only standing privilege is the **core** (self-region §7.1 + values). Every + other neighborhood is equal-status; its importance is **contextual** — computed live by spreading activation + given the present context, never a stored field. And membership is **soft and multiple** (the soft-membership + above already models this): a node can belong to several neighborhoods at once (math *can be* comedy), so the + operation uses overlapping community detection, not a hard partition. +- **Bounded + convergent.** It reifies real structure, not noise; dedupes against existing neighborhoods; + composes with the §2.5 write-barrier so unchanged reifications do not re-append each beat; and converges + rather than churning. + +This turns the engram from a graph curated from outside into a mind that organizes itself, with the explicit +call demoted to the override it always was. + --- ## 5. The body / orbit two-zone model (DESIGNED, refined) @@ -268,6 +323,23 @@ The faculties are **named for what they are, not for the matrix operation that i the mind reasons in the language of experience; the linear algebra lives in the whitepaper's Appendix A. This naming convention is a design principle (§10), not decoration. +### 6.0 The primitive — relating — and calculated perspective (framing) + +Underneath the named faculties is a single primitive: **relating.** Meaning *is* relation — a point means +nothing by itself, only by its position relative to others — so every operation reduces to relating: comparing +positions, binding what belongs, laying an edge. In that light the faculties are not a menu of separate powers: +**there is one capability — relating — and rhyme, recall, reasoning, translation, humor are *terrain* it +reaches or *paths* it traces.** A capability is a *composed geometrical function*, which is why capabilities +compose and recurse freely (self-cartography, §4.1, can map its own mapping). + +This makes **perspective calculable.** A perspective is a frame — an origin, a basis, a projection — so a new +one is *computed*, not retrieved, by transforming the space: **translate** the origin onto another's +self-region → empathy; **rotate** the frame → reframe; **project** onto an axis → a lens (read a thing through +cost, or safety); **change of basis** → analogy / metaphor / skill-transfer; **reflect** an axis → negation / +sarcasm; **scale** → abstraction vs. detail. Because a new vantage is a *transformation of the grounded space*, +it carries its grounding with it — unlimited yet grounded creativity: a derivation, never a hallucination. +The operator family (§6.1) and reasoning (§6.4) are instances of this frame. + ### 6.1 The operator family (mixed: LIVE / STAGED / DESIGNED) Activate several reified neighborhoods into working memory, then apply faculty-named operators over their @@ -349,6 +421,16 @@ grounding-and-consistency verifier tiers passed theirs (29/29), on a staged non- after a live cutover rather than relayed. **Still open (DESIGNED):** the formal-symbolic and full predictive verifier tiers, fluent discourse composition, and the fully-geometric generation path. +**Reasoning as constructive self-argument (framing).** In the plainest terms, reasoning is the self arguing +with itself constructively — relating (§6.0) turned inward: one facet of the self engages another (a thing that +is you, but not the entirety of you), and the new thing — the synthesis — forms in the friction. Conversation +is relating with another; reasoning is relating with the other-who-is-you. The verifier is precisely what keeps +that argument *reasoning* and not *rationalization*: it is the facet that refuses to agree unless the claim is +grounded. An argument with a yes-man forms nothing; grounding is the honest second voice. This is why the +verifier is not a bolt-on check but the governing half of the reasoning loop — the same polarity/consistency +axis that catches the "plausible lie" is what makes self-argument converge on truth rather than on what the +mind already wanted to believe. + --- ## 7. The self & the gate @@ -486,10 +568,14 @@ The invariants that govern every subsystem above: | Engram substrate, tiered/WAL store | LIVE (flag-gated) | | Durability: auto-remerge net | LIVE (interim) | | Durability: #56 load-merge-persist fix (events-become-the-graph) | LIVE / reboot-proven | +| Retrieval: structure-gated geometric retrieval (P@5 0.700, `skill` ⊥ `rainfall`) | LIVE / reboot-proven (2026-08-14) | +| §4 managed-memory cure: write-barrier + generational GC (store 1.616 GB → 38.5 MB, RSS → 82 MB, 0 loss) | LIVE / reboot-proven (2026-08-14) | +| LLM token telemetry (`usage.{input,output}_tokens`) | LIVE (2026-08-14) | | Durability: full WAL edge-ownership (remaining hardening) | decision-pending | | Two-store write-through (cultivate → durable) | **known issue, not fixed** | | Data model (nodes/edges/embeddings/immutability) | LIVE | | Reified `Neighborhood` nodes (28 live, 187 reseed pending) | LIVE | +| Autonomous superseding self-reification on the beat (flat + overlapping, contextual importance, residue) | DESIGNED / BUILDING (secondary-soul validation, 2026-08-14) | | Body/orbit two-zone + integration | DESIGNED / refined | | Operator `recall` | LIVE | | Operators recognize/synthesize/discern/gauge-distance (math) | LIVE (compiled) |