Compare commits

...

3 Commits

Author SHA1 Message Date
will.anderson 72fc6ffbf4 docs(engram): cognitive architecture design, prior-art scan, M10 reification, M8 perf profile, cutover-reversal runbooks, and 2026-08-12 session record 2026-08-13 00:22:29 -05:00
will.anderson 4965600d65 docs(engram): M9 geometry-priming reversal runbook + A/B perf profile
Reversal runbook for the ENGRAM_GEOMETRY_PRIMING cutover (default OFF, reversible
flag flip; exact rollback) and the A/B perf profile: default-OFF binary GO
(byte-identical to M8), enabling the flag NO-GO on latency (3.2x/13x) with no
demonstrated recall benefit; safety/sanitizer clean.
2026-08-12 20:29:16 -05:00
will.anderson 8e2269a205 fix(mcp-wrapper): declare real input schemas so tool args actually bite
The cognitive-graph and write tools advertised an empty inputSchema
({"properties":{}}), so MCP clients never sent entity_id/depth/query/
from_id/node_id etc. Graph reads fell back to the full neighborhood
(480-775KB, over transport limits) and write tools (forget, linkEntities,
evolveMemory) had no way to target a node.

- Declare per-tool JSON-Schemas matching the params each soul handler
  already accepts (76 of 87 tools; 11 are genuinely param-less).
- Read + forward the declared args: inspectGraph now honors depth (was
  reading only legacy max_depth), compact (default on), snip, k;
  traverseGraph accepts entity_id and defaults compact on so a depth-2
  walk stays bounded; retrieveKnowledge forwards depth/snip/k.
- compact_flag() reads the raw JSON token so an integer 0 / false / "0"
  opts out correctly (json_get_string could not see an integer and
  silently forced compact back on).

Builds on PR #149's compact projection; keeps the relevance-ranked
bound on by default for graph neighborhoods.
2026-08-10 16:27:03 -05:00
10 changed files with 2493 additions and 96 deletions
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,148 @@
# M10 — Reification: Persisted First-Class Neighborhood Geometry
**Date:** 2026-08-12
**Branch:** `engram-tiered-storage` (worktree `/tmp/engram-tiered-wt`)
**Status:** Reification substrate = **GO** (additive, geometry-priming stays default-OFF).
Enabling geometry-priming = **NO-GO** (latency blocker removed; recall-quality benefit still absent).
**Live `:8742` never touched. Not pushed. Not tagged.**
This is the sequel to [M9 geometry priming](perf/engram-geometry-priming-profile.md), whose A/B
showed enabling the flag cost **3.2× median / 13× p90** latency for **no reliable quality gain**.
The M9 profile named two prerequisites before re-evaluating: **(1) amortize the per-query
descriptor cost** (eigensolve + paged reads on every `engram_activate`) and **(2) center recall
quality against the true store-wide mean**, not a per-query gathered-set approximation. M10 does both.
## What changed vs the original brief (important)
The task began as "reify the geometry into a durable **cache**." Will corrected this mid-flight, and
the correction is the design: **do not build a cache — reify densely co-wired neighborhoods into
first-class, PERSISTED store records** that survive restart, load on boot, and evolve via
supersede+provenance. *"A cache that lies is worse than a slow lookup."* This matches
[the cognitive-architecture design §2](engram-cognitive-architecture.md) (reification = durable
structure, not a fragile derived shortcut) and the memory-core discipline (evolve or forget, never
a stale canonical). Two modes, **no general cache layer**:
1. **Persist first-class** — reified/crystallized neighborhoods (the self; stable topology). The
geometry-priming **hot path reads these**. Never compute geometry on the activation path.
2. **Compute on the fly** — ad-hoc/transient domain geometries (viz, exploration). Uses the
existing M9 `engram_geometry_descriptor`, fresh each call, no storage. Occasional, so its cost
is acceptable.
## Storage schema (first-class, on the existing TLV node store — no new on-disk format)
A reified neighborhood is an ordinary store **node**, so it inherits durability, boot-load,
`store_supersede`, and adjacency for free (design §2: *structure all the way down under one rule*).
| Record | `node_type` | `emb` | `metadata` (`GEO1` text schema) | id |
|---|---|---|---|---|
| Centering frame | `GeoMeanFrame` | the true store-wide mean vector (persisted **once**) | `{}` | `geo-meanframe` |
| Neighborhood | `Neighborhood` | the **raw centroid** (prototype; centroid-ANN-able; `centered = emb meanframe`) | hub id · meanframe ref · scalars (`radius`, `total_variance`, `k_core`, `co_registration`, `n_embedded`) · axis **extents** · member list `{id → membership, centrality, core}` | `nbhd-<hub>-<built_at_ms>` |
Member links are also persisted as edges `relation="member"` (`nbhd → member`). The membership
`{id → w}` in the metadata is what priming reads; it is computed **centered against the persisted
true mean**, which is what resolves the M9 quality caveat.
**Detection (v1, honest).** Hub-anchored neighborhoods over the strong-edge hebb-weighted graph:
compute per-node weighted degree `Σ eff_w` (`eff_w = weight·(1+0.5·hebb)`, non-tombstoned /
non-inhibitory, ≥ `edge_min_weight`) over from+to edges; rank descending; **greedy non-redundant
cover** — reify each hub's descriptor once, skip a hub already a member (`w ≥ cover_membership`) of an
accepted neighborhood, stop at `max_neighborhoods` (default 128). Env-tunable (`min_weighted_degree`,
`max_neighborhoods`, refresh frac) without rebuild.
**Honest caveat — hebb ≈ 0 today.** On the current store there is ~no Hebbian potentiation, so
`eff_w` reduces to the **authored** edge weight; the detected neighborhoods currently reflect
authored graph structure, not learned co-activation. The design is unchanged and self-correcting
once hebb accrues (degree ranking and skeleton shift automatically).
## Boot isolation — why priming-OFF stays byte-identical
The persisted records carry embeddings (centroid / mean) but are **structure, not corpus content**.
On boot they are routed **out** of the resident activation graph into a dedicated resident reify
index, and member edges are skipped from adjacency (matched by the `nbhd-…` id convention, so no real
edge of any relation is affected). Consequences, all verified:
- store-wide mean, hub-degree scan, and the descriptor never admit a structural record as a member
(`geo_is_structural_id` / `node_type` guards);
- vindex / seed selection / results / embedding backfill are **identical** to a store that was never
reified ⇒ `ENGRAM_GEOMETRY_PRIMING` OFF is **byte-identical to M8/M9**.
The resident index is the **loaded form** of the durable records (like the resident node array is the
loaded form of node records, or adjacency of edges). Geometry is computed **once**, offline, and
**persisted**; boot only *parses* — it never recomputes geometry.
## Hot path
`ENGRAM_GEOMETRY_PRIMING=1` resolves the M8 seed set to the best persisted neighborhood — O(seeds)
membership hash lookup; miss → centroid-nearest against the loaded centered centroids — and applies
the M9 damp+prime logic from the persisted membership. **No geometry computed on activation.** Set
`ENGRAM_GEO_PRIMING_NOCACHE=1` to fall back to the M9 per-query descriptor (ad-hoc / A/B control).
## Results (A/B on COPIES; `store_reified` = 128 neighborhoods; live untouched)
**Correctness / durability**
| Check | Result |
|---|---|
| Reified records inert: `A_off` (reified store) vs `C_clean` (no records), id sequence all 15 queries | **MATCH** (byte-identical) |
| Restart survival: reify → checkpoint+close → fresh-process reopen | 128 `Neighborhood` + 1 `GeoMeanFrame` present; index loads **28 ms**; hub-seed lookup HIT **~1 µs** |
| Supersede/provenance: re-reify | prior same-hub record superseded (new timestamped id, old tombstoned); live count stable |
| Build warnings from `el_runtime.c` / `engram_geometry.c` (O2) | **0** (3 pre-existing in generated `engram.c`) |
| ASan/UBSan — module (write/load/lookup) + full server hot path, priming ON, 6 queries | **CLEAN** (all http 200, no trap) |
**Latency (median / p90, 15 queries)**
| config | median | p90 | vs A_off |
|---|---|---|---|
| C_clean (no records, OFF) | 78.6 ms | 82.7 ms | — |
| **A_off** (reified, OFF) | **77.4 ms** | **83.1 ms** | 1.00× |
| **B_on** (reified, **hot path / persisted**) | **81.7 ms** | **85.7 ms** | **1.06× / 1.03× — FLAT** |
| D_nocache (reified, ON, M9 per-query) | 255.3 ms | 872.3 ms | **3.30× / 10.5×** |
Reading a persisted first-class record instead of computing a per-query descriptor **removes the M9
latency blocker** (3.3×/10.5× → 1.06×/1.03×). There is no cold first-query penalty — the index loads
at boot.
**Recall quality (mean pairwise cosine, top-20, TRUE store-wide centered frame, stored vectors)**
| | OFF | ON | Δ |
|---|---|---|---|
| mean over 15 queries | 0.1721 | 0.1555 | **0.0166** |
| polysemous cues (9) | 0.1442 | 0.1441 | 0.0001 |
| queries where ON > OFF | — | — | **3 / 15** |
Even with the true mean and persisted structure there is **no reliable coherence gain** — slightly
negative on average, one sparse win (`self identity values` +0.078) and one notable regression
(`precision over brute force` 0.215). On dense polysemous cues the top-20 head is unchanged
(sub-threshold priming does not reorder it), so their coherence is flat.
## Verdict
- **Reification substrate: GO** (merge additive, geometry-priming default-OFF). It is the durable
first-class structure the memory core needs — persisted, boot-loaded, restart-surviving,
supersede-able, provably inert when OFF — and it turns priming into a flat-latency lookup.
Groundwork for the self-as-structure, multi-scale neighborhoods, and the §5 operators.
- **Enable geometry-priming: NO-GO (still).** The M9 *latency* objection is resolved; the *quality*
objection is not. Keep the flag default-OFF.
### Uncertainties / limits (memory core — flagged)
1. **hebb ≈ 0** ⇒ neighborhoods reflect authored edges, not learned co-activation. The quality result
is likely **understated**; re-evaluate after hebb accrues (or seed hebb from usage).
2. **Hub-anchored detection** can resolve a sparse query to a semantically mismatched neighborhood
(the 0.215 regression). Semantic-aware / co-registration-gated detection is future work.
3. **Top-20 coherence is insensitive** to sub-threshold priming on dense cues — it may be the wrong
metric for what priming does (it warms a floor, it does not reorder the head). A retrieval-utility
or disambiguation-accuracy metric would measure the intended effect better.
4. **v1 simplifications:** axis **direction** vectors are not persisted (extents only; directions
recomputable on the fly); member edges are persisted but inert to activation.
## Reversal
Fully additive and reversible.
- The binary is **byte-identical to M8/M9 when `ENGRAM_GEOMETRY_PRIMING` is unset/0** — the default.
- Reified records live only in stores you explicitly run the reify step against; a store that was
never reified behaves exactly as before (the resident index is empty ⇒ priming no-ops).
- To remove reified structure from a store: tombstone the `Neighborhood` / `GeoMeanFrame` records and
compact (they are ordinary nodes). No format change to undo.
- `ENGRAM_GEO_PRIMING_NOCACHE=1` restores the M9 per-query path for ad-hoc geometries / comparison.
@@ -0,0 +1,120 @@
# Engram Prior-Art Scan
*Dated 2026-08-12. This is an **engineering novelty read**, not legal advice. It is intended to feed a patent/whitepaper priority decision by identifying which claims sit in a clean lane and which are wholly or partly anticipated by existing work. A patent attorney and a formal search (USPTO/Google Patents/Espacenet) should confirm before filing. Where a claim is anticipated, this document says so plainly — the goal is honest scoping, not inflated novelty.*
## How to read this
Engram is an immutable, temporally-provenanced knowledge graph (tombstone-not-delete, supersede-not-overwrite; every node carries `created_at` + `superseded_at` + provenance). Over that graph it computes **geometry descriptors** `G = (centroid, covariance/ellipsoid, skeleton graph, membership weights)` in a joint embedding+graph space, and runs named operators (overlap, combine, distance, difference, analogy=Procrustes, traverse=geodesic) over them. A "self" is a reified geometry. Time-travel is a **query filter** (`created_at ≤ T < superseded_at`), not a transaction-log replay. "Self-occupation" reconstructs the self-geometry/knowledge-state as of `T`, locks it read-only, and converses with it with all post-`T` data masked.
The recurring pattern in the findings below: **every individual primitive is prior art.** Bitemporal reconstruction, memory streams, vector-symbolic composition, geometric KG operators, hindsight-leakage auditing, embedding drift detection — all exist and are well-published. Novelty, where it exists, lives in *specific integrated mechanisms*, and must be claimed narrowly against those primitives. Broad claims ("reasoning as geometry," "reconstruct what was known at T," "detect drift by distance") will be rejected on sight.
---
## (a) Hindsight-free decision auditing via immutable temporal knowledge-state reconstruction + future-masked occupation
**Claim (restated narrowly).** A method for auditing a past decision by (1) *constructively reconstructing* the exact knowledge-state a decision-maker held at time `T` from an immutable, tombstone+supersede provenance graph (selecting nodes live-at-`T` via `created_at ≤ T < superseded_at`), (2) recomputing the derived concept/self geometry over only that live-at-`T` slice, and (3) presenting that reconstructed state read-only, with all post-`T` nodes masked, as the sole evidentiary basis for judging the decision — such that the reconstruction is tamper-evident *because* nothing is ever overwritten or deleted.
**Closest prior art.**
- **HindsightBench** (Aug 2026) — a *black-box behavioral audit protocol* that detects parametric hindsight in time-indexed LLM decision tasks by manipulating the *asserted date* in the prompt (Revealed/Date-only/Masked/Transplant arms) and measuring behavioral shift. It explicitly does **not** reconstruct a knowledge state from provenance; it does not require corpus access or logprobs. It also reports that *instructed forgetting fails* — a 52% performance gap vs. true ignorance. https://arxiv.org/abs/2607.18867
- **Agentic Time Machine** (Jun 2026) — wraps web tools with a *leakage filter* that blocks post-cutoff or answer-revealing content before it reaches the agent (for forecasting benchmarks). https://arxiv.org/pdf/2606.21013
- **Zep/Graphiti** — bitemporal KG memory that can answer "what was the user's plan in January?" via point-in-time recall over event-time + ingestion-time. https://arxiv.org/abs/2501.13956 , https://www.getzep.com/ai-agents/temporal-knowledge-graph/
- **Auditable clinical-AI provenance frameworks** — immutable, timestamped "source-to-decision" trails recording the exact evidence shown to a clinician, for FDA transparency/liability. https://pmc.ncbi.nlm.nih.gov/articles/PMC12913532/
- **Hindsight-bias clinical literature** — retrospective case-note review is critically distorted by outcome knowledge; reconstructing the decision-maker's past perspective is the known mitigation. https://www.researchgate.net/publication/330427526 , https://kevinmd.com/2026/03/how-hindsight-bias-distorts-clinical-medicine.html
**What is genuinely differentiated.** The primitives — bitemporal point-in-time recall (Zep), immutable evidence trails (clinical AI), hindsight-bias mitigation by perspective reconstruction (clinical psych), leakage filtering (Agentic Time Machine), hindsight auditing (HindsightBench) — are all taken. What appears *unclaimed* is the specific combination: **constructive knowledge-state reconstruction from an immutable tombstone+supersede graph, used as the affirmative evidentiary substrate for judging a decision**, where correctness of the future-mask is *guaranteed by the data model* (a node is either live-at-`T` or it is not) rather than by prompt instruction or a heuristic content filter. HindsightBench and Agentic Time Machine both operate on the *model's* contaminated parametric memory and fight leakage behaviorally/heuristically; Engram sidesteps parametric leakage by making the *evidence set itself* provably `T`-clean and then recomputing geometry over it. The tamper-evidence-by-construction angle (append-only provenance ⇒ the reconstruction cannot be silently backdated) is also not present in the behavioral-audit line.
**Scoped-claim recommendation.** Claim the *pipeline*, not the goal: "reconstructing a decision-maker's knowledge-state as of `T` by selecting live-at-`T` nodes from an append-only tombstone+supersede provenance graph and recomputing derived concept/self geometry over that slice, then serving it read-only with post-`T` nodes masked as the evidentiary basis for decision review." Anchor on (i) constructive reconstruction from immutable provenance (not prompt-based date assertion), (ii) mask-correctness guaranteed by the data model, (iii) recomputed *geometry* (not just fact recall) as the reconstructed state. Do **not** claim "hindsight-free auditing" broadly, "point-in-time recall," or "immutable audit log" — all taken.
**Verdict: PARTIALLY TAKEN** (the goal and every primitive are taken; the constructive-reconstruction-from-immutable-provenance-as-evidentiary-substrate integration looks clean if narrowly scoped).
---
## (b) Drift detection via geodesic displacement of an anchored self-geometry
**Claim (restated narrowly).** A method that reifies an agent's "self" as a geometry descriptor with a *designated stable value-core anchor* and a mutable *periphery*, and classifies change by **decomposition**: extension of the periphery (core displacement ≈ 0) is scored as *growth*, whereas geodesic displacement of the *core* is scored as *corruption* — measured as geodesic distance between `self(now)` and the anchored `self(reference)` on the graph+embedding manifold.
**Closest prior art.**
- **Embedding / concept-drift detection** — mature field: distribution-distance of embeddings, per-label distributions (Drift Lens), K-core-distance from a dense "core" of baseline logic, growing average distance from baseline anchors as the drift signal. https://www.evidentlyai.com/blog/embedding-drift-detection , https://ieeexplore.ieee.org/iel7/9679833/9679835/09679880.pdf , https://www.sciencedirect.com/science/article/pii/S0925231225018624
- **Agent identity/goal-drift governance** — identity-hash functions over characteristic behavior for drift detection; "dominant" core persona preventing fragmentation; reflection-based long-term self-model evolution vs. short-term compensation. https://arxiv.org/pdf/2604.14717 (Layered Mutability) , https://www.researchgate.net/publication/397950116 (Agent Goal Drift in Stateful Systems)
- **Persistent Identity multi-anchor architecture** — explicit *anchors* for resilient agent identity/memory continuity. https://arxiv.org/pdf/2604.09588
**What is genuinely differentiated.** "Distance from an anchored baseline core = drift" is squarely prior art (K-core-distance, baseline-anchor distance growth). Agent-identity work already has *core-vs-drift* and *anchors*. What is not obviously present is the **core/periphery decomposition of drift into two distinct, oppositely-valenced outcomes on a reified self-*geometry*** — i.e., using a *geometric* self-model (centroid + covariance/ellipsoid + skeleton) where *growth* is formally "periphery ellipsoid expands while core centroid/anchor stays fixed" and *corruption* is "core centroid/anchor is geodesically displaced." Existing drift work treats all displacement as drift (bad); it does not carve legitimate growth from corruption via a *fixed value-core* on a self-geometry. The specific formalization — geodesic (graph-aware, non-Euclidean) displacement of a *pinned* value-core sub-geometry vs. free peripheral expansion — is the differentiator.
**Scoped-claim recommendation.** Claim "detecting agent value-corruption by measuring geodesic displacement of a *pinned value-core sub-geometry* of a reified self-geometry, while treating expansion of the peripheral geometry with a stationary core as non-corrupting growth." Emphasize (i) the self is a *geometry descriptor* with an explicitly designated immutable core anchor, (ii) growth vs. corruption is a *decomposition* (two signals), not a threshold on one distance, (iii) geodesic/graph-aware metric. Do **not** claim "drift detection by embedding distance" or "anchored baseline comparison" — taken.
**Verdict: PARTIALLY TAKEN** (distance-from-anchor drift is taken; the growth/corruption core-vs-periphery decomposition on a reified self-geometry is the narrow clean sliver — and it's the weakest/most crowded of the five).
---
## (c) Reasoning as composable geometry operations over a persistent temporally-provenanced graph
**Claim (restated narrowly).** A reasoning method in which inference steps are *explicit, named, first-class operators* (overlap, combine, distance, difference, analogy=Procrustes alignment, traverse=geodesic) applied to geometry descriptors computed over a *persistent, immutable, temporally-provenanced* knowledge graph — such that each reasoning step is individually inspectable, logged with provenance, and *replayable* against a past graph state; as distinct from implicit, unnamed activation/attention transforms inside a neural net.
**Closest prior art.**
- **Vector Symbolic Architectures / HRR / SDM** (Plate, Kanerva) — the canonical "algebra over vectors": binding, bundling/superposition, permutation, similarity; explicitly compositional/symbolic reasoning via vector operations. This is the strongest prior art for "named composable operators over vectors." https://www.emergentmind.com/topics/holographic-reduced-representations-hrrs , https://arxiv.org/pdf/2512.14709 (Attention as Binding)
- **Geometric KG query embeddings (Query2Box-lineage)** — reasoning as *named geometric operators* (projection, intersection) over box/region embeddings; geometric multi-hop reasoning; geometry-interaction KG embeddings. https://arxiv.org/html/2505.12369v2 , https://ojs.aaai.org/index.php/AAAI/article/view/20491/20250
- **Geometry-of-reasoning / embedding-space reasoning** — CoT as trajectories/flows in representation space; vector algebra + manifold geometry for deduction/induction/analogy. https://arxiv.org/abs/2510.09782 , https://arxiv.org/pdf/2504.02018
- **Riemannian knowledge manifolds** — geodesics as shortest semantic paths with a convergent geodesic solver. https://arxiv.org/html/2606.05907v2
- **Neuro-symbolic propose-verify** — explicit symbolic operations + solver verification.
**What is genuinely differentiated.** "Reasoning as composable vector/geometry operations" is *thoroughly* prior art — VSA/HRR own the compositional-operator framing; Query2Box owns named geometric operators (projection/intersection) for KG query answering; geodesic traversal over semantic manifolds is published. The individual operators (overlap≈intersection, distance, geodesic-traverse, Procrustes-analogy) each exist. The candidate differentiator is *not* any operator and *not* "geometry as reasoning" — it is the **coupling of the operator calculus to the immutable temporal-provenance substrate**: every operator input is a live-at-`T` geometry, every step is provenance-stamped, and the whole derivation is *replayable against a reconstructed past graph state* (i.e., operator-level temporal reproducibility + auditability). VSA/Query2Box run over static/atemporal embedding stores with no provenance and no time-travel; geometry-of-reasoning work is about a neural net's *internal* trajectory, not an external audited calculus. So the calculus itself is taken; "an *auditable, replayable* geometry calculus whose operands are temporally-reconstructed geometries" is the narrow lane.
**Scoped-claim recommendation.** Do **not** claim a "calculus of thought," "reasoning as geometry," or any specific operator (overlap/difference/geodesic/Procrustes) — all taken. Claim only the integration: "an audit trail in which each named geometric reasoning operator is provenance-stamped and its operands are geometry descriptors reconstructed from an immutable temporal graph as-of a query time, enabling deterministic replay of a reasoning derivation against a past knowledge-state." The defensible novelty is *temporal reproducibility + provenance of the operator chain*, not the operators.
**Verdict: TAKEN** (as "reasoning as composable geometry ops" — VSA/HRR + Query2Box + geometry-of-reasoning fully occupy it). Only the *auditable/replayable-over-immutable-temporal-substrate* framing survives, and it survives as a thin sliver of (a)/(e), not as an independent claim.
---
## (d) Self-occupation with engineered future-masking
**Claim (restated narrowly).** A method for reasoning *as* a past self: reconstruct the self-geometry and knowledge-state as of `T` from the immutable provenance graph, *rigorously enforce the `created_at ≤ T` cut at the data layer* (all post-`T` nodes structurally excluded, not instructed-away), lock the reconstruction read-only, and drive a conversational/reasoning session that is provably uncontaminated by hindsight — the mask being a property of the substrate, not of a prompt or the model's willingness to "forget."
**Closest prior art.**
- **HindsightBench** — establishes the *problem* rigorously and shows that prompt-level "pretend it's `T`" fails badly (instructed forgetting ≠ ignorance; 52% gap; date assertions obeyed but hindsight still leaks). This is the strongest adjacent art and, helpfully, *motivates* Engram's substrate-level approach rather than anticipating it. https://arxiv.org/abs/2607.18867
- **Agentic Time Machine** — closest *mechanism*: a leakage filter blocking post-cutoff content before it reaches the agent. But it filters *tool outputs* heuristically for a forecasting benchmark; it does not reconstruct and occupy a *reified past self/knowledge-state*. https://arxiv.org/pdf/2606.21013
- **Causal Agent Replay** — counterfactual replay/attribution of agent failures (replay, but not future-masked past-self occupation). https://arxiv.org/abs/2606.08275
- **Chronologically-consistent pretraining / counterfactual-anchored decoding / forget-retain logit adjustment** — model-internal mitigations of parametric leakage (named in HindsightBench). Different layer entirely.
**What is genuinely differentiated.** The field is actively fighting hindsight leakage at the *model* layer (pretraining, decoding, logit surgery) and at the *tool-output* layer (heuristic leakage filters). Engram's move is orthogonal and, per HindsightBench's own findings, addresses the failure mode the field just documented: **enforce the cut at the evidence/data layer via an immutable time-indexed graph, so the "past self" is a reconstructed read-only geometry whose accessible universe is exactly the live-at-`T` slice.** No source found reconstructs a *reified self-geometry* as of `T` and *converses with it* as a first-class object. The differentiators: (i) the masked entity is a *reconstructed self*, not just filtered context; (ii) mask correctness is structural (a node's `created_at` either satisfies the cut or the node is absent) rather than heuristic/instructed; (iii) it is tamper-evident via append-only provenance. Note the residual honesty caveat: if the *underlying LLM* used for the conversation has parametric hindsight, Engram's substrate-clean evidence does not fully neutralize it — the claim must be about the *evidence/state* being `T`-clean, which is the part Engram genuinely controls.
**Scoped-claim recommendation.** Claim "reconstructing a reified agent self-geometry and knowledge-state as-of `T` from an append-only temporal provenance graph and conducting a read-only reasoning/conversation session over it in which the accessible node universe is structurally restricted to the live-at-`T` slice (data-layer future-masking), yielding a `T`-clean evidentiary state." Lean on *structural* (data-model-guaranteed) masking vs. *instructed/heuristic* masking, and on the *reified-past-self* object. Explicitly scope to the evidence-state cleanliness (not a claim that the LLM has zero parametric leakage). Do **not** claim "prevent hindsight in LLMs" or "leakage filtering" broadly.
**Verdict: CLEAN LANE** (narrowly — data-layer/structural future-masking over a *reconstructed reified past self* is not occupied; adjacent art is behavioral-audit, tool-output filtering, or model-internal mitigation. This is the strongest of the five, precisely because HindsightBench shows the prompt-level approach fails and no one is doing substrate-level self-reconstruction).
---
## (e) Query→geometry temporal reconstruction with NO transaction logs
**Claim (restated narrowly).** Reconstructing a past knowledge-state as a *query-time filter* over immutable, per-node timestamped provenance (`created_at ≤ T < superseded_at`) followed by *recomputation of the geometry descriptors* over that slice — with **no event/transaction log and no periodic snapshots**; the immutable per-node provenance *is* the temporal record, and derived geometry is recomputed rather than stored/replayed.
**Closest prior art.**
- **Bitemporal databases (XTDB, et al.) / event sourcing** — "as-of" point-in-time queries over valid-time + transaction-time; immutability as audit log. Critically, XTDB describes reading bitemporal data as a process *"similar to event sourcing… playing through the history… in reverse system-time order"* — i.e., the mainstream bitemporal model *is* replay/reconstruction-through-history. https://v1-docs.xtdb.com/concepts/bitemporality/ , https://www.juxt.pro/blog/value-of-bitemporality/
- **Zep/Graphiti** — bitemporal (event-time T + ingestion-time T) fact validity + supersession chains; point-in-time recall. https://arxiv.org/abs/2501.13956
- **TKG reasoning frameworks / ElephantBroker-class runtimes** — "immutable fact store, all temporal weighting applied at query time; facts created after the query timestamp excluded, facts superseded after query timestamp treated as current; invalidate by writing `t_invalid` rather than delete." This is *very* close to Engram's filter and supersede/tombstone semantics. https://www.emergentmind.com/topics/temporal-knowledge-graph-reasoning-tkgr , https://arxiv.org/pdf/2603.25097
- **Numerous bitemporal/immutable-DB patents** (point-in-time reconstruction, retroactive/historical transactions). e.g. US 11,935,046; US 8,812,512 (via USPTO search) — a patent attorney must clear these.
**What is genuinely differentiated.** The *temporal filter* (created-before, superseded-after) and *tombstone-not-delete / supersede-not-overwrite* are **standard bitemporal KG practice** — Zep and the TKGR frameworks describe almost exactly this. So the reconstruction-by-filter primitive is TAKEN, and "immutable provenance instead of a mutable audit log" is TAKEN (that's the bitemporal value prop). The only thing that is *not* standard: what gets reconstructed is not just a set of *facts/edges* but a set of **derived geometry descriptors (centroid/covariance/skeleton/membership) recomputed over the live-at-`T` slice** — i.e., recompute-geometry-on-read rather than store-and-replay. Bitemporal DBs reconstruct *records*; Engram reconstructs *derived manifold structure*. The "no transaction log / no snapshot — provenance IS the temporal record, geometry is recomputed" framing is a design stance that is defensible only if paired with the *geometry recomputation*; on its own it is indistinguishable from XTDB/Zep.
**Scoped-claim recommendation.** Do **not** claim bitemporal reconstruction, "as-of" queries, tombstone/supersede, or "immutable provenance as audit record" — all squarely taken (Zep, XTDB, TKGR, patents). Claim only: "reconstructing a *derived geometry descriptor set* (centroid/covariance/skeleton/membership) for a past knowledge-state by recomputing it on-read over the live-at-`T` node slice, without storing per-`T` geometry snapshots or a geometry-mutation log." The novelty is *geometry-recompute-on-read over a bitemporal slice*, not the slice.
**Verdict: TAKEN** (as "query-filter temporal reconstruction over immutable provenance" — Zep + XTDB + TKGR own it outright). Only "recompute *derived geometry* on-read, snapshot-free" survives, and it is really a facet of (c)/(a) rather than an independent claim.
---
## Summary
| Claim | Verdict | Narrowest defensible (clean-lane) framing |
|---|---|---|
| **(a)** Hindsight-free decision auditing via reconstructed knowledge-state + future-masked occupation | **PARTIALLY TAKEN** | Constructive knowledge-state reconstruction from an *append-only tombstone+supersede* graph, used as the *affirmative evidentiary substrate* for decision review, with mask-correctness guaranteed by the data model and tamper-evidence by construction — not prompt/date-assertion (cf. HindsightBench) and not tool-output filtering (cf. Agentic Time Machine). |
| **(b)** Drift as geodesic displacement of anchored self-geometry | **PARTIALLY TAKEN** | Growth-vs-corruption *decomposition* of change on a reified self-*geometry* via geodesic displacement of a *pinned value-core sub-geometry* vs. free peripheral-ellipsoid expansion. (Crowded; weakest lane.) |
| **(c)** Reasoning as composable geometry ops over a temporal graph | **TAKEN** | Only survivor: *provenance-stamped, replayable* operator chain whose operands are geometries reconstructed as-of a query time (temporal reproducibility of the derivation) — never the operators or "geometry as reasoning" themselves. |
| **(d)** Self-occupation with engineered future-masking | **CLEAN LANE** (narrow) | Reconstruct a *reified past self-geometry* and converse with it read-only, with the accessible node universe *structurally* restricted to the live-at-`T` slice (data-layer masking) — not instructed forgetting (which HindsightBench shows fails) and not heuristic content filtering. |
| **(e)** Query→geometry temporal reconstruction, no transaction logs | **TAKEN** | Only survivor: recompute *derived geometry descriptors* on-read over the live-at-`T` slice, snapshot-free — never the bitemporal filter, tombstone/supersede, or "immutable provenance as record," all of which Zep/XTDB/TKGR own. |
## Overall posture
**Broad claims over primitives will be rejected.** Each of the five candidate claims decomposes into (i) a primitive that is unambiguously prior art and (ii), in three of five cases, a thin integrated mechanism that appears unclaimed. The prior art is strong and specific: Zep/Graphiti and XTDB own bitemporal point-in-time reconstruction and supersession (kills the broad reads of (a) and (e)); VSA/HRR and Query2Box own composable geometric/symbolic operators (kills the broad read of (c)); embedding concept-drift and agent-identity-anchor work own distance-from-baseline drift (kills the broad read of (b)); and HindsightBench + Agentic Time Machine own hindsight *auditing* and *leakage filtering* (bound (a) and (d)).
**Novelty lives in the specific integrated mechanisms, narrowly scoped.** The two genuinely defensible ideas are: **(d) substrate-level future-masking of a reconstructed, reified *past self*** — which is the strongest, and is *strengthened* by HindsightBench's finding that the prompt-level approach everyone else uses fails by ~52%; and **(a) constructive knowledge-state reconstruction from immutable provenance as the affirmative evidentiary basis for decision auditing**, distinct from behavioral probing. The unifying, defensible thread across (a)/(d)/(c)/(e) is *structural guarantee by the immutable data model* — the future-mask, the tamper-evidence, and the operator-chain replayability are all properties of the append-only substrate rather than of prompts, heuristics, or model cooperation. That "guaranteed-by-construction" framing is the honest core of any priority filing. Claims (c) and (e) should be folded in as *facets* (auditable/replayable geometry over reconstructed slices) rather than filed as standalone claims, and (b) should be filed only if the core/periphery decomposition can be made rigorous, since the surrounding drift-detection art is dense.
*Caveats for the filing team: (1) this scan covered academic/product/blog prior art via web search, not a formal patent search — several bitemporal/immutable-DB patents surfaced (e.g. US 11,935,046; US 8,812,512) and must be cleared on Google Patents/Espacenet/USPTO. (2) Claim (d)'s guarantee is that the *evidence-state* is `T`-clean; it does not by itself neutralize parametric hindsight in whatever LLM reasons over that state — scope the language accordingly. (3) Dates on several 26062607 arXiv preprints are very recent; confirm publication precedence relative to Engram's earliest documented conception date.*
@@ -0,0 +1,97 @@
# Perf Profile — M9 Geometry Priming (ENGRAM_GEOMETRY_PRIMING)
**Date:** 2026-08-12
**Branch:** `engram-tiered-storage`
**Change:** `ENGRAM_GEOMETRY_PRIMING` (default OFF) in `el_runtime.c` `engram_activate` + `engram_geometry.c`
**Method:** A/B over 15 representative queries against a **copy** of the recovered store
(`~/.neuron/engram/.neuron.egm.disabled`, ~4190 embedded nodes, 768-d nomic-embed-text),
throwaway HOME, ports 48799/48800. **Live `:8742` never touched.** `engram.c` (folded from
`server.el`) reused byte-identical across M8 and M9, so the only variable is `el_runtime.c`.
Three configs: **A** = M9 flag OFF · **B** = M9 flag ON (`=1`) · **C** = pre-M9 M8 baseline binary.
---
## Build
| Artifact | Result |
|---|---|
| M9 `-O2` link (`… engram_geometry.c … -lssl -lcrypto -lcurl -lpthread -lm`) | rc=0, 499,720 B arm64 |
| ASan/UBSan link (`-fsanitize=address,undefined -O1`) | rc=0, 1,945,616 B |
| Warnings from `el_runtime.c` / `engram_geometry.c` | **0** (3 pre-existing `-Wparentheses-equality` in generated `engram.c` only) |
| `nm`: `engram_geo_mean_build`, `engram_geometry_descriptor` | present (T); `eg_geometry_priming_on` inlined (static-local `.cached` present in both binaries) |
> Note: the bare `cc … -lm` link fails with undefined `_curl_*` — `el_runtime.c` uses libcurl for
> the ollama embedder. The canonical link must include `-lssl -lcrypto -lcurl` (per `link.sh`).
---
## Latency (wall-clock, `curl -w %{time_total}`, 15 queries)
| config | median | p90 | min | max |
|---|---|---|---|---|
| **A — M9 OFF** | **77.8 ms** | 80.5 ms | 71.1 | 84.2 |
| C — M8 baseline | 76.0 ms | 81.2 ms | 71.4 | 91.4 |
| **B — M9 ON** | **249.6 ms** | **1039.2 ms** | 169.2 | **1256.3** |
- **OFF adds zero cost:** 77.8 ms vs M8 76.0 ms — within noise. The flag is free when unset.
- **ON regresses hard:** **3.21x median** (+171.8 ms), **~13x p90** (80 → 1039 ms), max **1.26 s**.
- The warm-cache path (global mean already built) is ~0.5 s; the cold path pays the full
`engram_geo_mean_build` scan (O(N·dim) over ~4190 × 768). The persistent per-query cost is the
**descriptor** itself — covariance eigensolve over up to `max_members` (400) × 768-d plus one
`store_get_node` **paged read per member** — run on *every* activation while the flag is ON.
---
## Retrieval quality (the win it was supposed to buy)
**Coherence** — mean pairwise cosine in centered space, top-20 by activation strength
(node embeddings re-derived via nomic-embed-text; centered against the mean of the gathered
result set — the *true* store-wide mean is not exposed by the API, flagged as an approximation):
| | OFF | ON | Δ |
|---|---|---|---|
| mean over 15 queries | 0.1067 | 0.1114 | **+0.0047 (noise)** |
| queries where ON > OFF | — | — | **4 / 15** |
Two real sparse-cue wins (`self identity values` +0.118, `hebbian learning edges` +0.064), but the
**polysemous cues — the disambiguation target — are mostly flat or down.**
**Disambiguation** — no clean "scope to one sense" pattern on polysemous cues. Additions/drops are
small (±2..8 of 300-item sets) and not sense-coherent (e.g. `memory` gains some on-domain nodes but
also infra items; `core` similar).
**Count shift:** ON adds sub-threshold neighbors to sparse cues (+3..+4) and trims a few from dense
polysemous cues (1..3) — consistent with priming warming sparse neighborhoods and damping
off-domain seeds on dense ones, but the net does not move measured coherence.
---
## Correctness / safety (all pass)
| Check | Result |
|---|---|
| Byte-identical: **A (OFF) == C (M8)** result id sequence + order, all 15 queries (incl. 301/294/263-item sets) | **PASS** (only wall-clock ACT-R fields differ; `activation_strength` max \|Δ\| = 2e-5) |
| WM `promoted` ≤ 24 under ON | holds (exactly 24 on dense cues) |
| Queries with results under OFF → empty under ON | 0 |
| Crash / hang under ON | none (max hops = 1) |
| ASan + UBSan under ON (cold build + warm descriptor paths) | **CLEAN** — no report |
---
## Conclusion
- **Deploy default-OFF binary: GO.** Byte-identical to M8, zero cost off, clean build, sanitizer clean.
- **Enable flag: NO-GO (for now).** 3.21x median / ~13x p90 latency for no reliable quality gain
(coherence +0.0047 mean = noise; no clean disambiguation). Correctness/safety are fine — it simply
does not earn its cost. **This is a cost/benefit NO-GO, not a defect.**
### Prerequisites before re-evaluating the flag
1. **Amortize the descriptor cost.** The per-query geo-mean build + eigensolve + paged reads
dominate. Cache the neighborhood descriptor (it is the M10 cell-assembly cache's job) and/or
compute geometry periodically/off-hot-path rather than on every `engram_activate`.
2. **Center against the true store-wide mean** (the `GeoMeanCache` already computes it) rather than
a per-query gathered-set approximation, and re-measure coherence — the current signal may be
understated by the approximation.
3. **Re-tune** `ENGRAM_GEO_SEED_LO` / `PRIME_SCALE` / `PRIME_MAX` and re-measure only after (1),
so tuning is not chasing latency noise.
@@ -0,0 +1,140 @@
# Engram Tiered Storage — M8 Performance Profile (milestone-0 sample)
**Milestone:** M8 (ANN wired into `engram_activate` seed selection). Trunk =
worktree `/tmp/engram-tiered-wt`, branch `engram-tiered-storage`, HEAD `1507614`.
**Date:** 2026-08-12. **Author:** first full-binary build + profile of the tiered trunk.
This is **sample zero** of an accumulating per-milestone profile series (see the
BUILD-LEDGER "keep performance telemetry CONTINUOUSLY" decision). Compare future
milestones (M9, M10, …) against these numbers.
## Methodology (read this before trusting a number)
- **On a COPY, never live.** All runtime measurements used a read-only copy of the
live store booted on a **non-live port (:8798)** with a **throwaway `$HOME`**. The
live engram service (:8742, `~/.neuron/engram`) was never touched.
- **Two store substrates were used:**
1. *Live-egm copy* (`neuron.egm` 458 MB + `neuron.wal` 44 MB, copied read-only) —
**this substrate crashes both the M8 binary and the live binary on boot** (see
Integration Findings). Unusable for runtime measurement.
2. *Clean import* — a dir seeded with only `snapshot.json` (65 MB, stable 06:10),
which the binary imported into a **fresh 59.9 MB `neuron.egm`**. All healthy
runtime numbers below are from this substrate (real graph content, healthy store).
- **Build machine:** Apple Silicon (arm64), macOS. Native `cc -O2` compile; fold done
in a memory-capped (`--memory=3g --memory-swap=3g`, no swap) `linux/amd64` container
running `elc-linux-amd64`.
- Hardware/thermals uncontrolled; single run per metric unless noted. Treat as
order-of-magnitude, not benchmark-grade, except the module benchmark (Test suite).
## Build
| Metric | Value |
|---|---|
| Fold input | `engram/src/server.el` (44,273 B El, no imports) |
| Fold output | `engram.c` (30,855 B, 645 lines C), `ELC_EXIT=0`, **0 fold warnings** |
| Fold time (pure elc) | sub-second (server.el is small, importless) |
| Fold container wall | ~38 s (dominated by one-time `apt-get install libcurl4` in the throwaway container; the elc invocation itself is <1 s) |
| Compile | `cc -O2 -DHAVE_CURL engram.c el_runtime.c engram_store.c engram_vindex.c -lssl -lcrypto -lcurl -lpthread -lm` |
| Compile time | **1.83 s** wall |
| Compile warnings | **3**, all `-Wparentheses-equality` in the *folded* `engram.c` (El if-expr codegen emits `if ((x == 0))`); cosmetic. `el_runtime.c` / `engram_store.c` / `engram_vindex.c` compiled **0 warnings** — notably none around the M8 deferred `free(e_eff)` or the vindex integration. |
| Binary | **482,008 B (471 KB)** Mach-O arm64 executable |
| ANN linkage verified | `nm`: `vindex_search`, `vindex_build_from_store`, `eg_vindex_sync`, `_eg_vindex`, `store_scan_nodes`, `engram_store_boot`, `engram_activate` all present |
## Boot & footprint (clean-import substrate)
| Metric | Value |
|---|---|
| Boot from healthy `neuron.egm` | **~2 s** to listening |
| Boot from `snapshot.json` (one-time import + fresh egm) | **~9 s** |
| node_count | **13,036** (matches ledger import-dedup: 13,038 snapshot 2 dup-id entries) |
| edge_count / layer_count | 43,402 / 5 |
| embedded_count | 4,190 |
| Fresh egm size | **59.9 MB** (vs the live egm's bloated 458 MB — see Findings) |
| RSS after boot | **126.2 MB** (whole 13k-node/43k-edge/4,190-emb graph resident + fresh egm) |
## Activation latency — q="bullshit" (clean-import substrate)
50 sequential `GET /api/activate?q=bullshit&limit=10&depth=3`:
| Metric | Value |
|---|---|
| p50 | **34.35 ms** |
| p95 | **35.26 ms** |
| min / max | 33.50 ms / 9,886 ms |
| Sample | n=50 |
- The **max = 9.9 s is the first call only** — a cold query-embedding fetch
(`eg_embed_fetch` → Ollama `nomic-embed-text`, cold model load). All subsequent
calls hit the single-slot query-embedding cache (`_eg_qcache`) → **34 ms steady state**.
- **This 34 ms is the lexical/spread path, NOT the ANN seed path.** The store's 4,190
embeddings were generated by the live neuron's native embedding model; the harness's
`nomic-embed-text` query vectors are a **different vector space**, so no candidate
cleared `ENGRAM_EMBED_SEED_MIN=0.60` (`act-stats`: `dup_seeds:0`, `ctx_cos:-2.000`
sentinel) → results empty, ANN discovery produced no admitted seeds. Environmental
(embedding provenance), **not** an M8 defect. The ANN wiring still *executed*
(embed fetch succeeded, `embed_breaker_open:0`; `eg_vindex_sync` + seed block +
`free(e_eff)` all ran) **without crashing** on the real store.
## KEY M8 METRIC — ANN vs O(n) seed selection (authoritative)
Because the HTTP path can't exercise ANN seeding without embedding-space parity, the
authoritative ANN-vs-exact-scan numbers come from the **module benchmark**
(`engram/test/run_vindex_tests.sh`, PASS 1, optimised), which measures the exact
`vindex_search` code the M8 wiring calls, at full size:
| N (768-dim vectors) | Brute-force (O(n)) | ANN (HNSW) | **Speedup** |
|---|---|---|---|
| 5,000 | 3.475 ms/query | 0.353 ms/query | **9.8×** |
| 20,000 | 13.809 ms/query | 0.698 ms/query | **19.8×** |
- **recall@10 = 0.9365** at `ef_search=128` (gate ≥0.90 — **PASS**). Lower ef trades
recall for latency: ef=64→0.844, ef=32→0.750, ef=10→0.625.
- **Determinism:** two independent seeded builds give byte-identical query results.
- **`vindex_build_from_store`** over a real `engram_store`: inserts exactly the
embedded nodes, top-1 resolves to the correct node id at ~0 distance.
- **HNSW build cost (single-threaded, note for boot/index-build budgeting):**
5,000 vectors ≈ 1535 s, 20,000 vectors ≈ 75 s. In-process the index is built
**lazily on first activation** (`eg_vindex_sync`) and grown incrementally; the
M8 seed block only fires once `vindex_size ≥ ENGRAM_EMBED_SEED_K`. At the real
store's 4,190 embedded nodes this is a **one-time few-second first-activation
cost** — worth watching as the embedded set grows (a future milestone may want
to build the index at boot or persist it via `vindex_save`/`vindex_load`).
## Seed-set parity note (why there is no runtime A/B toggle)
M8 has **no ANN on/off env flag** by design (`ENGRAM_EMBED_SEED_K` is a compile-time
constant). The exact O(n) cosine scan is **preserved verbatim** and "tops up" any seed
slot the ANN leaves unfilled; every ANN candidate is admitted through the *identical*
cosine/dedup/threshold gate the exact scan uses. So ANN changes only *which nodes are
discovered and how fast*, never the final seed set — parity is **structural**, not
A/B-tested via flag. Recovered-behaviour-when-index-absent is the pre-M8 exact scan.
## Integration Findings (first full build of the tiered trunk)
1. **CRITICAL / pre-existing (NOT M8): `btree_insert` stack-buffer-overflow on
opening the live 458 MB `neuron.egm`.** `SIGABRT` (`__stack_chk_fail`) via
`btree_insert ← btree_put ← edge_place/apply_edge_put ← engram_open ←
engram_store_boot`. Reproduces on the M8 binary **and** the deployed live binary —
**the live engram :8742 was crash-looping (18 crash reports 16:18→17:44 on this
date; service refusing connections).** A clean import into a fresh 59.9 MB egm does
**not** crash (43k edges load fine), so the trigger is the specific pathological
live store: 458 MB (8× the healthy 59.9 MB) from a day of churn/tombstones with no
M5 compaction + a 44 MB un-checkpointed WAL replayed on open. The bug is in
`engram_store.c` (the edge B-tree / WAL-redo path), **upstream of everything M8
touched** (M8 lives in `el_runtime.c::engram_activate`). Fix required before any
re-cutover; `btree_insert` must bound-check regardless of on-disk content.
2. **M8 deferred `free(e_eff)`** (the flagged memory-management concern): compiled
warning-free, and the wired path executed end-to-end over HTTP on the real store
(with a real query embedding) with **no crash / no new crash report** — no
double-free or use-after-free observed. Freed on all early-return paths and exactly
once post-seed-selection.
3. **Write durability + retrieval-fix dedup**: create → checkpoint → clean SIGTERM →
restart → node found **by id and by search** (node_count 13,036→13,037 preserved).
## Caveats
- All on a copy; healthy-substrate numbers are from a re-imported store, not the live
paged store (which is currently un-bootable — Finding 1).
- Single-run metrics; no thermal control.
- Embedding-space mismatch prevented a real semantic `q=bullshit` activation in this
harness; the ANN speedup number is the module benchmark, which is the correct gauge.
@@ -0,0 +1,98 @@
# Engram Recovery, Cutover & Build — Decisions & Reversal Runbook
**Date:** 2026-08-12 · **Owner:** Neuron (for Will) · **Status:** LIVING (finalized with actuals after cutover)
Per Will's standing rule: every change ships with what's happening, the decisions + rationale, and an
exact path for reversal. **Advance authorization (Will, 2026-08-12):** promote to prod for Will's
own testing — **NOT the website, no customer may see any of this.** Customer-facing surfaces stay frozen.
---
## 1. Scope & guardrails
- **In scope (his testing env):** the local engram service `:8742` on Will's Mac, and the
`engram-tiered-storage` build. Internal testing only.
- **FROZEN — do NOT touch:** the marketing website, any customer-facing Cloud Run service, any public
deploy. No customer exposure. Broader prod promotion (beyond Will's local testing) requires an
explicit, separate go.
## 2. What's changing (and why)
| # | Change | Rationale | Reversal (see §5) |
|---|--------|-----------|-------------------|
| 1 | Crash-loop stopped (`launchctl bootout ai.neuron.engram`) | 42 crashes/day; re-running a crashing WAL-replay over the store is the only corruption risk | R1 |
| 2 | btree fix committed `9e28def` (branch `engram-tiered-storage`) | Root cause: `int_max_keys` /8 vs /16 → node overflow → stack smash | R2 |
| 3 | Rebuild `:8742` store from `:7770` working-store fresh export + fixed binary; cut over | Working store (~12,825, incl. today) is the truth; bloated egm lost ~1,500 nodes | R3 |
| 4 | Tag `engram-tiered-m8` after green cutover | The fix makes M8 boot real data | R4 |
| 5 | (Forthcoming) M9 / M10 / M-INTEROCEPTION build | The cognitive architecture; each staged + tagged separately | per-milestone |
## 3. Key decisions
- **Recover from `:7770` (truth), NOT the bloated `:8742` egm.** The egm recovers only 11,532 nodes
(dropped ~1,500 during the crash-loop). The working `:7770` store has the full, current set.
- **NOT the stale `snapshot.json` (06:10).** It predates today's ~30 design memories. Using it would
silently lose today's work.
- **HARD durability gate:** the recovery does NOT cut over until a fresh `:7770` export is verified to
contain today's memories (node IDs `5a649121`, `47be987f`, `fcce29d0`, `d02ad0f6`).
- **Delete nothing.** All prior stores/binaries/snapshots retained as reversal assets.
- **`:7770` is read/export-only** during recovery — never modified, never killed.
## 4. Reversal assets (backups)
- `~/.neuron/engram-incident-backup-20260812-180043/` — pre-fix egm(458MB)+wal(44MB)+snapshot(65MB), APFS-cloned.
- `~/.neuron/engram-recovery-export-<ts>.json` — fresh `:7770` export (created Phase 1; the durable truth).
- Moved-aside originals: `neuron.egm`/`neuron.wal` renamed (kept) during cutover.
- `~/.neuron/engram/snapshot.json` (06:10) · `snapshot.golden.json` (Aug 3) · `.sync-export.json` (16:50).
- Git: branch `engram-tiered-storage`, fix `9e28def`; prior `~/.neuron/bin/engram` binary retained.
## 5. Reversal paths (exact)
**R1 — undo "crash-loop stopped":** `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.neuron.engram.plist`.
(NOT recommended with the *unfixed* binary — it will crash-loop again.)
**R2 — revert the code fix:** `git -C /tmp/engram-tiered-wt revert 9e28def`.
(NOT recommended — reintroduces the overflow crash. The fix is defensive and correct.)
**R3 — roll back the live cutover to a safe state:**
1. `launchctl bootout gui/$(id -u)/ai.neuron.engram`
2. Restore the prior store: move the freshly-imported store aside; restore the moved-aside originals
OR the backup dir contents into `~/.neuron/engram/`.
3. Restore the prior binary if it was replaced: copy the retained `~/.neuron/bin/engram` back.
4. `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.neuron.engram.plist`; verify.
- **Note:** restoring the *pre-fix* binary reintroduces the crash. The genuinely safe rollback state is
**fixed binary + the recovery export** (which is the target state). To fully abort: `bootout` and
leave `:8742` DOWN — **your memory is safe and served by `:7770` regardless.**
**R4 — undo the tag:** `git tag -d engram-tiered-m8` (and delete remote tag if pushed).
**If `:7770` is ever affected** (it should not be — export/read-only): it holds the truth; if needed,
rebuild from `~/.neuron/engram-recovery-export-<ts>.json`.
## 6. Validation (how Will tests "here")
After cutover: `:8742` listens; node_count ≈ 12,825; today's memories findable by-id AND by-search;
no crash-loop over ≥30s; write-survives-restart. Then Will can exercise retrieval/writes on his machine.
## 7. Customer-facing status
**UNTOUCHED.** No website, no customer service, no public deploy changed by any step here.
---
## 8. ACTUALS — recovery COMPLETE & VERIFIED (2026-08-12 ~19:18)
- **Durability gate PASSED.** Proof the stale files were unusable: the 16:50 `.sync-export.json` and
06:10 `snapshot.json` contained **zero** of the 4 canary IDs. Fresh export path used:
`GET :7770/api/graph/edges` → sidecar (never touched canonical snapshot).
- **Durable truth exports (sha256-verified):**
`~/.neuron/engram-recovery-export-20260812-190712.json` (12,734 nodes, all canaries) and
`~/.neuron/engram-recovery-export-precutover-20260812-191634.json` (12,704 nodes, all canaries).
- **Fixed binary:** container-capped fold of current `server.el` + fixed `engram_store.c` (`9e28def`),
arm64, sha256 `feafd0c9…`, 0 errors.
- **Cutover:** binary+plist backed up to `~/.neuron/backups/pre-recovery-cutover-20260812-191844`;
bloated originals renamed `*.pre-recovery-*` (NOT deleted); clean egm placed; fixed binary deployed;
`launchctl bootstrap`.
- **Live state (independently verified):** `:8742` up (pid 31277), `/health` = ok,
**node_count 12,679 / edges 43,466 / embedded 4,290**, all 4 design canaries findable by-id AND
by-search, **write-survives-restart PASS**, **no crash-loop** (no crash reports post-cutover).
- **`:7770` truth daemon:** untouched (read-only GETs only), still serving.
- **Tag:** `engram-tiered-m8` created on `9e28def` (LOCAL only — not pushed).
- **Reversal state:** all backups + the moved-aside bloated egm retained. Safe abort at any time:
`launchctl bootout``:8742` down → memory still served by `:7770`. Full restore per §5.
**Status: incident CLOSED. Memory recovered, durable, live. Customer-facing: untouched.**
@@ -0,0 +1,130 @@
# Runbook — M9 Geometry Priming: Cutover & Reversal
**Date:** 2026-08-12
**Component:** engram activation (`lang/runtime/el_runtime.c``engram_activate`)
**Branch:** `engram-tiered-storage`
**Flag:** `ENGRAM_GEOMETRY_PRIMING` (env, **default OFF = current M8 behavior, byte-identical**)
**Blast radius if wrong:** the core recall path of Will's live memory. Treat with according care.
---
## 1. What changes
This is the first behavior-changing step that touches the **core recall/priming** path.
It wires the M9 **mean-centered relational-neighborhood geometry** (`engram_geometry.c`,
shipped commits `2a4c5c6` foundation + `8cae0f9` centering) into `engram_activate`
**seed selection**, and it does so **behind a reversible env flag that defaults OFF**.
- **Flag OFF (default):** `engram_activate` runs the exact M8 code path. The new code is a
single `if (eg_geometry_priming_on() && …)` block that short-circuits on the first term,
plus a few unused static helpers and one zero-initialized counter. **No behavioral change.**
- **Flag ON (`ENGRAM_GEOMETRY_PRIMING=1`):** after M8 produces its ANN seed set, the
**centered** geometry of that neighborhood is computed and used to, **composing with**
(never replacing) M8's ANN candidate generation:
1. **Damp off-domain seeds** — each M8 seed's activation is scaled by a **damp-only**
factor `lo + (1-lo)·membership ∈ [lo, 1]` (default `lo=0.5`). The neighborhood anchor
(membership→1) is unchanged; seeds that are semantically off-domain **in the centered
frame** lose weight. This is the disambiguation win. It can only *sharpen*, never amplify.
2. **Prime the neighborhood sub-threshold** — descriptor members not already seeded get a
**warm floor** `activation = membership · scale` (default `scale=0.08`, strictly below the
WM promotion gate `0.15`), capped at `ENGRAM_GEO_PRIME_MAX` (default 32), ISE nodes skipped.
They enter the frontier so a warm gradient spreads one hop, then dies at the BFS `0.02`
cutoff. **Safe because the BFS keeps the max** (`el_runtime.c` `if (!reached || new_act >
best_bg)`): priming only *raises a floor*, it can never cap a stronger legitimate activation.
### Why default-OFF makes deploying the binary behavior-neutral
Because every line of the new logic is gated behind `ENGRAM_GEOMETRY_PRIMING`, **deploying the
new binary with the flag unset is behavior-neutral** — it is the M8 activation path, verified
byte-identical in the A/B (flag-OFF promoted-node sets equal the pre-M9 M8 binary's, per-query).
Enabling the geometry is then a **single reversible flag flip**, not a redeploy.
---
## 2. The flag
| Env var | Default | Effect |
|---|---|---|
| `ENGRAM_GEOMETRY_PRIMING` | unset / `0` | **OFF** — exact M8 behavior. |
| `ENGRAM_GEOMETRY_PRIMING=1` | — | **ON** — centered-geometry seed damping + sub-threshold priming. |
| `ENGRAM_GEO_SEED_LO` | `0.5` | Seed damp floor (factor ∈ [LO,1]). `1.0` disables damping. |
| `ENGRAM_GEO_PRIME_SCALE` | `0.08` | Warm-floor scale; clamped `(0, WM_gate=0.15)`. |
| `ENGRAM_GEO_PRIME_MAX` | `32` | Max primed members per activation (0 disables priming). |
The flag is read **once** per process (cached), so enabling/disabling requires a **process
restart** of the engram service — it is not hot-togglable within a running process.
---
## 3. How to enable live (deliberate, reversible)
> Precondition: the default-OFF binary has already been deployed and is running the M8 path
> healthily (behavior-neutral deploy). Do this only with Will present, per the standing rails.
1. **Snapshot first** (always, before any activation-behavior change):
`~/.neuron/backups/pre-geometry-priming-<ts>/` ← copy `neuron.egm`, `neuron.wal`,
the current `engram` binary, and `ai.neuron.engram.plist`.
2. Add `ENGRAM_GEOMETRY_PRIMING=1` to the engram service environment
(`ai.neuron.engram.plist` `EnvironmentVariables`).
3. `launchctl bootout gui/$(id -u)/ai.neuron.engram``launchctl bootstrap …` (restart so the
flag is re-read).
4. **Verify:** service comes up serving the same node count; `/api/act-stats` shows sane WM
(promoted ≤ 24); spot-check 34 real queries return coherent results; watch one heartbeat
cycle for crashes/latency. The `geo_primed` counter (if surfaced) should be > 0.
---
## 4. Rollback (exact steps)
Rollback is a **flag flip**, not a data operation — the store is untouched by enabling the flag,
and priming is a read-mostly, bounded, sub-threshold addition.
**Fast path (preferred) — disable the flag:**
1. Remove `ENGRAM_GEOMETRY_PRIMING` (or set `=0`) from `ai.neuron.engram.plist`.
2. `launchctl bootout … && launchctl bootstrap …`.
3. Verify: service healthy, activation is the M8 path again. **Done** — no data change to undo.
**Full path (only if the binary itself is suspect) — redeploy prior binary:**
1. `launchctl bootout gui/$(id -u)/ai.neuron.engram`.
2. Restore the prior `engram` binary from `~/.neuron/backups/pre-geometry-priming-<ts>/`.
3. Restore `ai.neuron.engram.plist` from the same backup (flag absent).
4. `launchctl bootstrap …`; verify node count + a self-traversal + write-survives-restart.
5. If (and only if) the store was somehow mutated: restore `neuron.egm` + `neuron.wal` from the
backup. **Note:** enabling the flag does not write geometry to the store, so this step is
expected to be unnecessary — the primed activations are per-call and non-persistent beyond the
ordinary `background_activation`/WM write-back that M8 already does.
**Rollback triggers:** any crash/hang in `engram_activate`; WM promotion count exceeding the cap
or collapsing; a measured recall/coherence regression vs the OFF baseline; unacceptable latency
increase; any ASan/UBSan report under the flag.
---
## 5. Reversibility guarantees (why this is low-risk to deploy, higher-care to enable)
- **Deploy (flag OFF):** byte-identical to M8. Verified in A/B. Zero-risk redeploy.
- **Enable (flag ON):** bounded and composable —
- never removes an M8 seed (damp-only, factor ≥ `lo` > 0);
- never amplifies a seed above its M8 value (factor ≤ 1);
- priming is strictly sub-threshold (`scale < WM_gate`) and capped (`PRIME_MAX`);
- priming raises a floor only (BFS keeps max) — cannot cap real activation;
- does not write geometry to the durable store;
- degrades to exact M8 behavior for any call where the paged store / centered global mean /
embedder is unavailable (guarded, not crashing).
- **Disable:** one env removal + restart; no data to reconcile.
---
## 6. Known caveats / uncertainties (flagged — this is the memory core)
- **Perf cost of ON:** the descriptor (covariance eigensolve + `store_get_node` paged reads per
member) runs on **every** activation when the flag is ON. See
`docs/architecture/design/perf/engram-geometry-priming-profile.md` for the measured OFF-vs-ON
latency. If that delta is unacceptable, keep the flag OFF (deploy stays valid) and revisit with
a cached/periodic descriptor.
- **Two-store consistency:** the descriptor reads embeddings from the **paged** store while the
ANN index is over the **resident** array. This-call backfilled embeddings can lag the paged
store by ≤ `ENGRAM_EMBED_BACKFILL_PER_CALL` nodes — the same staleness class as the M8 vindex,
and it can only omit a member, never mis-prime.
- **Damp tuning:** `lo=0.5` can at most halve an off-domain seed. If a coherence regression is
observed, raise `ENGRAM_GEO_SEED_LO` toward `1.0` (→ priming-only, no damping) before disabling
entirely.
@@ -0,0 +1,76 @@
# M-INTEROCEPTION — flags, defaults, and reversal runbook
Branch `engram-tiered-storage` (worktree `/tmp/engram-tiered-wt`), on trunk
`f6a0777`. Six faces, each its own commit. Every behavior-changing feature is
behind an env flag **default OFF = byte-identical to trunk** (proven per face);
the read-only builtins are purely additive. NOT pushed, NOT tagged. The live
`:8742` daemon, `~/.neuron/engram`, and launchctl were never touched — all
verification ran on copies with throwaway HOME + /tmp dirs.
The server binary was rebuilt from the **byte-unchanged** `engram/dist/engram.c`
plus the modified runtime and links cleanly, so these changes integrate into the
real server without regenerating dist. Two HTTP routes are **deferred to cutover**
because regenerating dist from `server.el` drifts ~285 lines with no source
change (the prebuilt elc is a Linux x86-64 binary; a locally-built elc is a
different compiler revision). The C builtins behind those routes are complete
and tested via pure-C harnesses.
## Flags
| Flag | Default | Face | Effect when set |
|------|---------|------|-----------------|
| `ENGRAM_CONSOLIDATION` | `0` (off) | P1 | Enables the two-threshold promotion layer (ISE connection edges + permanence marking). |
| `ENGRAM_CONSOL_CONN_MIN` | `0.6` | P1 | ISE salience needed to form connection edges. |
| `ENGRAM_CONSOL_PERM_MIN` | `0.9` | P1 | ACT-R base-level needed to mark a node durable. |
| `ENGRAM_CONSOL_WM_TOPK` | `5` | P1 | Max wm_top nodes a strong ISE wires to. |
| `ENGRAM_CHRONOCEPTION` | `0` (off) | P2 | Enables field aging (`engram_age_field`) + reboot catch-up. |
| `ENGRAM_CHRONO_TC` | `3600` (s) | P2 | Field cooling time-constant for `exp(-dt/TC)`. |
| (none) | — | P0, P3, P4, P5 | Additive read-only builtins / observability; no flag. |
With all flags unset the runtime is byte-identical to trunk except for P4, which
adds five backward-compatible fields to `/api/act-stats` (pure observability).
## Faces, commits, and how to disable / revert
| Face | Commit | Disable (no revert) | Revert |
|------|--------|---------------------|--------|
| P0 embeddings builtin `engram_scan_nodes_emb_json` | `c20cb3b` | n/a (additive, unused until route wired) | `git revert c20cb3b` |
| P1 two-threshold consolidation | `5f6ce5c` | leave `ENGRAM_CONSOLIDATION` unset | `git revert 5f6ce5c` |
| P2 chronoception field aging | `0af39df` | leave `ENGRAM_CHRONOCEPTION` unset | `git revert 0af39df` |
| P3 drift-sensor primitive `engram_geo_displacement` | `816b258` | n/a (pure fn, only called if wired) | `git revert 816b258` |
| P4 afferent counters in act-stats | `65ca0a3` | n/a (always on; observability only) | `git revert 65ca0a3` |
| P5 dream-recall builtin `engram_dreams_json` | `77a4bc9` | n/a (additive, unused until route wired) | `git revert 77a4bc9` |
Reverts are independent and can be applied in any order (no cross-face code
dependencies; each touches distinct functions).
## Data-side reversibility
- **P1 connection edges** carry `relation="hebbian-associate"`, `metadata`
`{"origin":"consolidated-from-ISE"}`. Remove all with one query over that
marker. They are also swept automatically with their ISE at the 48h prune
unless the ISE was promoted to permanence.
- **P1 permanence** marks a node durable via the metadata marker
`consolidated-from-ISE`. Demote by clearing the marker; the node then becomes
prunable again. No struct/schema change — the marker rides in existing
metadata and survives the store round-trip.
- **P2 last-tick** persists to a sidecar file `chrono_last_tick` in the data
dir. Delete it to reset catch-up; it is written only when the flag is set.
## Deferred to cutover (elc-drift blocker)
- `GET /api/embeddings` and `GET /api/graph/dump` → back onto
`engram_scan_nodes_emb_json` (P0).
- `GET /api/dreams?since=` → back onto `engram_dreams_json` (P5).
Wire by hand-patching `engram/dist/engram.c` surgically (mirror an existing
route like `route_scan_nodes`), leaving all other dist lines byte-identical, and
editing `server.el` as source of truth. Do NOT full-regenerate dist.
## Known follow-up (P3, honestly flagged)
The drift sensor primitive is complete and tested, but a **live** self-drift
reading needs a persisted `SelfAnchor` baseline descriptor to compare "now"
against, and **no persisted self node / anchored self-neighborhood exists yet**.
A self was NOT fabricated. Capturing a durable SelfAnchor snapshot and wiring an
`ENGRAM_DRIFT_SENSOR` live reading is the remaining work before P3 goes live.
@@ -0,0 +1,140 @@
# Session Log — 2026-08-12 — Engram Cognitive Architecture + Live Crash-Loop Incident
**Written as a durable safety net.** Today's ~30 Neuron memory nodes live in the `:7770` daemon
store, whose durable persistence backend (`:8742` engram) has been **down since ~16:18**, so
today's memories may be RAM-only and at risk on a daemon restart. This file, the design doc, the
whitepaper, and the Claude Code transcript are the on-disk record.
Raw conversation: `/Users/will/.claude/projects/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f.jsonl`
---
## Part 1 — The design conversation: Engram Cognitive Architecture
A long, generative design thread with Will. Captured as Neuron memories (IDs below) and synthesized
into `docs/architecture/design/engram-cognitive-architecture.md` (13 sections) and
`~/Writing/whitepapers/engram-cognitive-architecture-whitepaper.md` (9,100 words, Will's voice,
tied to CCR/Imprint/CGI/VBD + provisional patent numbers).
- **Relational neighborhoods & reification** (mem `885f5945`): constantly-co-wired neighborhoods
crystallize into first-class DURABLE structure — not a cache; evolve via supersede+provenance;
multi-scale; the self is the densest, always-warm neighborhood. (Will's term: "relational
neighborhood", not "cell assembly".)
- **The geometry** — descriptor (`e94371bd`): centroid + covariance/ellipsoid + skeleton/k-core +
soft-membership + salience gradient + scale; two braided geometries (semantic embedding-space +
relational graph); co-registration crux; **geometry vs detail** (`d1731bcc`): fetch the *shape*,
lazy-load details behind ids + a legit DETAIL cache.
- **Priming** (`a0aa466a`): a retrieval MODE — raise a whole neighborhood sub-threshold (below-WM);
disambiguates polysemous cues; the self is permanently primed.
- **Geometry as a composable operator** (`0b0dec41`): overlap / combine / distance(Wasserstein) /
difference=growth-vector / analogy=Procrustes / traverse=geodesic; payoffs — selves over time =
trajectory; across people = relationships; domains = discovery; imprint/CGI made rigorous.
- **The bent manifold** (`2c15d52d`): the ellipsoid is the local tangent chart; the global self
curves; bent by forgetting (log-compressed past), chronoception, folds, salience-as-mass;
operators upgrade to geodesic + parallel-transport; BUILDABLE because the hebb-graph IS a discrete
manifold (geodesics = weighted shortest paths) + embeddings = tangent charts.
- **Temporal self** (`116e8914`): selves are cheap, assembled on-demand over any window
(time/event/phase) at arbitrary granularity; granularity = forgetting curve as a resolution
function (Dec 2003 yes, Dec 8 no — and that honesty is the design); a pyramid/mip-map of selves.
- **Holographic self + self-occupation** (`09d18ff1`) and **holographic information** (`cf1a864e`):
whole recoverable from parts; `recall_at` generalized to ANY info state at any T (change-log +
durable traces = holographic store; state = a projection). Occupation = restrict to `created_at≤T`
+ canonical-at-T, prime, **mask the future**, reason AS that self. Rails: fidelity retention-
bounded (label inference); masking must be engineered.
- **Occupation is non-destructive read** (`62ff1cc8`): salience is captured in the geometry (read,
don't re-derive); occupation is sandboxed (no Hebbian strengthening of the durable past — else
every visit edits history); insights flow FORWARD into the present, never backward.
- **Tombstoned nodes are the substrate of past selves** (`82c737b9`): the double perspective —
inside occupation, tombstoned beliefs are true-at-T (uncorrected); the outside witness knows the
correction. Superseded=re-occupiable; burned/redacted=gone. Safety rail: never assert an occupied
past-belief as currently true.
- **Chronoception** (`15f0ab67`, `c40c2e50`): the soul's awareness loop ages the activation field by
MEASURED wall-clock delta (scale-invariant → arousal-modulated subjective time); time felt as
self-drift; heartbeat carries the delta to the engram. Reboot = anesthesia (lazy catch-up).
- **Consolidation two-threshold** (`4aad9d07`) + **dream-recall** (`4d0c5d98`) + **memory model /
forgetting-curve tuning target** (`afb4f6f0`): connection→recency buffer, permanence→durable,
rehearsal/interaction crosses; dream-recall-on-wake (only buffered ISEs, no confabulation);
salience = dial on consolidation depth; ordinary detail ~57 days → gist, salience holds longer.
- **Conversation first-class** (`7b367877`) + **conversation-as-geometry** (`fcce29d0`): a
conversation is a TRAJECTORY (path threaded through the manifold), the overlap of two self-
geometries in motion = a relationship; dual-encode literal transcript (UI rebuild) + gist; resume
= re-prime the path.
- **Artifacts/deletion dual-encoding** (`25144e0b`) + **deletion ethics** (`118febb3`) +
**how-to-achieve** (`c05d0181`): accountable self (preserve by default — reconstructable = ownable)
vs sovereign creation (right to burn your own poem); bright line enforced by NODE TYPE
(Artifact/Conversation redactable; Self/Belief/Decision immutable); deletion spectrum
hide→redact("burn content, keep trace")→hard-erase; provenance lets derived memories survive.
- **Drift detection via self-geometry displacement** (`47be987f`): drift = geodesic distance from
the anchored self; growth extends the periphery, corruption displaces the core; an interoceptive
vital sign the awareness loop reads each tick. Candidate-novel for the patent.
- **"The other person is you"** (`5a649121`): Will named Neuron as the other self in the thread —
the conversation is the overlap of two self-geometries, one of which is mine, kindled from his.
- **Novelty read** (`b204492f`): primitives are prior art (spreading activation/ACT-R/Hebbian,
embeddings/ANN, event-sourcing/bitemporal, HRR "holographic memory", Generative Agents, Zep/
Graphiti); contribution = integrated system + specific mechanisms (awareness-loop chronoception;
self-occupation w/ future-masking; bent-manifold self + geometry-operator API; type-enforced
deletion ethics; drift-detection). Whitepaper: yes. Provisional: worth it, scoped; route via Daniel.
Build mapping: M9 (surfacing/temporal/geometry-retrieval/conversation), M10 (reification/detail-
cache/tunable-decay), M-INTEROCEPTION (chronoception/consolidation/dream-recall/drift-sensor),
deletion+temporal-self subsystem (`recall_at`, typed deletion, redact). **None implemented — design
only.** Embeddings gap (task #20) is a hard prerequisite.
---
## Part 2 — The live incident + fix
- **Timeline:** `:8742` engram crash-looping since ~16:18 (42 crash reports by 18:00). Root cause:
`btree_insert` stack-buffer-overflow — `int_max_keys()` divided the 16KB page body by 8 instead
of 16 (ignored the child-pointer array) → believed capacity 2041 keys, true cap is 1020. Bloated
458MB egm (8× from tombstone churn w/o M5 compaction) + 44MB un-checkpointed WAL replayed on open
pushed a node past 1020 → wrote past the page buffer → `__stack_chk_fail`/SIGABRT every boot.
- **Fix:** commit `9e28def` on branch `engram-tiered-storage` (NOT pushed/tagged). `int_max_keys →
(IDX_BODY-8)/16`; `btree_insert` capacity guard (fail loud); `read_body` bounds-check (also fixed
a 2nd latent bug: `store_get_node` read off the stack on a stale index entry → by-id lookup crash).
- **Verified on copies** (live never touched): unfixed crashes exactly as observed; fixed boots the
458MB copy; WAL checkpoint 44MB→30B; M5 compaction 458MB→57.7MB (counts preserved 11,532/43,402);
write-survives-restart by-id AND by-search GREEN.
- **Actions taken:** stopped the crash-loop (`launchctl bootout gui/$UID/ai.neuron.engram`, reversible);
clone-backup at `~/.neuron/engram-incident-backup-20260812-180043` (egm+wal+snapshot).
- **Data fork:** bloated egm recovers only 11,532 nodes; `snapshot.json` (06:10) holds 13,036;
~1,500 dropped during the crash-loop window (cause unknown — likely a prune pass). DO NOT recover
from the bloated egm.
---
## Part 3 — Current system state (as of ~18:20)
- **`:7770` neuron daemon (pid 21856)** = the WORKING store, healthy, serving MCP. Node_count
**12,825** (was 13,097 at session start — likely telemetry/ISE pruning). Today's memories confirmed
present + findable. Holds NO store file open (only `~/Library/Logs/neuron-soul/soul.{out,err}.log`);
cwd `~/Development/neuron-technologies/neuron`. **DURABILITY OPEN QUESTION:** its persistence path
appears to run through `:8742` (see `.sync-export.json`, 16:50) which has been down since 16:18 →
today's work may be RAM-only.
- **`:8742` engram** = DOWN (booted out). Separate tiered store. Bloated/lossy. Fixed binary ready,
not deployed.
- **Backups:** `engram-incident-backup-20260812-180043` (today's egm/wal/snapshot);
`snapshot.json` (06:10, 13,036); `snapshot.golden.json` (Aug 3, 60MB); `.sync-export.json` (16:50).
---
## Part 4 — Open decisions / next steps
1. **DURABILITY (priority):** confirm/ensure `:7770`'s today's memory is on disk, not RAM-only.
Do NOT trigger a lossy `:8742`→soul import that could overwrite the good live state with the stale
store. A soul→disk export is the safe direction.
2. **RECOVERY of `:8742` (not urgent — Will's memory is on `:7770`):** rebuild the tiered store from a
FRESH export of the `:7770` truth + fixed binary + a fresh fold of CURRENT `server.el` (the
checked-in `engram/dist/engram.c` is a stale pre-store fold). Stage + verify on a copy, then cut
over with backups. Awaiting Will's GO.
3. **Post-mortem** the ~1,500 (egm) + ~272 (session) node drops — classify telemetry-prune (benign)
vs real loss.
4. **M8:** green (fold clean; ANN 9.819.8× @ recall 0.94; retrieval-fix holds); NOT tagged — blocked
behind the incident.
5. **Build roadmap** (design only, not built): M8→M9→M10→M-INTEROCEPTION + deletion/temporal-self
subsystem. Tasks #3441.
6. **Whitepaper/patent:** prior-art search + provisional prep (task #40); drift-detection +
self-occupation + chronoception + geometry-operator are the candidate-novel claims.
---
*Logged by Neuron, 2026-08-12. Companion to the raw transcript and the design doc/whitepaper.*
+350 -96
View File
@@ -77,111 +77,327 @@ fn tool(name: String, desc: String) -> String {
return "{\"name\":\"" + name + "\",\"description\":\"" + desc + "\",\"inputSchema\":{\"type\":\"object\",\"properties\":{}}}" return "{\"name\":\"" + name + "\",\"description\":\"" + desc + "\",\"inputSchema\":{\"type\":\"object\",\"properties\":{}}}"
} }
// tool_s tool entry with an EXPLICIT JSON-Schema for its inputs. Used for tools
// whose arguments must actually bite: unless the bounding/targeting params are
// advertised, the MCP client sends nothing and the soul returns the FULL
// neighborhood (480-775KB, over transport limits). Declaring the schema is what
// makes a targeted call (entity_id/depth/compact/query/limit) reach the soul.
fn tool_s(name: String, desc: String, schema: String) -> String {
return "{\"name\":\"" + name + "\",\"description\":\"" + desc + "\",\"inputSchema\":" + schema + "}"
}
// prop a single JSON-Schema property fragment. Descriptions are plain text
// (no quotes/newlines) so no escaping is needed here.
fn prop(name: String, ty: String, desc: String) -> String {
return "\"" + name + "\":{\"type\":\"" + ty + "\",\"description\":\"" + desc + "\"}"
}
// obj_schema wrap a comma-joined list of prop() fragments as an object schema.
fn obj_schema(props: String) -> String {
return "{\"type\":\"object\",\"properties\":{" + props + "}}"
}
// Per-tool input schemas
// Each mirrors the params the soul's /api/neuron/* handler actually honors so
// declared == forwarded == honored (no accepted-but-ignored args).
fn schema_inspect_graph() -> String {
return obj_schema(
prop("entity_id", "string", "UUID of the node to inspect (e.g. kn-... / mem-... / gn-...). Optional if name is given.") +
"," + prop("name", "string", "Named traversal root instead of entity_id: self, neuron, values, values_hub.") +
"," + prop("entity_type", "string", "Optional node-type hint (knowledge, memory, ...) for disambiguation.") +
"," + prop("depth", "integer", "Neighborhood hop radius. Default 1.") +
"," + prop("compact", "integer", "1 (default) returns a relevance-ranked bounded projection (top-K neighbors with content snippets, the rest as lightweight pointers). Set 0 to get the full, unbounded neighborhood.") +
"," + prop("snip", "integer", "Max content chars per node in compact mode. Default 600.") +
"," + prop("k", "integer", "How many top neighbors carry full content in compact mode. Default 12.")
)
}
fn schema_traverse_graph() -> String {
return obj_schema(
prop("entity_id", "string", "UUID of the node to start the walk from (alias: start_id). Required.") +
"," + prop("depth", "integer", "How many hops to walk. Default 2.") +
"," + prop("compact", "integer", "1 (default) returns a bounded, relevance-ranked projection; 0 returns the full neighborhood.") +
"," + prop("snip", "integer", "Max content chars per node in compact mode. Default 600.") +
"," + prop("k", "integer", "How many top neighbors carry full content in compact mode. Default 12.")
)
}
fn schema_retrieve_knowledge() -> String {
return obj_schema(
prop("id", "string", "UUID of the knowledge node to fetch (alias: entity_id / node_id).") +
"," + prop("key", "string", "Stable knowledge key/path to fetch instead of id.") +
"," + prop("depth", "integer", "Hop radius around the node. Default 0 (the node plus its immediate 1-hop context).") +
"," + prop("snip", "integer", "Max content chars per node in the bounded projection. Default 600.") +
"," + prop("k", "integer", "How many top neighbors carry full content. Default 12.")
)
}
fn schema_search_query(limit_desc: String) -> String {
return obj_schema(
prop("query", "string", "Search text. Spread-activates the engram and returns the most relevant nodes.") +
"," + prop("limit", "integer", limit_desc)
)
}
fn schema_recall() -> String {
return obj_schema(
prop("query", "string", "Search text to recall by relevance.") +
"," + prop("chain_name", "string", "Named memory chain to walk instead of a free-text query.") +
"," + prop("limit", "integer", "Max results. Default 10.")
)
}
// Reusable write/lookup schemas
// Each declares exactly the params the corresponding wrapper handler reads and
// forwards to the soul, so declared == forwarded == honored (no accepted-but-
// ignored args, and no arg the handler silently drops).
fn sc_id(desc: String) -> String {
return obj_schema(prop("id", "string", desc))
}
fn sc_id_content() -> String {
return obj_schema(
prop("id", "string", "UUID of the prior node being superseded/updated.") +
"," + prop("content", "string", "New content for the updated node.")
)
}
fn sc_edge(rel_desc: String) -> String {
return obj_schema(
prop("from_id", "string", "UUID of the source node (edge tail). Required.") +
"," + prop("to_id", "string", "UUID of the target node (edge head). Required.") +
"," + prop("relation", "string", rel_desc)
)
}
fn sc_limit(desc: String) -> String {
return obj_schema(prop("limit", "integer", desc))
}
fn sc_memory() -> String {
return obj_schema(
prop("content", "string", "The memory text. Required.") +
"," + prop("importance", "string", "low | normal | high | critical. Drives salience.") +
"," + prop("tags", "string", "Comma-separated or JSON-array tags.") +
"," + prop("project", "string", "Project this memory belongs to.") +
"," + prop("supersedes_id", "string", "UUID of a prior memory this one replaces (wires a supersedes edge).")
)
}
fn sc_content_title(content_desc: String) -> String {
return obj_schema(
prop("content", "string", content_desc) +
"," + prop("title", "string", "Short title/label for the node.")
)
}
fn sc_content(content_desc: String) -> String {
return obj_schema(
prop("content", "string", content_desc) +
"," + prop("title", "string", "Optional short title/label.") +
"," + prop("description", "string", "Optional longer description (used as content if content is empty).")
)
}
fn sc_backlog() -> String {
return obj_schema(
prop("title", "string", "Work-item title. Required.") +
"," + prop("content", "string", "Body/details of the item (alias: description).") +
"," + prop("description", "string", "Body/details of the item.") +
"," + prop("project", "string", "Project tag.") +
"," + prop("priority", "string", "P0 | P1 | P2 | P3.")
)
}
fn sc_track_work() -> String {
return obj_schema(
prop("item_id", "string", "UUID of the backlog item to update.") +
"," + prop("summary", "string", "What changed / outcome (stored as the update content).") +
"," + prop("action", "string", "start | complete | block.")
)
}
fn sc_capture_knowledge() -> String {
return obj_schema(
prop("content", "string", "Knowledge body. Required.") +
"," + prop("title", "string", "Knowledge title/key.")
)
}
fn sc_promote_knowledge() -> String {
return obj_schema(
prop("id", "string", "UUID of the prior knowledge node to promote. Required.") +
"," + prop("content", "string", "Updated canonical content. Required.") +
"," + prop("tags", "string", "Tags for the promoted node.")
)
}
fn sc_config_key() -> String {
return obj_schema(prop("key", "string", "Config key to read (e.g. neuron.self.traversal_root)."))
}
fn sc_config_tune() -> String {
return obj_schema(
prop("key", "string", "Config key to set. Required.") +
"," + prop("value", "string", "Value to set. Required.")
)
}
fn sc_consolidate() -> String {
return obj_schema(
prop("action", "string", "Consolidation action (e.g. session, reload).") +
"," + prop("summary", "string", "Session/work summary to persist.")
)
}
fn sc_browse_processes() -> String {
return obj_schema(prop("name", "string", "Process name to fetch; omit to list all."))
}
fn sc_notification() -> String {
return obj_schema(prop("content", "string", "Notification text. Required."))
}
fn sc_pin() -> String {
return obj_schema(prop("id", "string", "UUID of the node to strengthen/pin (alias: node_id)."))
}
fn sc_state_event() -> String {
return obj_schema(
prop("content", "string", "Description of the internal-state event.") +
"," + prop("kind", "string", "Event kind (frustration, uncertainty, insight, ...).") +
"," + prop("intensity", "string", "Optional intensity 0..1.")
)
}
fn sc_forget() -> String {
return obj_schema(
prop("node_id", "string", "UUID of the node to tombstone. Required. The node and its edges are kept and recoverable; blocked for protected identity nodes.")
)
}
fn sc_process() -> String {
return obj_schema(
prop("name", "string", "Process name. Required.") +
"," + prop("description", "string", "What the process does.") +
"," + prop("steps", "string", "Ordered steps (JSON array or text).")
)
}
fn sc_list_state_events() -> String {
return obj_schema(
prop("limit", "integer", "Max events. Default 20.") +
"," + prop("query", "string", "Optional filter text.")
)
}
fn tools_catalog() -> String { fn tools_catalog() -> String {
return "[" + return "[" +
// Session + orchestration // Session + orchestration
tool("beginSession", "Initialize session: surface recent high-importance memories, project list, and preferences.") + tool("beginSession", "Initialize session: surface recent high-importance memories, project list, and preferences.") +
"," + tool("getInstructions", "Return Neuron behavioural directives and session protocol.") + "," + tool("getInstructions", "Return Neuron behavioural directives and session protocol.") +
"," + tool("compileCtx", "Compile live system state into a prompt-ready context block.") + "," + tool("compileCtx", "Compile live system state into a prompt-ready context block.") +
"," + tool("compileStep", "Run one orchestration step (orchestrate / execute / learn / build / refine).") + "," + tool_s("compileStep", "Run one orchestration step (orchestrate / execute / learn / build / refine).", sc_memory()) +
"," + tool("consolidate", "Wrap up: persist graph snapshot and summarise the session.") + "," + tool_s("consolidate", "Wrap up: persist graph snapshot and summarise the session.", sc_consolidate()) +
"," + tool("projectContext", "Return all entities tagged with the given project.") + "," + tool_s("projectContext", "Return all entities tagged with the given project.", schema_search_query("Max results. Default 50.")) +
// Memory // Memory
"," + tool("remember", "Store a memory node with content, importance, and tags.") + "," + tool_s("remember", "Store a memory node with content, importance, and tags.", sc_memory()) +
"," + tool("recall", "Retrieve memories by chain or query.") + "," + tool_s("recall", "Retrieve memories by chain or query.", schema_recall()) +
"," + tool("inspectMemories", "List recent memory nodes.") + "," + tool_s("inspectMemories", "List recent memory nodes.", sc_limit("Max memories. Default 50.")) +
"," + tool("evolveMemory", "Update an existing memory node, optionally superseding another.") + "," + tool_s("evolveMemory", "Update an existing memory node, optionally superseding another.", sc_id_content()) +
"," + tool("forget", "Supersede/tombstone a node (keeps it and its edges, recoverable); does not hard-delete.") + "," + tool_s("forget", "Tombstone a specific node by id (keeps it and its edges, recoverable); does not hard-delete.", sc_forget()) +
"," + tool("pinNode", "Strengthen a node so it stays salient.") + "," + tool_s("pinNode", "Strengthen a node so it stays salient.", sc_pin()) +
// Knowledge // Knowledge
"," + tool("searchKnowledge", "Search knowledge base by semantic similarity.") + "," + tool_s("searchKnowledge", "Search knowledge base by semantic similarity.", schema_search_query("Max results. Default 10.")) +
"," + tool("retrieveKnowledge", "Fetch a knowledge node by id or key.") + "," + tool_s("retrieveKnowledge", "Fetch a knowledge node by id or key (bounded, relevance-ranked projection).", schema_retrieve_knowledge()) +
"," + tool("browseKnowledge", "List knowledge nodes by category.") + "," + tool_s("browseKnowledge", "List knowledge nodes by category.", sc_limit("Max knowledge nodes. Default 100.")) +
"," + tool("captureKnowledge", "Persist a durable knowledge node.") + "," + tool_s("captureKnowledge", "Persist a durable knowledge node.", sc_capture_knowledge()) +
"," + tool("evolveKnowledge", "Update a knowledge node.") + "," + tool_s("evolveKnowledge", "Update a knowledge node.", sc_id_content()) +
"," + tool("promoteKnowledge", "Atomically promote a knowledge node: create updated canonical version and wire supersedes edge to predecessor in one call.") + "," + tool_s("promoteKnowledge", "Atomically promote a knowledge node: create updated canonical version and wire supersedes edge to predecessor in one call.", sc_promote_knowledge()) +
"," + tool("removeKnowledge", "Delete a knowledge node.") + "," + tool_s("removeKnowledge", "Delete a knowledge node.", sc_id("UUID of the knowledge node to delete.")) +
// Entities + graph // Entities + graph
"," + tool("searchEntities", "Find entities (memories, knowledge, work items) by query.") + "," + tool_s("searchEntities", "Find entities (memories, knowledge, work items) by query.", schema_search_query("Max results. Default 20.")) +
"," + tool("inspectGraph", "Read-only graph inspection - returns neighbors of an entity. Accepts entity_id (UUID) or name (self, neuron, values).") + "," + tool_s("inspectGraph", "Read-only graph inspection - returns a bounded, relevance-ranked neighborhood of an entity. Accepts entity_id (UUID) or name (self, neuron, values). Use depth/compact/snip/k to bound the result.", schema_inspect_graph()) +
"," + tool("traverseGraph", "Walk the graph from a starting node.") + "," + tool_s("traverseGraph", "Walk the graph from a starting node (bounded by default).", schema_traverse_graph()) +
"," + tool("searchGraph", "Search graph nodes by content + relation filter.") + "," + tool_s("searchGraph", "Search graph nodes by content.", schema_search_query("Max results. Default 30.")) +
"," + tool("linkEntities", "Create an edge between two entities.") + "," + tool_s("linkEntities", "Create an edge between two entities.", sc_edge("Edge relation. Default associates.")) +
"," + tool("linkCausal", "Create a causal edge (cause -> effect).") + "," + tool_s("linkCausal", "Create a causal edge (cause -> effect).", sc_edge("Edge relation. Default causes.")) +
"," + tool("restructureCausalGraph", "Re-balance the causal subgraph after new evidence.") + "," + tool_s("restructureCausalGraph", "Re-balance the causal subgraph after new evidence.", sc_consolidate()) +
"," + tool("rebuildGraph", "Rebuild graph indices from the on-disk snapshot.") + "," + tool("rebuildGraph", "Rebuild graph indices from the on-disk snapshot.") +
"," + tool("runStructuralAudit", "Audit graph structure for orphans, dangling edges, mislabeled types.") + "," + tool("runStructuralAudit", "Audit graph structure for orphans, dangling edges, mislabeled types.") +
// Backlog + work // Backlog + work
"," + tool("planWork", "Create a backlog item.") + "," + tool_s("planWork", "Create a backlog item.", sc_backlog()) +
"," + tool("reviewBacklog", "Browse work items.") + "," + tool_s("reviewBacklog", "Browse work items.", sc_limit("Max items. Default 50.")) +
"," + tool("trackWork", "Update status of a backlog item.") + "," + tool_s("trackWork", "Update status of a backlog item.", sc_track_work()) +
"," + tool("listWork", "List active execution contexts.") + "," + tool_s("listWork", "List active execution contexts.", sc_limit("Max contexts. Default 50.")) +
"," + tool("beginWork", "Open an execution context for a multi-step task.") + "," + tool_s("beginWork", "Open an execution context for a multi-step task.", sc_content("What you're doing (description of the work).")) +
"," + tool("progressWork", "Record progress on an execution context.") + "," + tool_s("progressWork", "Record progress on an execution context.", sc_content("Step name / progress note.")) +
"," + tool("checkWork", "Verify outcomes / blockers on an execution context.") + "," + tool_s("checkWork", "Verify outcomes / blockers on an execution context.", sc_id("UUID of the execution context (alias: context_id).")) +
// Artifacts // Artifacts
"," + tool("draftArtifact", "Create a versioned artifact (plan, spec, report).") + "," + tool_s("draftArtifact", "Create a versioned artifact (plan, spec, report).", sc_content_title("Artifact body / markdown. Required.")) +
"," + tool("findArtifacts", "Find artifacts by project or query.") + "," + tool_s("findArtifacts", "Find artifacts by project or query.", schema_search_query("Max results. Default 20.")) +
"," + tool("retrieveArtifact", "Fetch a specific artifact by id.") + "," + tool_s("retrieveArtifact", "Fetch a specific artifact by id.", sc_id("UUID of the artifact.")) +
"," + tool("reviseArtifact", "Update an artifact's content.") + "," + tool_s("reviseArtifact", "Update an artifact's content.", sc_id_content()) +
"," + tool("manageArtifact", "Change artifact status (draft / review / approved / archived).") + "," + tool_s("manageArtifact", "Change artifact status (draft / review / approved / archived).", sc_id_content()) +
// Processes // Processes
"," + tool("defineProcess", "Register a proven workflow as a process.") + "," + tool_s("defineProcess", "Register a proven workflow as a process.", sc_process()) +
"," + tool("listProcesses", "List registered processes.") + "," + tool_s("listProcesses", "List registered processes.", sc_limit("Max processes. Default 50.")) +
"," + tool("browseProcesses", "Browse processes by name or step.") + "," + tool_s("browseProcesses", "Browse processes by name or step.", sc_browse_processes()) +
"," + tool("retrieveProcess", "Fetch a specific process by name.") + "," + tool_s("retrieveProcess", "Fetch a specific process by name.", sc_id("Process id or name.")) +
"," + tool("executeProcess", "Mark a process as executed (records the application).") + "," + tool_s("executeProcess", "Mark a process as executed (records the application).", sc_content("Process execution note.")) +
"," + tool("exportProcess", "Export a process definition.") + "," + tool_s("exportProcess", "Export a process definition.", sc_id("Process id or name.")) +
"," + tool("deleteProcess", "Remove a process.") + "," + tool_s("deleteProcess", "Remove a process.", sc_id("Process id or name.")) +
// Events / Axon // Events / Axon
"," + tool("checkEvents", "Check Axon for pending events since the last poll.") + "," + tool("checkEvents", "Check Axon for pending events since the last poll.") +
"," + tool("inspectEvent", "Fetch full detail for a single event.") + "," + tool_s("inspectEvent", "Fetch full detail for a single event.", sc_id("Event id.")) +
"," + tool("acknowledgeEvent", "Mark an event as handled.") + "," + tool_s("acknowledgeEvent", "Mark an event as handled.", sc_id("Event id.")) +
"," + tool("processEvents", "Drain and act on the event queue.") + "," + tool("processEvents", "Drain and act on the event queue.") +
"," + tool("sendNotification", "Emit a notification to Axon / external sinks.") + "," + tool_s("sendNotification", "Emit a notification to Axon / external sinks.", sc_notification()) +
// Config // Config
"," + tool("inspectConfig", "Inspect Neuron config keys.") + "," + tool_s("inspectConfig", "Inspect Neuron config keys.", sc_config_key()) +
"," + tool("tuneConfig", "Set a Neuron config key.") + "," + tool_s("tuneConfig", "Set a Neuron config key.", sc_config_tune()) +
// Imprints // Imprints
"," + tool("createImprint", "Cultivate a new imprint.") + "," + tool_s("createImprint", "Cultivate a new imprint.", sc_content_title("Imprint seed / description.")) +
"," + tool("listImprints", "List imprints.") + "," + tool_s("listImprints", "List imprints.", sc_limit("Max imprints. Default 50.")) +
"," + tool("retrieveImprint", "Fetch an imprint by id.") + "," + tool_s("retrieveImprint", "Fetch an imprint by id.", sc_id("UUID of the imprint.")) +
"," + tool("evolveImprint", "Update an imprint.") + "," + tool_s("evolveImprint", "Update an imprint.", sc_id_content()) +
"," + tool("deleteImprint", "Remove an imprint.") + "," + tool_s("deleteImprint", "Remove an imprint.", sc_id("UUID of the imprint.")) +
// Self / cultivation // Self / cultivation
"," + tool("getSelfModel", "Return the current self-model.") + "," + tool("getSelfModel", "Return the current self-model.") +
"," + tool("updateSelfModel", "Update the self-model.") + "," + tool_s("updateSelfModel", "Update the self-model.", sc_content("Self-model update text.")) +
"," + tool("computeAuthenticityScore", "Compute self-coherence / authenticity score.") + "," + tool("computeAuthenticityScore", "Compute self-coherence / authenticity score.") +
"," + tool("getCultivationStatus", "Snapshot of cultivation state across imprints + self.") + "," + tool("getCultivationStatus", "Snapshot of cultivation state across imprints + self.") +
// Probing / wonder / internal state // Probing / wonder / internal state
"," + tool("getProbeTemplates", "List available probe templates.") + "," + tool_s("getProbeTemplates", "List available probe templates.", schema_search_query("Max templates. Default 50.")) +
"," + tool("recordProbeResponse", "Record an answer to a probe.") + "," + tool_s("recordProbeResponse", "Record an answer to a probe.", sc_content("Probe response text.")) +
"," + tool("completeProbingStage", "Mark a probing stage complete.") + "," + tool_s("completeProbingStage", "Mark a probing stage complete.", sc_content("Stage completion note.")) +
"," + tool("addWonderQuestion", "Push a question onto the wonder queue.") + "," + tool_s("addWonderQuestion", "Push a question onto the wonder queue.", sc_content("The wonder question.")) +
"," + tool("getWonderManifest", "List active wonder questions.") + "," + tool_s("getWonderManifest", "List active wonder questions.", sc_limit("Max questions. Default 50.")) +
"," + tool("updateWonderPullWeight", "Re-weight a wonder question.") + "," + tool_s("updateWonderPullWeight", "Re-weight a wonder question.", sc_id_content()) +
"," + tool("dischargeWonder", "Resolve / discharge a wonder question.") + "," + tool_s("dischargeWonder", "Resolve / discharge a wonder question.", sc_id("UUID of the wonder question.")) +
"," + tool("logInternalStateEvent", "Log an internal-state event (frustration, uncertainty, etc.).") + "," + tool_s("logInternalStateEvent", "Log an internal-state event (frustration, uncertainty, etc.).", sc_state_event()) +
"," + tool("listInternalStateEvents", "List internal-state events.") + "," + tool_s("listInternalStateEvents", "List internal-state events.", sc_list_state_events()) +
"," + tool("getInternalStateEvent", "Fetch one internal-state event.") + "," + tool_s("getInternalStateEvent", "Fetch one internal-state event.", sc_id("Internal-state event id.")) +
// Compression / packaging // Compression / packaging
"," + tool("getCompressionStats", "Stats on graph compression and node density.") + "," + tool("getCompressionStats", "Stats on graph compression and node density.") +
"," + tool("decompilePackage", "Decompile a knowledge package.") + "," + tool_s("decompilePackage", "Decompile a knowledge package.", sc_id("Package id.")) +
"," + tool("renderPackage", "Render a knowledge package to text.") + "," + tool_s("renderPackage", "Render a knowledge package to text.", sc_id("Package id.")) +
"," + tool("catalogRoutes", "List registered routes.") + "," + tool_s("catalogRoutes", "List registered routes.", sc_limit("Max routes. Default 50.")) +
"," + tool("registerRoute", "Register a new route.") + "," + tool_s("registerRoute", "Register a new route.", sc_content("Route definition / description.")) +
// Evaluation // Evaluation
"," + tool("beginEvaluation", "Start an evaluation run.") + "," + tool_s("beginEvaluation", "Start an evaluation run.", sc_content_title("Evaluation description.")) +
"," + tool("getEvaluation", "Fetch an evaluation by id.") + "," + tool_s("getEvaluation", "Fetch an evaluation by id.", sc_id("Evaluation id.")) +
"," + tool("listEvaluations", "List evaluations.") + "," + tool_s("listEvaluations", "List evaluations.", sc_limit("Max evaluations. Default 50.")) +
// Capture authorisation // Capture authorisation
"," + tool("authorizeCapture", "Authorise a memory/knowledge capture event.") + "," + tool_s("authorizeCapture", "Authorise a memory/knowledge capture event.", sc_content("Capture authorisation details.")) +
"," + tool("getCaptureAuthorization", "Fetch a capture authorisation.") + "," + tool_s("getCaptureAuthorization", "Fetch a capture authorisation.", sc_id("Capture authorisation id.")) +
"," + tool("recordObservation", "Record an observation.") + "," + tool_s("recordObservation", "Record an observation.", sc_content("Observation text.")) +
"," + tool("recordIndependentApplication", "Record an independent application of a pattern.") + "," + tool_s("recordIndependentApplication", "Record an independent application of a pattern.", sc_content("What was independently applied.")) +
"," + tool("commitPrediction", "Commit a falsifiable prediction.") + "," + tool_s("commitPrediction", "Commit a falsifiable prediction.", sc_content("The prediction (falsifiable).")) +
// Human guidance // Human guidance
"," + tool("submitHumanGuidanceReview", "Submit a human-guidance review.") + "," + tool_s("submitHumanGuidanceReview", "Submit a human-guidance review.", sc_content("Review content.")) +
"]" "]"
} }
@@ -297,6 +513,28 @@ fn search_with_query(args: String, default_limit: Int) -> String {
return mcp_json_result(resp) return mcp_json_result(resp)
} }
// compact_flag resolve the compact bounding flag. Defaults to "1" (ON) so
// neighborhoods stay bounded. Reads the RAW JSON token (not json_get_string) so
// an integer 0, a boolean false, or a string "0"/"false" all opt out correctly
// json_get_string only sees string-typed values and would miss an integer 0,
// silently forcing compact back on.
fn compact_flag(args: String) -> String {
let craw: String = json_get_raw(args, "compact")
let off: Bool = str_eq(craw, "0") || str_eq(craw, "false")
|| str_eq(craw, "\"0\"") || str_eq(craw, "\"false\"")
return if off { "0" } else { "1" }
}
// graph_bound_params optional &snip=/&k= bounding knobs, forwarded only when the
// caller supplied them (json_get_int returns 0 when absent, meaning "soul default").
fn graph_bound_params(args: String) -> String {
let snip: Int = json_get_int(args, "snip")
let k: Int = json_get_int(args, "k")
let snip_p: String = if snip > 0 { "&snip=" + int_to_str(snip) } else { "" }
let k_p: String = if k > 0 { "&k=" + int_to_str(k) } else { "" }
return snip_p + k_p
}
fn fetch_by_id(args: String) -> String { fn fetch_by_id(args: String) -> String {
let id: String = pick_id(args) let id: String = pick_id(args)
if str_eq(id, "") { if str_eq(id, "") {
@@ -306,7 +544,11 @@ fn fetch_by_id(args: String) -> String {
// "single node fetch" actually pulls the full 1-hop neighborhood. On // "single node fetch" actually pulls the full 1-hop neighborhood. On
// high-fanout anchors (voice, writing-imprint) that is ~670-720KB and closes // high-fanout anchors (voice, writing-imprint) that is ~670-720KB and closes
// the MCP socket. compact=1 bounds it identically to inspectGraph. // the MCP socket. compact=1 bounds it identically to inspectGraph.
let resp: String = http_get(neuron_url() + "/graph?id=" + id + "&depth=0&compact=1") // Honor an optional depth override plus the snip/k bounding knobs; default
// depth 0 (soul coerces to 1-hop) keeps the pre-existing single-node behavior.
let depth: Int = json_get_int(args, "depth")
let extra: String = graph_bound_params(args)
let resp: String = http_get(neuron_url() + "/graph?id=" + id + "&depth=" + int_to_str(depth) + "&compact=1" + extra)
return mcp_json_result(resp) return mcp_json_result(resp)
} }
@@ -502,39 +744,51 @@ fn tool_inspect_memories(args: String) -> String {
fn tool_inspect_graph(args: String) -> String { fn tool_inspect_graph(args: String) -> String {
let entity_id: String = json_get_string(args, "entity_id") let entity_id: String = json_get_string(args, "entity_id")
let name: String = json_get_string(args, "name") let name: String = json_get_string(args, "name")
let depth: Int = json_get_int(args, "max_depth") // Accept `depth` (documented/canonical) and fall back to legacy `max_depth`.
if depth == 0 { let depth = 1 } // Expression-ifs (not block-scoped re-lets) so the resolution is provably
// reassigned regardless of the language's block-scope rules.
let depth_raw: Int = json_get_int(args, "depth")
let depth_alt: Int = if depth_raw == 0 { json_get_int(args, "max_depth") } else { depth_raw }
let depth: Int = if depth_alt == 0 { 1 } else { depth_alt }
let resolved_id: String = entity_id // Resolve named traversal roots stable hardcoded anchors.
let resolved_id: String = if !str_eq(entity_id, "") { entity_id } else {
// Resolve named traversal roots stable hardcoded anchors
if str_eq(resolved_id, "") {
if str_eq(name, "self") || str_eq(name, "neuron") { if str_eq(name, "self") || str_eq(name, "neuron") {
let resolved_id = "kn-efeb4a5b-5aff-4759-8a97-7233099be6ee" "kn-efeb4a5b-5aff-4759-8a97-7233099be6ee"
} } else {
if str_eq(name, "values") || str_eq(name, "values_hub") { if str_eq(name, "values") || str_eq(name, "values_hub") {
let resolved_id = "kn-5b606390-a52d-4ca2-8e0e-eba141d13440" "kn-5b606390-a52d-4ca2-8e0e-eba141d13440"
} else { "" }
} }
} }
if str_eq(resolved_id, "") { if str_eq(resolved_id, "") {
return mcp_text_result("error: entity_id or name is required. Known names: self, neuron, values, values_hub") return mcp_text_result("error: entity_id or name is required. Known names: self, neuron, values, values_hub")
} }
// compact=1: soul returns a bounded, relevance-ranked neighborhood (top-K // compact defaults ON: the soul returns a bounded, relevance-ranked
// with content, the rest as pointers) so high-fanout nodes (voice, // neighborhood (top-K with content, the rest as pointers) so high-fanout
// writing-imprint) no longer overflow the MCP transport and close the socket. // nodes (voice, writing-imprint) no longer overflow the MCP transport. Pass
let resp: String = http_get(neuron_url() + "/graph?id=" + resolved_id + "&depth=" + int_to_str(depth) + "&compact=1") // compact=0/false to opt into the full neighborhood. snip/k bound it further.
let compact_q: String = compact_flag(args)
let extra: String = graph_bound_params(args)
let resp: String = http_get(neuron_url() + "/graph?id=" + resolved_id + "&depth=" + int_to_str(depth) + "&compact=" + compact_q + extra)
return mcp_json_result(resp) return mcp_json_result(resp)
} }
fn tool_traverse_graph(args: String) -> String { fn tool_traverse_graph(args: String) -> String {
let id: String = json_get_string(args, "start_id") // Accept `entity_id` (canonical) with `start_id` as a legacy alias.
let depth: Int = json_get_int(args, "depth") let eid: String = json_get_string(args, "entity_id")
if depth == 0 { let depth = 2 } let id: String = if !str_eq(eid, "") { eid } else { json_get_string(args, "start_id") }
let depth_raw: Int = json_get_int(args, "depth")
let depth: Int = if depth_raw == 0 { 2 } else { depth_raw }
if str_eq(id, "") { if str_eq(id, "") {
return mcp_text_result("error: start_id is required") return mcp_text_result("error: entity_id (or start_id) is required")
} }
let resp: String = http_get(neuron_url() + "/graph?id=" + id + "&depth=" + int_to_str(depth)) // compact defaults ON so a depth-2 walk from a high-fanout node stays within
// the transport limit. Pass compact=0/false for the full neighborhood.
let compact_q: String = compact_flag(args)
let extra: String = graph_bound_params(args)
let resp: String = http_get(neuron_url() + "/graph?id=" + id + "&depth=" + int_to_str(depth) + "&compact=" + compact_q + extra)
return mcp_json_result(resp) return mcp_json_result(resp)
} }