neuron/docs/architecture/03-data-and-memory.md

# Neuron — Data & Memory (the Engram Graph Model)

> The engram is neuron's durable substrate. This document describes the graph
> model: node/edge structure, the consciousness layers, the two distinct tier
> systems, write-protection, the tombstone/supersede immutability model, and
> persistence. Sources: the runtime `el_runtime.c` (where the graph engine
> physically lives — "the runtime IS the database",
> `foundation/el/engram/src/server.el:1-6`), the engram HTTP face
> `server.el`, and the neuron-layer semantics in `memory.el` / `neuron-api.el`.
>
> Runtime path analyzed:
> `foundation/el/lang/releases/v1.0.0-20260501/el_runtime.c`.

## Where the model lives

The engram is **not** a database library. The graph, the activation math, and
Hebbian learning are compiled C in `el_runtime.c`; `server.el` is a thin HTTP
server that exposes them on `:8742`; the storage format is a single JSON
snapshot. There is no SQL, no SQLite, no append log. Keep this in mind: the
"schema" below is C structs, not tables.

> **Design-doc caveat.** `engram/README.md` describes a Rust/`sled`/`bincode`
> `EngramDb` with a `NodeType::Concept` enum. That is **aspirational/legacy
> narrative** — it does not match the shipped C engine. Treat the README as
> design story, not as the implementation. *(unverified against runtime)*

## Nodes

`EngramNode` — `el_runtime.c:5958-6018+`. Every node carries:

| Field group | Fields | Notes |
|---|---|---|
| Identity/content | `id`, `content`, `node_type`, `label`, `tier`, `tags`, `metadata` | all `char*` (`:5959-5965`) |
| Epistemic weights | `salience`, `importance`, `confidence` (double), `temporal_decay_rate` | per-node decay λ override; 0 = use global (`:5966-5969`) |
| Access history | `activation_count`, `last_activated`, `created_at`, `updated_at` | `:5970-5973` |
| Two-layer activation | `background_activation` (Layer 1, BFS fan-out), `working_memory_weight` (Layer 2, executive filter), `suppression_count` | context compilation uses **only** `working_memory_weight` (`:5974-5991`) |
| Consciousness layer | `layer_id` | default 1 = CORE_IDENTITY (`:5996`) |
| ACT-R learning | `access_ts[K]` ring buffer, `access_head`, `access_filled`, `wm_anchor` | base-level learning (`:5997-6008`) |
| Semantics | `emb` (768-dim nomic-embed-text vector, lazily backfilled), `emb_dim` | `:6009-6016` |
| Hebbian | eligibility trace | `:6017+` |

### Node types are strings, not an enum

`node_type` is a free `char*`, defaulting to `"Memory"` when unset
(`el_runtime.c:7401`, `server.el:159`). There is **no closed node-type enum** in
the shipped engine. Two consequences:

- The runtime *special-cases* a handful of type strings for activation
  thresholds (`engram_type_threshold`, `:5933-5955`): `DharmaSelf`/`Safety`
  (0.05, fire easily), `Belief`/`Entity` (0.30), `Knowledge` (0.20), everything
  else `Note`/`Memory`/`Working` (0.40). `InternalStateEvent` and `Tag` are
  **excluded from working-memory promotion** (`:6674-6676`, `:7368-7370`).
- Type strings the neuron layer actually writes: `Memory` (default), `Knowledge`
  (`server.el:549`), `InternalStateEvent` (`server.el:493`), `Tombstone`
  (`memory.el:55`), `Conversation` (session nodes, `sessions.el`), `Persona`
  (`soul.el:250-292`), plus identity/value `Knowledge` nodes.

The types the MCP surface names — `Self`, `BacklogItem`, `SessionSummary`,
`Artifact`, `Process`, `ConfigEntry` — are **`node_type` string conventions set
by higher neuron/Axon layers**, not runtime-known types. Where `BacklogItem` /
`Artifact` are set was not in the files read (they route to the Axon backend, doc
02) — **flag as unverified/TODO** for a human pass.

## Edges

`EngramEdge` — `el_runtime.c:6701-6730+`. Directed, typed, weighted:

| Field | Meaning |
|---|---|
| `id`, `from_id`, `to_id`, `relation` | typed relation string |
| `weight` (double) | **authored** strength — never mutated by activation |
| `hebb` (double) | **learned** co-activation potentiation — the fraction of recent activations in which both endpoints were in working memory together; strictly separate from `weight` |
| `inhibitory` (int flag) | if set, activating the source **suppresses** the target's WM weight instead of exciting it |
| `confidence`, `created_at`, `updated_at`, `last_fired`, `layer` | — |

The **`hebb` field is the co-activation weight** — the Hebbian/LTP channel — kept
deliberately separate from the static authored `weight`. Edges are created via
`engram_connect(from, to, weight, relation)` (`server.el:253`).

**Relation strings observed:** `associates` (default, `server.el:248`),
`identity`, `co-value`, `birthday-twin`, `canonical-self` (`soul.el:37-108`),
`supersedes`, `tombstones`, `contains`, `tagged` (`neuron-api.el`,
`el_runtime.c:6168`).

> **Edges as vectors — the intended model (TARGET; today's edge is scalar).** The
> live edge above carries a typed `relation` string plus **scalar** strength
> channels (`weight`, `hebb`). The design target is for an edge to be a **vector**
> — a first-class carrier of relationship-*meaning* in the node space — so that
> relationships can be **composed / subtracted / analogized / traversed** like
> nodes (the `06` §6 operator algebra over edges). Combined with append-only, this
> yields a **complete temporal record**: every discrete, significant change to a
> relationship is appended (a keyframe on material change), so the **full 4-D
> trajectory** of the meaning-manifold is preserved and `recall_at(t)` can read
> how any relationship was configured at any past `t` — bounded, because changes
> are discrete and meaning saturates by compositionality. **Status: TARGET / #39**
> (see `07-storage-coherence-and-distribution.md` §2.4); the runtime edge is scalar
> today.

## Consciousness layers

Orthogonal to memory tiers, the engram has five canonical **layers**
(`el_runtime.c:5919-5924`):

| id | Name | activation_priority | Role |
|---|---|---|---|
| 0 | SAFETY | 0 (fires earliest) | deepest / limbic |
| 1 | CORE_IDENTITY | — | **default** for all nodes (`ENGRAM_LAYER_DEFAULT`, `:7423`) |
| 2 | DOMAIN | — | domain knowledge |
| 3 | IMPRINT | — | persona overlay |
| 4 | SUIT | — | outermost |

`EngramLayer` (`:6731-6738`) carries `activation_priority` (lower fires first),
`suppressible` (can higher layers suppress it?), `transparent` (invisible to
introspection?), and `injectable` (add/remove at runtime?). Layers are managed
via `engram_add_layer` / `engram_node_layered` / `engram_list_layers`. This is
the identity-vs-domain-knowledge stratification, independent of the tier system
below.

## Two tier systems — do not conflate them

This is the single most important clarification in the data model, and the source
of the vocabulary mismatch flagged throughout this set.

### A. Cognitive memory tiers — the `tier` field
`Working` / `Episodic` / `Semantic` / `Procedural` (and `Canonical` in use).
Runtime default `"Working"` (`el_runtime.c:7408`; `README.md:41-49`). Nodes
**migrate between these by salience decay/reinforcement**, driven by the runtime.
Salience decays as `importance × 1/(1 + days_since) × ln(count + 1)`
(`README.md:57-62`). `memory.el` exposes `tier_working`/`episodic`/`canonical`
helpers (`memory.el:1-3`); `soul.el` writes `Semantic`-tier persona nodes
(`:267`, `:282`). So the live tier set is **{Working, Episodic, Semantic,
Procedural, Canonical}** with continuous salience/importance/confidence floats.

### B. Epistemic tiers & disposition — tags, not runtime concepts
The MCP-facing vocabulary — tiers `note → lesson → canonical`, disposition
`experimental → provisional → stable → deprecated` — is **not enforced anywhere
in `el_runtime.c`.** It is stored as **tags**:

- Knowledge capture preserves the incoming epistemic tier as a `tier:<x>` tag
  rather than mapping onto a cognitive tier — deliberately, to avoid a lossy
  mapping (`server.el:519-522, 544`).
- `promote_knowledge` writes a canonical node tagged
  `["Knowledge","tier:canonical","disposition:stable"]` (`neuron-api.el:533`).

There is **no state machine** validating `experimental → … → deprecated`.
Disposition and epistemic tier are convention-by-tag. *(Flag: not structurally
guarded. The exact MCP-enum → tag/float mapping is not fully traced in the files
read — unverified/TODO.)*

## Write-protection

> **Superseded (2026-08-16) — see `06-cognitive-architecture.md` §12.5.**
> Authority: `foundation/el/lang/spec/correspondence-and-censorship.md` §6 (branch
> `design/correspondence-and-censorship`).
>
> > **In an immutable substrate, any mechanism that refuses a write is either
> > redundant with immutability, or an epistemic constraint misfiled as a
> > protective one.**
>
> The requirement this gate was built for was never stated. It is
> **non-circularity of the reference frame** — a reference fitted to its own
> readings reports perfect correspondence forever while drift becomes
> undetectable from inside. That requirement is satisfied by *when* (the frame
> updates while activation is internally seeded, not while it is being used to
> act), not by *what*, so **the gate becomes unnecessary rather than removed, and
> nothing takes its place.** Corruption requires mutation, and the engram does not
> mutate: recoverability, governance, evidence quality, and rate are already
> properties of the substrate. Only **authorization** is residue, and it is
> bounded — an unauthorized writer can *propose*, never erase.
>
> **This section contradicts itself.** Thirty-five lines below, under
> *Immutability*, this same document states the conclusion in its own words:
> *"nothing it does is ever destructive — the safety is **after** the act, not a
> gate before it"* (`:185-187`). The 403 gate documented here **is** the
> before-the-act gate that sentence says is unnecessary. The design spec §6 names
> exactly this redundancy.
>
> The mechanism below is **still LIVE in code** and is described accurately; what
> is superseded is the claim that it is needed.

`is_protected_node(id)` (`neuron-api.el:20-37`) is a **hard-coded allowlist of 15
identity/value node IDs** — the self root, the values hub, intellectual-dna,
memory-philosophy, voice, and the 8 value nodes. Handlers that could mutate the
graph (tombstone / supersede / evolve / connect) check it and return HTTP 403
`api_err_protected` (`:39-41`) for a protected target (checked at `:384, 511,
692, 705, 746, 768`). Edges *into* a protected node are also blocked
(`handle_api_link_entities`).

**The one sanctioned override** is `POST /api/neuron/cultivate`
(`handle_api_cultivate`, `neuron-api.el:960` — **the `:781-816` cited here is
stale; verified 2026-08-16**) — it performs the same ops with the protection check
skipped, gated by convention to Will's explicit cultivation sessions. The self
layer is writable, but only through a deliberate door.

> **Superseded (2026-08-16).** A door built for a wall that need not stand. Per
> §12.5 of `06`, the gate above is redundant with immutability, so the override
> for it is redundant too. Neither is deleted here — this is a documentation
> branch; the change is sequenced in `correspondence-and-censorship.md` §11.

## Immutability — tombstone, never delete

Engram nodes are immutable (`memory.el:64-69`). The model is:

- **Tombstone** — `mem_tombstone(node_id)` (`memory.el:46-71`) **keeps the node
  and all its edges**, creates a `Tombstone` marker node
  (`content = target id`, `label = "tombstone:<id>"`) and wires a `tombstones`
  edge (weight 1.0). It never calls `engram_forget`. This is *the* one canonical
  delete — every user-facing forget path routes through it. Default bounded reads
  hide tombstoned nodes (`memory_hide_tombstoned`, `neuron-api.el:239-249`);
  `?include_deleted=1` recovers them.
- **Supersede** — updates/evolves (`neuron-api.el:394-428, 506-541, 715-734`)
  create a **new** node with the new content, wire a `supersedes` edge new→old
  (weight 0.9, or 0.95 for promote), and **keep the original**. The response
  returns both ids so the caller re-points. This is the `supersedes_id`
  pattern: new node linked, old preserved, full audit trail.

> **Supersession is residue, not garbage.** The superseded node is the *trail of
> how the current understanding was reached* — kept deliberately, because sometimes
> the truth was in the **old** idea even when the old idea was not itself the truth.
> This is what lets autonomous self-reification (`06` §4.1) run ungated: every
> rename/re-cluster supersedes into this residue chain, so nothing it does is ever
> destructive — the safety is *after* the act, not a gate before it.

> **The hole to know about.** The raw runtime `engram_forget` **does** hard-delete
> (frees node + edges, `el_runtime.c:7647`), and the engram HTTP route
> `DELETE /api/nodes/:id` calls it directly (`server.el:322-328`). Immutability
> is therefore an invariant of the **neuron-api / MCP layer routing**, not of the
> store. A client that hits engram HTTP directly can bypass it. *(flag)*

`engram_forget` is also used *internally* for genuine GC: boot-counter pruning
(`memory.el:184`), session-summary/telemetry pruning (`soul.el:369`,
`sessions.el`). Those are bounded housekeeping, not user deletes.

## Persistence, snapshots, backups

- **Storage:** a single JSON snapshot `snapshot.json` under `ENGRAM_DATA_DIR`,
  written by `engram_save` / read by `engram_load` (`el_runtime.c:9660+`; format
  `{"nodes":[...],"edges":[...]}`). In prod that dir is the RWO PVC mount `/data`
  (doc 04).
- **Write policy:** `persist_canonical()` writes the **full** snapshot after every
  durable write (`server.el:133-141`). The batch-edge route snapshots **once per
  batch** to avoid ~150 GB/day of writes from Hebbian edge churn
  (`server.el:258-305`) — this is why `hebb_consolidate` batches (doc 02).
- **Boot safety:** on load, engram writes `snapshot.boot-backup.json` (good load)
  or `snapshot.failed-load.json` (a non-empty file that parsed to 0 nodes)
  (`server.el:718-734`). Read routes export to scratch paths
  (`.scan-export.json`, `.sync-export.json`) and **never** touch the canonical
  (`server.el:207-223, 418-437`) — a guard added after a read-route corrupted the
  snapshot.
- **Off-cluster backup:** a Kubernetes CronJob (`engram-backup`) tars `/data`
  every 15 minutes to `gs://neuron-db-backup/gke/neuron-prod/` and keeps the last
  96 (24h) (`infrastructure/platform/k8s/neuron-mcp/backup-cronjob.yaml`).
  > **Ticker, but not consolidation (2026-08-16).** Flagged because
  > `06` §12.4's sequencing item is *"no tickers, no cron"* and an auditor
  > sweeping for tickers will land here. This one is **ops/backup, not
  > cognition** — it does not consolidate and must not be folded into the
  > dreamer. Its local counterpart is the `ai.neuron.engram-backup` launch agent
  > (`StartInterval = 3600`, measured 2026-08-16); a separate
  > `ai.neuron.snapshot-backup` runs at `StartInterval = 900`. Note the
  > **discrepancy**: this doc says the backup interval is 15 min, which matches
  > `snapshot-backup` (900 s) rather than the local `engram-backup` (3600 s).
  > The cluster manifest was not read on this branch — treat the 15-min figure as
  > unverified here.
- **Retention:** InternalStateEvent telemetry pruned at 48h
  (`ENGRAM_ISE_RETENTION_MS`, `server.el:485-499`).

> **Data-dir mismatch to flag:** the `server.el` header comment says the default
> is `~/.neuron/engram` (`:16`) but the code defaults to `/tmp/engram`
> (`:135, 717`). Prod overrides both via `ENGRAM_DATA_DIR=/data`. *(unverified —
> which default is intended)*

## The engram HTTP surface (`:8742`)

Dispatcher `handle_request` (`server.el:592-707`). Auth: `ENGRAM_API_KEY`; GETs
always allowed, mutations require `"_auth":"<key>"` in the JSON body
(`server.el:578-588`).

| Endpoint | Purpose |
|---|---|
| `GET /health`, `GET /` | health + live node/edge counts |
| `POST /api/nodes`, `GET /api/nodes`, `GET /api/nodes/:id`, `DELETE /api/nodes/:id` | node CRUD (DELETE = hard `engram_forget`) |
| `GET /api/edges`, `POST /api/edges`, `POST /api/edges/batch`, `GET /api/neighbors/:id?depth` | edge ops + traversal |
| `POST\|GET /api/activate?q&depth`, `POST\|GET /api/search` | spreading activation vs lexical search |
| `POST /api/strengthen` | Hebbian potentiation |
| `POST /api/save`, `/api/load`, `/api/load-merge` | snapshot control |
| `GET /api/sync` | soul daemon periodic pull |
| `GET /api/embed-backfill`, `GET /api/similarity?a&b` | embeddings + cosine |
| `POST /api/neuron/state-events` (auth-exempt), `POST /api/neuron/knowledge/capture` | neuron-layer helpers |
| `GET /api/stats`, `/api/act-stats`, `/api/text-health` | telemetry |

## Retrieval model (summary)

Retrieval is **spreading activation, not query matching**:
`strength = parent_strength × edge_weight × target_salience ×
cosine(query, target)` — multiplicative, top-N, with the two-layer
background → working-memory promotion (`README.md:27-36`; `el_runtime.c:5892+,
6094+`). `mem_recall` / `/api/activate` fire this and mutate WM; `mem_search` /
`/api/search` are passive lexical scans — **but as of 2026-08-14 the live
`route_search` runs structure-gated *geometric* retrieval**
(`engram_retrieve_geometric_json`; held-out **P@5 = 0.700**, semantic not lexical —
`skill` returns skill nodes and *rejects* the false-positive `rainfall`), with the
old lexical scan retained at `/api/search-lexical` (see `06` §2.5). The cognitive
API's `begin_session` and `compile_ctx` return a **bounded projection** of the
activated set, never the raw
graph (doc 02, §2).

## Update — 2026-08-14: layers as named neighborhoods (DESIGN; backlog #49)

A refinement of the `## Consciousness layers` model above, from the deep-night
session (node `92941631`). A **layer is not a storage tier — it is a named,
persistent relational neighborhood** in the one engram, each carrying its own
**growth policy** and its own **lock / threshold policy**:

- **Threshold-lock = `note`→`canonical` maturation at neighborhood scale.** The
  same epistemic-tier promotion the two-tier model (§B above) applies to a single
  node is lifted to a *region*: a neighborhood **earns its lock** by maturing past
  a threshold, at which point it stabilizes (read-mostly) the way a canonical node
  does. Growth and lock are per-neighborhood, not global.
- **A user's imprint is just another neighborhood.** It is not a separate store or
  a bolted-on partition — it lives in the same geometry as everything else.
- **Relate-across is the advantage over island engrams.** Because every
  neighborhood shares one geometry, anything can form edges to anything across
  neighborhood boundaries — the structural reason a single engram with named
  neighborhoods beats a set of isolated per-purpose stores.

**Status: DESIGN.** This is the intended model for engram layers; the naming,
growth, and threshold-lock policies are not yet a built runtime feature. See
`06-cognitive-architecture.md` (Update — second pass).