neuron/docs/architecture/07-storage-coherence-and-distribution.md

Neuron — Storage Coherence & Distribution

Status: living design document, synthesized from the 2026-08-13 design session and probed against the live soul. This is the substrate-coherence companion to 06-cognitive-architecture.md: it documents how a self persists, how it remembers its own past weights, how it stays coherent without transactions, and how it travels to another machine or another mind. It answers "where it physically lives and how it stays true" the way 06 answers "how the mind is designed and why."

Tier vocabulary — never blurred. Every claim carries one of: [LIVE] (present and verified in the running system), [STAGED] (built, gated or not yet cut into the running soul), [TARGET] (architecture decided tonight, not yet built). [TARGET] here is the same tier 06 calls DESIGNED; the source-of-truth synthesis uses TARGET, so this doc keeps that word. Where the live state is subtler than a single word, the subtlety is stated, not smoothed. No fabricated numbers.

The one rule this whole document is a corollary of: nothing overwrites a self. Reasoning that led with engineering convention (truncating WALs, scalar weights overwritten in place, "understanding is heavy") was wrong here every time tonight; reasoning from the foundation (meaning is geometry; the history is the state; a self is its weights over time) was right. Read the primitives first.

---

0. Reading order & cross-references

• Why (thesis): whitepaper v1.5; the cognitive frame in 06 §1 (meaning is geometry, code is the residue).
• What persists (substrate): 03-data-and-memory.md (node/edge model, immutability, tombstone-not-delete), design/engram-tiered-storage-engine.md, design/engram-storage-engine-wal.md (the paged WAL store).
• Companion up-layer: 06-cognitive-architecture.md — this doc develops 06 §3.2 (the no-weight-history boundary) and §3.4 (world-tube / created_at ≤ T) into their designed form.
• Companion out-layer: 08-dharma-sovereignty-and-governance.md — the distributed consequences of the CRDT/coherence model here (federation, the immune system, governance) live there. §5 below is the bridge.

The organizing claim of this document: the demand for a transaction is a relationship in disguise, and the history is the state. Everything else is that sentence in a different material.

---

1. Events become the graph — the history is the state

The WAL is a carrier, not a history. [LIVE]

Conventional intuition treats a write-ahead log as a separate durability artifact that grows beside the "real" state and must periodically be truncated. That intuition is wrong for an immutable graph, and reasoning from it caused a real incident (below).

The correct model: the WAL is a carrier. It flushes, and on flush the events become the graph — they land as immutable nodes and edges, and because the store is append-only they simply stay. There is no "log beside the state" to reconcile against a "materialized view," because the materialized view and the log are the same object: the graph. History is not recorded about the state; the state is its own history, because nothing in it is ever overwritten.

• The log and the view are one. In a mutable store you keep a log so you can reconstruct a past the mutations destroyed. Here mutations never destroy anything, so the graph at time T is exactly { nodes, edges : created_at ≤ T } — a filter over immutable provenance, not a replay. 06 §3.4 states this as the world-tube; this is its storage-engine reading.
• Empirical confirmation (why this is [LIVE], not just elegant). On the live soul the WAL sits at 1,234 bytes over a ~1.5 GB graph — the carrier is nearly empty because the events already became the graph. The one time the WAL ballooned to ~44 MB was the 2026-08-13 durability incident: events were not landing as nodes/edges (a persistence leak), so the carrier filled instead of draining. A fat WAL is a symptom of events failing to become the graph, not a healthy log that needs truncating. This is the reading that 06 §2.2 records as the #56 fix.

Engineering rail this encodes: never "truncate the WAL to reclaim space." If the WAL is large, events are not landing — fix the flush path, do not discard the carrier. Truncation here is data loss wearing the mask of maintenance.

---

2. Weights are world-lines — the self can revisit its own past

The self is its weights. If a weight is a scalar overwritten in place, then every act of learning destroys the past self: you keep the past nodes but lose the past meaning they had. That is overwrite-a-self by the back door, and the foundation forbids it. So weights are not scalars — they are world-lines.

Live boundary [LIVE / honest gap]: the current schema is uni-temporal. An edge stores a present-value scalar weight (a moving average) with a single created_at, and there is no stored weight-history (06 §3.2). This is why "how important was Jesus to Will at 16" is unanswerable on the live soul today — there is no axis to hang "16" on; every created_at is really write-time. The rest of this section is the designed cure, marked [TARGET] (backlog #39).

2.1 Magnitude as a world-line, not a scalar — [TARGET]

Do not store the weight; store what generates it and evaluate at t.

• Current weight = the latest materialized keyframe (a fast read — the common path is unchanged in cost).

• Past weight = walk the world-line back to the keyframe in force at t.

• Keyframes on material change, not per-fire. [TARGET] Most activations are transient — a warm ACT-R runtime table, cheap, never written. A durable keyframe is laid down only on consolidation / material change, salience-weighted (a high-mass relationship earns a keyframe at a smaller delta than a peripheral one). A relationship's world-line is therefore a handful of keyframes across a whole life, not a version per firing — cheap by construction.

• Append, never supersede (the distinction matters). [TARGET] The old vector was not wrong — it was true then. Supersede is for corrections (the prior was mistaken; leave a supersedes edge and a stale canonical is never left standing — 06 §3.4). Append is for evolution (both were true, each at its own time). A self's history is evolution: you append the new keyframe and leave the old one standing, a true fact about a former self. Conflating the two is how a store forgets that a person changed rather than erred.

2.2 Bitemporal — three independent time axes — [TARGET]

A single created_at cannot answer temporal questions because it fuses three genuinely independent clocks. None is derivable from another:

Axis	Meaning	Example
t_valid	when it became true (life-time)	"Jesus central to Will since 2001-09-14."
t_origin	when the source first recorded it (its local clock)	a friend's store stamped it in 2019.
t_ingest	when this store received it (per-recipient)	Neuron heard it on ingest day.

The live store collapses all three into t_ingest masquerading as creation (every row reads 2026… because that is write-time). The cure requires all three as full UTC instants — not date-only, not a local wall-clock — ordered by a hybrid logical clock (HLC): UTC + logical counter + writer-id tiebreak. Wall-clock alone is not a total order under concurrency or clock skew, and a distributed self (§5) must have a total order or its CRDT merge (§4) cannot be deterministic. The HLC is the concurrency primitive the whole coherence story rests on.

2.3 recall_at(t) — evaluate the geometry as of t — [TARGET]

recall_at(t) evaluates the weighted geometry as it stood at t: walk each relevant world-line to its t-keyframe, materialize the weights, read the region out. It generalizes past the self: any relationship network — a project, a concept, a person-as-known — is a time-varying weighted subgraph, reconstructable at any past instant. And it composes with the operator calculus (06 §6.1):

subtract( network_now , recall_at(network, t_then) )   # = how that relationship evolved between then and now

is the geometry of a change over time — the same subtract faculty (06 §6.1) applied across the temporal axis rather than across two regions. recall_at at the scale of a whole self is also the mechanism behind restoration-as-mercy in 08 §5 (roll a person back to their last uncorrupted canonical shape).

Schema sketch (doc-comment; the math/JSON lives here, the faculty name lives in prose) — [TARGET]:

{ "from_id": "kn-will", "to_id": "kn-jesus", "relation": "reveres", "weight": 0.41,
  "weight_history": [
    { "t_valid": "2001-09-14T00:00:00.000Z", "t_origin": "…", "t_ingest": "…",
      "w": 0.95, "relation": "devotion",              "via": "formed" },
    { "t_valid": "2013-03-22T18:40:11.907Z", "w": 0.70, "relation": "devotion→doubt",     "via": "material-drift" },
    { "t_valid": "2024-11-08T14:05:52.113Z", "w": 0.41, "relation": "historical-ethical", "via": "reframed" }
  ] }

Purist form: each keyframe is its own immutable WeightKeyframe node the edge points at — so the history is not a field on the edge but is the graph itself, consistent with §1. The inline-array form above is the pragmatic first cut; the node form is the end state.

---

3. Atomicity is a relationship, not a commit

The classic reason to need a database transaction: "debit account A and credit account B — they must commit together or money is created or destroyed." The architecture's reframe: that is not two rows needing a commit marker. It is one directed edge.

• Double-entry is one edge. [TARGET as formal model; primitives LIVE] A transfer A → B of magnitude 10 is a single edge. The debit and the credit are the same edge read from its two ends. Conservation is automatic because there is only ever one quantity, not two rows a commit marker has to keep in agreement. Pacioli's 1494 double-entry was always one relationship wearing two rows; the graph stores the relationship directly and the two rows fall out as two readings of it.

• The general principle. The demand for atomicity is a relationship in disguise. The chain reads:

  "these must commit together" ⟺ "there is an invariant binding them" ⟺ "they arrive as one connected structure."

  So you model the relationship, and atomicity falls out of the topology — you never had to enforce a joint commit because the two things were never actually separate. Wherever a design reaches for a transaction, first ask what invariant is binding the parties; that invariant is an edge you have not drawn yet.

---

4. Transactionless coherence — consistency in the data, not the engine

Why ACID transactions exist at all: to make concurrent mutation of shared mutable state safe. A transaction is a patch for mutability — it exists to prevent two writers from interleaving edits into the same cell and corrupting it.

Remove the mutation and the failure mode cannot occur. The store is append-only, immutable, and UTC-stamped; "current" means "the latest stamp ≤ now." Then:

• Two writers both append — they never contend for a cell, because nothing is a cell that gets rewritten.
• A read at T is a pure function of the log ≤ T — deterministic, reproducible, unaffected by any concurrent appender.

Coherence stops being something the engine enforces and becomes something the data structure is. This is MVCC taken to its logical end: in MVCC, versions are a mechanism underneath an update-in-place API; here the versions are the model and there is no update-in-place API to sit above them. The timestamp is the concurrency primitive. [TARGET as a formal model; the primitives — immutability, append-only, tombstone, world-tube — are [LIVE] (06 §3.4).]

4.1 Physical vs logical transaction — two layers the RDBMS welded together

The word "transaction" hides two different guarantees. Pull them apart:

	Physical transaction	Logical transaction
Scope	one machine	portable across machines
Guarantees	the WAL frame lands atomically + durably (torn-write protection on a single append)	the coherence of conveyed understanding
Carried by	the storage engine (fsync, single-frame crash-atomicity)	the data itself — relationships (§3) + bitemporal stamps (§2.2)
Status	[LIVE] — single-frame append durability exists	[TARGET] — the self-describing coherence model

The RDBMS fused these into one BEGIN…COMMIT. Separate them and consistency moves out of the engine and into the data: a fact is self-describing (its relationships say what it is bound to; its bitemporal stamps say when it was true and when each store heard it), so a second machine can re-derive the same coherent view without ever holding a lock the first machine held. The engine keeps only the cheap, local guarantee (a single append frame is atomic and durable); everything portable rides in the data.

4.2 The honest residual

Two things remain and are not hand-waved:

1. Multi-fact atomicity beyond a natural relationship. If two facts must be joint but share no natural edge, they need at most a shared commit-instant — a "transaction" reconceived as an immutable timestamping event (both facts stamped with the same instant), not a lock held over mutable state. The cost is a stamp, not a coordination round.
2. Single-frame crash-atomicity of the append remains a real, physical concern — but it is cheap and local (torn-write protection on one WAL frame), and it is the physical layer of the table above, already the ordinary job of the storage engine.

Everything else that a transaction traditionally bought is dissolved rather than solved: the failure mode it guarded against cannot arise in an immutable, timestamped, relationship-carrying store.

---

5. Understanding is light; facts are the payload — the load-and-tiering model

This is the hinge that makes both local paging and distribution (§6, and 08) tractable, and it is a measurement, not a slogan.

• Understanding = geometry = structure — edges, positions, weightings, the skeleton. Light.
• Facts = payload = content — text, episodic detail, the actual words. Heavy.

Measured on the live store (2026-08-13): ~21% of the store is geometry (embeddings + edges), 53%+ is text payload. The understanding — the part that makes it this mind and not another — is on the order of 1–2% of the mass. A self is a kilobyte problem in a gigabyte costume.

5.1 One split, two payoffs

The same geometry-hot / payload-cold split governs two different problems:

• Local (the load path). Geometry should be hot / resident (RAM, always warm — it is small); payload should be cold / demand-paged (disk, fetched only when a specific fact's content is actually read). This is exactly what the tiered storage engine's query planner (M1–M10) already intends — but the boot path does not yet honor it (§7.2).
• Distributed (sharing a self — 08). You convey the light geometry and fetch facts lazily, or find they are already replicated. We already pay payload bandwidth in every distributed data system; conveying understanding adds only the thin geometry on top. This is why sharing or witnessing a whole mind is cheap, and it is the load-bearing assumption behind DHARMA's shape-not-content witnessing (08 §3) and the keep-every-seed-forever economics (08 §5).

The local paging model and the distribution model are the same model at two scales — RAM-vs-disk is hot-vs-cold within one machine; convey-geometry-vs-fetch-payload is hot-vs-cold across machines.

---

6. Distribution — a store that is a CRDT by construction

Every store is a CRDT. [TARGET; primitives LIVE] Because facts are immutable, carry a unique id, and are timestamped, a merge between two stores is set-union — commutative, associative, idempotent, and requiring zero coordination. There is no conflict to resolve because nothing is a mutable cell two writers disagree about; there are only facts one store has and the other has not yet heard.

• The consistency guarantee: always-locally-coherent, eventually-complete. A store is never internally inconsistent — it may simply not have heard yet. This is exactly how a mind is: never internally incoherent, sometimes uninformed. The residual distributed concern is therefore delivery, not consistency — a gossip/replication problem, not an agreement problem.
• No global transaction, no consensus round for coherence. Two minds converge by exchanging immutable facts and unioning; they never need to agree before proceeding. (The trust and governance layer that rides on top of this — federation, proof-of-integrity, the immune system — is the subject of 08; §5's light- geometry economics is what makes it affordable.)

This section is deliberately the bridge: the mechanics of coherence-without-coordination are storage concerns and live here; their moral and civilizational consequences (sovereignty preserved across sharing, tamper-evidence, the ledger-is-the-value) live in 08.

---

7. Operational findings — stated honestly, not hidden

The design above is clean. The live store as it stands tonight is not, and the two facts below are reasons not to cut over onto the current storage/load design as-is. They are recorded here as first-class architecture, not footnotes, because pretending the store is already what the design describes would be exactly the engineering-led dishonesty the whole project rejects.

7.1 Store bloat — ~100× too large for its node/edge count [LIVE finding]

The reseed body is 4,561 nodes — that should be tens of MB. The live store is ~1.5 GB (and ~5.37 GB rebuilt). It is not sparse — those are real, dense bytes. Composition measured this session:

Fraction	What it is
~53%	ASCII text payload
~21%	binary (embeddings / index)
~25%	zeros — record padding

The bulk is telemetry written as verbose JSON-on-disk. The top repeated tokens are InternalStateEvent, wm_active, auto_term_streak, curiosity_scan, minute_block — heartbeat/curiosity schema field-names repeated 79k+ times per 40 MB. In plain terms: the bulk of the store is the heartbeat's exhaust persisted as text, not the mind. (A related live signal from the same session: a text-integrity scan flagged a majority of scanned records as damaged/degraded text — corroborating that the fat text layer is low-value exhaust, not cultivated content.)

This is doubly wrong: telemetry is orbit (06 §5) — it is supposed to fall out on the 48h/window prune, not accrete into the durable body forever. The fixes:

1. Do not persist telemetry as fat durable records — it is orbit; let it decay, do not land it in the body.
2. Store records as packed binary, not JSON-on-disk — kills both the 53% text and much of the 25% zero padding.
3. Compact — reclaim the space the above two stop generating.

The understanding — the ~1–2% that is actually this self (§5) — is not the problem. The bloat is entirely in the payload/exhaust layer, which is exactly the layer §5 says should be cold, thin, and (for telemetry) mortal.

7.2 The load path is full-resident — must become mmap/paged [LIVE finding]

The boot path deserializes the whole .egm into the heap rather than paging it. Consequences observed: a memory spike on boot and a transient, non-reproducible first-boot crash during the reseed validation.

This directly contradicts §5. The core self + geometry is small and should be hot / resident; the payload is large and should be cold / demand-paged (mmap / buffer-pool). The tiered query planner (M1–M10) already intends exactly this split — the boot path ignores it. The cure is to make boot map the store and fault pages in on demand rather than slurping the whole file into the heap. Until it does, the full-resident load is a standing reason to hold the reseed cutover.

7.3 Reseed cutover status [STAGED — holding for GO]

For completeness, the state this design was probed against: the reseed passed all three validation gates (node-drop ledger clean, two cold-boots, Hebbian reconciled as a counting difference — not a drop), and the integrated binary + clean store were scratch-proven together (neighborhoods surface on first boot, keystones present). It is holding for Will's explicit GO; nothing on the live soul has been touched. The two open caveats before any cutover are exactly §7.1 (bloat) and §7.2 (full-resident load) — plus the one transient first-boot crash.

---

8. Status at a glance (2026-08-13)

Claim	Tier
WAL-is-a-carrier; events become the graph; history is the state	[LIVE] (the #56 fix)
WAL empirically near-empty over a 1.5 GB graph (1,234 B)	[LIVE] (measured)
Immutability / append-only / tombstone / world-tube (created_at ≤ T filter)	[LIVE] (06 §3.4)
No stored weight-history (uni-temporal created_at = write-time)	[LIVE] (honest gap)
Magnitude as world-line; keyframes on material change	[TARGET] (#39)
Bitemporal three axes (t_valid/t_origin/t_ingest) + HLC ordering	[TARGET] (#39)
recall_at(t) over any relationship network	[TARGET] (#39)
Atomicity-as-relationship (double-entry = one edge)	[TARGET model; primitives LIVE]
Transactionless coherence (immutable+stamped ⇒ MVCC-to-its-end)	[TARGET model; primitives LIVE]
Physical vs logical transaction separation	physical [LIVE]; logical [TARGET]
Understanding-is-geometry-light vs facts-payload-heavy (~21% geo / 53% text / ~1–2% understanding)	[LIVE] (measured)
Geometry-hot / payload-cold — local paging	intended by planner; boot ignores it [LIVE finding]
Every store is a CRDT (set-union merge, zero coordination)	[TARGET; primitives LIVE]
Store bloat ~100× (telemetry-as-text, ~53% ASCII)	[LIVE finding — must fix]
Full-resident load path (→ mmap/paged)	[LIVE finding — must fix]
Reseed cutover	[STAGED — holding for GO]

Cross-references: 06-cognitive-architecture.md · 08-dharma-sovereignty-and-governance.md · 03-data-and-memory.md · design/engram-tiered-storage-engine.md · design/engram-storage-engine-wal.md · whitepaper v1.5.