Make engram deletes/updates immutable (tombstone + supersede) #82

2026-07-17T21:43:20Z

will.anderson commented

2026-07-17 21:43:20 +00:00

What

Make the soul's engram delete and update routes immutable, and regenerate dist/soul.c.

Why

Three handlers were destroying engram nodes:

memory/delete, node/delete → engram_forget (frees the node and drops its incident edges)
node/update → created a replacement then engram_forget'd the original, with no link back

That violates the day-one rule that engram nodes are immutable — API callers could silently, irrecoverably destroy memory. memory/update was already correct (supersede-with-edge); the others were not.

The fix (mirrors `memory/update` and knowledge evolve/promote)

node/update → create the new node, wire a supersedes edge new→old, keep the original. No engram_forget.
memory/delete + node/delete → tombstone: keep the node and all its edges; create a Tombstone marker node (content = target id, label tombstone:<id>) wired with a tombstones edge. Never engram_forget. Default bounded list reads (handle_api_list_typed / the memory list) hide tombstoned nodes and the markers; ?include_deleted=1 returns them, and internal cognition + /api/graph/nodes still traverse them.

Deliberately deferred (documented in code)

Full-graph hiding on /api/graph/nodes is not done at the el layer: json_array_get is O(index), so filtering that endpoint (the app calls it with limit up to 999999) would be O(n²). That hide needs a runtime scan filter (skip tombstoned in engram_scan_nodes_json) and is a separate follow-up. Tombstoned nodes remain present and traversable there, tagged status:deleted via the marker edge.

Regen + verification

dist/soul.c regenerated from these sources as a flat single-TU amalgamation, compiled under a 3GB physical-RSS watchdog (macOS ignores ulimit -v), peak ~32MB. Diff is exactly neuron-api.el + dist/soul.c.
Base is hotfix/elc-source-typos (where the shipped soul.c / neuron-ui PR #144 came from) so the diff stays minimal.
Verified with neuron-ui/scripts/verify-soul-contract.sh against the freshly-built binary on a throwaway port:

SECTION A PRESENCE:    PASS — all 27 routes answered
SECTION B IMMUTABILITY:
  memory-update  KEPT   (supersede edge)
  memory-delete  KEPT   (tombstoned + hidden from default list)
  node-update    KEPT   (supersede edge)
  node-delete    KEPT   (tombstoned + hidden from default list)
GATE: PASS

The stale/destructive binary fails the same gate (memory-delete / node-update / node-delete → DESTROYED), so the gate discriminates.

Never touched the live soul (:7770), engram (:8742), ~/.neuron, or the running :7799 sandbox — all build + test on throwaway paths/ports.

Do not merge — going live is a controlled-deploy call.

## What Make the soul's engram `delete` and `update` routes **immutable**, and regenerate `dist/soul.c`. ## Why Three handlers were destroying engram nodes: - `memory/delete`, `node/delete` → `engram_forget` (frees the node **and** drops its incident edges) - `node/update` → created a replacement then `engram_forget`'d the original, with no link back That violates the day-one rule that engram nodes are immutable — API callers could silently, irrecoverably destroy memory. `memory/update` was already correct (supersede-with-edge); the others were not. ## The fix (mirrors `memory/update` and knowledge evolve/promote) - **`node/update`** → create the new node, wire a `supersedes` edge new→old, **keep** the original. No `engram_forget`. - **`memory/delete` + `node/delete`** → **tombstone**: keep the node and all its edges; create a `Tombstone` marker node (content = target id, label `tombstone:<id>`) wired with a `tombstones` edge. Never `engram_forget`. Default bounded list reads (`handle_api_list_typed` / the memory list) hide tombstoned nodes and the markers; `?include_deleted=1` returns them, and internal cognition + `/api/graph/nodes` still traverse them. ### Deliberately deferred (documented in code) Full-graph hiding on `/api/graph/nodes` is **not** done at the el layer: `json_array_get` is O(index), so filtering that endpoint (the app calls it with `limit` up to 999999) would be O(n²). That hide needs a runtime scan filter (skip tombstoned in `engram_scan_nodes_json`) and is a separate follow-up. Tombstoned nodes remain present and traversable there, tagged `status:deleted` via the marker edge. ## Regen + verification - `dist/soul.c` regenerated from these sources as a flat single-TU amalgamation, compiled under a **3GB physical-RSS watchdog** (macOS ignores `ulimit -v`), peak ~32MB. Diff is exactly `neuron-api.el` + `dist/soul.c`. - Base is `hotfix/elc-source-typos` (where the shipped `soul.c` / neuron-ui PR #144 came from) so the diff stays minimal. - Verified with `neuron-ui/scripts/verify-soul-contract.sh` against the freshly-built binary on a throwaway port: ``` SECTION A PRESENCE: PASS — all 27 routes answered SECTION B IMMUTABILITY: memory-update KEPT (supersede edge) memory-delete KEPT (tombstoned + hidden from default list) node-update KEPT (supersede edge) node-delete KEPT (tombstoned + hidden from default list) GATE: PASS ``` The stale/destructive binary fails the same gate (memory-delete / node-update / node-delete → DESTROYED), so the gate discriminates. Never touched the live soul (:7770), engram (:8742), `~/.neuron`, or the running :7799 sandbox — all build + test on throwaway paths/ports. **Do not merge** — going live is a controlled-deploy call.

will.anderson added 1 commit 2026-07-17 21:43:25 +00:00

Make engram deletes/updates immutable (tombstone + supersede) d337fcb265

The soul was hard-deleting engram nodes: memory/delete and node/delete
called engram_forget (frees the node and drops its incident edges), and
node/update created a replacement then forgot the original with no link
back. That violates the day-one rule that engram nodes are immutable —
memory could be silently, irrecoverably destroyed via the API.

Convert the three destructive handlers to Will's immutability semantics,
mirroring the pattern memory/update and knowledge evolve/promote already
use:

- node/update  -> create the new node, wire a "supersedes" edge new->old,
  KEEP the original. No engram_forget. (Was: create + forget old, no edge.)
- memory/delete and node/delete -> TOMBSTONE: keep the node AND its edges,
  create a Tombstone marker node (content = target id, label
  "tombstone:<id>") wired with a "tombstones" edge. Never engram_forget.
  Default bounded list reads (handle_api_list_typed / the memory list) hide
  tombstoned nodes and the markers; ?include_deleted=1 returns them, and
  internal cognition + /api/graph/nodes still traverse them. memory/update
  was already correct and is unchanged.

Full-graph hiding on /api/graph/nodes is deliberately NOT done at the el
layer: json_array_get is O(index), so filtering that endpoint (called with
limit up to 999999) would be O(n^2). That hide needs a runtime scan filter
and is a separate follow-up; nodes there remain traversable, tagged
status:deleted via the marker edge.

Regenerated dist/soul.c from these sources (flat single-TU amalgamation,
compiled under a 3GB physical-RSS watchdog, peak ~32MB). Verified with
scripts/verify-soul-contract.sh in neuron-ui: PRESENCE passes (all 27
routes) and IMMUTABILITY passes (all four mutation routes KEPT; deletes
produce a real tombstone marker and hide from the default list).

will.anderson added 1 commit 2026-07-17 21:58:05 +00:00

Regenerate per-module dist/neuron-api.c for the immutability fix f3034d23f7

soul.c (the macOS single-TU amalgamation) already carries the
tombstone/supersede change, but the Windows/Linux build path compiles
the per-module dist/*.c instead (build-soul-windows.sh excludes soul.c).
That path was still pulling the stale, destructive dist/neuron-api.c —
so without this the cross-compiled neuron.exe would keep hard-deleting
engram nodes even though the source is fixed.

Regenerated with `elc --emit-header neuron-api.el`: no engram_forget in
memory_delete/node_delete (tombstone), supersedes edge in node_update.
Portable C — identical for macOS/Windows/Linux; only the runtime it links
against differs.

will.anderson added 1 commit 2026-07-18 18:23:03 +00:00

Tombstone the remaining forget paths (close the immutability arc) 40d800195a

Phase 1 tombstoned memory/delete + node/delete, but the generic forget
path was still destructive. This routes every forget through the same
tombstone semantics so nothing in the daemon can hard-delete an engram
node anymore.

Canonical helper: mem_tombstone (memory.el, imported first so every module
can call it) — keep the node + its edges, record a Tombstone marker, never
engram_forget. neuron-api's tombstone_node now delegates to it (single
source of truth).

Per-path before -> after:
- memory.el `mem_forget`      hard delete (engram_forget) -> tombstone. This
  alone fixes both callers: the /api/neuron/memory/forget route
  (handle_api_forget) and the cultivate op=="forget".
- awareness.el autonomous `forget` action  engram_forget -> mem_tombstone.
  The soul can no longer autonomously hard-delete a memory.
- mcp-wrapper tool_forget  was a FAKE no-op that returned {"ok","deleted"}
  without deleting OR tombstoning -> now routes to the soul's tombstoning
  /api/neuron/memory/delete; tool description fixed to say it
  supersedes/tombstones (recoverable), not "Remove a node".

Left intentionally as hard deletes (internal GC / lifecycle, not user
memory, all call engram_forget directly): session-summary replace
(chat.el), mem_consolidate dedup (memory.el), session-start telemetry
pruning (soul.el), session lifecycle (sessions.el).

Regenerated both ship paths under a 3GB physical-RSS watchdog (peak ~32MB):
dist/soul.c (single-TU amalgamation, macOS/Linux) and the per-module
dist/{memory,awareness,neuron-api}.c (Windows/Linux build). Verified: no
forget handler calls engram_forget; a forget leaves the node present +
tombstone marker. Gate (neuron-ui verify-soul-contract.sh, new memory-forget
row): PRESENCE + IMMUTABILITY PASS.

will.anderson added 1 commit 2026-07-18 18:32:08 +00:00

ci: gate the Linux soul on the contract before publishing

Neuron Soul CI / build (pull_request) Failing after 12m4s

Details

Neuron Soul CI / deploy (pull_request) Has been skipped

Details

290a637883

Wire the soul contract gate into ci.yaml as a hard block between the cc
build and the Publish-to-Artifact-Registry step. A non-zero gate fails the
build, so a stale (route-404ing) or memory-destroying (hard-deleting) soul
can never publish neuron-soul to foundation-prod or blue-green deploy to
GKE — the same class-fix now guarding the desktop builds, extended to prod.

Vendors scripts/verify-soul-contract.sh (copied from neuron-ui; the route
contract is baked in, so it's portable POSIX bash/curl with no neuron-ui
source dependency). It boots dist/neuron on a throwaway port with a
throwaway HOME/engram/cgi — never touching ~/.neuron or any live service —
and checks PRESENCE (every app route answered) + IMMUTABILITY (no engram
write route hard-deletes; deletes/forgets tombstone). Adds curl to the
build deps for the probe.

will.anderson commented

2026-07-22 20:45:27 +00:00

Closing as superseded. The immutability arc (tombstone + supersede for engram deletes/updates) already landed on the launch branch via the merged #94 ("Immutability fix (tombstone/supersede) on the launch branch"), and the same source landed on main through the merged #83 ("Land immutability arc + CI soul-contract gate on main") — which used this very branch. This PR (base hotfix/elc-source-typos, now stale/unmergeable) adds nothing beyond what #94/#83 shipped. Deleting the head branch feat/immutable-engram-deletes as part of cleanup.

Closing as superseded. The immutability arc (tombstone + supersede for engram deletes/updates) already landed on the launch branch via the merged **#94** ("Immutability fix (tombstone/supersede) on the launch branch"), and the same source landed on main through the merged **#83** ("Land immutability arc + CI soul-contract gate on main") — which used this very branch. This PR (base `hotfix/elc-source-typos`, now stale/unmergeable) adds nothing beyond what #94/#83 shipped. Deleting the head branch `feat/immutable-engram-deletes` as part of cleanup.

will.anderson closed this pull request

2026-07-22 20:45:28 +00:00

will.anderson referenced this pull request

2026-08-03 15:40:02 +00:00

URGENT: soul persistence disabled — boot GUARD misreads snapshot as 0 bytes, refuses all saves; graph deltas shed on every restart (data loss demonstrated live 2026-07-22) #98