el/lang/spec/runtime-ownership.md

El Runtime — Ownership and Capability ABI

Status: §0–§2 verified. §3 re-derived and built for the vector index (2026-08-16); not yet applied to the resident RAM graph. Date: 2026-08-16 Scope: lang/runtime/ — every El program (soul, engram, cgi-studio vessels) inherits this by rebuild. Nothing in this document is a change to any El program.

Note on §1's line numbers: they were read against a checkout that has since shifted by ~135 lines. Verified positions as of a67452f are in §2a.

---

0. The residual

Builtins own memory and reach process state directly.

That is the residual — the generator. Everything below labelled a "residue" is a deposit left by it. The distinction matters because we have spent significant effort removing deposits, and deposits regenerate.

A residue is fixed. A residual is eliminated. Fixing residues while the residual stands produces exactly the pattern observed on 2026-08-15/16: a run of individually-correct patches, each verified, followed by a new defect of the same shape in a different file.

---

1. The residues, measured

Each of these is a distinct merged or proposed fix. Each addresses one deposit. None addresses the residual.

residue	location	fix that was applied or proposed
state_get leaked its return value per call — 15 MB over 200k calls	builtin	el #140 (merged)
VIndex freed under a concurrent reader	el_runtime.c:9424	fb32d15 guard (merged 08:46:43)
_eg_vindex_seen realloc'd on a read path	el_runtime.c:9412	same guard
vindex_insert on a read path	el_runtime.c:9434, 9450	same guard
shared visited / epoch scratch stomped by concurrent searches	engram_vindex.c:79–81, 169–186, 195	proposed: move to per-search frame
nine append sites, none indexing → lazily-embedded nodes invisible	el_runtime.c:7806, 7988, 8148, 8224, 11526, 11731, 12050, 15295, 15312	"embed-gap #20", patched by making the read path catch up (9439 comment)

Measured: all file/line references above, read 2026-08-16. Crash frames engram_activate → eg_vindex_sync → vindex_insert → _realloc → _xzm_xzone_malloc_freelist_outlined are accounted for by rows 2–4.

Inferred, not yet verified: that the nine append sites do not share a single commit point. This needs one pass before Change C is sized.

---

2. Why these are one defect

eg_vindex_sync (el_runtime.c:9419) has exactly three callers, and all three are reads:

• engram_activate — 9802
• eg_knn_for_node — 13075 (its own header comment states "No writes.")
• engram_geo_reify_run_json — 13285

It mutates five process-global statics (9400–9404): _eg_vindex, _eg_vindex_dim, _eg_vindex_built_nc, _eg_vindex_seen, _eg_vindex_seen_cap.

Reads mutate because index maintenance was never given an owner on the write side. It got bolted onto reads, because a builtin could reach the globals — nothing prevented it. Likewise state_get leaked because a builtin owned the value it returned; nothing prevented that either.

The store is architecturally append-only and superseding. A read path that mutates contradicts that directly. The contradiction is expressible only because the ABI permits it.

---

2a. Verified positions and the fact §1 missed

Read directly at a67452f, 2026-08-16. §1's line numbers predate a ~135-line shift; these are current.

thing	§1 said	actually
five process-global statics	9400–9404	9535–9539
eg_vindex_seen_ensure realloc	9412	9547
eg_vindex_sync	9419	9554
vindex_free on a read path	9424	9559
vindex_insert on a read path	9434 / 9450	9569 (build) / 9585 (incremental)
caller: engram_activate_inner	9802	9939
caller: eg_knn_for_node	13075	13212
caller: engram_geo_reify_run_json	13285	13422
fb32d15 guard	—	lock 1602, depth 1631, eg_guard_enter 1636, http_worker acquire 1687, engram_activate wrapper 14097
VIndex scratch fields	79–81	79–81 ✓
search_layer race site	195	195 ✓

The structural fact §1 and §3 both missed: the index does not inherit the store's append-only property. vindex_insert rewires the NeighList links of already-existing elements and reallocs elems[] — so extending the index mutates the whole structure, not just its tail. This is why "make reads pure" is necessary but not sufficient, and why §3 needed a publication boundary rather than only a capability split. It is reproduced as a standing test (unsynchronized half, §5).

---

3. The change

(Re-derived 2026-08-16. The previous §3 — a runtime context struct carrying read/write capability pointers to every builtin — was written in mutable-store, C-ownership terms. It asked "who is permitted to mutate the shared thing?", which presupposes a shared mutable thing. The engram is immutable and recall is projection; what does not mutate needs no ownership discipline. So the question is not answered, it is dissolved. The implemented change is below.)

3.1 Three moves, in decreasing order of how much they dissolve

(1) Misfiled scratch is not shared state. visited / visit_epoch were never conceptually owned by the index — they are one traversal's local, hoisted into struct VIndex as an allocation optimisation. Nothing about them is derived geometry. They want neither a lock nor a capability nor a checkout pool: a pure function's scratch belongs to its call frame, and the fix is to put it back there. This is not "the capability model applied by hand to one global"; it is the deletion of a false ownership claim.

(2) const is the capability, and immutability hands it over for free. Once the scratch leaves the struct, search_layer reads the index and nothing else — so vindex_search can take a const VIndex*. That is precisely the teeth old-§3 wanted from capability pointers: a read path physically cannot call vindex_insert, and it is a compile error, not a review comment. It costs one qualifier rather than a new ABI swept across hundreds of builtins. The compiler enforces it on every future caller for the same reason.

The capability type was already in the language. It is spelled const.

(3) What remains is a publication problem, not an ownership problem. With scratch in the frame and reads const, one hazard survives, and it is real: HNSW insert is not an append. vindex_insert rewires the NeighList links of already-existing elements and reallocs elems[]. The store's append-only property does not transfer to the index derived from it. So a reader projecting against the index while its owner extends it is unsafe no matter how pure search is.

Immutability answers this too, and the answer is publication:

• eg_vindex_maintain — the sole mutator. Takes the boundary exclusively; never runs beside a reader.
• eg_vindex_view — returns a const VIndex* with the boundary held for read. N readers project concurrently; none can mutate.

A read path may demand that a current snapshot exist — that is a request to the owner, not a mutation by the reader. What it may not do is mutate the geometry it is projecting against. view / maintain is exactly that split, and it is why this replaces eg_vindex_sync rather than wrapping it.

Write-side owner. Index membership is owned by the event "an embedding became present on this ordinal" — not by node append, since a node without an embedding cannot be in a vector index at all. eg_vindex_note_embedded hooks the embedding-assignment sites: one O(log n) insert, no O(node_count) presence scan. This also retires the "STALENESS (honest tradeoff)" note in the old eg_vindex_sync, where a lazily-embedded older node stayed invisible to route_nearest / autoconnect until the next full rebuild.

3.2 What this does not claim

The resident RAM graph (g->nodes / g->edges) is a separate residue of the same residual and is untouched by this change. It is realloc'd in place (el_runtime.c:7618, 7629), so an awareness-thread reader holding EngramNode* n = &g->nodes[i] across a concurrent append holds a dangling pointer — and engram_activate_inner's embed-backfill writes n->emb through exactly such a pointer. It wants the same publication treatment the index just received. Until that lands, the fb32d15 guard stays (see §5).

---

4. Why this is not a large change

The old §4 argued that El owning its compiler makes a capability-ABI sweep mechanical, since elc generates every builtin call site. That argument was load-bearing only for the ABI, and the ABI is gone.

The constraint now travels with the type of the thing, not the shape of every call site — so no sweep is needed at all. Measured extent of the implemented change: two qualifiers (const VIndex* on vindex_search, propagated to engram_geometry_descriptor and engram_geo_reify_store), one struct field group relocated to a call frame, one rwlock, and three read call sites converted from eg_vindex_sync to view/release.

The payoff of owning the language is unchanged and is now cheaper: introduced once, enforced by the compiler on every future builtin, cannot subsequently be forgotten. Contrast the current state, where the same discipline was maintained by hand across hundreds of builtins and demonstrably failed at least six times.

---

5. What this deletes

Deleted (done, 2026-08-16):

• eg_vindex_sync — the function itself. Not renamed: split into eg_vindex_maintain (mutating, exclusive, sole owner) and eg_vindex_view (const, shared). A name that meant "read paths repair the index" had to stop existing.
• VIndex::visited / visit_epoch / visited_cap — the struct fields, visited_ensure, its call from elems_reserve, ix->visit_epoch = 0 in vindex_create, and free(ix->visited) in vindex_free.
• The proposed per-search scratch struct on the index (a checkout pool / VisitedListPool) — never built. The buffer is a plain frame local; a pool is machinery for an ownership question that no longer exists.
• The proposed reader-view / owner-handle split for VIndex specifically — superseded. const already is the reader view.
• EXPECT_RACE in run_vindex_concurrency_tests.sh — a knob that let a known defect ride as "expected". Replaced by four halves with real verdicts.

NOT deleted — the design doc was wrong about this one:

• fb32d15 (eg_guard_enter / engram_req_lock / _eg_req_depth). §5 originally called for its removal as "a lock protecting a mutation that ceases to exist." Measured, it guards two things, and only one of them ceases to exist. Its own comment names both: the RAM graph and _eg_vindex. The vindex justification is retired; the RAM-graph justification is independently load-bearing (§3.2), and removing the guard reintroduces the measured 11171→9579 edge-loss defect from 2026-08-14. Its comment has been narrowed to state the RAM graph only. Precondition for deleting it: the resident graph gets the same publication boundary the index just got.
• el #140's hand-patch. Left in place — the leak stops being expressible only under the abandoned capability-ABI §3, which is not what was built.

Ordering consequence (revised): the original ordering claim — "the residual lands first, the residues evaporate rather than get fixed" — did not survive contact. The residual here is not a single ABI that dissolves everything at once; it is a property (derived state is published, never edited) applied per structure. The index now has it. The RAM graph does not yet. Residues evaporate per structure, in the order the property is applied, and a residue whose structure has not been converted must be left standing, not deleted on the strength of the plan.

---

6. Sequencing

1. Read how builtins are declared and dispatched, to confirm the call sites are compiler-generated in one place. (This determines whether §4 holds. If dispatch is scattered, re-size before proceeding.)
2. Introduce the context type and capability types.
3. Codegen emits the context at every builtin call site.
4. Mechanical sweep of builtin signatures.
5. Move index maintenance behind the write capability; the three read callers take the read capability.
6. Delete the residue-fixes listed in §5.
7. One build of soul from el dev — which resolves the state_get leak and the crash together, rather than deploying a leak fix that reintroduces the crash.

---

7. Open questions

Answered 2026-08-16:

• Do the nine append sites share a commit point? Moot. The question was mis-aimed: node append is not the event that owns index membership, because a node without an embedding cannot be in a vector index. The five embedding-assignment sites are the real owner points (el_runtime.c:7091, 9839, 13362, 15002, plus snapshot-restore at 7951), and three of them carry the ordinal directly — which is all eg_vindex_note_embedded needs. The other two run before the node is resident, where the cold build picks it up.
• Does anything outside lang/runtime/ construct a second VIndex? No. Swept: the only constructors outside the runtime are engram/test/* and lang/runtime/vindex_bench.c, all single-threaded and index-private. Inside the runtime, engram_self_reify_beat_json builds a private index deliberately and never touches the shared boundary — that was already correct and is unchanged.
• Does the HTTP worker pool contend on the same globals? Yes, and it was never the whole story. Workers serialize against each other on engram_req_lock, but the awareness main thread does not take it at all — that is the gap fb32d15 closed. Now verified independent of that guard: the index boundary is its own rwlock, so worker/awareness contention on _eg_vindex is handled whether or not the request lock is held.

Still open:

• The resident RAM graph wants the same publication boundary (§3.2). Until it has one, fb32d15 cannot be deleted.
• eg_vindex_view holds the boundary for read across engram_geo_reify_store, which is a long pass. Correct, but it stalls the owner for that duration. If reify latency becomes a problem the answer is a refcounted snapshot, not a shorter lock.

---

7a. Evidence (measured 2026-08-16, engram/test/run_vindex_concurrency_tests.sh)

half	before	after
single — 3000 vectors, 1 thread, ASan+UBSan	clean	clean
readers — 4 readers, no writer, TSan	race at engram_vindex.c:195 (visited_reset ← vindex_search)	clean
unsynchronized — writer+reader, bare index, TSan	race	race, expected and permanent — now the proof the boundary must exist
published — owner + 4 readers through the boundary, TSan	(did not exist)	clean, all 3000 inserts landed

No recall regression: recall@10 = 0.9365 at ef_search=128 (gate ≥ 0.90); the determinism test still yields byte-identical results across two independent builds.

Builds locally: all seven engram runtime translation units compile -Wall -Wextra clean, and the full engram binary links (engram/dist/engram.c + runtime, arm64). The one pre-existing -Wcomment warning in el_runtime.c is present at a67452f too.

---

8. What this document is not

It is not an argument for a memory model in general, a garbage collector, process isolation between soul and engram, or a client/server split of the store. Each of those was considered and each addresses mutation that this change removes. They are answers to a question that stops being asked.