# El Runtime — Ownership and Capability ABI **Status:** §0–§2 verified. §3 re-derived and **built** for the vector index (2026-08-16); not yet applied to the resident RAM graph. **Date:** 2026-08-16 **Scope:** `lang/runtime/` — every El program (soul, engram, cgi-studio vessels) inherits this by rebuild. Nothing in this document is a change to any El *program*. **Note on §1's line numbers:** they were read against a checkout that has since shifted by ~135 lines. Verified positions as of `a67452f` are in §2a. --- ## 0. The residual > **Builtins own memory and reach process state directly.** That is the residual — the generator. Everything below labelled a "residue" is a deposit left by it. The distinction matters because we have spent significant effort removing deposits, and deposits regenerate. A residue is fixed. A residual is eliminated. Fixing residues while the residual stands produces exactly the pattern observed on 2026-08-15/16: a run of individually-correct patches, each verified, followed by a new defect of the same shape in a different file. --- ## 1. The residues, measured Each of these is a distinct merged or proposed fix. Each addresses one deposit. None addresses the residual. | residue | location | fix that was applied or proposed | |---|---|---| | `state_get` leaked its return value per call — 15 MB over 200k calls | builtin | el #140 (merged) | | VIndex freed under a concurrent reader | `el_runtime.c:9424` | `fb32d15` guard (merged 08:46:43) | | `_eg_vindex_seen` realloc'd on a read path | `el_runtime.c:9412` | same guard | | `vindex_insert` on a read path | `el_runtime.c:9434`, `9450` | same guard | | shared `visited` / epoch scratch stomped by concurrent searches | `engram_vindex.c:79–81`, `169–186`, `195` | ~~proposed:~~ **built** — moved to the call frame (§3.1(1), §5); TSan `readers` half clean (§7a) | | nine append sites, none indexing → lazily-embedded nodes invisible | `el_runtime.c:7806, 7988, 8148, 8224, 11526, 11731, 12050, 15295, 15312` | "embed-gap #20", patched by making the *read* path catch up (`9439` comment) | **Measured:** all file/line references above, read 2026-08-16. Crash frames `engram_activate → eg_vindex_sync → vindex_insert → _realloc → _xzm_xzone_malloc_freelist_outlined` are accounted for by rows 2–4. ~~**Inferred, not yet verified:** that the nine append sites do not share a single commit point. This needs one pass before Change C is sized.~~ **Moot — see §7.** The question was mis-aimed: node append is not the event that owns index membership, because a node without an embedding cannot be in a vector index. The five *embedding-assignment* sites are the real owner points. --- ## 2. Why these are one defect `eg_vindex_sync` (`el_runtime.c:9419`) has exactly three callers, and **all three are reads**: - `engram_activate` — `9802` - `eg_knn_for_node` — `13075` (its own header comment states *"No writes."*) - `engram_geo_reify_run_json` — `13285` It mutates five process-global statics (`9400–9404`): `_eg_vindex`, `_eg_vindex_dim`, `_eg_vindex_built_nc`, `_eg_vindex_seen`, `_eg_vindex_seen_cap`. Reads mutate because index maintenance was never given an owner on the write side. It got bolted onto reads, because a builtin *could* reach the globals — nothing prevented it. Likewise `state_get` leaked because a builtin *owned* the value it returned; nothing prevented that either. The store is architecturally append-only and superseding. A read path that mutates contradicts that directly. The contradiction is expressible only because the ABI permits it. --- ## 2a. Verified positions and the fact §1 missed Read directly at `a67452f`, 2026-08-16. §1's line numbers predate a ~135-line shift; these are current. | thing | §1 said | actually | |---|---|---| | five process-global statics | 9400–9404 | **9535–9539** | | `eg_vindex_seen_ensure` realloc | 9412 | **9547** | | `eg_vindex_sync` | 9419 | **9554** | | `vindex_free` on a read path | 9424 | **9559** | | `vindex_insert` on a read path | 9434 / 9450 | **9569** (build) / **9585** (incremental) | | caller: `engram_activate_inner` | 9802 | **9939** | | caller: `eg_knn_for_node` | 13075 | **13212** | | caller: `engram_geo_reify_run_json` | 13285 | **13422** | | `fb32d15` guard | — | lock **1602**, depth **1631**, `eg_guard_enter` **1636**, `http_worker` acquire **1687**, `engram_activate` wrapper **14097** | | VIndex scratch fields | 79–81 | **79–81** ✓ | | `search_layer` race site | 195 | **195** ✓ | **The structural fact §1 and §3 both missed:** *the index does not inherit the store's append-only property.* `vindex_insert` rewires the `NeighList` links of already-existing elements and reallocs `elems[]` — so extending the index mutates the whole structure, not just its tail. This is why "make reads pure" is necessary but **not sufficient**, and why §3 needed a publication boundary rather than only a capability split. It is reproduced as a standing test (`unsynchronized` half, §5). --- ## 3. The change *(Re-derived 2026-08-16. The previous §3 — a runtime context struct carrying read/write **capability pointers** to every builtin — was written in mutable-store, C-ownership terms. It asked "who is permitted to mutate the shared thing?", which presupposes a shared mutable thing. The engram is immutable and recall is projection; what does not mutate needs no ownership discipline. So the question is not answered, it is dissolved. The implemented change is below.)* ### 3.1 Three moves, in decreasing order of how much they dissolve **(1) Misfiled scratch is not shared state.** `visited` / `visit_epoch` were never conceptually owned by the index — they are one traversal's local, hoisted into `struct VIndex` as an allocation optimisation. Nothing about them is derived geometry. They want neither a lock nor a capability nor a checkout pool: a pure function's scratch belongs to its call frame, and the fix is to put it back there. This is not "the capability model applied by hand to one global"; it is the deletion of a false ownership claim. **(2) `const` is the capability, and immutability hands it over for free.** Once the scratch leaves the struct, `search_layer` reads the index and nothing else — so `vindex_search` can take a `const VIndex*`. That is *precisely* the teeth old-§3 wanted from capability pointers: a read path physically cannot call `vindex_insert`, and it is a **compile error**, not a review comment. It costs one qualifier rather than a new ABI swept across hundreds of builtins. The compiler enforces it on every future caller for the same reason. > The capability type was already in the language. It is spelled `const`. **(3) What remains is a publication problem, not an ownership problem.** With scratch in the frame and reads const, one hazard survives, and it is real: **HNSW insert is not an append.** `vindex_insert` rewires the `NeighList` links of *already-existing* elements and reallocs `elems[]`. The store's append-only property does **not** transfer to the index derived from it. So a reader projecting against the index while its owner extends it is unsafe no matter how pure search is. Immutability answers this too, and the answer is publication: - **`eg_vindex_maintain`** — the sole mutator. Takes the boundary exclusively; never runs beside a reader. - **`eg_vindex_view`** — returns a `const VIndex*` with the boundary held for read. N readers project concurrently; none can mutate. A read path may **demand that a current snapshot exist** — that is a request to the owner, not a mutation by the reader. What it may not do is mutate the geometry it is projecting against. `view` / `maintain` is exactly that split, and it is why this replaces `eg_vindex_sync` rather than wrapping it. **Write-side owner.** Index membership is owned by the event *"an embedding became present on this ordinal"* — not by node append, since a node without an embedding cannot be in a vector index at all. `eg_vindex_note_embedded` hooks the embedding-assignment sites: one O(log n) insert, no O(node_count) presence scan. This also retires the "STALENESS (honest tradeoff)" note in the old `eg_vindex_sync`, where a lazily-embedded *older* node stayed invisible to `route_nearest` / autoconnect until the next full rebuild. ### 3.2 What this does not claim The **resident RAM graph** (`g->nodes` / `g->edges`) is a *separate* residue of the same residual and is untouched by this change. It is realloc'd in place (`el_runtime.c:7618`, `7629`), so an awareness-thread reader holding `EngramNode* n = &g->nodes[i]` across a concurrent append holds a dangling pointer — and `engram_activate_inner`'s embed-backfill writes `n->emb` through exactly such a pointer. It wants the same publication treatment the index just received. Until that lands, the `fb32d15` guard stays (see §5). --- ## 4. Why this is not a large change The old §4 argued that El owning its compiler makes a capability-ABI sweep mechanical, since `elc` generates every builtin call site. That argument was load-bearing only for the ABI, and the ABI is gone. The constraint now travels with the **type of the thing**, not the shape of every call site — so no sweep is needed at all. Measured extent of the implemented change: two qualifiers (`const VIndex*` on `vindex_search`, propagated to `engram_geometry_descriptor` and `engram_geo_reify_store`), one struct field group relocated to a call frame, one rwlock, and three read call sites converted from `eg_vindex_sync` to `view`/`release`. The payoff of owning the language is unchanged and is now *cheaper*: introduced once, enforced by the compiler on every future builtin, cannot subsequently be forgotten. Contrast the current state, where the same discipline was maintained by hand across hundreds of builtins and demonstrably failed at least six times. --- ## 5. What this deletes **Deleted (done, 2026-08-16):** - `eg_vindex_sync` — the function itself. Not renamed: split into `eg_vindex_maintain` (mutating, exclusive, sole owner) and `eg_vindex_view` (const, shared). A name that meant "read paths repair the index" had to stop existing. - `VIndex::visited` / `visit_epoch` / `visited_cap` — the struct fields, `visited_ensure`, its call from `elems_reserve`, `ix->visit_epoch = 0` in `vindex_create`, and `free(ix->visited)` in `vindex_free`. - The **proposed** per-search scratch *struct on the index* (a checkout pool / `VisitedListPool`) — never built. The buffer is a plain frame local; a pool is machinery for an ownership question that no longer exists. - The **proposed** reader-view / owner-handle split for VIndex specifically — superseded. `const` already is the reader view. - `EXPECT_RACE` in `run_vindex_concurrency_tests.sh` — a knob that let a known defect ride as "expected". Replaced by four halves with real verdicts. **NOT deleted — the design doc was wrong about this one:** - `fb32d15` (`eg_guard_enter` / `engram_req_lock` / `_eg_req_depth`). §5 originally called for its removal as "a lock protecting a mutation that ceases to exist." **Measured, it guards two things, and only one of them ceases to exist.** Its own comment names both: the RAM graph *and* `_eg_vindex`. The vindex justification is retired; the RAM-graph justification is independently load-bearing (§3.2), and removing the guard reintroduces the measured 11171→9579 edge-loss defect from 2026-08-14. Its comment has been narrowed to state the RAM graph only. **Precondition for deleting it:** the resident graph gets the same publication boundary the index just got. - el #140's hand-patch. Left in place — the leak stops being *expressible* only under the abandoned capability-ABI §3, which is not what was built. **Ordering consequence (revised):** the original ordering claim — "the residual lands first, the residues evaporate rather than get fixed" — did not survive contact. The residual here is not a single ABI that dissolves everything at once; it is a *property* (derived state is published, never edited) applied per structure. The index now has it. The RAM graph does not yet. Residues evaporate **per structure, in the order the property is applied**, and a residue whose structure has not been converted must be left standing, not deleted on the strength of the plan. --- ## 6. Sequencing > **⚠ Steps 2–5 belong to the abandoned capability-ABI §3 and are superseded > (2026-08-16).** §3 was re-derived: the engram is immutable and recall is > projection, so *what does not mutate needs no ownership discipline* and the > question is dissolved rather than answered. There is no context type, no > capability type, and no codegen change — **`const` is the capability**, and the > constraint travels with the type of the thing rather than the shape of every call > site, so **no sweep is needed at all** (§4). Steps 1, 6 and 7 stand. Struck rather > than deleted, because the abandoned plan is why §4's cost argument is short. 1. **Read** how builtins are declared and dispatched, to confirm the call sites are compiler-generated in one place. *(This determines whether §4 holds. If dispatch is scattered, re-size before proceeding.)* 2. ~~Introduce the context type and capability types.~~ **Superseded** — `const`. 3. ~~Codegen emits the context at every builtin call site.~~ **Superseded** — no codegen change. 4. ~~Mechanical sweep of builtin signatures.~~ **Superseded** — the constraint travels with the type. 5. ~~Move index maintenance behind the write capability; the three read callers take the read capability.~~ **Done, differently:** `eg_vindex_maintain` (exclusive, sole mutator) / `eg_vindex_view` (`const VIndex*`, shared readers), with `eg_vindex_note_embedded` as the write-side owner. This is a **publication** boundary, not a capability split — HNSW insert is not an append, so purity alone was insufficient (§2a, §3.1(3)). 6. Delete the residue-fixes listed in §5. *(Partially done — see §5's "NOT deleted" list; a residue whose structure has not been converted must be left standing.)* 7. **One** build of soul from el dev — which resolves the `state_get` leak and the crash together, rather than deploying a leak fix that reintroduces the crash. --- ## 7. Open questions **Answered 2026-08-16:** - ~~Do the nine append sites share a commit point?~~ **Moot.** The question was mis-aimed: node append is not the event that owns index membership, because a node without an embedding cannot be in a vector index. The five *embedding-assignment* sites are the real owner points (`el_runtime.c:7091, 9839, 13362, 15002`, plus snapshot-restore at `7951`), and three of them carry the ordinal directly — which is all `eg_vindex_note_embedded` needs. The other two run before the node is resident, where the cold build picks it up. - ~~Does anything outside `lang/runtime/` construct a second `VIndex`?~~ **No.** Swept: the only constructors outside the runtime are `engram/test/*` and `lang/runtime/vindex_bench.c`, all single-threaded and index-private. Inside the runtime, `engram_self_reify_beat_json` builds a **private** index deliberately and never touches the shared boundary — that was already correct and is unchanged. - ~~Does the HTTP worker pool contend on the same globals?~~ **Yes, and it was never the whole story.** Workers serialize against each other on `engram_req_lock`, but the awareness main thread does not take it at all — that is the gap `fb32d15` closed. Now verified independent of that guard: the index boundary is its own rwlock, so worker/awareness contention on `_eg_vindex` is handled whether or not the request lock is held. **Still open:** - The resident RAM graph wants the same publication boundary (§3.2). Until it has one, `fb32d15` cannot be deleted. - `eg_vindex_view` holds the boundary for read across `engram_geo_reify_store`, which is a long pass. Correct, but it stalls the owner for that duration. If reify latency becomes a problem the answer is a refcounted snapshot, not a shorter lock. --- ## 7a. Evidence (measured 2026-08-16, `engram/test/run_vindex_concurrency_tests.sh`) | half | before | after | |---|---|---| | `single` — 3000 vectors, 1 thread, ASan+UBSan | clean | clean | | `readers` — 4 readers, no writer, TSan | **race** at `engram_vindex.c:195` (`visited_reset` ← `vindex_search`) | **clean** | | `unsynchronized` — writer+reader, bare index, TSan | race | **race, expected and permanent** — now the proof the boundary must exist | | `published` — owner + 4 readers through the boundary, TSan | *(did not exist)* | **clean**, all 3000 inserts landed | No recall regression: `recall@10 = 0.9365` at `ef_search=128` (gate ≥ 0.90); the determinism test still yields byte-identical results across two independent builds. Builds locally: all seven engram runtime translation units compile `-Wall -Wextra` clean, and the full engram binary links (`engram/dist/engram.c` + runtime, arm64). The one pre-existing `-Wcomment` warning in `el_runtime.c` is present at `a67452f` too. --- ## 8. What this document is not It is not an argument for a memory model in general, a garbage collector, process isolation between soul and engram, or a client/server split of the store. Each of those was considered and each addresses mutation that this change removes. They are answers to a question that stops being asked.