neuron-technologies/el
3
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases 5 Wiki Activity

singleton: guard the state, not the program's name #157

Merged
will.anderson merged 1 commits from fix/singleton-guards-the-state into dev 2026-08-17 00:57:50 +00:00
Conversation 0 Commits 1 Files Changed 8
+213 -49
will.anderson commented 2026-08-16 21:10:08 +00:00
Owner
Copy Link
Copy Source

The defect

el_singleton_acquire() guarded a filename, not a store.

The lock was $EL_SINGLETON_DIR|$TMPDIR|/tmp + /el-singleton-<program-name>.lock. It was keyed on the program's name and on a temp directory, and it never consulted the state it claimed to protect — while its own refusal message read:

Refusing to start a second instance against the same state.

It had not looked at any state. Measured on the pre-fix build, it failed in both directions:

  • False positive. A second engram started with a different ENGRAM_DATA_DIR was refused, naming an unrelated pid. Two instances against genuinely different stores could not coexist.
  • False negative — the dangerous one. TMPDIR=/tmp/other let a second instance start against the same data dir with no complaint. That is precisely the two-writer data-loss condition the guard exists to prevent, and the workaround was one environment variable.

The principle

Guard the thing, not the name. A lock that protects state must be keyed on the state.

Both failure directions are one error, not two: the identity of the resource had been replaced by a label for it. This is the same class of mistake as reducing a structure to a location.

The design, and why

The runtime cannot know generically which environment variable holds an arbitrary program's state, so the program block has to say. Three options were weighed:

Option Verdict
(a) lock lives inside the guarded dir chosen (mechanism)
(b) fixed lock dir, name carries a hash of the canonical state path rejected
(c) extend program so the singleton names its state chosen (declaration) — (a) is impossible without it

(b) was rejected because it does not actually fix the bug. Hashing the state into the name leaves $EL_SINGLETON_DIR/$TMPDIR in the location, so the false negative survives: point the lock dir somewhere else and the second instance still starts. Closing that would mean hard-coding a lock directory and writing a canonicaliser whose string comparison has to be right — machinery that (a) gets from the filesystem for free.

(a) is the whole fix, in one placement decision:

<state>/.el-singleton-<id>.lock
  • Same directory ⇒ same file ⇒ same inode ⇒ the flock contends. There is no TMPDIR in the key, so there is nothing left to change to get past it. $EL_SINGLETON_DIR is deleted, not deprecated.
  • Different directories ⇒ different files ⇒ no contention. Two stores are two stores; they were never in conflict and are no longer treated as if they were.
  • Different spellings of one directory — trailing slash, x/../x, a symlink — collapse to one inode in the kernel's own path walk, so they contend without this code comparing strings. realpath() is used only so the diagnostic names one directory in one spelling; the decision never depends on it and is correct if it fails.
  • An unguardable state (missing, read-only) is a refusal, not a fallback. Starting unguarded against the store the guard protects is the failure being removed.

(c), minimally: guards: is an expression, not a string. The engram's data dir is owned by engram_resolve_data_dir() — it owns both the $ENGRAM_DATA_DIR read and the $HOME/.neuron/engram fallback. Spec §18.4 already forbids giving that path a second owner. A string-valued guards: would force the default to be written down twice, and a guard that resolved the path its own way could lock a directory the program never writes to. An expression lets the block point at the existing owner:

program "engram" {
    singleton: "engram"
    guards:    engram_resolve_data_dir()
    ...
}

guards: is mandatory. singleton: without it is now a compile error (#error in the emitted C). Emitting a name-keyed lock instead would be emitting the defect — and it fails silently in the direction that loses data. Engram was the only singleton: in the tree, so nothing else moves.

Kept

  • Still an flock, still reports the holder's pid (added because pkill -f silently failed to match a stale process that went on answering probes).
  • Still no "delete the lock file to get unstuck" ritual — the kernel drops the lock on crash and on SIGKILL. A lock file left inside a cp -Rc'd data dir is inert: it carries no lock, only a stale pid string the next holder overwrites. Demonstrated below.

Changed

The FATAL message is now true. It says "the same state" only because the lock it failed to take is in that state, and it names the state it checked. A diagnostic asserting a check that did not happen is worse than none — it is what let the name-keyed version read as correct for as long as it did.

Measurement

CI is not the bar; the local build is. Built exactly as documented, with the rebuilt dist/platform/elc (self-host fixpoint verified byte-identical):

cd lang && ./dist/platform/elc ../engram/src/server.el > /tmp/x.c
cc -std=c11 -O1 -I runtime -I$(brew --prefix openssl@3)/include \
   -L$(brew --prefix openssl@3)/lib -o /tmp/xnew /tmp/x.c \
   runtime/el_runtime.c runtime/el_seed.c runtime/engram_*.c runtime/eg_cosine_batch*.c \
   -lcurl -lssl -lcrypto -lpthread -lm

Emitted call:

static void __el_program_init(void) {
  el_singleton_acquire(EL_STR("engram"), engram_resolve_data_dir());

Run on two APFS clones of the live store (cp -Rc ~/.neuron/engram dataA / dataB, 34 948 nodes each) on ports 19001–19012. Production (:8742, ~/.neuron/engram) was never bound, written, or signalled.

Summary

# Case Correct Before After
1 same data dir, same TMPDIR refuse refused ✅ refused ✅
2 same data dir, different TMPDIR refuse started ❌ refused ✅
3 different data dirs, same TMPDIR both start refused ❌ both started ✅
4 same dir, different spellings refuse trailing-slash / .. refused only because TMPDIR matched; symlink + different TMPDIR started ❌ all three refused ✅

Case 4's "passes" before the fix were passing for the wrong reason — the name matched, not the store — which case 4c exposes by varying TMPDIR at the same time.

Case 1 — same data dir, same TMPDIR → REFUSED (still works)

--- holder pid: 2971 ---
exit=1
[el] FATAL: another instance of 'engram' is already running (pid 2971).
[el]        state: /private/tmp/.../lab/dataA
[el]        lock:  /private/tmp/.../lab/dataA/.el-singleton-engram.lock
[el]        Refusing to start a second instance against the same
[el]        state. Two writers against one store is data loss, not a
[el]        warning. Stop the running one and VERIFY it is gone
[el]        (ps -p 2971) before retrying — or point this instance at a
[el]        different state, which is permitted and is not refused.

Case 2 — same data dir, DIFFERENT TMPDIR → REFUSED (the bug)

Before (ENGRAM_DATA_DIR=…/dataA TMPDIR=…/tmpB) — the second instance started:

--- holder pid: 176 ---
exit=124
[http] listening on [::]:19004 (dual-stack)

After — refused, and the message names the store, not the temp dir:

--- holder pid: 3004 ---
exit=1
[el] FATAL: another instance of 'engram' is already running (pid 3004).
[el]        state: /private/tmp/.../lab/dataA
[el]        lock:  /private/tmp/.../lab/dataA/.el-singleton-engram.lock
[el]        Refusing to start a second instance against the same
[el]        state. Two writers against one store is data loss, not a
[el]        warning. Stop the running one and VERIFY it is gone
[el]        (ps -p 3004) before retrying — or point this instance at a
[el]        different state, which is permitted and is not refused.

Case 3 — different data dirs, same TMPDIR → BOTH START

Before — the second was refused, naming an unrelated pid:

--- A pid 251 alive? YES ---
--- B pid 265 alive? NO ---
--- A probe: {"status":"ok",...,"node_count":34948,"edge_count":43467} ---
--- B probe: NO-RESPONSE ---
--- B stderr: ---
[el] FATAL: another instance of 'engram' is already running (pid 251).
[el]        lock: /private/tmp/.../lab/tmpA/el-singleton-engram.lock
[el]        Refusing to start a second instance against the same
[el]        state. Stop the running one and VERIFY it is gone
[el]        (ps -p <pid>) before retrying.

After — both alive, both serving, both answering from their own store:

--- A pid 3035 alive? YES ---
--- B pid 3052 alive? YES ---
--- A probe: {"status":"ok","engine":"engram-runtime-native","node_count":34948,"edge_count":43467} ---
--- B probe: {"status":"ok","engine":"engram-runtime-native","node_count":34948,"edge_count":43467} ---
--- B stderr: ---
[http] listening on [::]:19006 (dual-stack)

Case 4 — equivalent-but-differently-spelled paths → REFUSED

Holder is on …/lab/dataA. Note 4c also varies TMPDIR — before the fix it started (exit=124, [http] listening on [::]:19010); the two that did refuse before did so only because the name matched.

--- holder pid: 3094 ---
-- 4a: trailing slash  /private/tmp/.../lab/dataA/ --
exit=1
[el] FATAL: another instance of 'engram' is already running (pid 3094).
[el]        state: /private/tmp/.../lab/dataA
[el]        lock:  /private/tmp/.../lab/dataA/.el-singleton-engram.lock
[el]        Refusing to start a second instance against the same
[el]        state. Two writers against one store is data loss, not a
[el]        warning. Stop the running one and VERIFY it is gone
[el]        (ps -p 3094) before retrying — or point this instance at a
[el]        different state, which is permitted and is not refused.

-- 4b: dot-dot round trip  /private/tmp/.../lab/dataA/../dataA --
exit=1
[el] FATAL: another instance of 'engram' is already running (pid 3094).
[el]        state: /private/tmp/.../lab/dataA
[el]        lock:  /private/tmp/.../lab/dataA/.el-singleton-engram.lock
[el]        Refusing to start a second instance against the same
[el]        state. Two writers against one store is data loss, not a
[el]        warning. Stop the running one and VERIFY it is gone
[el]        (ps -p 3094) before retrying — or point this instance at a
[el]        different state, which is permitted and is not refused.

-- 4c: symlink  /private/tmp/.../lab/dataA-link  (+ different TMPDIR) --
exit=1
[el] FATAL: another instance of 'engram' is already running (pid 3094).
[el]        state: /private/tmp/.../lab/dataA
[el]        lock:  /private/tmp/.../lab/dataA/.el-singleton-engram.lock
[el]        Refusing to start a second instance against the same
[el]        state. Two writers against one store is data loss, not a
[el]        warning. Stop the running one and VERIFY it is gone
[el]        (ps -p 3094) before retrying — or point this instance at a
[el]        different state, which is permitted and is not refused.

All three spellings report the same canonical state and the same canonical lock path as the holder — the guard resolved them to one thing.

Two more, because they are properties this fix must not break

No unstick ritual. After kill -9 of the holder, the lock file is still on disk carrying the dead pid, and the next start just works:

$ ls -la .../dataA/.el-singleton-engram.lock
-rw-r--r--  1 will  wheel  5 Aug 16 16:07 .../dataA/.el-singleton-engram.lock
contents (pid of the dead holder): 3094

$ ENGRAM_DATA_DIR=.../dataA ENGRAM_BIND=:19011 /tmp/xnew
[http] listening on [::]:19011 (dual-stack)

Unguardable state refuses instead of starting unguarded (read-only dir):

[el] FATAL: singleton 'engram': cannot open the lock inside the state it guards.
[el]        state: /private/tmp/.../lab/ro
[el]        lock:  /private/tmp/.../lab/ro/.el-singleton-engram.lock (Permission denied)
[el]        The guarded directory must exist and be writable. Refusing to
[el]        start unguarded against it.

Compile-time refusal of a guardless singleton:

$ ./dist/platform/elc /tmp/noguards.el
#error "singleton 'demo' declares no `guards:` — a singleton must name the state it protects, e.g. `guards: engram_resolve_data_dir()` (spec 18.2)"

Toolchain

lang/dist/platform/elc is rebuilt, because the parser and codegen changed. Self-host fixpoint verified byte-identical (elc-new recompiling elc-cli.el reproduces its own input .c exactly). tests/native/{test_core,test_compiler,test_string,test_json}.el all still compile under it.

lang/AGENTS.md's compiler-rebuild recipe is corrected in passing: linking el_runtime.c alone no longer resolves (ld fails on engram_store / engram_vindex / eg_cosine_batch / el_seed symbols).

Files

  • lang/runtime/el_runtime.c / .h — el_singleton_acquire(id, state); lock moved inside the state; el_singleton_dir() and $EL_SINGLETON_DIR deleted; messages made true.
  • lang/el-compiler/src/parser.el — guards: <expr> in the program block.
  • lang/el-compiler/src/codegen.el — passes it as the second argument; #error when absent.
  • lang/spec/language.md — §18.1 grammar, §18.2 rewritten around the principle and the measured failure table, §18.4 cross-reference.
  • engram/src/server.el — guards: engram_resolve_data_dir().
  • lang/dist/platform/elc, lang/AGENTS.md.
## The defect `el_singleton_acquire()` guarded a filename, not a store. The lock was `$EL_SINGLETON_DIR|$TMPDIR|/tmp` + `/el-singleton-<program-name>.lock`. It was keyed on the program's **name** and on a **temp directory**, and it never consulted the state it claimed to protect — while its own refusal message read: > `Refusing to start a second instance against the same state.` It had not looked at any state. Measured on the pre-fix build, it failed in **both** directions: - **False positive.** A second engram started with a different `ENGRAM_DATA_DIR` was refused, naming an unrelated pid. Two instances against genuinely different stores could not coexist. - **False negative — the dangerous one.** `TMPDIR=/tmp/other` let a second instance start against the **same** data dir with no complaint. That is precisely the two-writer data-loss condition the guard exists to prevent, and the workaround was one environment variable. ## The principle > **Guard the thing, not the name.** A lock that protects state must be keyed on the state. Both failure directions are one error, not two: the identity of the resource had been replaced by a label for it. This is the same class of mistake as reducing a structure to a location. ## The design, and why The runtime cannot know generically which environment variable holds an arbitrary program's state, so the `program` block has to say. Three options were weighed: | Option | Verdict | |---|---| | **(a)** lock lives inside the guarded dir | **chosen (mechanism)** | | (b) fixed lock dir, name carries a hash of the canonical state path | rejected | | **(c)** extend `program` so the singleton names its state | **chosen (declaration)** — (a) is impossible without it | **(b) was rejected because it does not actually fix the bug.** Hashing the state into the *name* leaves `$EL_SINGLETON_DIR`/`$TMPDIR` in the *location*, so the false negative survives: point the lock dir somewhere else and the second instance still starts. Closing that would mean hard-coding a lock directory *and* writing a canonicaliser whose string comparison has to be right — machinery that (a) gets from the filesystem for free. **(a) is the whole fix, in one placement decision:** ``` <state>/.el-singleton-<id>.lock ``` - **Same directory** ⇒ same file ⇒ same inode ⇒ the `flock` contends. There is no `TMPDIR` in the key, so there is nothing left to change to get past it. `$EL_SINGLETON_DIR` is deleted, not deprecated. - **Different directories** ⇒ different files ⇒ no contention. Two stores are two stores; they were never in conflict and are no longer treated as if they were. - **Different spellings of one directory** — trailing slash, `x/../x`, a symlink — collapse to one inode in the kernel's own path walk, so they contend *without this code comparing strings*. `realpath()` is used only so the diagnostic names one directory in one spelling; the decision never depends on it and is correct if it fails. - **An unguardable state** (missing, read-only) is a **refusal**, not a fallback. Starting unguarded against the store the guard protects is the failure being removed. **(c), minimally: `guards:` is an expression, not a string.** The engram's data dir is owned by `engram_resolve_data_dir()` — it owns both the `$ENGRAM_DATA_DIR` read and the `$HOME/.neuron/engram` fallback. Spec §18.4 already forbids giving that path a second owner. A string-valued `guards:` would force the default to be written down twice, and a guard that resolved the path its own way could lock a directory the program never writes to. An expression lets the block *point at* the existing owner: ```el program "engram" { singleton: "engram" guards: engram_resolve_data_dir() ... } ``` **`guards:` is mandatory.** `singleton:` without it is now a compile error (`#error` in the emitted C). Emitting a name-keyed lock instead would be emitting the defect — and it fails silently in the direction that loses data. Engram was the only `singleton:` in the tree, so nothing else moves. ### Kept - Still an `flock`, still reports the **holder's pid** (added because `pkill -f` silently failed to match a stale process that went on answering probes). - Still **no "delete the lock file to get unstuck" ritual** — the kernel drops the lock on crash and on `SIGKILL`. A lock file left inside a `cp -Rc`'d data dir is inert: it carries no lock, only a stale pid string the next holder overwrites. Demonstrated below. ### Changed The FATAL message is now **true**. It says "the same state" only because the lock it failed to take is *in* that state, and it names the state it checked. A diagnostic asserting a check that did not happen is worse than none — it is what let the name-keyed version read as correct for as long as it did. ## Measurement CI is not the bar; the local build is. Built exactly as documented, with the rebuilt `dist/platform/elc` (self-host fixpoint verified byte-identical): ``` cd lang && ./dist/platform/elc ../engram/src/server.el > /tmp/x.c cc -std=c11 -O1 -I runtime -I$(brew --prefix openssl@3)/include \ -L$(brew --prefix openssl@3)/lib -o /tmp/xnew /tmp/x.c \ runtime/el_runtime.c runtime/el_seed.c runtime/engram_*.c runtime/eg_cosine_batch*.c \ -lcurl -lssl -lcrypto -lpthread -lm ``` Emitted call: ```c static void __el_program_init(void) { el_singleton_acquire(EL_STR("engram"), engram_resolve_data_dir()); ``` Run on two APFS clones of the live store (`cp -Rc ~/.neuron/engram dataA` / `dataB`, 34 948 nodes each) on ports 19001–19012. Production (`:8742`, `~/.neuron/engram`) was never bound, written, or signalled. ### Summary | # | Case | Correct | Before | After | |---|---|---|---|---| | 1 | same data dir, same `TMPDIR` | refuse | refused ✅ | **refused ✅** | | 2 | same data dir, **different** `TMPDIR` | refuse | **started ❌** | **refused ✅** | | 3 | **different** data dirs, same `TMPDIR` | both start | **refused ❌** | **both started ✅** | | 4 | same dir, different spellings | refuse | trailing-slash / `..` refused only because `TMPDIR` matched; symlink + different `TMPDIR` **started ❌** | **all three refused ✅** | Case 4's "passes" before the fix were passing for the wrong reason — the name matched, not the store — which case 4c exposes by varying `TMPDIR` at the same time. ### Case 1 — same data dir, same TMPDIR → REFUSED (still works) ``` --- holder pid: 2971 --- exit=1 [el] FATAL: another instance of 'engram' is already running (pid 2971). [el] state: /private/tmp/.../lab/dataA [el] lock: /private/tmp/.../lab/dataA/.el-singleton-engram.lock [el] Refusing to start a second instance against the same [el] state. Two writers against one store is data loss, not a [el] warning. Stop the running one and VERIFY it is gone [el] (ps -p 2971) before retrying — or point this instance at a [el] different state, which is permitted and is not refused. ``` ### Case 2 — same data dir, DIFFERENT TMPDIR → REFUSED (the bug) Before (`ENGRAM_DATA_DIR=…/dataA TMPDIR=…/tmpB`) — the second instance **started**: ``` --- holder pid: 176 --- exit=124 [http] listening on [::]:19004 (dual-stack) ``` After — refused, and the message names the store, not the temp dir: ``` --- holder pid: 3004 --- exit=1 [el] FATAL: another instance of 'engram' is already running (pid 3004). [el] state: /private/tmp/.../lab/dataA [el] lock: /private/tmp/.../lab/dataA/.el-singleton-engram.lock [el] Refusing to start a second instance against the same [el] state. Two writers against one store is data loss, not a [el] warning. Stop the running one and VERIFY it is gone [el] (ps -p 3004) before retrying — or point this instance at a [el] different state, which is permitted and is not refused. ``` ### Case 3 — different data dirs, same TMPDIR → BOTH START Before — the second was refused, naming an unrelated pid: ``` --- A pid 251 alive? YES --- --- B pid 265 alive? NO --- --- A probe: {"status":"ok",...,"node_count":34948,"edge_count":43467} --- --- B probe: NO-RESPONSE --- --- B stderr: --- [el] FATAL: another instance of 'engram' is already running (pid 251). [el] lock: /private/tmp/.../lab/tmpA/el-singleton-engram.lock [el] Refusing to start a second instance against the same [el] state. Stop the running one and VERIFY it is gone [el] (ps -p <pid>) before retrying. ``` After — both alive, both serving, both answering from their own store: ``` --- A pid 3035 alive? YES --- --- B pid 3052 alive? YES --- --- A probe: {"status":"ok","engine":"engram-runtime-native","node_count":34948,"edge_count":43467} --- --- B probe: {"status":"ok","engine":"engram-runtime-native","node_count":34948,"edge_count":43467} --- --- B stderr: --- [http] listening on [::]:19006 (dual-stack) ``` ### Case 4 — equivalent-but-differently-spelled paths → REFUSED Holder is on `…/lab/dataA`. Note 4c also varies `TMPDIR` — before the fix it **started** (`exit=124`, `[http] listening on [::]:19010`); the two that did refuse before did so only because the *name* matched. ``` --- holder pid: 3094 --- -- 4a: trailing slash /private/tmp/.../lab/dataA/ -- exit=1 [el] FATAL: another instance of 'engram' is already running (pid 3094). [el] state: /private/tmp/.../lab/dataA [el] lock: /private/tmp/.../lab/dataA/.el-singleton-engram.lock [el] Refusing to start a second instance against the same [el] state. Two writers against one store is data loss, not a [el] warning. Stop the running one and VERIFY it is gone [el] (ps -p 3094) before retrying — or point this instance at a [el] different state, which is permitted and is not refused. -- 4b: dot-dot round trip /private/tmp/.../lab/dataA/../dataA -- exit=1 [el] FATAL: another instance of 'engram' is already running (pid 3094). [el] state: /private/tmp/.../lab/dataA [el] lock: /private/tmp/.../lab/dataA/.el-singleton-engram.lock [el] Refusing to start a second instance against the same [el] state. Two writers against one store is data loss, not a [el] warning. Stop the running one and VERIFY it is gone [el] (ps -p 3094) before retrying — or point this instance at a [el] different state, which is permitted and is not refused. -- 4c: symlink /private/tmp/.../lab/dataA-link (+ different TMPDIR) -- exit=1 [el] FATAL: another instance of 'engram' is already running (pid 3094). [el] state: /private/tmp/.../lab/dataA [el] lock: /private/tmp/.../lab/dataA/.el-singleton-engram.lock [el] Refusing to start a second instance against the same [el] state. Two writers against one store is data loss, not a [el] warning. Stop the running one and VERIFY it is gone [el] (ps -p 3094) before retrying — or point this instance at a [el] different state, which is permitted and is not refused. ``` All three spellings report the **same canonical state and the same canonical lock path** as the holder — the guard resolved them to one thing. ### Two more, because they are properties this fix must not break **No unstick ritual.** After `kill -9` of the holder, the lock file is still on disk carrying the dead pid, and the next start just works: ``` $ ls -la .../dataA/.el-singleton-engram.lock -rw-r--r-- 1 will wheel 5 Aug 16 16:07 .../dataA/.el-singleton-engram.lock contents (pid of the dead holder): 3094 $ ENGRAM_DATA_DIR=.../dataA ENGRAM_BIND=:19011 /tmp/xnew [http] listening on [::]:19011 (dual-stack) ``` **Unguardable state refuses instead of starting unguarded** (read-only dir): ``` [el] FATAL: singleton 'engram': cannot open the lock inside the state it guards. [el] state: /private/tmp/.../lab/ro [el] lock: /private/tmp/.../lab/ro/.el-singleton-engram.lock (Permission denied) [el] The guarded directory must exist and be writable. Refusing to [el] start unguarded against it. ``` **Compile-time refusal of a guardless singleton:** ``` $ ./dist/platform/elc /tmp/noguards.el #error "singleton 'demo' declares no `guards:` — a singleton must name the state it protects, e.g. `guards: engram_resolve_data_dir()` (spec 18.2)" ``` ## Toolchain `lang/dist/platform/elc` is rebuilt, because the parser and codegen changed. Self-host fixpoint verified byte-identical (`elc-new` recompiling `elc-cli.el` reproduces its own input `.c` exactly). `tests/native/{test_core,test_compiler,test_string,test_json}.el` all still compile under it. `lang/AGENTS.md`'s compiler-rebuild recipe is corrected in passing: linking `el_runtime.c` alone no longer resolves (`ld` fails on `engram_store` / `engram_vindex` / `eg_cosine_batch` / `el_seed` symbols). ## Files - `lang/runtime/el_runtime.c` / `.h` — `el_singleton_acquire(id, state)`; lock moved inside the state; `el_singleton_dir()` and `$EL_SINGLETON_DIR` deleted; messages made true. - `lang/el-compiler/src/parser.el` — `guards: <expr>` in the `program` block. - `lang/el-compiler/src/codegen.el` — passes it as the second argument; `#error` when absent. - `lang/spec/language.md` — §18.1 grammar, §18.2 rewritten around the principle and the measured failure table, §18.4 cross-reference. - `engram/src/server.el` — `guards: engram_resolve_data_dir()`. - `lang/dist/platform/elc`, `lang/AGENTS.md`.
will.anderson added 1 commit 2026-08-16 21:10:10 +00:00
singleton: guard the state, not the program's name
El SDK CI - dev / build-and-test (pull_request) Failing after 4m6s
Details
45325f7391 
The singleton lock protected a filename, not a store. It was keyed on
$EL_SINGLETON_DIR|$TMPDIR|/tmp + /el-singleton-<program>.lock — the
program's NAME and a temp directory — and never consulted the state it
claimed to protect, while its own refusal message read "Refusing to start
a second instance against the same state."

Measured, it failed in both directions. A second engram against a
DIFFERENT data dir was refused, naming the first's pid. And
TMPDIR=/tmp/other let a second engram start against the SAME data dir
with no complaint — the two-writer data-loss condition the guard exists
to prevent, defeated by one environment variable.

Both are one error: the identity of the resource had been replaced by a
label for it.

The lock now lives inside the state it guards —
<state>/.el-singleton-<id>.lock — and the program block says what that
state is. Same directory is the same file is the same inode, so it
contends and there is no TMPDIR left in the key to change. Different
directories are different files, so they don't. Different spellings of
one directory (trailing slash, x/../x, symlink) collapse in the kernel's
own path walk, so they contend without this code comparing strings;
canonicalisation is for the message, never the decision.

`guards:` is an expression so a program can point at the resolver that
already owns its path — guards: engram_resolve_data_dir() — instead of
restating that resolver's default, which is the two-owners defect spec
18.4 exists to prevent. A `singleton:` without `guards:` is now a compile
error; emitting a name-keyed lock instead would be emitting the defect.

Kept: the flock (the kernel drops it on crash and SIGKILL, so there is
still no "delete the lock file to get unstuck" ritual — a stale file
inside a copied data dir is inert), and the holder's pid in the message.
Changed: the message is true. It says "the same state" because the lock
it failed to take is in that state, and it names the state it checked.
An unguardable state (missing, read-only) now refuses rather than
starting unguarded.

Also corrects lang/AGENTS.md's compiler rebuild line, which had gone
stale: linking el_runtime.c alone no longer resolves.
will.anderson merged commit e1bc6fe944 into dev 2026-08-17 00:57:50 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
BETA-CRITICAL
On the beta critical path — must resolve before ship
 HELD
Blocked pending an external decision (counsel / dual sign-off)
 ORTHOGONAL
Parallel workstream (testing/hardening) — safe to assign off critical path
 POST-BETA
Important but not blocking the beta ship
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
andre.almeida (Andre Almeida) tim.lingo (Tim Lingo) will.anderson (Will Anderson)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/el#157
Delete Branch "fix/singleton-guards-the-state"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?