Give cross-cutting concerns an owner instead of a convention #145

2026-08-16T16:39:39Z

will.anderson commented

2026-08-16 16:39:39 +00:00

Step 0 first: the premise was wrong

I was asked to confirm or refute "el has no middleware/effect mechanism" before building on it. It is refuted. El has a boundary seam, it is already load-bearing, and I found it by reading lang/el-compiler/src/ rather than inferring from code shape:

codegen.el cg_fn() injects engram_boundary_beat(<fn name>) at the entry of every @manager/@accessor function. The source comment states the intent exactly: "auto-emit at the decorated-fn boundary … so a decorated op self-reports with ZERO hand-written instrumentation in its body." That is middleware.
Decorators take arguments and stack (@route("/p","GET") @manager fn f()), parsed into a decorators list.
dharma_emit from a non-@manager fn emits a #error into the generated C — compile-time policy, not lint.
cgi and service are program-level declarative blocks; cgi injects el_cgi_init at the head of main(), and the block kind drives capability enforcement.

So the correct move was not to invent a mechanism but to generalize the one that already existed. Inventing a parallel one would have duplicated working machinery.

The real gap is narrower, and is now recorded in the spec: the seam is prologue-only, and its callee is a fixed builtin chosen by the compiler rather than derived from the decorator. That is precisely why the two concerns needing an epilogue (durability) or per-route policy (auth) degraded into 62 and 10 hand-written call sites.

Also: lang/spec/language.md was stale. It claimed "Decorators with structural meaning today: none" and listed VBD enforcement as planned; both have been implemented for some time. It did not mention service, decorator arguments, stacking, @route, or engram_boundary_beat at all. Corrected here.

What this adds

One new keyword. A program block — the third program-level declarative block. cgi and service declare what a program may do; program declares what it is.

program "engram" {
    singleton: "engram"
    env ENGRAM_BIND: String = ":8742"
    env GUIDE_PORT:  Int    = "8771"
    env ENGRAM_API_KEY: String required
}

Process identity

singleton takes an exclusive flock injected as the first statement of main() and refuses a second start, reporting the holder's pid.

A lock, not a pidfile — the kernel releases it on death including SIGKILL, so there is no stale-lock state and therefore no "delete the lock file to get unstuck" ritual, which would itself be a convention.
It reports the pid because "already running" is not actionable and a pid is. This is the direct answer to a stale process surviving a pkill -f and going on answering probes.

Configuration

env entries resolve once at startup — environment wins, declaration supplies the fallback — then validate as a whole, reporting every problem in one run rather than costing one restart per variable. config("X") for an undeclared X is fatal, because an advisory schema is just another convention. Programs with no program block are unaffected, so migration is per-program.

Only program became a keyword. config and env could not — both are real identifiers in this tree (lang/runtime/env.el:70, epm/src/registry.el:23). The block's fields are read as identifier token values by its own parse loop, so they stay usable everywhere else.

Evidence

Singleton, on the real engram binary — instance 1's pid agrees across three independent read-backs (lsof on the bound port, the shell's job pid, and the lock file contents, all 5946):

eng2    5946 will    4u  IPv6 ...  TCP *:19100 (LISTEN)
shell bg pid = 5946
lock file contains = 5946

[el] FATAL: another instance of 'engram' is already running (pid 5946).
exit=1

The refused instance left the data directory empty — it refused before any side effect. That is the contrast that matters: today a second engram whose bind() fails merely returns from http_serve, after replaying the WAL and writing boot-time backup files, and then exits 0 — indistinguishable from a clean run. That is how two instances came to share one data dir.

SIGKILL the holder and the next start succeeds with no cleanup, proving the no-stale-state property.

Configuration — three bad values, one run:

[el] config: ENGRAM_ISE_RETENTION_MS is declared Int but its value is "abc"
[el] config: GUIDE_PORT is declared Int but its value is "nope"
[el] config: GUIDE_NGL is declared Int but its value is "x"
[el] FATAL: program 'engram' has 3 invalid configuration entries. Refusing to start.
exit=1

Migration completeness, by bijection on the generated C — 18 config() reads, 18 declarations, nothing read without a declaration and nothing declared without a read.

Self-hosting fixpoint verified byte-identical, before the rebase and again after rebuilding against merged dev. test_compiler 82/82.

The ENGRAM_DATA_DIR bug, fixed

The clearest instance of the defect. Read at six sites. Five were dead — let dir_raw = env("ENGRAM_DATA_DIR") immediately shadowed on the next line by engram_resolve_data_dir(). The sixth was live and defaulted to /tmp/engram, contradicting the canonical resolver's $HOME/.neuron/engram — and its consumer is the pre-destructive reseed backup, so with ENGRAM_DATA_DIR unset the safety copy was written to ephemeral storage while the store it protected lived elsewhere.

It is deliberately not declared in the program block, and the source says why: engram_resolve_data_dir() already owns it, and a second declaration would give it two owners that can disagree — recreating the exact defect being removed. The rule recorded in the spec: a variable belongs in the block when the block would be its only owner.

Designed, deliberately not implemented

lang/spec/language.md §19 specifies durability as an epilogue effect (@durable, with wrapping rather than prefixing, coalescing N mutations into one fsync, and a #error for a mutating fn that lacks the decorator) and request auth as a route effect (auth: on @route, checked before dispatch, defaulting to required — defaulting to public would preserve the current silent-forget failure mode, and a default that preserves the defect is not a fix).

Not implemented because of collision, not difficulty: both land in el_runtime.c and engram/src/server.el paths under concurrent modification by the VIndex and geometry/transduce work. Editing files whose correctness is currently under repair is how you get a false conclusion about whose change broke what. The prerequisite for both — lifting the seam from prologue-only to prologue/epilogue — is independent of both collisions and can land first.

Corrections to the brief's premises

neuron/soul.el and neuron/routes.el do not exist in this repo. Zero hits for soul anywhere in the worktree. Soul lives in a separate repo, so its 10 env vars were not migrated. Engram has 20 distinct env vars, not 12 (18 configuration + ENGRAM_DATA_DIR + HOME).
engram.memguard does not exist. Zero hits repo-wide; the binary is engram.
The documented build recipe is stale: lang/AGENTS.md says link only runtime/el_runtime.c, but the link now also needs el_seed.c plus the engram translation units. README.md is worse — it references lang/el-compiler/runtime/, which does not exist.

Note on the bootstrap binary

lang/dist/platform/elc is re-stamped, because server.el now declares a program block the previously committed compiler cannot parse — without it the tree is source the repo's own compiler rejects. This is the documented re-stamp from BOOTSTRAP.md, and its precondition (verified fixpoint) is met.

Two pre-existing test failures are unchanged and are not from this work — confirmed by rebuilding them against the original runtime: test_env's "state_keys returns JSON array" fails identically before and after, and test_json/test_state fail to link on symbols never prototyped (json_build_array, state_has) — the same class of gap as config(), which this branch fixed because it blocked the build.

Follow-ups worth filing

ENGRAM_API_KEY keeps its permissive empty default, which disables auth entirely. Pre-existing; making it required is a behaviour change and out of scope, but it is the obvious hardening.
http_serve returning instead of exiting on bind failure is still there. singleton covers the duplicate-instance case, but a genuine port conflict still exits 0.

## Step 0 first: the premise was wrong I was asked to confirm or refute "el has no middleware/effect mechanism" before building on it. **It is refuted.** El has a boundary seam, it is already load-bearing, and I found it by reading `lang/el-compiler/src/` rather than inferring from code shape: - `codegen.el` `cg_fn()` injects `engram_boundary_beat(<fn name>)` at the entry of every `@manager`/`@accessor` function. The source comment states the intent exactly: *"auto-emit at the decorated-fn boundary … so a decorated op self-reports with ZERO hand-written instrumentation in its body."* That is middleware. - Decorators take arguments and stack (`@route("/p","GET") @manager fn f()`), parsed into a `decorators` list. - `dharma_emit` from a non-`@manager` fn emits a `#error` into the generated C — compile-time policy, not lint. - `cgi` and `service` are program-level declarative blocks; `cgi` injects `el_cgi_init` at the head of `main()`, and the block kind drives capability enforcement. So the correct move was **not to invent a mechanism but to generalize the one that already existed.** Inventing a parallel one would have duplicated working machinery. The real gap is narrower, and is now recorded in the spec: the seam is **prologue-only**, and its callee is a **fixed builtin** chosen by the compiler rather than derived from the decorator. That is precisely why the two concerns needing an epilogue (durability) or per-route policy (auth) degraded into 62 and 10 hand-written call sites. Also: `lang/spec/language.md` was **stale**. It claimed "Decorators with structural meaning today: none" and listed VBD enforcement as planned; both have been implemented for some time. It did not mention `service`, decorator arguments, stacking, `@route`, or `engram_boundary_beat` at all. Corrected here. ## What this adds One new keyword. A `program` block — the third program-level declarative block. `cgi` and `service` declare what a program *may do*; `program` declares what it *is*. ```el program "engram" { singleton: "engram" env ENGRAM_BIND: String = ":8742" env GUIDE_PORT: Int = "8771" env ENGRAM_API_KEY: String required } ``` ### Process identity `singleton` takes an exclusive `flock` injected as the **first statement of `main()`** and refuses a second start, reporting the holder's pid. - **A lock, not a pidfile** — the kernel releases it on death including `SIGKILL`, so there is no stale-lock state and therefore no "delete the lock file to get unstuck" ritual, which would itself be a convention. - **It reports the pid** because "already running" is not actionable and a pid is. This is the direct answer to a stale process surviving a `pkill -f` and going on answering probes. ### Configuration `env` entries resolve once at startup — environment wins, declaration supplies the fallback — then validate as a whole, reporting **every** problem in one run rather than costing one restart per variable. `config("X")` for an undeclared `X` is fatal, because an advisory schema is just another convention. Programs with no `program` block are unaffected, so migration is per-program. Only `program` became a keyword. `config` and `env` **could not** — both are real identifiers in this tree (`lang/runtime/env.el:70`, `epm/src/registry.el:23`). The block's fields are read as identifier token *values* by its own parse loop, so they stay usable everywhere else. ## Evidence **Singleton, on the real engram binary** — instance 1's pid agrees across three independent read-backs (`lsof` on the bound port, the shell's job pid, and the lock file contents, all 5946): ``` eng2 5946 will 4u IPv6 ... TCP *:19100 (LISTEN) shell bg pid = 5946 lock file contains = 5946 [el] FATAL: another instance of 'engram' is already running (pid 5946). exit=1 ``` The refused instance left the data directory **empty** — it refused before any side effect. That is the contrast that matters: today a second engram whose `bind()` fails merely *returns* from `http_serve`, **after** replaying the WAL and writing boot-time backup files, and then exits **0** — indistinguishable from a clean run. That is how two instances came to share one data dir. `SIGKILL` the holder and the next start succeeds with no cleanup, proving the no-stale-state property. **Configuration** — three bad values, one run: ``` [el] config: ENGRAM_ISE_RETENTION_MS is declared Int but its value is "abc" [el] config: GUIDE_PORT is declared Int but its value is "nope" [el] config: GUIDE_NGL is declared Int but its value is "x" [el] FATAL: program 'engram' has 3 invalid configuration entries. Refusing to start. exit=1 ``` **Migration completeness, by bijection on the generated C** — 18 `config()` reads, 18 declarations, nothing read without a declaration and nothing declared without a read. **Self-hosting fixpoint** verified byte-identical, before the rebase and again after rebuilding against merged `dev`. `test_compiler` 82/82. ## The ENGRAM_DATA_DIR bug, fixed The clearest instance of the defect. Read at six sites. Five were **dead** — `let dir_raw = env("ENGRAM_DATA_DIR")` immediately shadowed on the next line by `engram_resolve_data_dir()`. The sixth was live and defaulted to `/tmp/engram`, contradicting the canonical resolver's `$HOME/.neuron/engram` — and its consumer is the **pre-destructive reseed backup**, so with `ENGRAM_DATA_DIR` unset the safety copy was written to ephemeral storage while the store it protected lived elsewhere. It is deliberately **not** declared in the `program` block, and the source says why: `engram_resolve_data_dir()` already owns it, and a second declaration would give it two owners that can disagree — recreating the exact defect being removed. The rule recorded in the spec: *a variable belongs in the block when the block would be its only owner.* ## Designed, deliberately not implemented `lang/spec/language.md` §19 specifies **durability as an epilogue effect** (`@durable`, with wrapping rather than prefixing, coalescing N mutations into one fsync, and a `#error` for a mutating fn that lacks the decorator) and **request auth as a route effect** (`auth:` on `@route`, checked before dispatch, defaulting to `required` — defaulting to public would preserve the current silent-forget failure mode, and a default that preserves the defect is not a fix). Not implemented because of **collision, not difficulty**: both land in `el_runtime.c` and `engram/src/server.el` paths under concurrent modification by the VIndex and geometry/`transduce` work. Editing files whose correctness is currently under repair is how you get a false conclusion about whose change broke what. The prerequisite for both — lifting the seam from prologue-only to prologue/epilogue — is independent of both collisions and can land first. ## Corrections to the brief's premises - **`neuron/soul.el` and `neuron/routes.el` do not exist in this repo.** Zero hits for `soul` anywhere in the worktree. Soul lives in a separate repo, so its 10 env vars were **not** migrated. Engram has **20** distinct env vars, not 12 (18 configuration + `ENGRAM_DATA_DIR` + `HOME`). - **`engram.memguard` does not exist.** Zero hits repo-wide; the binary is `engram`. - The documented build recipe is **stale**: `lang/AGENTS.md` says link only `runtime/el_runtime.c`, but the link now also needs `el_seed.c` plus the engram translation units. `README.md` is worse — it references `lang/el-compiler/runtime/`, which does not exist. ## Note on the bootstrap binary `lang/dist/platform/elc` is re-stamped, because `server.el` now declares a `program` block the previously committed compiler cannot parse — without it the tree is source the repo's own compiler rejects. This is the documented re-stamp from `BOOTSTRAP.md`, and its precondition (verified fixpoint) is met. Two pre-existing test failures are unchanged and are **not** from this work — confirmed by rebuilding them against the original runtime: `test_env`'s "state_keys returns JSON array" fails identically before and after, and `test_json`/`test_state` fail to link on symbols never prototyped (`json_build_array`, `state_has`) — the same class of gap as `config()`, which this branch fixed because it blocked the build. ## Follow-ups worth filing - `ENGRAM_API_KEY` keeps its permissive empty default, which **disables auth entirely**. Pre-existing; making it `required` is a behaviour change and out of scope, but it is the obvious hardening. - `http_serve` returning instead of exiting on bind failure is still there. `singleton` covers the duplicate-instance case, but a genuine port conflict still exits 0.

will.anderson added 4 commits 2026-08-16 16:39:42 +00:00

lang: give cross-cutting concerns an owner instead of a convention 8ae163e8e5

El's units of encapsulation are the function and the module. Neither can hold
a concern that belongs to the process, so each one had been expressed the only
way it could be -- as a convention: call this at every site. Conventions of
that shape do not hold. Measured here: zero process-identity guards at any
layer, 20 environment variables each with its default written inline at the
read site, 62 persist call sites, 10 per-route auth checks. One absence, four
times.

Step 0 first, because the premise was wrong. El was believed to have no
middleware or effect mechanism. It has one, and it is already load-bearing:
codegen injects engram_boundary_beat at the entry of every @manager/@accessor
fn, decorators take arguments and stack, dharma_emit from a non-@manager fn is
a #error, and the cgi block injects el_cgi_init at the head of main(). So the
correct move was not to invent a mechanism but to generalize the seam that
already existed. The real gap is narrower and is now recorded: the seam is
prologue-only and its callee is a fixed builtin.

Adds a `program` block -- the third program-level declarative block. cgi and
service declare what a program may do; program declares what it is.

  program "engram" {
      singleton: "engram"
      env ENGRAM_BIND: String = ":8742"
      env GUIDE_PORT:  Int    = "8771"
  }

singleton takes an exclusive flock before any user statement runs and refuses a
second start, reporting the holder's pid. It is a lock rather than a pidfile so
the kernel releases it on death including SIGKILL -- no stale state, and so no
"delete the lock file to get unstuck" ritual, which would itself be a
convention. It reports the pid because "already running" is not actionable; a
pid is. That is the direct answer to a stale process surviving a pkill and
going on answering probes.

env entries resolve once at startup -- environment wins, declaration supplies
the fallback -- and validate as a whole, reporting every problem at once rather
than costing one restart per variable. config("X") for an undeclared X is
fatal, because an advisory schema is just another convention. Programs without
a program block are unaffected, so migration is per-program.

Only one keyword is added. `config` and `env` could not become keywords -- both
are real identifiers in the tree -- so the block's fields are read as
identifier token values by its own parse loop and stay usable everywhere else.

The init function is emitted at the block site and called from main() rather
than inlined into main(). The live backend is codegen_streaming, which emits in
source order and cannot hold the entry list alive until main(); this way only a
single bool has to survive.

Also fixes: config() was defined in el_runtime.c but never prototyped in
el_runtime.h, so any el program calling it failed to compile under C99.

Spec: section 18 documents what shipped. Section 9 is corrected -- it claimed
decorators had no structural meaning, which has not been true for some time.
Section 19 designs durability-as-an-epilogue-effect and route authorization
and states plainly why neither is implemented here: both land in files under
concurrent modification, and the prerequisite for both is lifting the seam
from prologue-only to prologue/epilogue.

Self-hosting fixpoint verified byte-identical.

lang: re-stamp the bootstrap compiler so the tree can compile its own source b305b49f40

server.el declares a `program` block, which the previously committed elc cannot
parse. Without this the tree is internally inconsistent: source in the repo that
the compiler in the repo rejects.

This is the documented re-stamp from BOOTSTRAP.md / AGENTS.md, and its
precondition is met -- the self-hosting fixpoint was verified byte-identical
(stage3 output == stage2 output) both before installing and again with the
installed binary. tests/native/test_compiler.el passes 82/82 against it.

Two pre-existing failures are unchanged and are NOT from this work, confirmed
by rebuilding them against the original runtime: test_env's
"state_keys returns JSON array" fails identically before and after, and
test_json/test_state fail to link on symbols (json_build_array, state_has) that
were never prototyped -- the same class of gap as config(), which this branch
fixed because it blocked the build.

engram: declare configuration once instead of at every read site c18abf799c

Migrates engram to the `program` block. 18 configuration variables that each
carried their default inline at the point of use now declare it in one place,
and engram declares itself a singleton.

The read sites lose their defaults entirely: `let v = env("X")` followed by
`if str_eq(v,"") { "default" } else { v }` collapses to `config("X")`. The
guide_env_or(key, dflt) helper is deleted -- its whole job was supplying a
per-site default, which is the thing being removed.

Fixes ENGRAM_DATA_DIR, which was the clearest instance of the defect. It was
read at six sites. Five were dead: `let dir_raw = env("ENGRAM_DATA_DIR")`
immediately shadowed on the next line by `engram_resolve_data_dir()`. The sixth
was live and defaulted to /tmp/engram, contradicting the canonical resolver's
$HOME/.neuron/engram -- and its consumer is the pre-destructive reseed backup,
so with ENGRAM_DATA_DIR unset the safety copy was written to ephemeral storage
while the store it protected lived elsewhere. All six now go through
engram_resolve_data_dir().

ENGRAM_DATA_DIR is deliberately NOT declared in the program block, and the
source says why: engram_resolve_data_dir() already owns it, and a second
declaration would give it two owners that can disagree -- recreating the exact
defect being removed here. A variable belongs in the block when the block would
be its only owner. HOME stays a raw env() read; it is an environment fact, not
configuration.

singleton: "engram" matters more than it looks. Today a second engram whose
bind() fails merely returns from http_serve -- after it has already replayed
the WAL and written boot-time backup files -- and then exits 0, indistinguishable
from a clean run. That is how two instances came to share one data dir. Verified
that the second instance now refuses before any side effect: with instance 1
holding the lock (lsof pid, shell pid, and lock file contents all agreeing at
5946), the second start named that pid, exited 1, and left the data directory
untouched.

Verified by bijection on the generated C: 18 config() reads, 18 declarations,
no read without a declaration and no declaration without a read. Three bad Int
values are reported in a single run rather than costing one restart each.

ENGRAM_API_KEY keeps its permissive empty default, which disables auth -- that
is pre-existing behaviour and changing it is out of scope. The source marks
making it `required` as the obvious hardening follow-up.

lang: rebuild the bootstrap compiler against merged dev

El SDK CI - dev / build-and-test (pull_request) Failing after 4m46s

Details

26af149aa1

The binary was stamped before dev advanced (vindex publication landed in
el_runtime.c and engram_vindex.c). Rebuilt against the merged runtime so the
committed compiler matches the runtime it ships beside. Fixpoint re-verified
byte-identical; test_compiler 82/82; engram/src/server.el still compiles and
still emits its 18 config declarations.

will.anderson merged commit 616815b2ab into dev

2026-08-16 16:57:54 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#145