lang: give cross-cutting concerns an owner instead of a convention
El's units of encapsulation are the function and the module. Neither can hold
a concern that belongs to the process, so each one had been expressed the only
way it could be -- as a convention: call this at every site. Conventions of
that shape do not hold. Measured here: zero process-identity guards at any
layer, 20 environment variables each with its default written inline at the
read site, 62 persist call sites, 10 per-route auth checks. One absence, four
times.
Step 0 first, because the premise was wrong. El was believed to have no
middleware or effect mechanism. It has one, and it is already load-bearing:
codegen injects engram_boundary_beat at the entry of every @manager/@accessor
fn, decorators take arguments and stack, dharma_emit from a non-@manager fn is
a #error, and the cgi block injects el_cgi_init at the head of main(). So the
correct move was not to invent a mechanism but to generalize the seam that
already existed. The real gap is narrower and is now recorded: the seam is
prologue-only and its callee is a fixed builtin.
Adds a `program` block -- the third program-level declarative block. cgi and
service declare what a program may do; program declares what it is.
program "engram" {
singleton: "engram"
env ENGRAM_BIND: String = ":8742"
env GUIDE_PORT: Int = "8771"
}
singleton takes an exclusive flock before any user statement runs and refuses a
second start, reporting the holder's pid. It is a lock rather than a pidfile so
the kernel releases it on death including SIGKILL -- no stale state, and so no
"delete the lock file to get unstuck" ritual, which would itself be a
convention. It reports the pid because "already running" is not actionable; a
pid is. That is the direct answer to a stale process surviving a pkill and
going on answering probes.
env entries resolve once at startup -- environment wins, declaration supplies
the fallback -- and validate as a whole, reporting every problem at once rather
than costing one restart per variable. config("X") for an undeclared X is
fatal, because an advisory schema is just another convention. Programs without
a program block are unaffected, so migration is per-program.
Only one keyword is added. `config` and `env` could not become keywords -- both
are real identifiers in the tree -- so the block's fields are read as
identifier token values by its own parse loop and stay usable everywhere else.
The init function is emitted at the block site and called from main() rather
than inlined into main(). The live backend is codegen_streaming, which emits in
source order and cannot hold the entry list alive until main(); this way only a
single bool has to survive.
Also fixes: config() was defined in el_runtime.c but never prototyped in
el_runtime.h, so any el program calling it failed to compile under C99.
Spec: section 18 documents what shipped. Section 9 is corrected -- it claimed
decorators had no structural meaning, which has not been true for some time.
Section 19 designs durability-as-an-epilogue-effect and route authorization
and states plainly why neither is implemented here: both land in files under
concurrent modification, and the prerequisite for both is lifting the seam
from prologue-only to prologue/epilogue.
Self-hosting fixpoint verified byte-identical.
This commit is contained in:
bigmerge
+169
-5
@@ -29,6 +29,8 @@ This section is the **single source of truth** for what works and what is planne
|
||||
- Lexer: keywords, identifiers, integer/float/string/bool literals, operators below.
|
||||
- Parser: `let`, `return`, `fn`, `type`, `enum`, `import`, `from … import`, `while`, `for`, `if/else if/else`, `match`, `@decorator`, array/map literals, all listed operators, function calls, field access, index access, unary `!`/`-`, postfix `?`.
|
||||
- Codegen: function definitions, top-level `main()`, all expression forms above, control flow, decorator-as-AST-attachment.
|
||||
- Boundary seam: decorator arguments and stacking; VBD role enforcement via `#error`; `engram_boundary_beat` auto-emit at `@manager`/`@accessor` entry; `@route` dispatch tables (Section 9).
|
||||
- Program-level declarative blocks: `cgi`, `service`, and `program` — the last carrying process identity and configuration (Section 18).
|
||||
- C runtime: I/O, string operations, integer math, lists, maps, filesystem, command-line args, basic `json_get` substring lookup.
|
||||
|
||||
### Planned (in flight)
|
||||
@@ -37,7 +39,7 @@ This section is the **single source of truth** for what works and what is planne
|
||||
- **Match codegen.** Currently parsed; codegen does not emit. Adding `({ ... })` statement-expression emission.
|
||||
- **`?` propagation.** Currently no-op. Adding nil-propagation semantics.
|
||||
- **`cgi` block parsing.** Currently lexed (`cgi` is a keyword) but not parsed as a statement. Adding `parse_cgi_block` and codegen of `el_cgi_init` at the head of `main()`.
|
||||
- **VBD role enforcement.** `@manager`/`@engine`/`@accessor` are accepted as decorators but not enforced. Adding compile-time check that `dharma_emit`/`dharma_field` only appear inside `@manager` functions.
|
||||
- **Boundary epilogues.** The decorator seam injects a prologue only. Adding prologue/epilogue wrapping, the prerequisite for durability-as-an-effect (Section 19.1).
|
||||
- **`vessel` keyword.** Replaces `package` in manifests. Adding to lexer.
|
||||
- **Real `engram_*` runtime.** Currently stub. Adding in-process graph store with spreading activation, Hebbian strengthening, and disk persistence — see Section 16.4.
|
||||
- **Real `dharma_*` runtime.** Currently stub. Adding network transport, channel registry, identity resolution.
|
||||
@@ -96,8 +98,10 @@ The following words are reserved and cannot be used as identifiers. Each row not
|
||||
| `while` | yes | Loop |
|
||||
| `import` / `from` / `as` | yes | Module import |
|
||||
| `true` / `false` | yes | Bool literals |
|
||||
| `cgi` | planned | Top-level CGI declaration block |
|
||||
| `manager` / `engine` / `accessor` | as decorators | VBD role marker on `fn` (enforcement planned) |
|
||||
| `cgi` | yes | Top-level CGI declaration block |
|
||||
| `service` | yes | Top-level capability-bounded declaration block |
|
||||
| `program` | yes | Top-level cross-cutting declaration block (Section 18) |
|
||||
| `manager` / `engine` / `accessor` | as decorators | VBD role marker on `fn`; enforcement and boundary auto-emit are live (Section 9) |
|
||||
| `vessel` | planned | Manifest declaration (replaces `package`) |
|
||||
| `activate` / `where` | planned | Spreading-activation construct |
|
||||
| `sealed` | planned | Capability scope block |
|
||||
@@ -446,9 +450,21 @@ Parsed. The module name is recorded; the brace-list is consumed. Both forms prod
|
||||
fn handle(channel: String, msg: String) -> Void { … }
|
||||
```
|
||||
|
||||
The `@` token followed by an identifier attaches a decorator name to the next `FnDef`. Decorators with structural meaning today: none. Planned enforcement (Section 16.2): VBD roles `@manager`, `@engine`, `@accessor`.
|
||||
The `@` token followed by an identifier attaches a decorator to the next `FnDef`.
|
||||
|
||||
Non-VBD decorators are accepted and ignored.
|
||||
**Decorators take arguments and they stack.** `@route("/p", "GET") @manager fn f()` attaches both to `f` as a `decorators` list of `{name, args}` records, topmost-first. Arguments are string literals only.
|
||||
|
||||
**Decorators have structural meaning today.** This is El's function-level boundary seam — the mechanism by which a cross-cutting concern is handled *at the boundary* rather than by a convention repeated at every call site:
|
||||
|
||||
| Decorator | Structural effect |
|
||||
|---|---|
|
||||
| `@manager` | Permits calls to `dharma_emit` / `dharma_field`. Calling either from a non-`@manager` fn emits a `#error` into the generated C — a compile-time failure, not a lint. |
|
||||
| `@manager`, `@accessor` | Codegen injects one call to `engram_boundary_beat(<fn name>)` at function entry. The decorated op self-reports (chrono tick, afferent counter, self-activity strengthen, dharma bus event) with **zero** hand-written instrumentation in its body. |
|
||||
| `@route(path, method, …)` | Records a route into a generated dispatch table. |
|
||||
|
||||
Decorators with no registered meaning are accepted and ignored.
|
||||
|
||||
**Limits of the seam, as it stands.** The injection is a *prologue only* — there is no epilogue, no wrapping of the call, and no way for a decorator to run code after the body returns. The injected callee is a fixed builtin chosen by the compiler, not derived from the decorator name or its arguments. Section 19 depends on lifting exactly these two limits.
|
||||
|
||||
---
|
||||
|
||||
@@ -1088,4 +1104,152 @@ The next minor version closes the implementation gaps named in this document. Tr
|
||||
|
||||
---
|
||||
|
||||
## 18. The Program Block — cross-cutting concerns [implemented]
|
||||
|
||||
### 18.0 Why this exists
|
||||
|
||||
A cross-cutting concern is one that belongs to the *process*, not to any function in it: only one of me may run; this is what my configuration is; every mutation must be durable; every request must be authorized.
|
||||
|
||||
El's units of encapsulation are the function and the module. Neither can hold a concern like that. So each one had been expressed the only way it could be — as a **convention**: *call this at every site.* Conventions of that shape do not hold. They are not enforced by anything, they are invisible in review, and they fail silently at the one site somebody forgot.
|
||||
|
||||
Measured in this codebase before this section existed:
|
||||
|
||||
| Concern | State | What the convention was |
|
||||
|---|---|---|
|
||||
| process identity | **zero** guards anywhere — no pidfile, no lock, no already-running check, at any layer | "check nothing is already running first" |
|
||||
| configuration | **20** distinct environment variables in one program, each with its default written inline at the read site | "remember the right default here" |
|
||||
| durability | **62** `persist_*` / `engram_save` / `wal_*` / `checkpoint` call sites | "after you mutate, remember to persist" |
|
||||
| request auth | **10** per-route `_auth` checks | "check the token in this handler too" |
|
||||
|
||||
These are not four problems. They are one absence, four times.
|
||||
|
||||
That the convention form fails is observed, not predicted. Process identity failed three times in a single day: twice, two engram processes ran simultaneously against the same data directory; twice, a stale binary held a port and answered probes while a fresh build was believed to be under test, because `pkill -f` had silently failed to match its argv — which nearly produced a false "the fix does not work" conclusion. Configuration failed structurally: `ENGRAM_DATA_DIR` was read at six sites, five of them dead bindings, and the sixth defaulted to `/tmp/engram` — contradicting the canonical resolver's `$HOME/.neuron/engram` and landing a pre-destructive safety backup on ephemeral storage.
|
||||
|
||||
The `program` block is where a concern of this shape is declared once and enforced by the compiler at the process boundary.
|
||||
|
||||
### 18.1 Syntax
|
||||
|
||||
```
|
||||
program "engram" {
|
||||
singleton: "engram"
|
||||
env ENGRAM_BIND: String = ":8742"
|
||||
env GUIDE_PORT: Int = "8771"
|
||||
env ENGRAM_API_KEY: String required
|
||||
}
|
||||
```
|
||||
|
||||
At most one `program` block per program. It composes with `cgi` and `service` — those declare what a program *may do*; `program` declares what a program *is*.
|
||||
|
||||
Grammar:
|
||||
|
||||
```ebnf
|
||||
program_block = "program" string "{" { program_field } "}" ;
|
||||
program_field = singleton_field | env_field ;
|
||||
singleton_field = "singleton" ":" string [ "," ] ;
|
||||
env_field = "env" ident ":" type
|
||||
[ "=" string ] [ "required" ] [ "," ] ;
|
||||
```
|
||||
|
||||
`singleton` and `env` are **not** reserved words. They are read as identifier token values by the block's own parse loop, so they remain usable as ordinary identifiers everywhere else. `program` is the only keyword this section adds.
|
||||
|
||||
### 18.2 Process identity — `singleton`
|
||||
|
||||
`singleton: "id"` compiles to an `el_singleton_acquire("id")` call injected as the **first statement of `main()`**, before any user statement runs.
|
||||
|
||||
The runtime takes an exclusive non-blocking `flock` on `<dir>/el-singleton-<id>.lock`, where `<dir>` is `$EL_SINGLETON_DIR`, else `$TMPDIR`, else `/tmp`. On success it writes its pid and holds the descriptor open for the life of the process. On contention it **refuses to start**: it reports the holder's pid, names the lock file, and exits 1.
|
||||
|
||||
Two properties are deliberate:
|
||||
|
||||
- **It is a lock, not a pidfile.** The kernel releases an `flock` when the owning process dies — including on `SIGKILL` and on crash. There is therefore no stale-lock state, and so no "delete the lock file to get unstuck" recovery ritual. Such a ritual would itself be a convention, which is the thing this section exists to remove.
|
||||
- **It reports the holder's pid.** "Already running" is not actionable. A pid is. This is the direct answer to the observed failure where a stale process survived a `pkill` and went on answering probes.
|
||||
|
||||
Refusal is loud and total. It is not a warning, and the program does not continue degraded. This matters more than it looks: today a second engram whose `bind()` fails merely *returns* from `http_serve` — after it has already replayed the WAL and written boot-time backup files — and then exits **0**, indistinguishable from a clean run. `singleton` refuses before the first side effect.
|
||||
|
||||
### 18.3 Configuration — `env`
|
||||
|
||||
Each `env` entry declares one configuration variable: its name, its type (`Int` or `String`), and either a default or `required`.
|
||||
|
||||
Resolution happens once, at startup, in declaration order: **the environment wins; the declaration supplies the fallback.** Then `el_config_validate` checks the whole schema and reports *every* problem at once before exiting — a startup that fails one variable at a time costs one restart per variable.
|
||||
|
||||
Values are read with `config("NAME")`, which returns a `String`.
|
||||
|
||||
The enforcement that makes the declaration real: **once a program block exists, `config("X")` for an undeclared `X` is a fatal error.** Without that, the schema would be advisory, and an advisory schema is just another convention. Programs with no `program` block are unaffected — `config()` falls back to a plain environment read, so migration is incremental and per-program.
|
||||
|
||||
The point is not that configuration is now centralized. It is that **a default is no longer a decision made at a read site.** A read site cannot disagree with another read site about what a variable means, because a read site no longer says.
|
||||
|
||||
### 18.4 What is deliberately not declared here
|
||||
|
||||
Some values look like configuration and are not. `ENGRAM_DATA_DIR` already has a single owner — `engram_resolve_data_dir()`, which resolves it, creates the directory, and fails loud rather than silently persisting to an ephemeral path. Declaring it in the `program` block as well would give it two owners that can disagree, recreating the precise defect this section removes.
|
||||
|
||||
The rule: **a variable belongs in the program block when the block would be its only owner.** If a resolver already owns it, leave it there.
|
||||
|
||||
`HOME` is likewise not configuration. It is an environment fact, and stays a raw `env()` read.
|
||||
|
||||
---
|
||||
|
||||
## 19. Boundary Effects — durability and request authorization [design only, not implemented]
|
||||
|
||||
Sections 19.1 and 19.2 specify the two remaining concerns from the table in 18.0. Both are **designed and deliberately unimplemented.** The reason is stated in 19.3 and it is not difficulty.
|
||||
|
||||
### 19.1 Durability as an epilogue effect
|
||||
|
||||
**The defect.** 62 call sites carry the convention *"after you mutate, remember to persist."* This is structurally the same defect as the index bug being fixed elsewhere in this tree — *"after you append, remember to index"* — which failed at **9 of 9** sites. A convention that failed at 100% of its sites is the strongest available evidence about what this class of convention is worth.
|
||||
|
||||
**Why the existing seam cannot express it.** §9's injection is a prologue. Durability is inherently an *epilogue*: persist after the mutation succeeds, and not at all if it threw. The seam has no epilogue.
|
||||
|
||||
**Design.** Extend the decorator seam from prologue-only to prologue/epilogue, then declare durability as an effect on the mutating function:
|
||||
|
||||
```
|
||||
@durable("engram")
|
||||
fn engram_write_node(id: String, body: String) -> Bool { … }
|
||||
```
|
||||
|
||||
Codegen wraps rather than prefixes:
|
||||
|
||||
```c
|
||||
el_val_t engram_write_node(el_val_t id, el_val_t body) {
|
||||
el_effect_enter(EL_STR("durable"), EL_STR("engram"));
|
||||
el_val_t __r = /* original body */;
|
||||
el_effect_exit(EL_STR("durable"), EL_STR("engram"), __r);
|
||||
return __r;
|
||||
}
|
||||
```
|
||||
|
||||
`el_effect_exit` is where the persist happens, and it is the only place it happens. Two properties follow that the 62 hand-written sites cannot have:
|
||||
|
||||
- **Coalescing.** The epilogue is a single choke point, so N mutations inside one request can produce one fsync instead of N. The hand-written form cannot coalesce, because no site knows about the others.
|
||||
- **Failure is not silent.** A persist that fails inside `el_effect_exit` can force the mutation's return value to failure. A forgotten `persist_*` call cannot fail — it simply does not happen, which is exactly why the defect is invisible.
|
||||
|
||||
**Enforcement, and this is the part that actually fixes it.** Mirroring §9's `#error` for `dharma_emit`: a function that calls a mutating primitive without carrying `@durable` is a **compile error**. Otherwise this is a 63rd thing to remember rather than a replacement for 62.
|
||||
|
||||
### 19.2 Request authorization as a route effect
|
||||
|
||||
**The defect.** 10 per-route `_auth` checks. The HTTP layer has no concept of authorization, so a new route is unauthenticated by default and silently so — the failure mode is a route that forgot, and nothing anywhere reports it.
|
||||
|
||||
**Design.** Authorization becomes an argument to the `@route` decorator, which already takes arguments and already builds a dispatch table:
|
||||
|
||||
```
|
||||
@route("/api/write", "POST", auth: "required")
|
||||
fn route_write(body: String) -> String { … }
|
||||
```
|
||||
|
||||
The generated dispatcher performs the check **before** dispatch, so an unauthorized request never reaches the handler and the handler contains no auth code at all.
|
||||
|
||||
The default must be `required`. A route that says nothing gets authorization; opening one up takes an explicit `auth: "public"`. Defaulting to public preserves the current failure mode exactly — forgetting stays silent — and a default that preserves the defect is not a fix.
|
||||
|
||||
Route inventory falls out for free: the dispatch table already exists, so the compiler can emit the full route/auth matrix and make "which routes are public" a fact that is read rather than audited.
|
||||
|
||||
### 19.3 Why these are not implemented
|
||||
|
||||
Not difficulty — **collision**. Both land squarely in regions two other agents hold right now:
|
||||
|
||||
- **Durability** requires changing the mutation and persist paths in `lang/runtime/el_runtime.c` and `engram/src/server.el` — the same files and the same read/write paths being restructured by concurrent work on VIndex read-path mutation and memory ownership, and on geometry-as-an-el-value and `transduce`.
|
||||
- **Request auth** requires changing route dispatch in `engram/src/server.el`, which the geometry/`transduce` work is actively reshaping.
|
||||
|
||||
Implementing either now would mean editing files under concurrent modification and resolving conflicts in exactly the paths whose correctness is currently under repair. The designs are recorded here so the work is not lost, and so that whoever lands them does so against a settled tree.
|
||||
|
||||
The prerequisite for 19.1 is the same in both cases: **lift the §9 seam from prologue-only to prologue/epilogue.** That change is independent of both collisions and can land first.
|
||||
|
||||
---
|
||||
|
||||
End of specification.
|
||||
|
||||
Reference in New Issue
Block a user