bigmerge
45325f7391
El SDK CI - dev / build-and-test (pull_request) Failing after 4m6s
The singleton lock protected a filename, not a store. It was keyed on $EL_SINGLETON_DIR|$TMPDIR|/tmp + /el-singleton-<program>.lock — the program's NAME and a temp directory — and never consulted the state it claimed to protect, while its own refusal message read "Refusing to start a second instance against the same state." Measured, it failed in both directions. A second engram against a DIFFERENT data dir was refused, naming the first's pid. And TMPDIR=/tmp/other let a second engram start against the SAME data dir with no complaint — the two-writer data-loss condition the guard exists to prevent, defeated by one environment variable. Both are one error: the identity of the resource had been replaced by a label for it. The lock now lives inside the state it guards — <state>/.el-singleton-<id>.lock — and the program block says what that state is. Same directory is the same file is the same inode, so it contends and there is no TMPDIR left in the key to change. Different directories are different files, so they don't. Different spellings of one directory (trailing slash, x/../x, symlink) collapse in the kernel's own path walk, so they contend without this code comparing strings; canonicalisation is for the message, never the decision. `guards:` is an expression so a program can point at the resolver that already owns its path — guards: engram_resolve_data_dir() — instead of restating that resolver's default, which is the two-owners defect spec 18.4 exists to prevent. A `singleton:` without `guards:` is now a compile error; emitting a name-keyed lock instead would be emitting the defect. Kept: the flock (the kernel drops it on crash and SIGKILL, so there is still no "delete the lock file to get unstuck" ritual — a stale file inside a copied data dir is inert), and the holder's pid in the message. Changed: the message is true. It says "the same state" because the lock it failed to take is in that state, and it names the state it checked. An unguardable state (missing, read-only) now refuses rather than starting unguarded. Also corrects lang/AGENTS.md's compiler rebuild line, which had gone stale: linking el_runtime.c alone no longer resolves.
204 lines
12 KiB
Markdown
204 lines
12 KiB
Markdown
# El Language — Agent Guide
|
|
|
|
El is a self-hosting, statically-typed language that compiles to C. This file orients agents that work on El itself or on programs written in El.
|
|
|
|
---
|
|
|
|
## Current work in this worktree — the API reshape / decorated seam (IN PROGRESS, 2026-08-14)
|
|
|
|
This is the `api-reshape` worktree. The build here reshapes Neuron's external
|
|
surface and how it is *declared* — proven on isolated dev-port clones only; **live
|
|
prod engram `:8742` is untouched and nothing is promoted.** Full framing lives in
|
|
`neuron/docs/architecture/06-cognitive-architecture.md` (Update — 2026-08-14 deep
|
|
night) and `02-components.md §5`.
|
|
|
|
- **Surface collapse.** The ~90 noun-organized CRUD MCP tools collapse to a few
|
|
**geometry ops** — `read` (the *vantage-read*: re-origin + salience/recency +
|
|
an **aperture** → a bounded slice, curing the whole-self dump), `write`,
|
|
`relate`, `supersede` (evolve/tombstone/promote, never a hard delete) — plus the
|
|
agentic primitives `think`/`attend`/`learn`/`ground`/`assert`. The old noun is a
|
|
`type` parameter. Implemented in `tools/api-reshape/surface.el` with a parity
|
|
harness (`parity.sh`); aperture proven to bound output. ~~**Not yet:** compiled
|
|
into the MCP server~~ — **shipped (verified 2026-08-16): the live MCP surface is
|
|
exactly these nine ops** (`read` · `write` · `relate` · `supersede` · `think` ·
|
|
`attend` · `assert` · `ground` · `learn`); the ~87-tool surface is gone.
|
|
`attend` absorbed `getInstructions` / `beginSession`'s active-context sweep /
|
|
`checkEvents` — those are **gone, not gapped**. Still outstanding: hot-swap,
|
|
all-alias dispatch.
|
|
|
|
> **⚠ Two of those primitives are the wrong shape, and it is documented
|
|
> (2026-08-16).** `think({seeds, faculty})` treats **faculties as parameters**;
|
|
> they are **operations** — `reason` changes the estimate (a read), `induce`
|
|
> changes the parameters, `abduce` changes the *structure* (a write
|
|
> `GeoGradient` cannot express). And `ground` mints a `grounded-by` edge, but
|
|
> **grounding is not a subsystem — it IS the edge weight**: a property *of* a
|
|
> relation, not a relation *between* nodes. Authority:
|
|
> `lang/spec/correspondence-and-censorship.md`. Do not re-derive it; if you think
|
|
> a section is wrong, say so with a measurement.
|
|
- **Decorated seam.** `@route(path,method,…)` makes codegen synthesize
|
|
`el_route_dispatch` (replacing the hand-written `handle_request` if-else) —
|
|
proven decorate→serve on `:8951`. `@manager`/`@engine`/`@accessor` are **parsed
|
|
but structurally inert** in the shipped compiler today; the `@route` codegen
|
|
lives on the **unmerged branch `feat/el-route-decorators`**. Telemetry-emit and
|
|
dharma-bus auto-wiring at the boundary are **staged, not shipped**. In-process,
|
|
an `@accessor` reaches the engram via **`engram_*` builtins**, not `http_get`.
|
|
|
|
**Do not edit** the protected build sources while this is in flight:
|
|
`el-compiler/src/codegen.el`, `el-compiler/runtime/el_seed.c` (and the archived
|
|
`legacy/el_runtime.c`), the `runtime/engram_*.c` boot files, and `surface.el`
|
|
(when present in the reshape tree) — these are owned by the build agents.
|
|
|
|
---
|
|
|
|
## What El Is
|
|
|
|
El compiles `.el` source → C → native binary. Every El value is `el_val_t` (int64_t). Strings are heap pointers cast through int64_t. The compiler is written in El (self-hosting).
|
|
|
|
**The compiler pipeline:**
|
|
```
|
|
elc-cli.el
|
|
└─ imports: compiler.el
|
|
└─ imports: lexer.el, parser.el, codegen.el, codegen-js.el
|
|
```
|
|
|
|
The canonical compiler binary is `dist/platform/elc`. It was produced by running an earlier version of itself on `elc-cli.el`.
|
|
|
|
---
|
|
|
|
## The Two Layers — Know Which One You're In
|
|
|
|
### Layer 1: El programs (`.el` files)
|
|
|
|
This is where almost all work belongs. El programs are source files that get compiled by `elc`. New library functions, application logic, and language-level utilities all go here as `.el` files.
|
|
|
|
**Do not add C code when El can express it.** If functionality can be built from existing El primitives (string ops, `exec`, `fs_read/write`, `http_post`, etc.), write it in El.
|
|
|
|
### Layer 2: The C seed (`runtime/el_seed.c`)
|
|
|
|
This is the self-contained C OS-boundary layer. It provides the `__`-prefixed primitives that compiled El programs call: libcurl HTTP, pthreads, filesystem I/O, arena allocation, etc. It is **not generated** — it is maintained by hand.
|
|
|
|
The runtime is native El (`runtime/*.el`) over a C OS-boundary. **Status (verified 2026-08-15):** the migration to a seed-only boundary is *in progress, not done*. Two files exist:
|
|
- `runtime/el_runtime.c` (~860 KB) — **LIVE**. Holds the engram store (`EngramStore engram_global`) plus the `http_*`/`json_*`/`state_*`/`engram_*` impls. It is the authoritative single-file link target for the compiler, and `tools/install.sh` compiles it into `libel.a`. This is where a new C builtin's *implementation* must currently live to be linkable.
|
|
- `runtime/el_seed.c` — the intended hand-maintained `__`-prefixed seed (thin wrappers over the above). It is compiled alongside `el_runtime.c` by `tools/install.sh`, but does **not** compile standalone yet (see the build-path caveat under "Rebuilding the Compiler").
|
|
|
|
**Only edit these when you genuinely need OS-level access** (raw sockets, GPU calls, new libcurl features, a new engram store op). For everything else, write El.
|
|
|
|
When you add a C builtin (verbatim-emit recipe — the El name is emitted as the exact C symbol; `builtin_arity` is an arity guard only, not a dispatch table):
|
|
1. Implement the C function in `el_runtime.c` (and declare it in `el_runtime.h`).
|
|
2. Add a `__`-prefixed thin wrapper in `el_seed.c` and declare it in `el_seed.h`.
|
|
3. Add the name to `builtin_arity` in `el-compiler/src/codegen.el` — add **both** the plain and `__`-prefixed spellings.
|
|
4. Rebuild the elc binary (see below) and confirm the self-host fixpoint is byte-identical.
|
|
5. **Prove it with a NEGATIVE CONTROL.** Show the test FAILING on a build without your change, then passing with it. A test that has never been seen to fail has proven nothing.
|
|
|
|
> **Step 5 is not optional, and step 4 does not cover it.** The fixpoint proves the *compiler reproduces itself*. It says nothing whatsoever about whether your builtin works. A recipe ending at "byte-identical" reads as complete while having verified nothing about the thing just added — which is why this file, until 2026-08-16, produced builtins with no tests at all.
|
|
>
|
|
> Measured cost of the omission (2026-08-16): `engram_node_set_emb`, `engram_curiosity_json` and `dream_set_handler` were all added in one session with zero tests. Separately, a UTF-8 fix was written, tested, and **the test passed on the unpatched build too** — the defect was elsewhere entirely, and only building the pre-fix binary exposed it. Without a negative control that fix would have merged as verified.
|
|
>
|
|
> Two shapes that pass while proving nothing, both hit the same day:
|
|
> - A test that never exercises your change (the route supplied a default that bypassed the code under test).
|
|
> - An induction that loses a race. `curl --max-time` on a large response left *both* builds alive; only `SO_LINGER 0` — a genuine RST, so the peer is provably gone — reproduced the failure. Six of ten attempts is not a control.
|
|
>
|
|
> Before every probe, confirm **your** process bound the port (`lsof -nP -iTCP:<port>`, match the PID). A stale instance answering on the port has silently produced false results here more than once, and `pkill -f` does not reliably match an argv like `./engram`.
|
|
|
|
Worked example: the `engram_assert_json` (op_assert seam) and `engram_node_full_in`/`engram_connect_in` (purview write-side) primitives added 2026-08-15 follow exactly this recipe.
|
|
|
|
---
|
|
|
|
## Rebuilding the Compiler
|
|
|
|
After changing any `.el` source in `el-compiler/src/` (run from the `lang/` dir):
|
|
|
|
```bash
|
|
# 1. Stage2: current elc compiles the (modified) compiler to C
|
|
./dist/platform/elc elc-cli.el > elc-new.c
|
|
# 2. Build the new compiler. Link the WHOLE runtime set, not el_runtime.c alone:
|
|
# el_runtime.c calls into engram_store / engram_vindex / eg_cosine_batch and
|
|
# wraps el_seed.c, so a one-file link fails at `ld` with undefined symbols
|
|
# (verified 2026-08-16 — the previous single-file line in this doc is stale).
|
|
cc -std=c11 -O2 -I runtime -I$(brew --prefix openssl@3)/include \
|
|
-L$(brew --prefix openssl@3)/lib \
|
|
-o dist/platform/elc-new \
|
|
elc-new.c runtime/el_runtime.c runtime/el_seed.c \
|
|
runtime/engram_cognition.c runtime/engram_geometry.c runtime/engram_reason.c \
|
|
runtime/engram_store.c runtime/engram_verify.c runtime/engram_vindex.c \
|
|
runtime/eg_cosine_batch.c runtime/eg_cosine_batch_strategy_cpu.c \
|
|
-lcurl -lssl -lcrypto -lpthread -lm
|
|
# 3. Verify self-hosting FIXPOINT (stage3 == stage2 output, byte-identical):
|
|
./dist/platform/elc-new elc-cli.el > elc-verify.c
|
|
diff elc-new.c elc-verify.c # must be identical
|
|
mv dist/platform/elc-new dist/platform/elc
|
|
```
|
|
|
|
> **Build-path caveat (verified 2026-08-15).** `el_seed.c` is the intended hand-maintained OS-boundary seed, but it does **not** compile standalone under modern clang: it wraps ~16 unprefixed `el_runtime.c` symbols (`http_serve`, `json_*`, `state_*`, `http_response`) without prototypes, and clang treats implicit declarations as errors (C99+). The productionised install (`tools/install.sh`) builds `libel.a` from **both** `el_seed.o` + `el_runtime.o` together, which is why linking succeeds there. To make `el_seed.c` build on its own, add prototypes for those symbols (or `#include "el_runtime.h"`, reconciling the `__http_serve` return-type mismatch first). Until then, `el_runtime.c` is the authoritative single-file link target for the compiler.
|
|
|
|
After changing `el_seed.c` only (no El source changes), rebuild downstream programs but do NOT need to rebuild the compiler binary itself — the seed is linked at the application level, not the compiler level.
|
|
|
|
---
|
|
|
|
## How El Programs Are Built
|
|
|
|
Each El application has a `build.sh` that:
|
|
1. Concatenates all `.el` source files (stripping `import` lines)
|
|
2. Runs `elc` to produce a `.c` file
|
|
3. Runs `cc` linking against `el_seed.c`
|
|
|
|
Example (cgi-studio daemon):
|
|
```bash
|
|
cd products/cgi-studio/el-daemon
|
|
./build.sh
|
|
```
|
|
|
|
When you add a new `.el` file to an application, add it to that application's `build.sh` concat list.
|
|
|
|
---
|
|
|
|
## Parallelism in El
|
|
|
|
El is single-threaded at the application level. Parallelism is achieved through subprocess fan-out:
|
|
|
|
```el
|
|
// Pattern: write payloads to temp files, exec bash script with & and wait,
|
|
// read results back from temp files.
|
|
fn http_post_parallel(urls: [String], bodies: [String]) -> [String] {
|
|
// ... bash fan-out via exec() ...
|
|
}
|
|
```
|
|
|
|
Use `exec()` (blocking) or `exec_bg()` (fire-and-forget) with shell scripts to run concurrent work. There is no goroutine or async/await — parallelism goes through the OS process layer.
|
|
|
|
---
|
|
|
|
## Key Files
|
|
|
|
| Path | What it is |
|
|
|------|-----------|
|
|
| `dist/platform/elc` | Canonical compiler binary (arm64 Mac) |
|
|
| `el-compiler/src/codegen.el` | Code generator — builtin arity table lives here |
|
|
| `el-compiler/src/lexer.el` | Lexer |
|
|
| `el-compiler/src/parser.el` | Parser |
|
|
| `runtime/el_seed.c` | Self-contained C OS-boundary layer (replaces el_runtime.c) |
|
|
| `runtime/el_seed.h` | Seed header (C function declarations) |
|
|
| `spec/language.md` | Language specification |
|
|
| `BOOTSTRAP.md` | How to recover the compiler from scratch |
|
|
| `elc-cli.el` | Compiler entry point |
|
|
| `elc-combined.el` | Pre-merged single-file compiler (used during early bootstrap) |
|
|
|
|
---
|
|
|
|
## HTTP Timeout
|
|
|
|
The El HTTP client (libcurl) defaults to **60 seconds**. Override per-process via `EL_HTTP_TIMEOUT_MS` env var. Set it before spawning any subprocess that makes long API calls:
|
|
|
|
```el
|
|
exec("EL_HTTP_TIMEOUT_MS=300000 " + SOME_BIN + " " + args + " 2>&1")
|
|
```
|
|
|
|
---
|
|
|
|
## Rules
|
|
|
|
- New library functions → write in El
|
|
- New OS/hardware primitives → write in C and register in `codegen.el` arity table
|
|
- Never edit `dist/platform/elc` directly — always rebuild from source
|
|
- Never modify `el_seed.c` to add functionality that El can express
|