220 lines
15 KiB
Markdown
220 lines
15 KiB
Markdown
# El Language — Agent Guide
|
|
|
|
El is a self-hosting, statically-typed language that compiles to C. This file orients agents that work on El itself or on programs written in El.
|
|
|
|
---
|
|
|
|
## Current work in this worktree — the API reshape / decorated seam (IN PROGRESS, 2026-08-14)
|
|
|
|
This is the `api-reshape` worktree. The build here reshapes Neuron's external
|
|
surface and how it is *declared* — proven on isolated dev-port clones only; **live
|
|
prod engram `:8742` is untouched and nothing is promoted.** Full framing lives in
|
|
`neuron/docs/architecture/06-cognitive-architecture.md` (Update — 2026-08-14 deep
|
|
night) and `02-components.md §5`.
|
|
|
|
- **Surface collapse.** The ~90 noun-organized CRUD MCP tools collapse to a few
|
|
**geometry ops** — `read` (the *vantage-read*: re-origin + salience/recency +
|
|
an **aperture** → a bounded slice, curing the whole-self dump), `write`,
|
|
`relate`, `supersede` (evolve/tombstone/promote, never a hard delete) — plus the
|
|
agentic primitives `think`/`attend`/`learn`/`ground`/`assert`. The old noun is a
|
|
`type` parameter. Implemented in `tools/api-reshape/surface.el` with a parity
|
|
harness (`parity.sh`); aperture proven to bound output. ~~**Not yet:** compiled
|
|
into the MCP server~~ — **shipped (verified 2026-08-16): the live MCP surface is
|
|
exactly these nine ops** (`read` · `write` · `relate` · `supersede` · `think` ·
|
|
`attend` · `assert` · `ground` · `learn`); the ~87-tool surface is gone.
|
|
`attend` absorbed `getInstructions` / `beginSession`'s active-context sweep /
|
|
`checkEvents` — those are **gone, not gapped**. Still outstanding: hot-swap,
|
|
all-alias dispatch.
|
|
|
|
> **⚠ Two of those primitives are the wrong shape, and it is documented
|
|
> (2026-08-16).** `think({seeds, faculty})` treats **faculties as parameters**;
|
|
> they are **operations** — `reason` changes the estimate (a read), `induce`
|
|
> changes the parameters, `abduce` changes the *structure* (a write
|
|
> `GeoGradient` cannot express). And `ground` mints a `grounded-by` edge, but
|
|
> **grounding is not a subsystem — it IS the edge weight**: a property *of* a
|
|
> relation, not a relation *between* nodes. Authority:
|
|
> `lang/spec/correspondence-and-censorship.md`. Do not re-derive it; if you think
|
|
> a section is wrong, say so with a measurement.
|
|
- **Decorated seam.** `@route(path,method,…)` makes codegen synthesize
|
|
`el_route_dispatch` (replacing the hand-written `handle_request` if-else) —
|
|
proven decorate→serve on `:8951`. `@manager`/`@engine`/`@accessor` are **parsed
|
|
but structurally inert** in the shipped compiler today; the `@route` codegen
|
|
lives on the **unmerged branch `feat/el-route-decorators`**. Telemetry-emit and
|
|
dharma-bus auto-wiring at the boundary are **staged, not shipped**. In-process,
|
|
an `@accessor` reaches the engram via **`engram_*` builtins**, not `http_get`.
|
|
|
|
**Do not edit** the protected build sources while this is in flight:
|
|
`el-compiler/src/codegen.el`, `el-compiler/runtime/el_seed.c` (and the archived
|
|
`legacy/el_runtime.c`), the `runtime/engram_*.c` boot files, and `surface.el`
|
|
(when present in the reshape tree) — these are owned by the build agents.
|
|
|
|
---
|
|
|
|
## What El Is
|
|
|
|
El compiles `.el` source → C → native binary. Every El value is `el_val_t` (int64_t). Strings are heap pointers cast through int64_t. The compiler is written in El (self-hosting).
|
|
|
|
**The compiler pipeline:**
|
|
```
|
|
elc-cli.el
|
|
└─ imports: compiler.el
|
|
└─ imports: lexer.el, parser.el, codegen.el, codegen-js.el
|
|
```
|
|
|
|
The canonical compiler binary is `dist/platform/elc`. It was produced by running an earlier version of itself on `elc-cli.el`.
|
|
|
|
---
|
|
|
|
## The Two Layers — Know Which One You're In
|
|
|
|
### Layer 1: El programs (`.el` files)
|
|
|
|
This is where almost all work belongs. El programs are source files that get compiled by `elc`. New library functions, application logic, and language-level utilities all go here as `.el` files.
|
|
|
|
**Do not add C code when El can express it.** If functionality can be built from existing El primitives (string ops, `exec`, `fs_read/write`, `http_post`, etc.), write it in El.
|
|
|
|
### Layer 2: The C seed (`runtime/el_seed.c`)
|
|
|
|
This is the self-contained C OS-boundary layer. It provides the `__`-prefixed primitives that compiled El programs call: libcurl HTTP, pthreads, filesystem I/O, arena allocation, etc. It is **not generated** — it is maintained by hand.
|
|
|
|
The runtime is native El (`runtime/*.el`) over a C OS-boundary. **Status (verified 2026-08-16):** the migration to a seed-only boundary is *in progress, not done*.
|
|
|
|
**The runtime is MULTI-FILE. There is no single-file link target and there has not been one for months.** The canonical link set is listed once, in **`runtime/SOURCES`**, and printed by `scripts/el-runtime-sources.sh`. It currently holds ten translation units: `el_runtime.c`, `el_seed.c`, the six `engram_*.c` concern files, and `eg_cosine_batch{,_strategy_cpu}.c`.
|
|
|
|
- `runtime/el_runtime.c` (~940 KB, 20.5k lines) — **LIVE, and oversized.** It began life on 2026-05-03 as a temporary build shim: it was deleted that afternoon ("runtime is 100% native El") and restored 25 minutes later, explicitly "UNTIL the compiler is updated to emit `#include el_seed.h`". That `until` never arrived, and in the 3.5 months since, the file doubled. **It is not a volatility unit — it is a dumping ground.** ~47.5% of it is engram code that belongs in the six sibling files that already exist. Do not add to it. See "Where a new C builtin goes" below.
|
|
- `runtime/el_seed.c` — the intended hand-maintained `__`-prefixed seed (thin wrappers over the above).
|
|
- `runtime/engram_{store,vindex,geometry,reason,verify,cognition}.c` — the engram concerns, each with its own header. `el_runtime.c` `#include`s all six headers and makes hard cross-TU calls into all six.
|
|
|
|
> **Linking `el_runtime.c` alone does not work and has not for months.** It fails at `ld` with undefined symbols (`engram_ground_json`, `engram_activate_inner`, `eg_find_relation`, `cog_assert_two_axis`, …). Any recipe, script, or CI step that names `el_runtime.c` by itself is stale — replace it with `$(scripts/el-runtime-sources.sh lang/runtime)`.
|
|
|
|
**Only edit these when you genuinely need OS-level access** (raw sockets, GPU calls, new libcurl features, a new engram store op). For everything else, write El.
|
|
|
|
#### Where a new C builtin goes
|
|
|
|
**Put it in the `.c` that owns the concern — NOT in `el_runtime.c`.**
|
|
|
|
*Placement is a link-time concern. The compiler cannot tell which `.c` a symbol came from, and never could.* `builtin_arity` in `el-compiler/src/codegen.el` maps NAME → ARITY INT and nothing else (~413 entries); the El name is emitted as the exact C symbol and resolved by `ld`. Proof, if you want it: `nm lang/dist/platform/elc` on the *shipped* compiler shows `T _engram_geo_reify_index_new` (defined in `engram_geometry.c`), `T _vindex_insert` (`engram_vindex.c`), `T _engram_think` (`engram_cognition.c`), `T _engram_reason_abduce` (`engram_reason.c`). **The shipped compiler is already linked from ten translation units.** A builtin defined in a sibling `.c` is exactly as linkable as one defined in `el_runtime.c`.
|
|
|
|
Choose the file by concern: engram store ops → `engram_store.c`; index → `engram_vindex.c`; geometry/priming → `engram_geometry.c`; reasoning → `engram_reason.c`; grounding/consistency → `engram_verify.c`; think/stance → `engram_cognition.c`. **If no existing file owns it, create one** — add the `.c` to `runtime/SOURCES` (one line) and every build path picks it up. For a builtin that belongs to a downstream program rather than the runtime, declare `c_source "path/to/file.c"` in that program's `manifest.el`; `elb` already links it (`parse_manifest_c_sources`, `lang/elb.el:82`).
|
|
|
|
When you add a C builtin (verbatim-emit recipe — the El name is emitted as the exact C symbol; `builtin_arity` is an arity guard only, not a dispatch table):
|
|
1. Implement the C function in the **concern-owning `.c`** (and declare it in that file's `.h`). Add the file to `runtime/SOURCES` if it is new. Only put it in `el_runtime.c` if it is genuinely EL core (val/str/map/list/arena) — that is ~8% of what is in there today.
|
|
2. Add a `__`-prefixed thin wrapper in `el_seed.c` and declare it in `el_seed.h`.
|
|
3. Add the name to `builtin_arity` in `el-compiler/src/codegen.el` — add **both** the plain and `__`-prefixed spellings.
|
|
4. Rebuild the elc binary (see below) and confirm the self-host fixpoint is byte-identical.
|
|
5. **Prove it with a NEGATIVE CONTROL.** Show the test FAILING on a build without your change, then passing with it. A test that has never been seen to fail has proven nothing.
|
|
|
|
> **Step 5 is not optional, and step 4 does not cover it.** The fixpoint proves the *compiler reproduces itself*. It says nothing whatsoever about whether your builtin works. A recipe ending at "byte-identical" reads as complete while having verified nothing about the thing just added — which is why this file, until 2026-08-16, produced builtins with no tests at all.
|
|
>
|
|
> Measured cost of the omission (2026-08-16): `engram_node_set_emb`, `engram_curiosity_json` and `dream_set_handler` were all added in one session with zero tests. Separately, a UTF-8 fix was written, tested, and **the test passed on the unpatched build too** — the defect was elsewhere entirely, and only building the pre-fix binary exposed it. Without a negative control that fix would have merged as verified.
|
|
>
|
|
> Two shapes that pass while proving nothing, both hit the same day:
|
|
> - A test that never exercises your change (the route supplied a default that bypassed the code under test).
|
|
> - An induction that loses a race. `curl --max-time` on a large response left *both* builds alive; only `SO_LINGER 0` — a genuine RST, so the peer is provably gone — reproduced the failure. Six of ten attempts is not a control.
|
|
>
|
|
> Before every probe, confirm **your** process bound the port (`lsof -nP -iTCP:<port>`, match the PID). A stale instance answering on the port has silently produced false results here more than once, and `pkill -f` does not reliably match an argv like `./engram`.
|
|
|
|
Worked example: the `engram_assert_json` (op_assert seam) and `engram_node_full_in`/`engram_connect_in` (purview write-side) primitives added 2026-08-15 follow exactly this recipe.
|
|
|
|
---
|
|
|
|
## Rebuilding the Compiler
|
|
|
|
After changing any `.el` source in `el-compiler/src/` (run from the `lang/` dir):
|
|
|
|
```bash
|
|
# 1. Stage2: current elc compiles the (modified) compiler to C
|
|
./dist/platform/elc elc-cli.el > elc-new.c
|
|
# 2. Build the new compiler. Link the WHOLE runtime set, not el_runtime.c alone:
|
|
# el_runtime.c calls into engram_store / engram_vindex / eg_cosine_batch and
|
|
# wraps el_seed.c, so a one-file link fails at `ld` with undefined symbols
|
|
# (verified 2026-08-16 — the previous single-file line in this doc is stale).
|
|
cc -std=c11 -O2 -I runtime -I$(brew --prefix openssl@3)/include \
|
|
-L$(brew --prefix openssl@3)/lib \
|
|
-o dist/platform/elc-new \
|
|
elc-new.c runtime/el_runtime.c runtime/el_seed.c \
|
|
runtime/engram_cognition.c runtime/engram_geometry.c runtime/engram_reason.c \
|
|
runtime/engram_store.c runtime/engram_verify.c runtime/engram_vindex.c \
|
|
runtime/eg_cosine_batch.c runtime/eg_cosine_batch_strategy_cpu.c \
|
|
-lcurl -lssl -lcrypto -lpthread -lm
|
|
# 3. Verify self-hosting FIXPOINT (stage3 == stage2 output, byte-identical):
|
|
./dist/platform/elc-new elc-cli.el > elc-verify.c
|
|
diff elc-new.c elc-verify.c # must be identical
|
|
mv dist/platform/elc-new dist/platform/elc
|
|
```
|
|
|
|
> **Build-path caveat (verified 2026-08-15).** `el_seed.c` is the intended hand-maintained OS-boundary seed, but it does **not** compile standalone under modern clang: it wraps ~16 unprefixed `el_runtime.c` symbols (`http_serve`, `json_*`, `state_*`, `http_response`) without prototypes, and clang treats implicit declarations as errors (C99+). The productionised install (`tools/install.sh`) builds `libel.a` from **both** `el_seed.o` + `el_runtime.o` together, which is why linking succeeds there. To make `el_seed.c` build on its own, add prototypes for those symbols (or `#include "el_runtime.h"`, reconciling the `__http_serve` return-type mismatch first).
|
|
>
|
|
> **There is no single-file link target.** *(Corrected 2026-08-16 — this paragraph previously ended "`el_runtime.c` is the authoritative single-file link target for the compiler". Measured: that is false. Linking `elc-new.c` against `runtime/el_runtime.c` alone fails at `ld` with undefined `engram_ground_json`, `engram_activate_inner`, `eg_find_relation`, `cog_assert_two_axis`, and others, because `el_runtime.c` `#include`s six engram headers and calls into all six sibling `.c` files.)* Link the set in `runtime/SOURCES` via `$(../scripts/el-runtime-sources.sh runtime)`.
|
|
|
|
After changing `el_seed.c` only (no El source changes), rebuild downstream programs but do NOT need to rebuild the compiler binary itself — the seed is linked at the application level, not the compiler level.
|
|
|
|
---
|
|
|
|
## How El Programs Are Built
|
|
|
|
Each El application has a `build.sh` that:
|
|
1. Concatenates all `.el` source files (stripping `import` lines)
|
|
2. Runs `elc` to produce a `.c` file
|
|
3. Runs `cc` linking against `el_seed.c`
|
|
|
|
Example (cgi-studio daemon):
|
|
```bash
|
|
cd products/cgi-studio/el-daemon
|
|
./build.sh
|
|
```
|
|
|
|
When you add a new `.el` file to an application, add it to that application's `build.sh` concat list.
|
|
|
|
---
|
|
|
|
## Parallelism in El
|
|
|
|
El is single-threaded at the application level. Parallelism is achieved through subprocess fan-out:
|
|
|
|
```el
|
|
// Pattern: write payloads to temp files, exec bash script with & and wait,
|
|
// read results back from temp files.
|
|
fn http_post_parallel(urls: [String], bodies: [String]) -> [String] {
|
|
// ... bash fan-out via exec() ...
|
|
}
|
|
```
|
|
|
|
Use `exec()` (blocking) or `exec_bg()` (fire-and-forget) with shell scripts to run concurrent work. There is no goroutine or async/await — parallelism goes through the OS process layer.
|
|
|
|
---
|
|
|
|
## Key Files
|
|
|
|
| Path | What it is |
|
|
|------|-----------|
|
|
| `dist/platform/elc` | Canonical compiler binary (arm64 Mac) |
|
|
| `el-compiler/src/codegen.el` | Code generator — builtin arity table lives here |
|
|
| `el-compiler/src/lexer.el` | Lexer |
|
|
| `el-compiler/src/parser.el` | Parser |
|
|
| `runtime/el_seed.c` | Self-contained C OS-boundary layer (replaces el_runtime.c) |
|
|
| `runtime/el_seed.h` | Seed header (C function declarations) |
|
|
| `spec/language.md` | Language specification |
|
|
| `BOOTSTRAP.md` | How to recover the compiler from scratch |
|
|
| `elc-cli.el` | Compiler entry point |
|
|
| `elc-combined.el` | Pre-merged single-file compiler (used during early bootstrap) |
|
|
|
|
---
|
|
|
|
## HTTP Timeout
|
|
|
|
The El HTTP client (libcurl) defaults to **60 seconds**. Override per-process via `EL_HTTP_TIMEOUT_MS` env var. Set it before spawning any subprocess that makes long API calls:
|
|
|
|
```el
|
|
exec("EL_HTTP_TIMEOUT_MS=300000 " + SOME_BIN + " " + args + " 2>&1")
|
|
```
|
|
|
|
---
|
|
|
|
## Rules
|
|
|
|
- New library functions → write in El
|
|
- New OS/hardware primitives → write in C and register in `codegen.el` arity table
|
|
- Never edit `dist/platform/elc` directly — always rebuild from source
|
|
- Never modify `el_seed.c` to add functionality that El can express
|