will.anderson
4bff40fa4a
High-fanout identity anchors (voice, writing-imprint, self-root) have ~670KB
neighborhoods. inspect_graph returned the full traversal, which overflowed the
MCP client's context and socket-closed the wrapper mid self-load -- the soul
could not traverse its own identity graph.
handle_api_inspect_graph gains an opt-in `compact` projection (compact=1|true):
the neighborhood is relevance-ranked, the top K (default 12) keep a UTF-8-safe
content snippet (default snip=600), and the remainder collapse to lightweight
{id,label,node_type,tier,edge,pointer:true} stubs. This bounds the voice node
from 669,799B -> 25,353B (HTTP 200, valid JSON) and the wrapper's soul-load no
longer socket-closes. New helpers: api_compact_neighbors, api_neigh_full,
api_neigh_pointer, api_neigh_rank, api_neigh_better, api_float_or.
The flag is gated: ABSENT it, the response is byte-identical to the old plain
traversal, so the studio app (which never sends it) is unaffected. The MCP
wrapper (mcp-wrapper/src/main.el) appends &compact=1 on its inspectGraph and
fetch-by-id paths.
dist/soul.c is REGENERATED so CI ships the fix: CI compiles the committed
single-TU dist/soul.c directly (running elb/elc on the Linux runner OOM-kills
it), so an .el-only change would build the OLD behavior. Regenerated and verified
on macOS -- compiles with the CI cc line (0 errors) and, on a throwaway soul over
a copy of the live snapshot, serves compact ~25KB / non-compact ~670KB. The regen
also syncs the amalgamation to this branch's .el sources, which had drifted
several self-review commits ahead of the previously-committed soul.c.
Docs: docs/architecture/00-05 added; 01/02/05 corrected so the relevance-ranked
inspect_graph projection reads as committed source, not an in-flight concern.
166 lines
8.9 KiB
Markdown
166 lines
8.9 KiB
Markdown
# Neuron — El & the Build Pipeline
|
|
|
|
> The soul and the engram are written in **El**, a self-hosted language that
|
|
> compiles to C11. This document covers the language layer, the
|
|
> amalgamation → `soul.c` → binary pipeline, how the soul is composed from its
|
|
> layers, and the compile-time capability gates. Sources: `manifest.el`,
|
|
> `soul.el`, `dist/soul.c`, the El toolchain under `foundation/el/`
|
|
> (`elc.c`, `elb.el`, `BOOTSTRAP.md`), and `.gitea/workflows/`.
|
|
|
|
## The El language layer
|
|
|
|
El is a compiled, Lisp-family language transpiled to C11. Every El program links
|
|
a shared runtime, `el_runtime.c` / `el_runtime.h`, which implements **all
|
|
builtins**: the engram graph engine (`engram_*`), HTTP (`http_*`), JSON
|
|
(`json_*`), crypto, time, LLM calls, and DHARMA primitives (`el_runtime.h`,
|
|
`BOOTSTRAP.md:599-644`). The runtime also provides an arena allocator (server
|
|
mode) and ARC refcounting. Practically: **the runtime is both the standard
|
|
library and the database** — the graph physically lives in `el_runtime.c`, and El
|
|
source files are the orchestration/logic on top.
|
|
|
|
A recurring texture in the source is workaround comments for codegen quirks
|
|
(e.g. broken `%`/`*` operators). These are El-compiler maturity issues, not
|
|
architecture — but they explain some of the hand-rolled arithmetic in
|
|
`awareness.el`/`memory.el`.
|
|
|
|
## The toolchain: `elc`, `elb`, `el_runtime`
|
|
|
|
| Tool | What it is | Role |
|
|
|---|---|---|
|
|
| `elc` | the El compiler, **self-hosted** (written in El) | compiles one El translation unit → C11. Import resolution is textual, depth-first, dedup'd — it inlines all imports into one string and emits forward decls for every fn (`BOOTSTRAP.md:927-936`). |
|
|
| `elb` | the build coordinator (`elb.el`, ~367 lines) | reads `manifest.el`, walks the import graph, does **incremental** separate compilation using `.elh` header files (`extern fn` decls), links the final binary (".NET-style incremental build", `BOOTSTRAP.md:886, 916-925`). |
|
|
| `el_runtime.c/.h` | the C runtime | linked by every compiled El binary; implements all builtins and the graph engine. |
|
|
|
|
The `.elh` files present in this repo (`soul.elh`, `memory.elh`,
|
|
`neuron-api.elh`, `routes.elh`, …) are **auto-generated headers** (`elc
|
|
--emit-header`) — the `extern fn` interface each module exports. They are the
|
|
contract surface `elb` uses for incremental builds, and they double as a concise
|
|
map of each module's public functions.
|
|
|
|
### Self-hosting fixed point
|
|
|
|
`elc` is bootstrapped from a seed binary (`dist/platform/elc`, Mach-O arm64) and
|
|
verified by a **fixed-point self-recompile**: the compiler must compile its own
|
|
source to a byte-identical binary (`BOOTSTRAP.md:7-58, 801-816`). Pipeline:
|
|
`elc-cli.el → compiler.el → lexer/parser/codegen.el`.
|
|
|
|
## Building the soul: `.el → elc → .c → cc → binary`
|
|
|
|
The concrete pipeline (mirrored in the engram build, `engram/src/server.el:8-11`):
|
|
|
|
```
|
|
soul.el (+ imports)
|
|
│ elc (self-hosted El→C11, inlines imports)
|
|
▼
|
|
dist/soul.c (~31,300 lines — single amalgamated translation unit)
|
|
│ cc -std=c11 -O2 soul.c el_runtime.c
|
|
▼
|
|
dist/neuron (native binary)
|
|
```
|
|
|
|
### Why `dist/soul.c` is committed
|
|
|
|
`dist/soul.c` is the authoritative combined translation unit, **regenerated on
|
|
macOS by running `elb`**. It is checked into the repo on purpose: CI compiles it
|
|
**directly** and skips `elb` entirely (`ci.yaml`). The reason is operational, not
|
|
aesthetic —
|
|
|
|
- `elb` succeeds on arm64/macOS `ld`, but **fails on Linux** (duplicate strong
|
|
symbols), and
|
|
- `elc` uses 24GB+ virtual memory, which **OOM-kills the 16GB CI runner**.
|
|
|
|
So the pattern is: **compile on the Mac, commit the amalgamation, and let Linux
|
|
CI do only the cheap `cc` step.** `dist/` also holds the per-module `.c` outputs
|
|
(`memory.c`, `awareness.c`, `chat.c`, the NLG morphology tables, …) —
|
|
intermediate artifacts of the same process.
|
|
|
|
> **Mechanism note (observed during the self-load regen).** The single-TU
|
|
> `dist/soul.c` is produced by running `elc` over the **flattened import set** —
|
|
> every module source in `soul.el`'s transitive import graph, concatenated with
|
|
> `import` lines stripped, compiled in one pass (`elc` hoists forward decls for
|
|
> all functions, so concat order doesn't affect correctness). `elb` on its own
|
|
> emits **per-module `.c` + a linked binary**, not the combined `soul.c`; it is
|
|
> the separate-compilation coordinator, and `elc soul.el` alone yields only the
|
|
> soul module. Because the amalgamation is regenerated only on demand, it can lag
|
|
> the `.el` sources: this PR regenerated it after it had fallen behind several
|
|
> source commits, and folded in the `inspect_graph` relevance-ranked projection
|
|
> (the `compact=1` self-load fix, doc 02) so CI ships it.
|
|
|
|
## How the soul is composed (layer stack)
|
|
|
|
`manifest.el` declares the build:
|
|
|
|
```
|
|
package "neuron" { version "0.1.0" edition "2026" }
|
|
build { entry "soul.el" }
|
|
```
|
|
|
|
The comment in `manifest.el:8-16` documents the intended **layer composition
|
|
order**: a base layer `../foundation/nlg` (the NLG engine — 31-language
|
|
morphology, grammar, realizer, semantics) with the **soul layer** (`soul.el`)
|
|
injected on top. New layers are added by importing them in `soul.el` before the
|
|
soul's own code. *(The `../foundation/nlg` path is the manifest's stated NLG base;
|
|
the NLG sources compile into the `dist/*.c` morphology/grammar tables seen in the
|
|
tree.)*
|
|
|
|
`soul.el` itself imports, in order (`soul.el:1-10`): `elp.el`, `memory.el`,
|
|
`safety.el`, `stewardship.el`, `imprint.el`, `awareness.el`, `chat.el`,
|
|
`studio.el`, `elp-input.el`, `routes.el` — then declares the `cgi "neuron-soul"`
|
|
identity block (`:12-17`): `dharma_id`, `principal`, `network`, and
|
|
`engram: http://localhost:8742`. Because `elc` inlines imports depth-first, this
|
|
import list *is* the amalgamation order that produces `dist/soul.c`.
|
|
|
|
The `cgi` block is not just metadata — it sets the program's **capability tier**
|
|
(next section).
|
|
|
|
## Compile-time capability gates
|
|
|
|
El's codegen classifies each program by its top-level declaration and **enforces
|
|
capabilities at compile time** (`BOOTSTRAP.md:958-965`):
|
|
|
|
| Declaration | Tier | Allowed |
|
|
|---|---|---|
|
|
| `cgi { … }` | full | everything — `llm_call_agentic`, `llm_register_tool`, `dharma_emit`, `dharma_field`, LLM, DHARMA |
|
|
| `service { … }` | restricted | no `llm_call_agentic` / `llm_register_tool` / `dharma_emit` / `dharma_field` |
|
|
| neither | utility | no DHARMA, no LLM |
|
|
|
|
A program that calls a capability its tier forbids **fails to compile**: codegen
|
|
emits a C `#error` naming the forbidding call, so the downstream `cc` aborts.
|
|
This is the **primary hard gate** in the build — capability escalation is caught
|
|
by the compiler, not at runtime. The soul is a `cgi`, so it gets the full tier;
|
|
`engram` is declared without `cgi`/`service` semantics that would grant LLM
|
|
access (it is a store).
|
|
|
|
## Verification gates
|
|
|
|
| Gate | Where | What it checks |
|
|
|---|---|---|
|
|
| Capability tier | El codegen (`BOOTSTRAP.md:958`) | no capability escalation; hard `#error` at compile |
|
|
| Self-hosting fixed point | `elc` bootstrap (`BOOTSTRAP.md:801`) | compiler reproduces itself byte-identically |
|
|
| `test_soul_guard.el` | `tests/` | the genesis `safe_to_seed` boot guard — a sparse/oversized snapshot must not clobber the graph |
|
|
| `test_layer_contract.el` | `tests/` | JSON interface shapes between composition-stack layers that `layered_cycle` depends on (e.g. `safety_screen` always returns an `action` field) |
|
|
| other `tests/*.el` | `tests/` | `test_sessions.el`, `test_safety.el`, `test_bell_safety.el`, `test_layered_cycle.el`, `test_imprint.el`, `test_stewardship.el`, `test_api_define_process.el`, … |
|
|
| CI smoke test | `ci.yaml` | `dist/neuron --help` runs |
|
|
|
|
> **Flag (unverified/TODO).** CI (`ci.yaml`) runs only the `cc` compile + the
|
|
> `dist/neuron --help` smoke test — it does **not** invoke the `tests/*.el`
|
|
> soul-guard / layer-contract suites, and `.githooks/` is empty. Whether these
|
|
> tests are gated anywhere (a pre-merge hook, a separate workflow, or manual
|
|
> discipline on the Mac before regenerating `soul.c`) is **not evident in the
|
|
> files read**. This is the most important build-integrity gap to confirm with a
|
|
> human: the contract tests exist but their enforcement point is unproven.
|
|
|
|
## Practical consequences for a contributor
|
|
|
|
- **You cannot rebuild the whole soul on Linux/CI.** Regenerate `dist/soul.c` on
|
|
a Mac (`elb`), commit it, then CI compiles it. Changing an `.el` file without
|
|
regenerating `soul.c` ships nothing.
|
|
- **The `.elh` files are your API map.** To see what a module exposes, read its
|
|
`.elh` — it's the generated `extern fn` list.
|
|
- **Memory/activation behavior often can't be changed from this repo.** The
|
|
volatile numeric core is in `foundation/el` `el_runtime.c`. Doc 01, Divergence
|
|
6 explains why this is the sharpest edge in the architecture.
|
|
- **The engram is a separate repo.** It is cloned and compiled by CI
|
|
(`Dockerfile`, `.gitea/workflows/`), not vendored here. Its source of truth is
|
|
`foundation/el/engram`.
|