bigmerge
4abeed6bee
The docs described a mind made of subsystems — a grounding subsystem, a wonder manifest, a dreamer on a beat, faculties as arguments to one call. Each of those is a supervisor invented for something that should be a property of the substrate, and two of the documents carrying them are load-bearing for a build agent: cognitive-architecture.design.md says "a build agent executes from this doc", and tools/api-reshape/README.md marks the refuted shapes PROVEN on a live clone. Corrections carried, per lang/spec/correspondence-and-censorship.md (PR #149) and lang/spec/runtime-ownership.md: - Grounding is not a subsystem — it IS the edge weight. grounded-by as a relation type should not exist; grounding is a property of a relation, not a relation between nodes. Never computed on demand. - Faculties are operations, not parameters. reason changes the estimate, induce changes the parameters, abduce changes the structure — a write, which GeoGradient cannot express. A write is not a parameter of a read. - Wonder is the boundary, not a manifest. Curiosity is wonder crystallized at a nucleation site: one thing at two phases. Removed wonder from the operator table in AGENTS.md. - Consolidation is ambient, not scheduled. A brain has no cron job. The presence of a ticker is the diagnostic. - co_registration is deprecated — it averaged a per-edge property into a region scalar, so opposing sites cancelled. GeoEdge.discord replaces it. Nothing new may read it. - In an immutable substrate, any mechanism that refuses a write is either redundant with immutability or an epistemic constraint misfiled as a protective one. The two design docs are marked superseded-in-part with the refutation at the point each claim is made, not rewritten. Preserving what was argued down is the point of an immutable record. Also measured and corrected while verifying the above: engram/README.md documented a Rust engram-core crate on sled with "flat cosine scan until scale demands HNSW" — there is no Rust in engram/ and HNSW is the index; lang/releases/ no longer exists, so both README.md and AGENTS.md pointed at a deleted path for the authored runtime; language.md listed the engram_* and http_* runtimes as stubs. Added language.md §20 for geometry-as-a-value, realizers and transduce (#144), which had landed with no spec coverage. Documentation only. No .c, .h, or .el file is touched.
184 lines
17 KiB
Markdown
184 lines
17 KiB
Markdown
# El
|
||
|
||
**A self-hosting, statically-typed language that compiles to C — built around a graph-native runtime instead of a database driver.**
|
||
|
||
El is the execution substrate for the Neuron agent runtime, the DHARMA network, and the Engram knowledge graph. This repository is the monorepo for the whole stack: the language itself, the graph memory engine it's built to talk to natively, and the tools (package manager, IDE, UI framework, diagramming) built on top of it.
|
||
|
||
---
|
||
|
||
## Why El exists
|
||
|
||
Every other language treats persistent, associative state as something you reach for through a driver — a SQL client, an ORM, a Redis library bolted on from outside. El inverts that: graph operations (`engram_*`) are runtime primitives, on the same footing as string or list operations. There is no separate database driver because the database is not separate.
|
||
|
||
El has four defining properties:
|
||
|
||
1. **Self-hosting compiler.** The compiler (`lexer.el`, `parser.el`, `codegen.el`, `compiler.el`) is written in El. It compiles El source to C, which `cc` compiles against a fixed runtime into a native binary. A Rust genesis compiler bootstrapped the first iteration; the self-hosted binary at `lang/dist/platform/elc` has been the canonical compiler ever since — every binary in `dist/platform/` was produced by an earlier version of itself compiling `el-compiler/src/`. The chain is auditable: source is the ground truth, not the binary. See [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) for the full recovery path if that binary is ever lost.
|
||
2. **C compilation target.** Every compiled program is plain C11. Every El value is `el_val_t` (`int64_t`); strings are heap pointers cast through it. Functions become C functions; top-level statements become `main()`.
|
||
3. **Graph-native runtime.** The runtime provides first-class graph operations over an in-process Engram store — no separate DB driver, no ORM.
|
||
4. **DHARMA-aware identity.** A `cgi` block declares a program's DHARMA identity at compile time. The runtime resolves identity before user code runs, so `dharma_*` calls have a stable principal and channel surface throughout.
|
||
|
||
---
|
||
|
||
## Architecture map
|
||
|
||
```
|
||
┌─────────────┐
|
||
│ lang │ El compiler + C runtime
|
||
│ (El itself) │ everything below is written in it,
|
||
└──────┬──────┘ or compiles down through it
|
||
│
|
||
┌─────────────┼─────────────┐
|
||
│ │ │
|
||
┌──────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
|
||
│ engram │ │ epm │ │ ide │
|
||
│ graph/mem │ │ package │ │ editor + │
|
||
│ substrate │ │ manager │ │ LSP │
|
||
└──────┬─────┘ └───────────┘ └───────────┘
|
||
│
|
||
┌───────┼────────────────┬─────────────────────┐
|
||
│ │ │ │
|
||
┌─────▼───┐ ┌─▼──────────┐ ┌──▼──────────┐ ┌─────▼──────┐
|
||
│ elp │ │ ql │ │ ui │ │ arbor │
|
||
│ NLG / │ │engram-el. │ |spreading- │ |arbor │
|
||
│ 31 langs│ │studio+tests│ |activation UI│ |diagram lang│
|
||
└─────────┘ └────────────┘ └─────────────┘ └────────────┘
|
||
```
|
||
|
||
`lang` is the foundation — the compiler and C runtime everything else builds on. `engram` is the graph-native memory/state engine that gives El its identity (property 3 above). Everything else is either a tool for working with El (`epm`, `ide`) or a system built on top of Engram's graph model (`elp`, `ql`, `ui`, `arbor`).
|
||
|
||
---
|
||
|
||
## Repository layout
|
||
|
||
### [lang/](lang/) — the El language
|
||
|
||
The compiler and runtime. Self-hosting: `elc-cli.el` → `compiler.el` → `lexer.el` / `parser.el` / `codegen.el` / `codegen-js.el`, textually inlined and compiled in one pass. Compiles to C11 and links against `el-compiler/runtime/el_seed.c`, a hand-maintained OS-boundary layer (libcurl HTTP, pthreads, filesystem, arena allocation) — everything else in the runtime is native El (`runtime/*.el`).
|
||
|
||
Two layers to know: **El programs** (`.el` files — where nearly all work belongs) and **the C seed** (`el_seed.c` — edit only for genuine OS-level access; never re-implement what El can already express).
|
||
|
||
Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented, as are the `program` block with `singleton:` and declared configuration ([§18](lang/spec/language.md)), and **geometry as a first-class value** with El-declarable realizers and `transduce` ([§20](lang/spec/language.md)). In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), and boundary epilogues. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language.
|
||
|
||
**Signal enters as geometry.** Until 2026-08-16 nodes took text and geometry was *derived* from it, which made text the mandatory entry medium: any non-text modality had to be described in prose first, so the geometry being reasoned over was the geometry **of the description, not of the signal**. `Geometry` is now an ordinary El value carrying its own width, and a realizer is an ordinary El function resolved by name through `dlsym` — so admitting a new modality never requires a runtime patch. Worked, self-checking example: [`lang/examples/transduce.el`](lang/examples/transduce.el).
|
||
|
||
Key docs: [AGENTS.md](lang/AGENTS.md) (agent-facing orientation), [BOOTSTRAP.md](lang/BOOTSTRAP.md) (compiler recovery from scratch), [spec/language.md](lang/spec/language.md), [spec/codegen-js.md](lang/spec/codegen-js.md).
|
||
|
||
### [engram/](engram/) — graph intelligence substrate
|
||
|
||
**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. The engine is **C11** (`lang/runtime/engram_{store,geometry,reason,cognition,verify,vindex}.{c,h}`); the server is **El** (`engram/src/server.el`).
|
||
|
||
The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop, gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works. **Activation conducts through well-grounded relations because the weight *is* the groundedness** — nothing filters the traversal; grounded inference falls out of spreading.
|
||
|
||
Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay** — importance × recency-decay × log(activation_count). Forgetting is adaptive pruning, not a bug. Nothing is mutated and nothing is hard-deleted: writes are additive, corrections are supersessions, removals are tombstones — which is what makes supersession an audit trail rather than an edit log.
|
||
|
||
On disk: a paged store (superblock + mirror, slotted 16 KiB pages, self-describing TLV records, B+-tree primary and adjacency indexes), magic `ENGST01`. Vector search is an **HNSW** index published behind a read/write boundary — `eg_vindex_view` returns a `const VIndex*` to N concurrent readers, `eg_vindex_maintain` is the sole mutator. `recall@10 = 0.9365` at `ef_search=128`.
|
||
|
||
> **Doc correction, 2026-08-16.** The previous revision of this paragraph, and most of `engram/README.md`, described a Rust `engram-core` crate backed by `sled` with "flat cosine scan… until scale demands an HNSW layer." **Measured: there is no Rust in `engram/`** — no `.rs` files, no `Cargo.toml`, no `crates/` — and `sled` appears nowhere in the tree. HNSW has been the vector index for some time.
|
||
|
||
Full design rationale, the cognition surface, and the standing corrections: [engram/README.md](engram/README.md).
|
||
|
||
### [elp/](elp/) — EL Projector
|
||
|
||
*(Formerly "EL Language Processor" / "Engram Language Protocol"; renamed **EL Projector** 2026-08-15.)* Neuron's **efferent** organ: the native realizer that *projects* understanding onto a surface via `plan(frame) → realize(spec, profile)`, where **a surface is a profile** and language is one profile among many (text, speech, music, image). Projection, not diffusion — generation *from* an owned, understood signature, never the averaging of a stolen corpus.
|
||
|
||
Its flagship profile is a bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*` → `grammar` → `realizer` → `semantics` → `elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages.
|
||
|
||
### [epm/](epm/) — El Package Manager
|
||
|
||
Manages **vessels** (El's package unit): publish, install, resolve dependencies. Vessels are stored in Engram as graph nodes, not files in a registry index — `epm` reads the local `manifest.el`, talks to Engram over HTTP, and writes resolved vessels to `.epm/vessels/`. Source: `registry.el`, `install.el`, `update.el`, `manifest.el`.
|
||
|
||
### [ide/](ide/) — El IDE
|
||
|
||
Three vessels: **el-ide-server** (HTTP backend — file ops, build/run, LSP bridge, plugin host, settings), **el-lsp** (the language server — completion, hover, diagnostics, outline, format, type graph), and **el-plugin-host** (first-party plugin lifecycle: install/remove/enable/disable). `ide/projects/` and `ide/examples/` hold sample projects, including the canonical `hello-friends` first-program walkthrough.
|
||
|
||
### [ql/](ql/) — engram-el
|
||
|
||
The El-native integration layer for a *live* Engram server — not a library (no importable modules, no build artifact), a set of standalone `.el` programs run directly via `el run-file`. Three components: **Studio** (`studio/studio.el`, a full terminal graph explorer), a **Hebbian field-model** proof of concept, and El builtin / LLM-builtin smoke test suites. This is the reference for correct patterns when an El program uses Engram as its substrate. Spec: [ql/spec/elql.md](ql/spec/elql.md).
|
||
|
||
### [ui/](ui/) — el-ui
|
||
|
||
A frontend framework where **component state is an Engram graph and reactivity is spreading activation** — not virtual-DOM diffing (React), Proxy-based dependency tracking (Vue), or compile-time analysis (Svelte). Re-renders are activated and propagated the same way associative memory retrieval works in `engram/`.
|
||
|
||
~15 vessels covering the full frontend surface: `el-platform` (env/fs/network/clock abstraction), `el-config`, `el-html` (SSR emit primitives), `el-layout`, `el-style` (design tokens/themes), `el-i18n`, `el-auth` / `el-identity` (JWT, sessions, OAuth PKCE — Engram-native), `el-services` (REST/gRPC/WebSocket bindings), `el-aop` (`@authenticate`/`@authorize`/`@cache`/`@rate_limit` decorators), `el-secrets`, `el-graph` (graph rendering/editor), `el-publish` (App Store / Play Store automation), and `el-ui-compiler` (El→JS component compiler; currently a stub pending a JS backend in `elc`). Spec: [ui/spec/framework.md](ui/spec/framework.md).
|
||
|
||
### [arbor/](arbor/) — diagram language
|
||
|
||
A `.arbor` diagram language and toolchain: `arbor-core` (NodeId/shape/edge-kind types), `arbor-parse` (recursive-descent parser), `arbor-diagram` (IR + Mermaid serializer + architecture-diagram builders), `arbor-layout` (hierarchical layout — rank assignment, positioning, group bounds), `arbor-render` (SVG renderer), `arbor-cli`. (The architecture map above is the kind of diagram this is for.)
|
||
|
||
---
|
||
|
||
## Getting started
|
||
|
||
Install the El SDK from the latest release:
|
||
|
||
```bash
|
||
bash lang/install.sh
|
||
# EL_VERSION=v1.0.0 bash lang/install.sh # pin a specific release tag
|
||
# EL_PREFIX=/opt/el bash lang/install.sh # custom install prefix
|
||
```
|
||
|
||
Or build the compiler from source and verify the self-hosting chain:
|
||
|
||
```bash
|
||
cd lang
|
||
./dist/platform/elc elc-cli.el > elc-new.c
|
||
cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \
|
||
-o dist/platform/elc-new \
|
||
elc-new.c el-compiler/runtime/el_seed.c
|
||
|
||
# Confirm the new binary reproduces itself exactly
|
||
./dist/platform/elc-new elc-cli.el > elc-verify.c
|
||
diff elc-new.c elc-verify.c # should be identical
|
||
|
||
mv dist/platform/elc-new dist/platform/elc
|
||
```
|
||
|
||
Run your first program:
|
||
|
||
```bash
|
||
./lang/dist/platform/elc lang/examples/hello.el > hello.c
|
||
cc -std=c11 -I lang/el-compiler/runtime -lcurl -lpthread \
|
||
-o hello hello.c lang/el-compiler/runtime/el_seed.c
|
||
./hello
|
||
```
|
||
|
||
More examples in [lang/examples/](lang/examples/), including a full starter project at `lang/examples/hello-project/`.
|
||
|
||
If the compiler binary is ever lost or corrupted, [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) is the authoritative recovery path.
|
||
|
||
---
|
||
|
||
## Cognition — and the standing corrections
|
||
|
||
The engram carries a live cognition surface: `think` (a directed traversal-read returning a **gradient**, never a point), plus `ground`, `assert`, `attend`, and the correspondence-beat. Two specs govern it, and both are authoritative over anything else in this repo that disagrees:
|
||
|
||
- **[lang/spec/correspondence-and-censorship.md](lang/spec/correspondence-and-censorship.md)** — grounding, wonder, curiosity, dreaming. *(Lands with PR #149.)*
|
||
- **[lang/spec/runtime-ownership.md](lang/spec/runtime-ownership.md)** — ownership, the capability ABI that was dissolved, and the vector-index publication boundary.
|
||
|
||
**Do not re-derive them.** Every earlier version of the first was wrong in an instructive way and each correction was argued down. If a section looks wrong, say so with a measurement rather than editing it.
|
||
|
||
The corrections, in brief:
|
||
|
||
- **Grounding is not a subsystem — it IS the edge weight.** One quantity, not two fields. `grounded-by` as a relation *type* should not exist: grounding is a property *of* a relation, not a relation *between* nodes. It is never computed on demand; computing-and-writing a score makes reads write, which is the `eg_vindex_sync` defect one level up.
|
||
- **Faculties are operations, not parameters.** `reason` changes the estimate (a read); `induce` changes the parameters (the correspondence-beat, which exists and works); `abduce` changes the structure (a write the current `GeoGradient` signature cannot express). A write is not a parameter of a read.
|
||
- **Wonder is the boundary, not a manifest.** Any structure at all has an edge. There are about six wonders, the same for everyone, and they never close. **Curiosity is wonder crystallized** at a nucleation site — one thing at two phases, not two objects.
|
||
- **Consolidation is ambient, not scheduled. A brain has no cron job.** The presence of a ticker is the diagnostic. Measured 2026-08-16: consolidation has **ten implementations**. `soul.el`'s continuous loop is the one with the correct shape; the rest fold into it.
|
||
- **In an immutable substrate, any mechanism that refuses a write is either redundant with immutability, or an epistemic constraint misfiled as a protective one.**
|
||
|
||
[engram/spec/cognitive-architecture.design.md](engram/spec/cognitive-architecture.design.md) is the original design and is **superseded in part** — it is retained, with the refuted claims marked inline at the point each is made, because preserving what was argued down is the point of an immutable record.
|
||
|
||
---
|
||
|
||
## Development workflow
|
||
|
||
Branching follows `dev → stage → main`: work lands on `dev`, promotes to `stage` for integration testing, and is promoted to `main` for release (visible directly in the git history of this repo). CI is defined per-subproject under `.gitea/workflows/` — `lang`/`epm`/`ide` share the root pipeline; `engram` and `ql` carry their own (`ci-dev`, `ci-stage`, and a release workflow each).
|
||
|
||
- Language/runtime specs live at `*/spec/*.md` (`lang/spec/`, `ql/spec/`, `ui/spec/`) and are the single source of truth for implemented-vs-planned status — code and docs are expected to agree with the spec's status markers, not the other way around.
|
||
- Agent-facing orientation guides live at `*/AGENTS.md` (currently `lang/AGENTS.md`); more subprojects may grow their own as they need agent-specific conventions documented.
|
||
- **A release is a git tag, not a folder** (`el-runtime-vX.Y.Z` on this repo). *(Corrected 2026-08-16: this line said "tagged releases live under `lang/releases/`, each with its own `RELEASE.md`." **Measured: `lang/releases/` does not exist** — the restructure named in `AGENTS.md` landed, and the authored runtime is at `lang/runtime/`.)*
|
||
|
||
---
|
||
|
||
## Status
|
||
|
||
This is an actively developed, internal monorepo — not yet published under an open license. Treat everything here as proprietary to Neuron Technologies unless told otherwise.
|