Andre Botelho Rodrigues Almeida
b97b644799
El SDK CI - dev / build-and-test (pull_request) Successful in 8m18s
155 lines
12 KiB
Markdown
155 lines
12 KiB
Markdown
# El
|
||
|
||
**A self-hosting, statically-typed language that compiles to C — built around a graph-native runtime instead of a database driver.**
|
||
|
||
El is the execution substrate for the Neuron agent runtime, the DHARMA network, and the Engram knowledge graph. This repository is the monorepo for the whole stack: the language itself, the graph memory engine it's built to talk to natively, and the tools (package manager, IDE, UI framework, diagramming) built on top of it.
|
||
|
||
---
|
||
|
||
## Why El exists
|
||
|
||
Every other language treats persistent, associative state as something you reach for through a driver — a SQL client, an ORM, a Redis library bolted on from outside. El inverts that: graph operations (`engram_*`) are runtime primitives, on the same footing as string or list operations. There is no separate database driver because the database is not separate.
|
||
|
||
El has four defining properties:
|
||
|
||
1. **Self-hosting compiler.** The compiler (`lexer.el`, `parser.el`, `codegen.el`, `compiler.el`) is written in El. It compiles El source to C, which `cc` compiles against a fixed runtime into a native binary. A Rust genesis compiler bootstrapped the first iteration; the self-hosted binary at `lang/dist/platform/elc` has been the canonical compiler ever since — every binary in `dist/platform/` was produced by an earlier version of itself compiling `el-compiler/src/`. The chain is auditable: source is the ground truth, not the binary. See [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) for the full recovery path if that binary is ever lost.
|
||
2. **C compilation target.** Every compiled program is plain C11. Every El value is `el_val_t` (`int64_t`); strings are heap pointers cast through it. Functions become C functions; top-level statements become `main()`.
|
||
3. **Graph-native runtime.** The runtime provides first-class graph operations over an in-process Engram store — no separate DB driver, no ORM.
|
||
4. **DHARMA-aware identity.** A `cgi` block declares a program's DHARMA identity at compile time. The runtime resolves identity before user code runs, so `dharma_*` calls have a stable principal and channel surface throughout.
|
||
|
||
---
|
||
|
||
## Architecture map
|
||
|
||
```
|
||
┌─────────────┐
|
||
│ lang │ El compiler + C runtime
|
||
│ (El itself) │ everything below is written in it,
|
||
└──────┬──────┘ or compiles down through it
|
||
│
|
||
┌─────────────┼─────────────┐
|
||
│ │ │
|
||
┌──────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
|
||
│ engram │ │ epm │ │ ide │
|
||
│ graph/mem │ │ package │ │ editor + │
|
||
│ substrate │ │ manager │ │ LSP │
|
||
└──────┬─────┘ └───────────┘ └───────────┘
|
||
│
|
||
┌───────┼────────────────┬─────────────────────┐
|
||
│ │ │ │
|
||
┌─────▼───┐ ┌─▼──────────┐ ┌──▼──────────┐ ┌─────▼──────┐
|
||
│ elp │ │ ql │ │ ui │ │ arbor │
|
||
│ NLG / │ │engram-el. │ |spreading- │ |arbor │
|
||
│ 31 langs│ │studio+tests│ |activation UI│ |diagram lang│
|
||
└─────────┘ └────────────┘ └─────────────┘ └────────────┘
|
||
```
|
||
|
||
`lang` is the foundation — the compiler and C runtime everything else builds on. `engram` is the graph-native memory/state engine that gives El its identity (property 3 above). Everything else is either a tool for working with El (`epm`, `ide`) or a system built on top of Engram's graph model (`elp`, `ql`, `ui`, `arbor`).
|
||
|
||
---
|
||
|
||
## Repository layout
|
||
|
||
### [lang/](lang/) — the El language
|
||
|
||
The compiler and runtime. Self-hosting: `elc-cli.el` → `compiler.el` → `lexer.el` / `parser.el` / `codegen.el` / `codegen-js.el`, textually inlined and compiled in one pass. Compiles to C11 and links against `el-compiler/runtime/el_seed.c`, a hand-maintained OS-boundary layer (libcurl HTTP, pthreads, filesystem, arena allocation) — everything else in the runtime is native El (`runtime/*.el`).
|
||
|
||
Two layers to know: **El programs** (`.el` files — where nearly all work belongs) and **the C seed** (`el_seed.c` — edit only for genuine OS-level access; never re-implement what El can already express).
|
||
|
||
Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented. In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), the real `engram_*` and `dharma_*` runtimes (currently stubs), and libcurl-backed `http_get`/`http_post`/`http_serve`. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language.
|
||
|
||
Key docs: [AGENTS.md](lang/AGENTS.md) (agent-facing orientation), [BOOTSTRAP.md](lang/BOOTSTRAP.md) (compiler recovery from scratch), [spec/language.md](lang/spec/language.md), [spec/codegen-js.md](lang/spec/codegen-js.md).
|
||
|
||
### [engram/](engram/) — graph intelligence substrate
|
||
|
||
**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. Rust core (`engram-core`, `engram-ffi`) exposed to El and other languages (Kotlin, TypeScript/WASM, Go bindings).
|
||
|
||
The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop (`strength = parent_strength × edge_weight × target_salience × cosine_sim`), gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works.
|
||
|
||
Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay** — `importance × recency-decay × log(activation_count)`. Forgetting is adaptive pruning, not a bug: unreinforced memories stop competing for attention without being deleted.
|
||
|
||
Backed by `sled` (embedded, local-first, no daemon) with flat cosine scan for vector search — deliberately simple until scale demands an HNSW layer. Full API and design rationale in [engram/README.md](engram/README.md).
|
||
|
||
### [elp/](elp/) — Engram Language Protocol
|
||
|
||
Bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*` → `grammar` → `realizer` → `semantics` → `elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages.
|
||
|
||
### [epm/](epm/) — El Package Manager
|
||
|
||
Manages **vessels** (El's package unit): publish, install, resolve dependencies. Vessels are stored in Engram as graph nodes, not files in a registry index — `epm` reads the local `manifest.el`, talks to Engram over HTTP, and writes resolved vessels to `.epm/vessels/`. Source: `registry.el`, `install.el`, `update.el`, `manifest.el`.
|
||
|
||
### [ide/](ide/) — El IDE
|
||
|
||
Three vessels: **el-ide-server** (HTTP backend — file ops, build/run, LSP bridge, plugin host, settings), **el-lsp** (the language server — completion, hover, diagnostics, outline, format, type graph), and **el-plugin-host** (first-party plugin lifecycle: install/remove/enable/disable). `ide/projects/` and `ide/examples/` hold sample projects, including the canonical `hello-friends` first-program walkthrough.
|
||
|
||
### [ql/](ql/) — engram-el
|
||
|
||
The El-native integration layer for a *live* Engram server — not a library (no importable modules, no build artifact), a set of standalone `.el` programs run directly via `el run-file`. Three components: **Studio** (`studio/studio.el`, a full terminal graph explorer), a **Hebbian field-model** proof of concept, and El builtin / LLM-builtin smoke test suites. This is the reference for correct patterns when an El program uses Engram as its substrate. Spec: [ql/spec/elql.md](ql/spec/elql.md).
|
||
|
||
### [ui/](ui/) — el-ui
|
||
|
||
A frontend framework where **component state is an Engram graph and reactivity is spreading activation** — not virtual-DOM diffing (React), Proxy-based dependency tracking (Vue), or compile-time analysis (Svelte). Re-renders are activated and propagated the same way associative memory retrieval works in `engram/`.
|
||
|
||
~15 vessels covering the full frontend surface: `el-platform` (env/fs/network/clock abstraction), `el-config`, `el-html` (SSR emit primitives), `el-layout`, `el-style` (design tokens/themes), `el-i18n`, `el-auth` / `el-identity` (JWT, sessions, OAuth PKCE — Engram-native), `el-services` (REST/gRPC/WebSocket bindings), `el-aop` (`@authenticate`/`@authorize`/`@cache`/`@rate_limit` decorators), `el-secrets`, `el-graph` (graph rendering/editor), `el-publish` (App Store / Play Store automation), and `el-ui-compiler` (El→JS component compiler; currently a stub pending a JS backend in `elc`). Spec: [ui/spec/framework.md](ui/spec/framework.md).
|
||
|
||
### [arbor/](arbor/) — diagram language
|
||
|
||
A `.arbor` diagram language and toolchain: `arbor-core` (NodeId/shape/edge-kind types), `arbor-parse` (recursive-descent parser), `arbor-diagram` (IR + Mermaid serializer + architecture-diagram builders), `arbor-layout` (hierarchical layout — rank assignment, positioning, group bounds), `arbor-render` (SVG renderer), `arbor-cli`. (The architecture map above is the kind of diagram this is for.)
|
||
|
||
---
|
||
|
||
## Getting started
|
||
|
||
Install the El SDK from the latest release:
|
||
|
||
```bash
|
||
bash lang/install.sh
|
||
# EL_VERSION=v1.0.0 bash lang/install.sh # pin a specific release tag
|
||
# EL_PREFIX=/opt/el bash lang/install.sh # custom install prefix
|
||
```
|
||
|
||
Or build the compiler from source and verify the self-hosting chain:
|
||
|
||
```bash
|
||
cd lang
|
||
./dist/platform/elc elc-cli.el > elc-new.c
|
||
cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \
|
||
-o dist/platform/elc-new \
|
||
elc-new.c el-compiler/runtime/el_seed.c
|
||
|
||
# Confirm the new binary reproduces itself exactly
|
||
./dist/platform/elc-new elc-cli.el > elc-verify.c
|
||
diff elc-new.c elc-verify.c # should be identical
|
||
|
||
mv dist/platform/elc-new dist/platform/elc
|
||
```
|
||
|
||
Run your first program:
|
||
|
||
```bash
|
||
./lang/dist/platform/elc lang/examples/hello.el > hello.c
|
||
cc -std=c11 -I lang/el-compiler/runtime -lcurl -lpthread \
|
||
-o hello hello.c lang/el-compiler/runtime/el_seed.c
|
||
./hello
|
||
```
|
||
|
||
More examples in [lang/examples/](lang/examples/), including a full starter project at `lang/examples/hello-project/`.
|
||
|
||
If the compiler binary is ever lost or corrupted, [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) is the authoritative recovery path.
|
||
|
||
---
|
||
|
||
## Development workflow
|
||
|
||
Branching follows `dev → stage → main`: work lands on `dev`, promotes to `stage` for integration testing, and is promoted to `main` for release (visible directly in the git history of this repo). CI is defined per-subproject under `.gitea/workflows/` — `lang`/`epm`/`ide` share the root pipeline; `engram` and `ql` carry their own (`ci-dev`, `ci-stage`, and a release workflow each).
|
||
|
||
- Language/runtime specs live at `*/spec/*.md` (`lang/spec/`, `ql/spec/`, `ui/spec/`) and are the single source of truth for implemented-vs-planned status — code and docs are expected to agree with the spec's status markers, not the other way around.
|
||
- Agent-facing orientation guides live at `*/AGENTS.md` (currently `lang/AGENTS.md`); more subprojects may grow their own as they need agent-specific conventions documented.
|
||
- Tagged releases live under `lang/releases/`, each with its own `RELEASE.md`.
|
||
|
||
---
|
||
|
||
## Status
|
||
|
||
This is an actively developed, internal monorepo — not yet published under an open license. Treat everything here as proprietary to Neuron Technologies unless told otherwise.
|