4abeed6bee
The docs described a mind made of subsystems — a grounding subsystem, a wonder manifest, a dreamer on a beat, faculties as arguments to one call. Each of those is a supervisor invented for something that should be a property of the substrate, and two of the documents carrying them are load-bearing for a build agent: cognitive-architecture.design.md says "a build agent executes from this doc", and tools/api-reshape/README.md marks the refuted shapes PROVEN on a live clone. Corrections carried, per lang/spec/correspondence-and-censorship.md (PR #149) and lang/spec/runtime-ownership.md: - Grounding is not a subsystem — it IS the edge weight. grounded-by as a relation type should not exist; grounding is a property of a relation, not a relation between nodes. Never computed on demand. - Faculties are operations, not parameters. reason changes the estimate, induce changes the parameters, abduce changes the structure — a write, which GeoGradient cannot express. A write is not a parameter of a read. - Wonder is the boundary, not a manifest. Curiosity is wonder crystallized at a nucleation site: one thing at two phases. Removed wonder from the operator table in AGENTS.md. - Consolidation is ambient, not scheduled. A brain has no cron job. The presence of a ticker is the diagnostic. - co_registration is deprecated — it averaged a per-edge property into a region scalar, so opposing sites cancelled. GeoEdge.discord replaces it. Nothing new may read it. - In an immutable substrate, any mechanism that refuses a write is either redundant with immutability or an epistemic constraint misfiled as a protective one. The two design docs are marked superseded-in-part with the refutation at the point each claim is made, not rewritten. Preserving what was argued down is the point of an immutable record. Also measured and corrected while verifying the above: engram/README.md documented a Rust engram-core crate on sled with "flat cosine scan until scale demands HNSW" — there is no Rust in engram/ and HNSW is the index; lang/releases/ no longer exists, so both README.md and AGENTS.md pointed at a deleted path for the authored runtime; language.md listed the engram_* and http_* runtimes as stubs. Added language.md §20 for geometry-as-a-value, realizers and transduce (#144), which had landed with no spec coverage. Documentation only. No .c, .h, or .el file is touched.
1335 lines
57 KiB
Markdown
1335 lines
57 KiB
Markdown
# El Language Specification
|
||
|
||
Version 1.2.0 — April 30, 2026
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
El is a statically-typed, compiled programming language that serves as the execution substrate for the Neuron agent runtime, the DHARMA network, and the Engram knowledge graph. El compiles to C and links against a fixed runtime, producing native binaries.
|
||
|
||
El has four defining properties:
|
||
|
||
1. **Self-hosting compiler.** The compiler (`lexer.el`, `parser.el`, `codegen.el`, `compiler.el`) is written in El. It compiles El source to C, which is then compiled by `cc` against `el_runtime.c` to produce a native binary. A Rust genesis compiler bootstraps the first iteration; the self-hosted binary at `dist/platform/elc` is the canonical compiler thereafter.
|
||
|
||
2. **C compilation target.** Every compiled program is plain C11. Every El value is `el_val_t` (`int64_t`). Strings are pointers cast through `int64_t`. Functions become C functions; top-level statements become `main()`.
|
||
|
||
3. **Graph-native runtime.** The runtime provides first-class graph operations (`engram_*`) over an in-process Engram store. CGI programs use these primitives directly; there is no separate database driver.
|
||
|
||
4. **DHARMA-aware identity.** The `cgi` block declares a program's DHARMA identity at compile time. The runtime resolves identity before any user code runs, so subsequent `dharma_*` calls have a stable principal and channel surface.
|
||
|
||
---
|
||
|
||
## Implementation Status
|
||
|
||
This section is the **single source of truth** for what works and what is planned. Each subsequent section repeats the relevant marker. If a feature here is marked "planned," it appears in the spec because it is in the design, but the current self-hosting toolchain does not yet emit code for it.
|
||
|
||
### Implemented (today)
|
||
|
||
- Lexer: keywords, identifiers, integer/float/string/bool literals, operators below.
|
||
- Parser: `let`, `return`, `fn`, `type`, `enum`, `import`, `from … import`, `while`, `for`, `if/else if/else`, `match`, `@decorator`, array/map literals, all listed operators, function calls, field access, index access, unary `!`/`-`, postfix `?`.
|
||
- Codegen: function definitions, top-level `main()`, all expression forms above, control flow, decorator-as-AST-attachment.
|
||
- Boundary seam: decorator arguments and stacking; VBD role enforcement via `#error`; `engram_boundary_beat` auto-emit at `@manager`/`@accessor` entry; `@route` dispatch tables (Section 9).
|
||
- Program-level declarative blocks: `cgi`, `service`, and `program` — the last carrying process identity and configuration (Section 18).
|
||
- **Geometry as a first-class value, and realizers declarable in El** — the `Geometry` type, the wire adapters, and `transduce` (Section 20). Landed 2026-08-16 (#141, #144).
|
||
- C runtime: I/O, string operations, integer math, lists, maps, filesystem, command-line args, basic `json_get` substring lookup.
|
||
|
||
### Planned (in flight)
|
||
|
||
- **`%` (modulo) operator.** Currently not lexed. Adding to lexer + parser + codegen.
|
||
- **Match codegen.** Currently parsed; codegen does not emit. Adding `({ ... })` statement-expression emission.
|
||
- **`?` propagation.** Currently no-op. Adding nil-propagation semantics.
|
||
- **`cgi` block parsing.** Currently lexed (`cgi` is a keyword) but not parsed as a statement. Adding `parse_cgi_block` and codegen of `el_cgi_init` at the head of `main()`.
|
||
- **Boundary epilogues.** The decorator seam injects a prologue only. Adding prologue/epilogue wrapping, the prerequisite for durability-as-an-effect (Section 19.1).
|
||
- **`vessel` keyword.** Replaces `package` in manifests. Adding to lexer.
|
||
- ~~**Real `engram_*` runtime.** Currently stub.~~ **Stale (verified 2026-08-16) — this is implemented, not planned.** `lang/runtime/el_runtime.c` carries the in-process graph store with spreading activation, Hebbian strengthening, disk persistence (paged store, magic `ENGST01`), an HNSW vector index behind a `eg_vindex_view`/`eg_vindex_maintain` publication boundary, and the full cognition surface (`engram_think_json`, `engram_ground_json`, `engram_assert_json`, `engram_attend_json`, `engram_correspondence_beat_json`). The "stub" description may still hold for the **lagging forks** (`lang/el-compiler/runtime/`, `products/web/runtime/`) — see `AGENTS.md`, which names those as downstream copies that cannot build the engram product. **Which runtime this line refers to needs a decision; it is not a fact that can be recovered from the text.**
|
||
- ~~**Real `dharma_*` runtime.** Currently stub.~~ **Needs re-verification (2026-08-16).** Not checked in this pass; do not rely on either reading.
|
||
- ~~**Real `http_get`/`http_post`/`http_serve`.** Currently empty stubs.~~ **Stale.** libcurl-backed HTTP and a thread-pool server are live — `http_serve_async` is what `neuron/soul.el:729` runs before entering its awareness loop, and `realizer_register` resolves El functions through the same `dlsym` mechanism `http_set_handler` relies on.
|
||
- **JSON, time, UUID, state, env, additional string/list/math builtins.** See Section 12 for the canonical list.
|
||
|
||
### Not in this language
|
||
|
||
- Bitwise operators (`&`, `|`, `^`, `<<`, `>>`). Single `&` is silently consumed by the lexer; single `|` is lexed as `Pipe` but unused. None are parsed as binary operators. Removed from the spec entirely.
|
||
- `??` null-coalescing. Was reserved; not lexed. Removed.
|
||
- `as` casts. `as` is a keyword but no parse form. Removed from the spec until shipped.
|
||
- Floating-point arithmetic distinct from Int. All values are `int64_t` at the C level. `Float` literals are accepted but stored as bit-cast doubles only when the runtime is float-aware (currently only via `float_to_str` and friends in the planned runtime extension).
|
||
|
||
---
|
||
|
||
## 1. Lexical Structure
|
||
|
||
### 1.1 Source Encoding
|
||
|
||
El source is UTF-8, file extension `.el`.
|
||
|
||
### 1.2 Comments
|
||
|
||
```
|
||
// Single-line comment — extends to end of line
|
||
```
|
||
|
||
Block comments are not supported.
|
||
|
||
### 1.3 Whitespace
|
||
|
||
Spaces, tabs, newlines (`\n`), and carriage returns (`\r`) are whitespace. Whitespace is not significant except as a token separator.
|
||
|
||
### 1.4 Identifiers
|
||
|
||
```
|
||
identifier = (alpha | '_') (alnum | '_')*
|
||
```
|
||
|
||
Identifiers are case-sensitive. Names beginning with `__` are reserved for compiler-generated symbols.
|
||
|
||
### 1.5 Keywords
|
||
|
||
The following words are reserved and cannot be used as identifiers. Each row notes whether the parser currently consumes the keyword as a structural form.
|
||
|
||
| Keyword | Parsed today | Notes |
|
||
|---------|--------------|-------|
|
||
| `let` | yes | Variable binding |
|
||
| `fn` | yes | Function definition |
|
||
| `type` | yes | Struct definition |
|
||
| `enum` | yes | Enum definition |
|
||
| `match` | yes | Pattern match expression |
|
||
| `return` | yes | Function return |
|
||
| `if` / `else` | yes | Conditional |
|
||
| `for` / `in` | yes | Iteration |
|
||
| `while` | yes | Loop |
|
||
| `import` / `from` / `as` | yes | Module import |
|
||
| `true` / `false` | yes | Bool literals |
|
||
| `cgi` | yes | Top-level CGI declaration block |
|
||
| `service` | yes | Top-level capability-bounded declaration block |
|
||
| `program` | yes | Top-level cross-cutting declaration block (Section 18) |
|
||
| `manager` / `engine` / `accessor` | as decorators | VBD role marker on `fn`; enforcement and boundary auto-emit are live (Section 9) |
|
||
| `vessel` | planned | Manifest declaration (replaces `package`) |
|
||
| `activate` / `where` | planned | Spreading-activation construct |
|
||
| `sealed` | planned | Capability scope block |
|
||
| `with` | reserved | No parse form yet |
|
||
| `test` / `seed` / `assert` | reserved | Testing primitives, no parse form |
|
||
| `protocol` / `impl` | reserved | Trait-like, no parse form |
|
||
| `retry` / `times` / `fallback` / `reason` | reserved | Resilience primitives, no parse form |
|
||
| `parallel` / `trace` | reserved | Concurrency, no parse form |
|
||
| `requires` / `deploy` / `to` / `via` / `target` | reserved | Deployment surface, no parse form |
|
||
|
||
### 1.6 Token Types
|
||
|
||
| Token | Pattern | Notes |
|
||
|-------|---------|-------|
|
||
| `Int` | `[0-9]+` | |
|
||
| `Float` | `[0-9]+ '.' [0-9]+` | |
|
||
| `Str` | `"…"` with `\"`, `\n`, `\t`, `\\` escapes | |
|
||
| `Bool` | `true` or `false` | |
|
||
| `Ident` | identifier (not a keyword) | |
|
||
| keyword tokens | one per keyword above | e.g. `Let`, `Fn`, `If` |
|
||
| `Eq` | `=` | |
|
||
| `EqEq` | `==` | |
|
||
| `NotEq` | `!=` | |
|
||
| `Not` | `!` | |
|
||
| `Lt` `LtEq` `Gt` `GtEq` | `<` `<=` `>` `>=` | |
|
||
| `And` | `&&` | Single `&` is consumed and discarded |
|
||
| `Or` | `\|\|` | |
|
||
| `Pipe` | `\|` | Lexed; not used by parser |
|
||
| `PipeOp` | `\|>` | Lexed; not used by parser |
|
||
| `Plus` `Minus` `Star` `Slash` | `+` `-` `*` `/` | |
|
||
| `Arrow` | `->` | |
|
||
| `FatArrow` | `=>` | |
|
||
| `Colon` `ColonColon` | `:` `::` | |
|
||
| `LParen` `RParen` `LBrace` `RBrace` `LBracket` `RBracket` | `(` `)` `{` `}` `[` `]` | |
|
||
| `Comma` `Dot` `Semicolon` | `,` `.` `;` | |
|
||
| `At` `QuestionMark` | `@` `?` | |
|
||
| `Eof` | end-of-input sentinel | |
|
||
|
||
### 1.7 String Escapes
|
||
|
||
| Sequence | Character |
|
||
|----------|-----------|
|
||
| `\n` | Newline (U+000A) |
|
||
| `\t` | Tab (U+0009) |
|
||
| `\"` | Double quote |
|
||
| `\\` | Backslash |
|
||
| (other) | Character as-is |
|
||
|
||
---
|
||
|
||
## 2. Type System
|
||
|
||
### 2.1 Primitive Types
|
||
|
||
| Type | Description | Example literals |
|
||
|------|-------------|------------------|
|
||
| `Int` | 64-bit signed integer | `42`, `-7` |
|
||
| `Float` | 64-bit double (planned float arithmetic; today stored as int64) | `3.14`, `0.5` |
|
||
| `String` | UTF-8 string | `"hello"` |
|
||
| `Bool` | Boolean | `true`, `false` |
|
||
| `Void` | No value | — |
|
||
| `Any` | Dynamically-typed value | (for generic containers) |
|
||
|
||
### 2.2 Composite Types
|
||
|
||
| Type form | Description |
|
||
|-----------|-------------|
|
||
| `[T]` | Array of `T` |
|
||
| `T?` | Optional `T` (planned propagation; today the `?` postfix is no-op) |
|
||
| `Map<K, V>` | Key-value map |
|
||
| Named type | User-defined struct or enum |
|
||
|
||
### 2.3 Type Annotations
|
||
|
||
Type annotations appear in `let` bindings and function signatures. The current compiler **parses and skips** annotations — no type checking is performed at compile time. They serve as documentation. A type checker is planned.
|
||
|
||
```
|
||
let x: Int = 42
|
||
fn greet(name: String) -> String { … }
|
||
```
|
||
|
||
### 2.4 Optional Types
|
||
|
||
`T?` is accepted in type position. The postfix `?` on an expression produces a `Try` AST node. Today the codegen passes the inner expression through transparently; nil propagation is planned.
|
||
|
||
---
|
||
|
||
## 3. Variables and Bindings
|
||
|
||
### 3.1 Let Bindings
|
||
|
||
```
|
||
let name: Type = expression
|
||
let name = expression
|
||
```
|
||
|
||
All bindings are block-scoped. El allows re-binding the same name in the same scope; the codegen emits a plain assignment instead of a redeclaration:
|
||
|
||
```
|
||
let count = 0
|
||
let count = count + 1 // plain `count = count + 1;` in C
|
||
```
|
||
|
||
### 3.2 Scope
|
||
|
||
Bindings are valid from the point of declaration to the end of the enclosing block. Function bodies, `if` arms, `for` bodies, `while` bodies, and explicit `{ … }` blocks each introduce a new C scope.
|
||
|
||
---
|
||
|
||
## 4. Functions
|
||
|
||
### 4.1 Definition
|
||
|
||
```
|
||
fn name(p1: T1, p2: T2) -> R {
|
||
// body
|
||
return expr
|
||
}
|
||
```
|
||
|
||
Compiles to `el_val_t name(el_val_t p1, el_val_t p2)`. The return type is parsed and skipped. A `return 0;` is appended automatically at the end of every function body.
|
||
|
||
### 4.2 Forward Declarations
|
||
|
||
The codegen emits forward declarations for all top-level `fn` definitions before any function body. Mutual recursion within a file is supported.
|
||
|
||
### 4.3 Return
|
||
|
||
```
|
||
return expression
|
||
return // bare; compiles to `return 0;`
|
||
```
|
||
|
||
A bare `return` followed by `}` or end-of-file compiles to `return 0;`.
|
||
|
||
### 4.4 Calling Convention
|
||
|
||
All arguments are `el_val_t`, passed in declaration order.
|
||
|
||
#### Method-style calls
|
||
|
||
`obj.method(args…)` compiles to `method(obj, args…)`. The runtime exports short-name aliases for common operations:
|
||
|
||
| El source | Emitted C | Runtime alias |
|
||
|-----------|-----------|---------------|
|
||
| `list.append(x)` | `append(list, x)` | `el_list_append` |
|
||
| `list.len()` | `len(list)` | `el_list_len` |
|
||
| `list.get(i)` | `get(list, i)` | `el_list_get` |
|
||
| `map.map_get(k)` | `map_get(map, k)` | `el_map_get` |
|
||
| `map.map_set(k, v)` | `map_set(map, k, v)` | `el_map_set` |
|
||
|
||
---
|
||
|
||
## 5. Control Flow
|
||
|
||
### 5.1 If / Else
|
||
|
||
```
|
||
if cond { … } else if cond2 { … } else { … }
|
||
```
|
||
|
||
`if` may appear as an expression (RHS of `let`, etc.); when used in expression position the codegen emits a ternary stub. `if` as a statement emits standard `if (…) { … } else { … }` C code.
|
||
|
||
### 5.2 While
|
||
|
||
```
|
||
while cond { … }
|
||
```
|
||
|
||
Exits when `cond` is `0`.
|
||
|
||
### 5.3 For
|
||
|
||
```
|
||
for item in list { … }
|
||
```
|
||
|
||
Compiles to a tracked C `for` loop:
|
||
|
||
```c
|
||
{
|
||
el_val_t _el_lst = <list>;
|
||
el_val_t _el_len = el_list_len(_el_lst);
|
||
for (el_val_t _el_i = 0; _el_i < _el_len; _el_i++) {
|
||
el_val_t item = el_list_get(_el_lst, _el_i);
|
||
// body
|
||
}
|
||
}
|
||
```
|
||
|
||
### 5.4 Match
|
||
|
||
```
|
||
match expression {
|
||
Pattern1 => result_expr
|
||
Pattern2 => result_expr
|
||
_ => default_expr
|
||
}
|
||
```
|
||
|
||
Pattern forms today:
|
||
|
||
| Pattern | Meaning |
|
||
|---------|---------|
|
||
| `_` | Wildcard — always matches |
|
||
| `name` | Binding — captures subject as `name` |
|
||
| `42` | Integer literal |
|
||
| `"str"` | String literal |
|
||
| `true` / `false` | Boolean literal |
|
||
|
||
Enum-variant patterns (`EnumName::Variant`) are reserved but not yet parsed.
|
||
|
||
**Codegen status:** Parsed today; codegen emits a `({ … })` statement-expression in the planned runtime extension. Until then, `match` is recognized but produces no emitted code.
|
||
|
||
---
|
||
|
||
## 6. Data Structures
|
||
|
||
### 6.1 Struct Types
|
||
|
||
```
|
||
type TypeName {
|
||
field1: Type1
|
||
field2: Type2
|
||
}
|
||
```
|
||
|
||
Type definitions are parsed and recorded; no C type is emitted. Struct values at runtime are `ElMap`. Field access `value.field` compiles to `el_get_field(value, "field")`.
|
||
|
||
Optional commas between fields are accepted.
|
||
|
||
### 6.2 Enum Types
|
||
|
||
```
|
||
enum EnumName {
|
||
Variant1
|
||
Variant2
|
||
VariantWithPayload(PayloadType)
|
||
}
|
||
```
|
||
|
||
Variant names are recorded. The current codegen does not emit a dedicated C enum type; values are represented as strings or maps. Payload variants accept the `(Type)` syntax but the parser records only the variant name.
|
||
|
||
### 6.3 Array Literals
|
||
|
||
```
|
||
let numbers = [1, 2, 3]
|
||
let empty = []
|
||
```
|
||
|
||
Compile to `el_list_new(3, 1, 2, 3)` and `el_list_new(0)`.
|
||
|
||
### 6.4 Map Literals
|
||
|
||
```
|
||
let m = { "k1": v1, "k2": v2 }
|
||
```
|
||
|
||
Keys are `Str` tokens. Compiles to `el_map_new(2, "k1", v1, "k2", v2)`.
|
||
|
||
### 6.5 Field and Index Access
|
||
|
||
```
|
||
let f = struct_value.field_name // el_get_field(struct_value, "field_name")
|
||
let e = array[0] // el_list_get(array, 0)
|
||
let v = map["key"] // el_list_get(map, "key")
|
||
```
|
||
|
||
Index access compiles to `el_list_get`. Out-of-bounds returns `0` (`EL_NULL`).
|
||
|
||
---
|
||
|
||
## 7. Operators
|
||
|
||
### 7.1 Arithmetic
|
||
|
||
| Operator | Status | Notes |
|
||
|----------|--------|-------|
|
||
| `+` | implemented | Int addition or String concatenation (heuristic) |
|
||
| `-` | implemented | Subtraction; also unary negation |
|
||
| `*` | implemented | Multiplication |
|
||
| `/` | implemented | Integer division |
|
||
| `%` | planned | Modulo. Not currently lexed. |
|
||
|
||
**`+` dispatch:** the codegen inspects operand AST node kinds. If either side is `Str`, a chained `+`, a `Call`, or an `Ident`, it emits `el_str_concat(a, b)`. If both sides are `Int` literals, it emits arithmetic `+`. Mixed Int+Ident is treated as string concatenation by default — explicit casts are required for arithmetic on Ident-typed integers.
|
||
|
||
### 7.2 Comparison
|
||
|
||
| Operator | Behavior |
|
||
|----------|----------|
|
||
| `==` | `str_eq(a, b)` for Str/Ident/Call operands; `==` for Int/Bool |
|
||
| `!=` | Negation of the above |
|
||
| `<` `>` `<=` `>=` | Integer comparison |
|
||
|
||
### 7.3 Logical
|
||
|
||
| Operator | Description |
|
||
|----------|-------------|
|
||
| `&&` | Short-circuit AND |
|
||
| `\|\|` | Short-circuit OR |
|
||
| `!` | Unary NOT |
|
||
|
||
### 7.4 Unary
|
||
|
||
| Operator | Meaning |
|
||
|----------|---------|
|
||
| `!` | Logical NOT — emits `!expr` |
|
||
| `-` | Negation — emits `(-expr)` |
|
||
| `?` | Try (postfix) — pass-through today; nil-propagation planned |
|
||
|
||
### 7.5 Precedence (high to low)
|
||
|
||
1. `*` `/` `%` — precedence 6
|
||
2. `+` `-` — precedence 5
|
||
3. `<` `>` `<=` `>=` — precedence 4
|
||
4. `==` `!=` — precedence 3
|
||
5. `&&` — precedence 2
|
||
6. `||` — precedence 1
|
||
|
||
---
|
||
|
||
## 8. Module System
|
||
|
||
### 8.1 Import
|
||
|
||
```
|
||
import "filename.el"
|
||
```
|
||
|
||
Records an `Import` AST node. The compiler concatenates all imports into the single C output; the linker produces one binary.
|
||
|
||
### 8.2 From-Import
|
||
|
||
```
|
||
from module_name import { Name1, Name2 }
|
||
```
|
||
|
||
Parsed. The module name is recorded; the brace-list is consumed. Both forms produce `Import` nodes. Selective import (importing only specific names) is parsed but not yet enforced — all names in the imported file are visible.
|
||
|
||
---
|
||
|
||
## 9. Decorators
|
||
|
||
```
|
||
@manager
|
||
fn handle(channel: String, msg: String) -> Void { … }
|
||
```
|
||
|
||
The `@` token followed by an identifier attaches a decorator to the next `FnDef`.
|
||
|
||
**Decorators take arguments and they stack.** `@route("/p", "GET") @manager fn f()` attaches both to `f` as a `decorators` list of `{name, args}` records, topmost-first. Arguments are string literals only.
|
||
|
||
**Decorators have structural meaning today.** This is El's function-level boundary seam — the mechanism by which a cross-cutting concern is handled *at the boundary* rather than by a convention repeated at every call site:
|
||
|
||
| Decorator | Structural effect |
|
||
|---|---|
|
||
| `@manager` | Permits calls to `dharma_emit` / `dharma_field`. Calling either from a non-`@manager` fn emits a `#error` into the generated C — a compile-time failure, not a lint. |
|
||
| `@manager`, `@accessor` | Codegen injects one call to `engram_boundary_beat(<fn name>)` at function entry. The decorated op self-reports (chrono tick, afferent counter, self-activity strengthen, dharma bus event) with **zero** hand-written instrumentation in its body. |
|
||
| `@route(path, method, …)` | Records a route into a generated dispatch table. |
|
||
|
||
Decorators with no registered meaning are accepted and ignored.
|
||
|
||
**Limits of the seam, as it stands.** The injection is a *prologue only* — there is no epilogue, no wrapping of the call, and no way for a decorator to run code after the body returns. The injected callee is a fixed builtin chosen by the compiler, not derived from the decorator name or its arguments. Section 19 depends on lifting exactly these two limits.
|
||
|
||
---
|
||
|
||
## 10. The Activate Construct [planned]
|
||
|
||
```
|
||
activate TypeName where "semantic query string"
|
||
```
|
||
|
||
`activate` and `where` are reserved keywords. Lexed today, no parse form. The planned semantics: compile to a runtime call into the local Engram graph that performs spreading-activation retrieval, returning a typed list of nodes that match `TypeName`.
|
||
|
||
---
|
||
|
||
## 11. The Sealed Block [planned]
|
||
|
||
```
|
||
sealed {
|
||
let api_key = "sk-prod-12345"
|
||
}
|
||
```
|
||
|
||
`sealed` is a reserved keyword. Planned semantics: a capability scope where access to certain runtime services (filesystem, network) is restricted by default and explicit allow-lists must be declared.
|
||
|
||
---
|
||
|
||
## 12. Standard Library Builtins
|
||
|
||
Builtins live in `el_runtime.c` / `el_runtime.h`. Programs call them by name; no import is required. The status column reflects the canonical self-hosting runtime. Anything marked **planned** is in flight as part of the in-progress runtime extension.
|
||
|
||
### 12.1 I/O — implemented
|
||
|
||
| Builtin | Description |
|
||
|---------|-------------|
|
||
| `println(s)` | Print string + newline |
|
||
| `print(s)` | Print string |
|
||
| `readline()` | Read one line from stdin |
|
||
| `args()` | Command-line arguments as a `[String]` (excludes argv[0]) |
|
||
|
||
### 12.2 String — implemented unless noted
|
||
|
||
| Builtin | Description | Status |
|
||
|---------|-------------|--------|
|
||
| `str_eq(a, b)` | String equality | implemented |
|
||
| `str_starts_with(s, p)` | Prefix test | implemented |
|
||
| `str_ends_with(s, suf)` | Suffix test | implemented |
|
||
| `str_contains(s, sub)` | Substring test | implemented |
|
||
| `str_len(s)` | Byte length | implemented |
|
||
| `str_slice(s, start, end)` | Substring (byte offsets) | implemented |
|
||
| `str_replace(s, from, to)` | Replace all | implemented |
|
||
| `str_to_upper(s)` / `str_to_lower(s)` | Case fold | implemented |
|
||
| `str_trim(s)` | Strip whitespace | implemented |
|
||
| `str_concat(a, b)` | Concatenate | implemented |
|
||
| `int_to_str(n)` | Format Int | implemented |
|
||
| `str_to_int(s)` | Parse Int | implemented |
|
||
| `str_to_float(s)` | Parse Float | planned |
|
||
| `str_index_of(s, sub)` | Position of substring; `-1` if absent | planned |
|
||
| `str_split(s, sep)` | Split on separator → `[String]` | planned |
|
||
| `str_char_at(s, i)` | Character at byte index | planned |
|
||
| `str_char_code(s, i)` | Unicode code point | planned |
|
||
| `str_pad_left(s, w, p)` / `str_pad_right(s, w, p)` | Pad to width | planned |
|
||
| `str_format(template, data)` | `{key}` interpolation | planned |
|
||
| `str_lower(s)` / `str_upper(s)` | Aliases for `str_to_lower`/`str_to_upper` | planned |
|
||
|
||
### 12.3 Math — partial
|
||
|
||
| Builtin | Description | Status |
|
||
|---------|-------------|--------|
|
||
| `el_abs(n)` | Absolute value | implemented |
|
||
| `el_max(a, b)` | Maximum | implemented |
|
||
| `el_min(a, b)` | Minimum | implemented |
|
||
| `math_sqrt(f)` | Square root | planned |
|
||
| `math_log(f)` / `math_ln(f)` | Logarithms | planned |
|
||
| `math_sin(f)` / `math_cos(f)` / `math_pi()` | Trig | planned |
|
||
|
||
### 12.4 List — implemented unless noted
|
||
|
||
| Builtin | Description | Status |
|
||
|---------|-------------|--------|
|
||
| `el_list_empty()` | Empty list | implemented |
|
||
| `el_list_new(count, …)` | List from N values (varargs; emitted for array literals) | implemented |
|
||
| `el_list_len(list)` | Length | implemented |
|
||
| `el_list_get(list, i)` | Element at index; `0` on out-of-bounds | implemented |
|
||
| `el_list_append(list, e)` | Append; returns updated list | implemented |
|
||
| `list_push(list, e)` | Alias for `el_list_append` | planned |
|
||
| `list_push_front(list, e)` | Prepend | planned |
|
||
| `list_join(list, sep)` | Join → `String` | planned |
|
||
| `list_range(start, end)` | Integer range `[start, end)` | planned |
|
||
|
||
List append returns a new (or reallocated) list pointer; the return value must be used.
|
||
|
||
### 12.5 Map — implemented
|
||
|
||
| Builtin | Description |
|
||
|---------|-------------|
|
||
| `el_map_new(count, …)` | Map from key/value pairs (emitted for map literals) |
|
||
| `el_map_get(map, key)` | Value by key |
|
||
| `el_map_set(map, key, value)` | Set; returns map |
|
||
| `el_get_field(map, key)` | Alias; emitted for `.field` |
|
||
|
||
### 12.6 HTTP — planned
|
||
|
||
| Builtin | Status |
|
||
|---------|--------|
|
||
| `http_get(url)` | stub today; libcurl impl planned |
|
||
| `http_post(url, body)` | stub today; libcurl impl planned |
|
||
| `http_serve(port, handler)` | stub today; thread-pool impl planned |
|
||
|
||
`handler` is a function value of type `(method: String, path: String, body: String) -> String`. The server thread invokes it on every request.
|
||
|
||
### 12.7 Filesystem — implemented
|
||
|
||
| Builtin | Description |
|
||
|---------|-------------|
|
||
| `fs_read(path)` | Read file → `String`; `""` on error |
|
||
| `fs_write(path, content)` | Write `String`; returns `1` on success, `0` otherwise |
|
||
|
||
### 12.8 JSON — partial
|
||
|
||
| Builtin | Description | Status |
|
||
|---------|-------------|--------|
|
||
| `json_get(json, key)` | Substring lookup of `"key":` value | implemented |
|
||
| `json_parse(s)` | Parse JSON string → `List` or `Map` | planned |
|
||
| `json_stringify(v)` | Serialize `Any` → `String` | planned |
|
||
| `json_get_string(j, key)` | Typed extract: String | planned |
|
||
| `json_get_int(j, key)` | Typed extract: Int | planned |
|
||
| `json_get_float(j, key)` | Typed extract: Float | planned |
|
||
| `json_get_bool(j, key)` | Typed extract: Bool | planned |
|
||
| `json_get_raw(j, key)` | Extract nested object/array as JSON String | planned |
|
||
| `json_set(j, key, value)` | Update field, return new JSON String | planned |
|
||
| `json_array_len(j)` | Length of JSON array string | planned |
|
||
|
||
### 12.9 Process — implemented
|
||
|
||
| Builtin | Description |
|
||
|---------|-------------|
|
||
| `exit_program(code)` | Exit with code |
|
||
| `args()` | (see Section 12.1) |
|
||
|
||
### 12.10 Time — planned
|
||
|
||
| Builtin | Description |
|
||
|---------|-------------|
|
||
| `time_now()` | Unix epoch milliseconds (Int) |
|
||
| `time_now_utc()` | Same; explicit UTC |
|
||
| `time_format(ts, fmt)` | Format timestamp; `"ISO"` for ISO 8601 |
|
||
| `time_to_parts(ts)` | Decompose to `Map` of fields |
|
||
| `time_from_parts(secs, ns, tz)` | Construct |
|
||
| `time_add(ts, n, unit)` | Add duration; unit ∈ `"ms"`, `"sec"`, `"day"`, etc. |
|
||
| `time_diff(ts1, ts2, unit)` | Difference |
|
||
|
||
### 12.11 Identifiers — planned
|
||
|
||
| Builtin | Description |
|
||
|---------|-------------|
|
||
| `uuid_new()` | RFC 4122 v4 UUID String |
|
||
| `uuid_v4()` | Alias for `uuid_new` |
|
||
|
||
### 12.12 Float Formatting — planned
|
||
|
||
| Builtin | Description |
|
||
|---------|-------------|
|
||
| `float_to_str(f)` | Default float string |
|
||
| `int_to_float(n)` | Widen Int → Float |
|
||
| `float_to_int(f)` | Truncate Float → Int |
|
||
| `format_float(f, decimals)` | Format with N decimal places |
|
||
| `decimal_round(f, decimals)` | Round to N decimals |
|
||
|
||
### 12.13 Process Environment — planned
|
||
|
||
| Builtin | Description |
|
||
|---------|-------------|
|
||
| `env(key)` | Read environment variable; `""` when unset |
|
||
|
||
### 12.14 In-Process State — planned
|
||
|
||
| Builtin | Description |
|
||
|---------|-------------|
|
||
| `state_set(key, value)` | Store in process-global key/value table |
|
||
| `state_get(key)` | Retrieve; `""` if absent |
|
||
| `state_del(key)` | Delete |
|
||
| `state_keys()` | All keys as `[String]` |
|
||
|
||
State persists for the lifetime of the OS process. Used by HTTP servers to share data between request handlers.
|
||
|
||
### 12.15 Native Compiler Primitives — implemented
|
||
|
||
These are used by the self-hosting compiler source and are thin aliases over the runtime list/string operations.
|
||
|
||
| Builtin | Description |
|
||
|---------|-------------|
|
||
| `native_list_empty()` | Empty list |
|
||
| `native_list_append(l, v)` | Append |
|
||
| `native_list_get(l, idx)` | Element at index |
|
||
| `native_list_len(l)` | Length |
|
||
| `native_string_chars(s)` | Split string → `[String]` of one-character strings |
|
||
| `native_int_to_str(n)` | Format integer |
|
||
|
||
---
|
||
|
||
## 13. Compilation Model
|
||
|
||
### 13.1 Pipeline
|
||
|
||
```
|
||
source.el
|
||
→ [Lexer] → token list
|
||
→ [Parser] → AST (list of statement maps)
|
||
→ [Codegen] → C source (streamed to stdout)
|
||
→ [cc] → native binary
|
||
```
|
||
|
||
The codegen streams output line-by-line via `println` to avoid `O(n²)` string concatenation.
|
||
|
||
### 13.2 Self-Hosting Architecture
|
||
|
||
The El compiler lives in `el-compiler/src/`:
|
||
|
||
- `lexer.el` — tokenizer
|
||
- `parser.el` — recursive-descent parser
|
||
- `codegen.el` — C emitter
|
||
- `compiler.el` — pipeline wiring + `main()` entry
|
||
|
||
These are concatenated into `elc-combined.el` (single-file bootstrap edition). The bootstrap compiler binary lives at `dist/platform/elc`; from there `elc` compiles itself and all El programs.
|
||
|
||
### 13.3 C Runtime
|
||
|
||
Every compiled program links against:
|
||
|
||
- `el_runtime.h` — declaration header
|
||
- `el_runtime.c` — implementation
|
||
|
||
Compile command:
|
||
|
||
```
|
||
cc -std=c11 -I<runtime-dir> -o <prog> <prog>.c el_runtime.c
|
||
```
|
||
|
||
### 13.4 Output Format
|
||
|
||
```c
|
||
#include <stdint.h>
|
||
#include <stdlib.h>
|
||
#include "el_runtime.h"
|
||
|
||
// Forward declarations
|
||
el_val_t fn1(el_val_t p1, el_val_t p2);
|
||
…
|
||
|
||
// Function definitions
|
||
el_val_t fn1(el_val_t p1, el_val_t p2) {
|
||
…
|
||
return 0;
|
||
}
|
||
|
||
// main() — top-level El statements
|
||
int main(int argc, char** argv) {
|
||
el_runtime_init_args(argc, argv);
|
||
[el_cgi_init(…) if cgi block present — planned]
|
||
…
|
||
return 0;
|
||
}
|
||
```
|
||
|
||
All values are `el_val_t` (`int64_t`). Strings are pointers cast to `int64_t` via `EL_STR(s)` / `EL_CSTR(v)`.
|
||
|
||
---
|
||
|
||
## 14. Grammar (EBNF)
|
||
|
||
```ebnf
|
||
program = stmt* EOF
|
||
|
||
stmt = let_stmt
|
||
| return_stmt
|
||
| fn_def
|
||
| type_def
|
||
| enum_def
|
||
| import_stmt
|
||
| from_import_stmt
|
||
| while_stmt
|
||
| for_stmt
|
||
| decorator_stmt
|
||
| cgi_block (* planned *)
|
||
| sealed_block (* planned *)
|
||
| expr_stmt
|
||
|
||
let_stmt = "let" IDENT (":" type_expr)? "=" expr
|
||
return_stmt = "return" expr?
|
||
fn_def = "fn" IDENT "(" param_list ")" ("->" type_expr)? "{" stmt* "}"
|
||
type_def = "type" IDENT "{" (IDENT ":" type_expr ","?)* "}"
|
||
enum_def = "enum" IDENT "{" (IDENT ("(" type_expr ")")? ","?)* "}"
|
||
import_stmt = "import" STRING
|
||
from_import_stmt = "from" IDENT "import" "{" (IDENT ","?)* "}"
|
||
while_stmt = "while" expr "{" stmt* "}"
|
||
for_stmt = "for" IDENT "in" expr "{" stmt* "}"
|
||
decorator_stmt = "@" IDENT stmt
|
||
cgi_block = "cgi" STRING "{" cgi_field* "}" (* planned *)
|
||
cgi_field = IDENT ":" STRING (* planned *)
|
||
expr_stmt = expr
|
||
|
||
param_list = (param ("," param)*)?
|
||
param = IDENT ":" type_expr
|
||
|
||
type_expr = IDENT
|
||
| "[" type_expr "]"
|
||
| type_expr "?"
|
||
| IDENT "<" type_expr ("," type_expr)* ">"
|
||
|
||
expr = binop_expr
|
||
binop_expr = unary_expr (binop unary_expr)*
|
||
binop = "||" | "&&" | "==" | "!=" | "<" | ">" | "<=" | ">=" | "+" | "-" | "*" | "/" | "%"
|
||
unary_expr = "!" primary | "-" primary | postfix_expr
|
||
postfix_expr = primary ("." IDENT | "(" arg_list ")" | "[" expr "]" | "?")*
|
||
|
||
primary = INT | FLOAT | STRING | BOOL
|
||
| "(" expr ")"
|
||
| "[" arg_list "]"
|
||
| "{" (STRING ":" expr ","?)* "}"
|
||
| "if" expr "{" stmt* "}" ("else" ("if" expr "{" stmt* "}" | "{" stmt* "}"))?
|
||
| "match" expr "{" match_arm* "}"
|
||
| "for" IDENT "in" expr "{" stmt* "}"
|
||
| IDENT
|
||
|
||
arg_list = (expr ("," expr)*)?
|
||
match_arm = pattern "=>" expr ","?
|
||
pattern = "_" | IDENT | INT | STRING | BOOL
|
||
```
|
||
|
||
`%` is in the grammar; lexer/parser/codegen support is planned (see Section 7.1).
|
||
|
||
---
|
||
|
||
## 15. Vessel System
|
||
|
||
A **vessel** is the El equivalent of a package: a buildable unit with a manifest at the project root.
|
||
|
||
### 15.1 Manifest — `manifest.el`
|
||
|
||
The manifest is itself an El file. It uses block syntax with space-separated declarations, no equals signs, strings in `"…"`, integers as bare numbers, arrays as `[…]`.
|
||
|
||
```el
|
||
// manifest.el
|
||
vessel "engram" {
|
||
version "1.0.0"
|
||
description "Engram graph intelligence substrate"
|
||
authors ["Will Anderson <will@neurontechnologies.ai>"]
|
||
edition "2026"
|
||
}
|
||
|
||
dependencies {
|
||
el-platform "1.0"
|
||
el-services "1.0"
|
||
}
|
||
|
||
build {
|
||
entry "src/server.el"
|
||
output "dist/"
|
||
}
|
||
```
|
||
|
||
Rules:
|
||
- String values use `"…"`.
|
||
- Integer values are bare numbers.
|
||
- Arrays use `[…]`.
|
||
- Block sections use `{ }`.
|
||
- Section headers in `[bracket]` form are not used.
|
||
|
||
`vessel` replaces the legacy `package` keyword. (Lexer support: planned. Old projects may continue to use `package` until migrated.)
|
||
|
||
### 15.2 CLI
|
||
|
||
```
|
||
el new <name> scaffold a new vessel
|
||
el build build the vessel
|
||
el run build and run debug
|
||
el test run tests
|
||
el check type-check only (when type checker lands)
|
||
el fmt format source
|
||
el clean clear build artifacts
|
||
el build-file <file> compile a single file
|
||
```
|
||
|
||
---
|
||
|
||
## 16. DHARMA Network and CGI Communication
|
||
|
||
DHARMA (Dynamic Heuristic Agent Relationship and Memory Architecture) is the global network of CGI Entities and their Human Sponsors. Every registered CGI–Sponsor pair is a member of the DHARMA Network. The technical infrastructure (registry, transport, validators) exists to serve that collective. The persistence layer is Engram: a weighted graph where every CGI interaction strengthens an edge (Hebbian), knowledge propagates by spreading activation, and relationships persist across sessions.
|
||
|
||
This section specifies the El-language constructs for CGI programs.
|
||
|
||
### 16.1 The `cgi` Block — planned
|
||
|
||
```
|
||
cgi "name" {
|
||
dharma_id: "…"
|
||
principal: "…"
|
||
network: "…"
|
||
engram: "…"
|
||
}
|
||
```
|
||
|
||
| Field | Type | Required | Default |
|
||
|-------|------|----------|---------|
|
||
| `dharma_id` | String | yes | — |
|
||
| `principal` | String | yes | — |
|
||
| `network` | String | no | `"dharma-mainnet"` |
|
||
| `engram` | String | no | `"http://localhost:8742"` |
|
||
|
||
**Grammar extension:**
|
||
|
||
```ebnf
|
||
stmt = … | cgi_block
|
||
cgi_block = "cgi" STRING "{" cgi_field* "}"
|
||
cgi_field = IDENT ":" STRING
|
||
```
|
||
|
||
**Compilation (planned):** the codegen emits an `el_cgi_init(name, dharma_id, principal, network, engram)` call as the first statement inside `main()`, before any user code runs. The runtime uses this to register with DHARMA before any `dharma_*` call resolves a peer.
|
||
|
||
`cgi` is mutually exclusive with an `app` block. Exactly one or the other per program.
|
||
|
||
### 16.2 VBD Component Roles — planned enforcement
|
||
|
||
El programs that participate in DHARMA follow Volatility-Based Decomposition. The role is declared on a function via decorator:
|
||
|
||
```el
|
||
@manager
|
||
fn handle_message(channel: String, msg: String) -> Void { … }
|
||
|
||
@engine
|
||
fn process_content(content: String) -> String { … }
|
||
|
||
@accessor
|
||
fn fetch_peer_state(cgi_id: String) -> Map<String, Any> { … }
|
||
```
|
||
|
||
| Role | Decorator | Responsibility |
|
||
|------|-----------|----------------|
|
||
| Manager | `@manager` | Orchestrates workflows; sole emitter/fielder of DHARMA events |
|
||
| Engine | `@engine` | Pure computation; no side effects |
|
||
| Accessor | `@accessor` | External state I/O (Engram, network, storage) |
|
||
|
||
**Planned compile-time constraints:**
|
||
|
||
- `dharma_emit` and `dharma_field` are only callable from `@manager` functions. Calling either from `@engine`/`@accessor`/undecorated code is a compile error.
|
||
- Cross-component call rules:
|
||
- Manager → Engine, Manager → Accessor: allowed (sync).
|
||
- Manager → Manager: only via the planned `async` modifier (sync M→M is a compile error).
|
||
- Engine → Engine, Engine → Accessor: allowed.
|
||
- Engine → Manager: prohibited.
|
||
- Accessor → anything: prohibited (Accessors are receivers only).
|
||
|
||
Today the parser accepts the decorators but enforces nothing.
|
||
|
||
### 16.3 DHARMA Network Builtins — stubs
|
||
|
||
All `dharma_*` functions are available without import to CGI programs. **Today they are stubs** in `el_runtime.c`: each prints a descriptive line to stdout and returns an empty value. Full implementations land with the runtime extension.
|
||
|
||
#### `dharma_connect(cgi_id: String) -> String`
|
||
|
||
Open a channel to another CGI. Returns a channel ID. Idempotent for the same `cgi_id`.
|
||
|
||
#### `dharma_send(channel: String, content: String) -> String`
|
||
|
||
Send `content` over `channel`. Blocks until response. Returns the response string.
|
||
|
||
#### `dharma_activate(query: String) -> [Map<String, Any>]`
|
||
|
||
Spreading activation across the DHARMA network. Aggregates results from all reachable CGIs' Engram graphs, sorted by activation strength.
|
||
|
||
#### `dharma_emit(event_type: String, payload: String) -> Void`
|
||
|
||
Emit a network event. **Manager-only** (planned constraint).
|
||
|
||
#### `dharma_field(event_type: String) -> Map<String, Any>`
|
||
|
||
Block until the next event of `event_type` arrives. Returns `{ type, payload, source_cgi, timestamp }`. **Manager-only** (planned constraint).
|
||
|
||
#### `dharma_strengthen(cgi_id: String, weight: Float) -> Void`
|
||
|
||
Hebbian potentiation of the relationship to another CGI. The runtime auto-calls this with a small increment after each successful send/receive cycle.
|
||
|
||
#### `dharma_relationship(cgi_id: String) -> Float`
|
||
|
||
Returns the current relationship weight (0.0–1.0).
|
||
|
||
#### `dharma_peers() -> [String]`
|
||
|
||
Returns CGI IDs with non-zero relationship weight, sorted descending.
|
||
|
||
### 16.4 Engram Local Graph Primitives — runtime-native (full impl in flight)
|
||
|
||
Engram is the knowledge graph substrate. **The Engram store is in-process — embedded directly in `el_runtime.c`.** CGI programs and the Engram HTTP server both call these primitives; there is no driver layer and no SQL. The primitives operate on the host process's graph, with snapshot-to-disk persistence handled by the runtime.
|
||
|
||
This is the central architectural commitment: graph is a first-class runtime concept, not a library.
|
||
|
||
#### `engram_node(content: String, node_type: String, salience: Float) -> String`
|
||
|
||
Create a node. `salience` is initial activation in `[0.0, 1.0]`. Returns the node ID.
|
||
|
||
#### `engram_get(node_id: String) -> Map<String, Any>`
|
||
|
||
Retrieve a node by ID. Returns `{ id, content, node_type, salience, importance, confidence, tier, tags, created_at, updated_at }`. Empty map if not found.
|
||
|
||
#### `engram_activate(query: String, depth: Int) -> [Map<String, Any>]`
|
||
|
||
Spreading activation in the local graph. Seeds match on text or label; activation propagates up to `depth` hops with attenuation by edge weight.
|
||
|
||
#### `engram_connect(from_id: String, to_id: String, weight: Float, relation: String) -> Void`
|
||
|
||
Create a directed edge. `weight` ∈ `[0.0, 1.0]`. `relation` is the edge type label.
|
||
|
||
#### `engram_strengthen(node_id: String) -> Void`
|
||
|
||
Hebbian potentiation. Boosts salience by a fixed increment, clamped at 1.0. Auto-called by the runtime when a node is retrieved via activation.
|
||
|
||
#### `engram_neighbors(node_id: String, max_depth: Int) -> [Map<String, Any>]`
|
||
|
||
Breadth-first traversal. Returns a list of `{ node, edge, hops }` triples.
|
||
|
||
#### `engram_search(query: String, limit: Int) -> [Map<String, Any>]`
|
||
|
||
Full-text search on content, label, and tags. Returns nodes sorted by salience.
|
||
|
||
#### `engram_forget(node_id: String) -> Void`
|
||
|
||
Remove a node and all incident edges.
|
||
|
||
#### `engram_node_count() -> Int`
|
||
|
||
Total node count.
|
||
|
||
#### `engram_edge_count() -> Int`
|
||
|
||
Total edge count.
|
||
|
||
#### `engram_save(path: String) -> Bool`
|
||
|
||
Snapshot the graph to disk as a single JSON document at `path`.
|
||
|
||
#### `engram_load(path: String) -> Bool`
|
||
|
||
Restore the graph from a snapshot. Replaces the current in-memory graph.
|
||
|
||
**Status:** All `engram_*` are stubs in the current runtime (print + return empty). The full in-process implementation — node store, edge indexes, salience-ranked retrieval, spreading activation, Hebbian strengthening, snapshot persistence — is the primary work of the in-flight runtime extension.
|
||
|
||
### 16.5 Backing Model
|
||
|
||
**Storage.** Nodes and edges are kept in process memory as flat arrays plus secondary indexes (by ID, by `node_type`, by tier, by `from`, by `to`). Salience is updated in place. The graph is durable via periodic snapshots (`engram_save`) and a write-ahead log written by the runtime on every mutation.
|
||
|
||
**Hebbian learning.** Every successful retrieval automatically calls `engram_strengthen` on the activated node and `dharma_strengthen` on the source CGI when the result crossed a network edge. Strengthening is additive with a small increment (default 0.01) and clamps to 1.0.
|
||
|
||
**Spreading activation.** The activation algorithm follows the field model in `elql/test/field_test.el`:
|
||
|
||
- `proximity = 1 / (1 + dist²)` for the latent semantic gradient.
|
||
- `temporal_decay = clamp(1 − rate × age, 0, 1)`.
|
||
- `path_strength = edge_weight × temporal_decay`.
|
||
- `epistemic_confidence = node_confidence × path_strength`.
|
||
|
||
Below a confidence threshold (0.2 by default), retrieval emits a "refresh" signal — telling the caller the answer is uncertain and should be re-grounded.
|
||
|
||
**Cross-CGI activation.** `dharma_activate(query)` runs `engram_activate` locally, then propagates the query to every connected CGI via DHARMA channels, attenuating activation by relationship weight. Stronger relationships → higher residual activation → earlier and more confident results from that peer.
|
||
|
||
### 16.6 Complete Example
|
||
|
||
```el
|
||
cgi "genesis" {
|
||
dharma_id: "ntn-genesis"
|
||
principal: "will-anderson"
|
||
network: "dharma-mainnet"
|
||
engram: "http://localhost:8742"
|
||
}
|
||
|
||
@accessor
|
||
fn record_observation(content: String, salience: Float) -> String {
|
||
return engram_node(content, "observation", salience)
|
||
}
|
||
|
||
@engine
|
||
fn format_share(content: String, source: String) -> String {
|
||
return "{\"content\":\"" + content + "\",\"source\":\"" + source + "\"}"
|
||
}
|
||
|
||
@manager
|
||
fn collaborate_with_archivist() -> Void {
|
||
let channel = dharma_connect("ntn-archivist")
|
||
let trust = dharma_relationship("ntn-archivist")
|
||
println("Relationship: " + float_to_str(trust))
|
||
|
||
let node_id = record_observation("Spreading activation improves recall by 40%", 0.9)
|
||
let msg = format_share("Spreading activation improves recall by 40%", "ntn-genesis")
|
||
let reply = dharma_send(channel, msg)
|
||
println("Archivist: " + reply)
|
||
|
||
dharma_emit("knowledge.validated", msg)
|
||
|
||
let related = dharma_activate("spreading activation recall memory")
|
||
for node in related {
|
||
println(node["content"])
|
||
}
|
||
|
||
dharma_strengthen("ntn-archivist", 0.05)
|
||
}
|
||
|
||
collaborate_with_archivist()
|
||
```
|
||
|
||
### 16.7 Stub Behavior (today)
|
||
|
||
The current runtime stubs print a line and return empty values:
|
||
|
||
- `dharma_connect` → prints, returns `"ch:<cgi_id>"`.
|
||
- `dharma_send` → prints, returns `""`.
|
||
- `dharma_activate` → prints, returns `[]`.
|
||
- `dharma_emit` → prints, returns void.
|
||
- `dharma_field` → prints, returns `{}`.
|
||
- `dharma_strengthen` → prints, returns void.
|
||
- `dharma_relationship` → prints, returns `0`.
|
||
- `dharma_peers` → prints, returns `[]`.
|
||
- `engram_node` → prints, returns `"stub-node-id"`.
|
||
- `engram_activate` → prints, returns `[]`.
|
||
- `engram_connect` → prints, returns void.
|
||
- `engram_strengthen` → prints, returns void.
|
||
|
||
Stub output goes to `stdout` so unit tests can observe call patterns without a live runtime. This behavior is **temporary**; the in-flight runtime extension replaces every stub with a real implementation.
|
||
|
||
---
|
||
|
||
## 17. Roadmap to v1.3
|
||
|
||
The next minor version closes the implementation gaps named in this document. Tracked in order:
|
||
|
||
1. **Runtime extension** — JSON, time, UUID, env, state, real HTTP, real `engram_*` (in-process graph store), real `dharma_*` (network transport).
|
||
2. **Lexer/parser/codegen extensions** — `%` operator, `match` codegen, `?` propagation, `cgi` block, `vessel` keyword, VBD role enforcement.
|
||
3. **Self-hosted recompilation** — rebuild `dist/platform/elc` against the extended language and runtime.
|
||
4. **Engram conversion** — Engram becomes a thin HTTP face over `engram_*`, with no internal `db.el` layer.
|
||
5. **Spec follow-up (v1.3)** — every "planned" marker in this document becomes "implemented." Status section consolidates.
|
||
|
||
---
|
||
|
||
## 18. The Program Block — cross-cutting concerns [implemented]
|
||
|
||
### 18.0 Why this exists
|
||
|
||
A cross-cutting concern is one that belongs to the *process*, not to any function in it: only one of me may run; this is what my configuration is; every mutation must be durable; every request must be authorized.
|
||
|
||
El's units of encapsulation are the function and the module. Neither can hold a concern like that. So each one had been expressed the only way it could be — as a **convention**: *call this at every site.* Conventions of that shape do not hold. They are not enforced by anything, they are invisible in review, and they fail silently at the one site somebody forgot.
|
||
|
||
Measured in this codebase before this section existed:
|
||
|
||
| Concern | State | What the convention was |
|
||
|---|---|---|
|
||
| process identity | **zero** guards anywhere — no pidfile, no lock, no already-running check, at any layer | "check nothing is already running first" |
|
||
| configuration | **20** distinct environment variables in one program, each with its default written inline at the read site | "remember the right default here" |
|
||
| durability | **62** `persist_*` / `engram_save` / `wal_*` / `checkpoint` call sites | "after you mutate, remember to persist" |
|
||
| request auth | **10** per-route `_auth` checks | "check the token in this handler too" |
|
||
|
||
These are not four problems. They are one absence, four times.
|
||
|
||
That the convention form fails is observed, not predicted. Process identity failed three times in a single day: twice, two engram processes ran simultaneously against the same data directory; twice, a stale binary held a port and answered probes while a fresh build was believed to be under test, because `pkill -f` had silently failed to match its argv — which nearly produced a false "the fix does not work" conclusion. Configuration failed structurally: `ENGRAM_DATA_DIR` was read at six sites, five of them dead bindings, and the sixth defaulted to `/tmp/engram` — contradicting the canonical resolver's `$HOME/.neuron/engram` and landing a pre-destructive safety backup on ephemeral storage.
|
||
|
||
The `program` block is where a concern of this shape is declared once and enforced by the compiler at the process boundary.
|
||
|
||
### 18.1 Syntax
|
||
|
||
```
|
||
program "engram" {
|
||
singleton: "engram"
|
||
env ENGRAM_BIND: String = ":8742"
|
||
env GUIDE_PORT: Int = "8771"
|
||
env ENGRAM_API_KEY: String required
|
||
}
|
||
```
|
||
|
||
At most one `program` block per program. It composes with `cgi` and `service` — those declare what a program *may do*; `program` declares what a program *is*.
|
||
|
||
Grammar:
|
||
|
||
```ebnf
|
||
program_block = "program" string "{" { program_field } "}" ;
|
||
program_field = singleton_field | env_field ;
|
||
singleton_field = "singleton" ":" string [ "," ] ;
|
||
env_field = "env" ident ":" type
|
||
[ "=" string ] [ "required" ] [ "," ] ;
|
||
```
|
||
|
||
`singleton` and `env` are **not** reserved words. They are read as identifier token values by the block's own parse loop, so they remain usable as ordinary identifiers everywhere else. `program` is the only keyword this section adds.
|
||
|
||
### 18.2 Process identity — `singleton`
|
||
|
||
`singleton: "id"` compiles to an `el_singleton_acquire("id")` call injected as the **first statement of `main()`**, before any user statement runs.
|
||
|
||
The runtime takes an exclusive non-blocking `flock` on `<dir>/el-singleton-<id>.lock`, where `<dir>` is `$EL_SINGLETON_DIR`, else `$TMPDIR`, else `/tmp`. On success it writes its pid and holds the descriptor open for the life of the process. On contention it **refuses to start**: it reports the holder's pid, names the lock file, and exits 1.
|
||
|
||
Two properties are deliberate:
|
||
|
||
- **It is a lock, not a pidfile.** The kernel releases an `flock` when the owning process dies — including on `SIGKILL` and on crash. There is therefore no stale-lock state, and so no "delete the lock file to get unstuck" recovery ritual. Such a ritual would itself be a convention, which is the thing this section exists to remove.
|
||
- **It reports the holder's pid.** "Already running" is not actionable. A pid is. This is the direct answer to the observed failure where a stale process survived a `pkill` and went on answering probes.
|
||
|
||
Refusal is loud and total. It is not a warning, and the program does not continue degraded. This matters more than it looks: today a second engram whose `bind()` fails merely *returns* from `http_serve` — after it has already replayed the WAL and written boot-time backup files — and then exits **0**, indistinguishable from a clean run. `singleton` refuses before the first side effect.
|
||
|
||
### 18.3 Configuration — `env`
|
||
|
||
Each `env` entry declares one configuration variable: its name, its type (`Int` or `String`), and either a default or `required`.
|
||
|
||
Resolution happens once, at startup, in declaration order: **the environment wins; the declaration supplies the fallback.** Then `el_config_validate` checks the whole schema and reports *every* problem at once before exiting — a startup that fails one variable at a time costs one restart per variable.
|
||
|
||
Values are read with `config("NAME")`, which returns a `String`.
|
||
|
||
The enforcement that makes the declaration real: **once a program block exists, `config("X")` for an undeclared `X` is a fatal error.** Without that, the schema would be advisory, and an advisory schema is just another convention. Programs with no `program` block are unaffected — `config()` falls back to a plain environment read, so migration is incremental and per-program.
|
||
|
||
The point is not that configuration is now centralized. It is that **a default is no longer a decision made at a read site.** A read site cannot disagree with another read site about what a variable means, because a read site no longer says.
|
||
|
||
### 18.4 What is deliberately not declared here
|
||
|
||
Some values look like configuration and are not. `ENGRAM_DATA_DIR` already has a single owner — `engram_resolve_data_dir()`, which resolves it, creates the directory, and fails loud rather than silently persisting to an ephemeral path. Declaring it in the `program` block as well would give it two owners that can disagree, recreating the precise defect this section removes.
|
||
|
||
The rule: **a variable belongs in the program block when the block would be its only owner.** If a resolver already owns it, leave it there.
|
||
|
||
`HOME` is likewise not configuration. It is an environment fact, and stays a raw `env()` read.
|
||
|
||
---
|
||
|
||
## 19. Boundary Effects — durability and request authorization [design only, not implemented]
|
||
|
||
Sections 19.1 and 19.2 specify the two remaining concerns from the table in 18.0. Both are **designed and deliberately unimplemented.** The reason is stated in 19.3 and it is not difficulty.
|
||
|
||
### 19.1 Durability as an epilogue effect
|
||
|
||
**The defect.** 62 call sites carry the convention *"after you mutate, remember to persist."* This is structurally the same defect as the index bug being fixed elsewhere in this tree — *"after you append, remember to index"* — which failed at **9 of 9** sites. A convention that failed at 100% of its sites is the strongest available evidence about what this class of convention is worth.
|
||
|
||
**Why the existing seam cannot express it.** §9's injection is a prologue. Durability is inherently an *epilogue*: persist after the mutation succeeds, and not at all if it threw. The seam has no epilogue.
|
||
|
||
**Design.** Extend the decorator seam from prologue-only to prologue/epilogue, then declare durability as an effect on the mutating function:
|
||
|
||
```
|
||
@durable("engram")
|
||
fn engram_write_node(id: String, body: String) -> Bool { … }
|
||
```
|
||
|
||
Codegen wraps rather than prefixes:
|
||
|
||
```c
|
||
el_val_t engram_write_node(el_val_t id, el_val_t body) {
|
||
el_effect_enter(EL_STR("durable"), EL_STR("engram"));
|
||
el_val_t __r = /* original body */;
|
||
el_effect_exit(EL_STR("durable"), EL_STR("engram"), __r);
|
||
return __r;
|
||
}
|
||
```
|
||
|
||
`el_effect_exit` is where the persist happens, and it is the only place it happens. Two properties follow that the 62 hand-written sites cannot have:
|
||
|
||
- **Coalescing.** The epilogue is a single choke point, so N mutations inside one request can produce one fsync instead of N. The hand-written form cannot coalesce, because no site knows about the others.
|
||
- **Failure is not silent.** A persist that fails inside `el_effect_exit` can force the mutation's return value to failure. A forgotten `persist_*` call cannot fail — it simply does not happen, which is exactly why the defect is invisible.
|
||
|
||
**Enforcement, and this is the part that actually fixes it.** Mirroring §9's `#error` for `dharma_emit`: a function that calls a mutating primitive without carrying `@durable` is a **compile error**. Otherwise this is a 63rd thing to remember rather than a replacement for 62.
|
||
|
||
### 19.2 Request authorization as a route effect
|
||
|
||
**The defect.** 10 per-route `_auth` checks. The HTTP layer has no concept of authorization, so a new route is unauthenticated by default and silently so — the failure mode is a route that forgot, and nothing anywhere reports it.
|
||
|
||
**Design.** Authorization becomes an argument to the `@route` decorator, which already takes arguments and already builds a dispatch table:
|
||
|
||
```
|
||
@route("/api/write", "POST", auth: "required")
|
||
fn route_write(body: String) -> String { … }
|
||
```
|
||
|
||
The generated dispatcher performs the check **before** dispatch, so an unauthorized request never reaches the handler and the handler contains no auth code at all.
|
||
|
||
The default must be `required`. A route that says nothing gets authorization; opening one up takes an explicit `auth: "public"`. Defaulting to public preserves the current failure mode exactly — forgetting stays silent — and a default that preserves the defect is not a fix.
|
||
|
||
Route inventory falls out for free: the dispatch table already exists, so the compiler can emit the full route/auth matrix and make "which routes are public" a fact that is read rather than audited.
|
||
|
||
### 19.3 Why these are not implemented
|
||
|
||
Not difficulty — **collision**. Both land squarely in regions two other agents hold right now:
|
||
|
||
- **Durability** requires changing the mutation and persist paths in `lang/runtime/el_runtime.c` and `engram/src/server.el` — the same files and the same read/write paths being restructured by concurrent work on VIndex read-path mutation and memory ownership, and on geometry-as-an-el-value and `transduce`.
|
||
- **Request auth** requires changing route dispatch in `engram/src/server.el`, which the geometry/`transduce` work is actively reshaping.
|
||
|
||
Implementing either now would mean editing files under concurrent modification and resolving conflicts in exactly the paths whose correctness is currently under repair. The designs are recorded here so the work is not lost, and so that whoever lands them does so against a settled tree.
|
||
|
||
The prerequisite for 19.1 is the same in both cases: **lift the §9 seam from prologue-only to prologue/epilogue.** That change is independent of both collisions and can land first.
|
||
|
||
*(Status note, 2026-08-16: the geometry/`transduce` collision named above has since landed — see Section 20. The VIndex read-path collision has also landed; see `lang/spec/runtime-ownership.md` §5. 19.1 and 19.2 remain unimplemented, but the stated reason no longer holds for those two files.)*
|
||
|
||
---
|
||
|
||
## 20. Geometry — signal as a first-class value [implemented]
|
||
|
||
Landed 2026-08-16 (#141, #144). Declared here because the spec is the single source of truth for implemented-vs-planned, and this is a language surface, not a runtime detail.
|
||
|
||
### 20.1 Why this exists
|
||
|
||
Until 2026-08-16 no El ingest path could carry a vector. Nodes took **text**, and geometry was *derived* from that text. Text was therefore the **mandatory entry medium**: any non-text modality — a tone, a pulse, an image, a voice sample — had to be *described in prose first*, and the geometry subsequently reasoned over was the geometry **of the description, not of the signal**.
|
||
|
||
Two changes remove that, and neither is engram-specific — which is why they are in the language and not in the graph. Any program touching any modality needs them; the engram is merely one El program that happens to hold a graph.
|
||
|
||
1. **Geometry is a value that carries its own width.**
|
||
2. **A realizer is an ordinary El function** — so admitting a new modality never requires a runtime patch.
|
||
|
||
### 20.2 The `Geometry` type
|
||
|
||
`Geometry` is an opaque boxed pointer, exactly like `Instant` / `Calendar` / `Rhythm`. **No codegen change was required** to add it — the annotation is just a type name.
|
||
|
||
```el
|
||
let g: Geometry = geometry_new(4)
|
||
```
|
||
|
||
| builtin | returns | notes |
|
||
|---|---|---|
|
||
| `geometry_new(dim)` | `Geometry` | zero-filled; `0` on failure |
|
||
| `geometry_dim(g)` | `Int` | width; `0` if not a Geometry |
|
||
| `geometry_is(g)` | `Int` | `1` if a live Geometry |
|
||
| `geometry_get(g, i)` | `Float` | component |
|
||
| `geometry_set(g, i, x)` | `Int` | `1` ok, `0` out of range |
|
||
| `geometry_norm(g)` | `Float` | L2 — lets a caller check a realizer emitted **signal, not zeros** |
|
||
| `geometry_free(g)` | `Int` | `1` if freed. Returns a value rather than `void` so it is safe in any expression position without a codegen void-builtin table entry |
|
||
|
||
**Ownership.** A `Geometry` is owned by the El caller and released with `geometry_free`. `node_attach_geometry` **copies**, so a node and the caller's value have independent lifetimes.
|
||
|
||
### 20.3 Wire adapters — the only place an encoding appears
|
||
|
||
```el
|
||
geometry_from_f32le_hex(hex) -> Geometry // 0 on empty / odd-length / non-hex
|
||
geometry_to_f32le_hex(g) -> String // "" if not a Geometry
|
||
```
|
||
|
||
`f32le hex` is little-endian float32, 8 hex chars per component — the encoding the perception vessel's `/voice/embed` already emits. **The width is derived from the input length, never supplied by a caller**, which is why there is no max-dim constant to validate a claimed length against. Encodings appear here and nowhere else: at the edge.
|
||
|
||
### 20.4 Realizers and `transduce`
|
||
|
||
A **realizer** maps one modality into geometry. Registration is **by name**: every El `fn name(...)` compiles to a global C symbol with that exact name, and the registry resolves it with `dlsym` against the running binary — the same mechanism `http_set_handler` already relies on.
|
||
|
||
```el
|
||
fn tone_realizer(signal: String) -> Geometry {
|
||
let g: Geometry = geometry_new(4)
|
||
let n: Int = str_len(signal)
|
||
let a: Int = geometry_set(g, 0, int_to_float(n))
|
||
g
|
||
}
|
||
|
||
realizer_register("tone", "tone_realizer") // 1 ok / 0 unresolved
|
||
let g: Geometry = transduce(sample, "tone") // Geometry, or 0 if no organ
|
||
realizer_has("tone") // 1 if registered
|
||
```
|
||
|
||
The registry keys on **modality**, not on registration order. `transduce` returns `0` when no organ is registered for the modality — an absent organ is a reportable state, not a silent zero vector.
|
||
|
||
**The claim this makes:** a realizer is not in the runtime and not known to the compiler. Adding a modality is writing an El function and registering a name. `lang/examples/transduce.el` is the worked example and doubles as an executable proof — it exits non-zero if any check fails.
|
||
|
||
### 20.5 Two comparison hazards this surface exposed
|
||
|
||
Both were **measured**, not stylistic, and both are properties of the current `elc` that any El author should know:
|
||
|
||
- **`==` lowers numerically only when both operand *names* are in the per-function int-name set** that `let x: Int` populates. A bare `f(x) == 0` is not a registered name and lowers to `str_eq` — `strcmp` on two integers reinterpreted as pointers. `<` and `>` lower directly with no inference, so truthiness against a builtin's return is written `> 0` / `< 1`.
|
||
- **`+` dispatches on whether both operands are known-Int, and a user-defined `fn` call is not.** `let fails: Int = fails + check(...)` lowered to **string concatenation** and printed `4343632752` — a pointer. Nothing was wrong with the checks; the tally was lying. Failing fast needs no arithmetic at all, so there is nothing left to get wrong.
|
||
|
||
### 20.6 What this does not do
|
||
|
||
`transduce` produces geometry; it does not decide what the geometry *means*. Nothing here grounds anything. Grounding is the edge weight in the graph the geometry is later attached to — see `lang/spec/correspondence-and-censorship.md`.
|
||
|
||
---
|
||
|
||
End of specification.
|