Archive Rust bootstrap — El compiler is now self-hosting
This commit is contained in:
Will Anderson
+427
@@ -0,0 +1,427 @@
|
||||
# ELVM — El Virtual Machine Specification
|
||||
|
||||
Version 1.0.0 — April 29, 2026
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
ELVM is the execution substrate for El programs. It is a stack-based virtual machine that executes JSON-serialized bytecode instructions stored in a typed binary container format. ELVM has four defining properties:
|
||||
|
||||
1. **Binary container with semantic header.** The `.elc` file format begins with a 4-byte magic identifier (`ELVM`), a 4-byte version field, and an 8-byte payload length — making every artifact self-describing and parseable without external schema.
|
||||
|
||||
2. **JSON-serialized instruction encoding.** The bytecode payload is a JSON array of instruction objects. Each instruction is human-inspectable with standard JSON tools, language-portable, and schema-free. The JSON encoding is the canonical on-disk format; no separate binary encoding exists.
|
||||
|
||||
3. **Builtin dispatch architecture.** Function calls resolve first against a runtime builtin dispatch table, then against the user-defined function table. New builtins are added to the dispatch table without recompiling existing programs. The execution model treats builtins and user functions identically from the instruction side.
|
||||
|
||||
4. **Self-hosting bootstrap.** A Rust genesis compiler produces ELVM bytecode. That bytecode, when executed by ELVM, runs the El compiler written in El, which produces ELVM bytecode for all subsequent compilation. The VM is the only persistent execution engine.
|
||||
|
||||
---
|
||||
|
||||
## 1. Binary Container Format
|
||||
|
||||
### 1.1 Header
|
||||
|
||||
Every `.elc` file begins with a 16-byte header:
|
||||
|
||||
```
|
||||
Offset Size Type Field
|
||||
────── ───── ──────────────── ──────────────────────────────────────────
|
||||
0 4 [u8; 4] Magic: b"ELVM"
|
||||
4 4 u32 little-endian Version: currently 1
|
||||
8 8 u64 little-endian Payload length in bytes
|
||||
```
|
||||
|
||||
### 1.2 Payload
|
||||
|
||||
Immediately following the 16-byte header is the payload: a UTF-8 JSON array encoding a `Vec<Bytecode>` instruction sequence.
|
||||
|
||||
```
|
||||
[instruction, instruction, ...]
|
||||
```
|
||||
|
||||
The payload length field in the header equals the exact byte count of the JSON payload. Bytes beyond `header_size + payload_length` are ignored (reserved for future trailer use).
|
||||
|
||||
### 1.3 Legacy Format
|
||||
|
||||
Files without the `ELVM` magic header are treated as raw JSON bytecode (no header). This is the legacy format produced by early compiler versions. The VM accepts both formats via the `deserialize_all` entry point.
|
||||
|
||||
### 1.4 Identification
|
||||
|
||||
The magic bytes `ELVM` (hex: `45 4C 56 4D`) uniquely identify El bytecode containers. Any tool that encounters these bytes at offset 0 may assume the 16-byte header format described above.
|
||||
|
||||
---
|
||||
|
||||
## 2. Value Types
|
||||
|
||||
The VM stack holds `Value` instances of the following types:
|
||||
|
||||
| Variant | Rust encoding | Description |
|
||||
|---------|--------------|-------------|
|
||||
| `Int(i64)` | 64-bit signed integer | All integer arithmetic |
|
||||
| `Float(f64)` | IEEE 754 double | Floating-point arithmetic |
|
||||
| `Str(String)` | UTF-8 string | Text and identifiers |
|
||||
| `Bool(bool)` | Boolean | Logical conditions |
|
||||
| `Nil` | Unit | Absent value, uninitialized |
|
||||
| `List(Vec<Value>)` | Ordered sequence | Arrays, activation results |
|
||||
| `Map(Vec<(String, Value)>)` | Ordered key-value pairs | Struct instances, JSON objects |
|
||||
| `ResultOk(Box<Value>)` | Success variant | Fallible operation success |
|
||||
| `ResultErr(Box<Value>)` | Error variant | Fallible operation failure |
|
||||
| `Struct { type_name, fields }` | Named struct | Typed record with field list |
|
||||
|
||||
Maps use `Vec<(String, Value)>` rather than a hash map to preserve insertion order and remain serialization-friendly.
|
||||
|
||||
---
|
||||
|
||||
## 3. Instruction Set
|
||||
|
||||
### 3.1 JSON Encoding
|
||||
|
||||
Each instruction serializes as either a JSON string (no-argument instructions) or a JSON object (instructions with arguments):
|
||||
|
||||
```json
|
||||
"Halt"
|
||||
"Pop"
|
||||
{"Push": {"Int": 42}}
|
||||
{"Push": {"Str": "hello"}}
|
||||
{"Push": "Nil"}
|
||||
{"LoadLocal": "x"}
|
||||
{"StoreLocal": "x"}
|
||||
{"Call": {"name": "println", "arity": 1}}
|
||||
{"Jump": 3}
|
||||
{"JumpIfNot": -2}
|
||||
```
|
||||
|
||||
### 3.2 Stack Instructions
|
||||
|
||||
| Instruction | JSON | Stack effect | Description |
|
||||
|-------------|------|-------------|-------------|
|
||||
| `Push(v)` | `{"Push": <value>}` | `-- v` | Push constant value |
|
||||
| `Pop` | `"Pop"` | `v --` | Discard top of stack |
|
||||
| `Dup` | `"Dup"` | `v -- v v` | Duplicate top of stack |
|
||||
|
||||
### 3.3 Arithmetic Instructions
|
||||
|
||||
All arithmetic instructions pop two operands and push the result. For binary operations, operands are popped as `(b, a)` — `b` is popped first (top of stack), `a` is popped second. The operation computes `a OP b`.
|
||||
|
||||
| Instruction | JSON | Description |
|
||||
|-------------|------|-------------|
|
||||
| `Add` | `"Add"` | `a + b` (Int/Float/Str concatenation) |
|
||||
| `Sub` | `"Sub"` | `a - b` |
|
||||
| `Mul` | `"Mul"` | `a * b` |
|
||||
| `Div` | `"Div"` | `a / b` (Int: integer division; division by zero returns Nil) |
|
||||
| `Mod` | `"Mod"` | `a % b` |
|
||||
| `BitAnd` | `"BitAnd"` | `a & b` (Int only) |
|
||||
| `BitOr` | `"BitOr"` | `a \| b` (Int only) |
|
||||
| `BitXor` | `"BitXor"` | `a ^ b` (Int only) |
|
||||
| `BitNot` | `"BitNot"` | `~a` (Int only, unary) |
|
||||
| `Shl` | `"Shl"` | `a << b` (Int only) |
|
||||
| `Shr` | `"Shr"` | `a >> b` (Int only) |
|
||||
|
||||
**Type promotion:** When one operand is `Int` and the other is `Float`, the `Int` is widened to `Float` before the operation.
|
||||
|
||||
### 3.4 Comparison Instructions
|
||||
|
||||
Each pops `(b, a)` and pushes `Bool`:
|
||||
|
||||
| Instruction | JSON | Description |
|
||||
|-------------|------|-------------|
|
||||
| `Eq` | `"Eq"` | `a == b` |
|
||||
| `NotEq` | `"NotEq"` | `a != b` |
|
||||
| `Lt` | `"Lt"` | `a < b` |
|
||||
| `Gt` | `"Gt"` | `a > b` |
|
||||
| `LtEq` | `"LtEq"` | `a <= b` |
|
||||
| `GtEq` | `"GtEq"` | `a >= b` |
|
||||
|
||||
**Ordering semantics:** Numeric types use natural ordering. Strings use lexicographic ordering. Mixed numeric types promote Int to Float. All other mixed-type comparisons return `Equal` (neither less nor greater).
|
||||
|
||||
### 3.5 Logical Instructions
|
||||
|
||||
| Instruction | JSON | Description |
|
||||
|-------------|------|-------------|
|
||||
| `And` | `"And"` | Pop `(b, a)`, push `Bool(a && b)` |
|
||||
| `Or` | `"Or"` | Pop `(b, a)`, push `Bool(a \|\| b)` |
|
||||
| `Not` | `"Not"` | Pop `a`, push `Bool(!a)` |
|
||||
|
||||
Logical `And` and `Or` do not short-circuit — both operands are always evaluated before the instruction executes.
|
||||
|
||||
### 3.6 Local Variable Instructions
|
||||
|
||||
| Instruction | JSON | Description |
|
||||
|-------------|------|-------------|
|
||||
| `LoadLocal(name)` | `{"LoadLocal": "name"}` | Push value of local variable; pushes `Nil` if not set |
|
||||
| `StoreLocal(name)` | `{"StoreLocal": "name"}` | Pop top of stack into local variable |
|
||||
|
||||
Local variables are stored in a `HashMap<String, Value>` per call frame. The `StoreLocal` instruction overwrites the slot unconditionally, implementing El's variable shadowing/mutation model.
|
||||
|
||||
### 3.7 Function Call Instructions
|
||||
|
||||
| Instruction | JSON | Description |
|
||||
|-------------|------|-------------|
|
||||
| `Call { name, arity }` | `{"Call": {"name": "fn", "arity": 2}}` | Call function with `arity` arguments |
|
||||
| `Return` | `"Return"` | Return from current frame |
|
||||
|
||||
**Call dispatch sequence:**
|
||||
|
||||
1. The `Call` instruction attempts `builtins::dispatch(name, arity, stack, args, global_state)`.
|
||||
2. If the builtin dispatcher returns `NotBuiltin`, the VM looks up `name` in the function table.
|
||||
3. If found, the VM saves the current frame (return IP + locals snapshot) to the call stack and sets IP to the function's entry point.
|
||||
4. If not found, `Nil` is pushed (graceful failure).
|
||||
|
||||
**Function table construction:** Before execution begins, the VM scans the entire instruction array for `Push(Int(n))` followed immediately by `StoreLocal("__fn_name")`. Each such pair registers function `name` at entry IP `n`. This is the function registration protocol that the codegen emits for every `fn` definition.
|
||||
|
||||
**Return protocol:** `Return` pops the top call frame, restores `saved_locals`, and sets IP to `return_ip`. The return value remains on the stack.
|
||||
|
||||
### 3.8 Control Flow Instructions
|
||||
|
||||
All jump offsets are signed 32-bit integers relative to the instruction *after* the jump instruction (i.e., the new IP = current IP + 1 + offset).
|
||||
|
||||
| Instruction | JSON | Description |
|
||||
|-------------|------|-------------|
|
||||
| `Jump(offset)` | `{"Jump": 3}` | Unconditional jump |
|
||||
| `JumpIf(offset)` | `{"JumpIf": 2}` | Jump if top of stack is `Bool(true)`; pops the value |
|
||||
| `JumpIfNot(offset)` | `{"JumpIfNot": -5}` | Jump if top of stack is not `Bool(true)`; pops the value |
|
||||
|
||||
A positive offset jumps forward; a negative offset jumps backward. An offset of `0` jumps to the instruction immediately following the jump (a no-op equivalent).
|
||||
|
||||
### 3.9 Field and Indexing Instructions
|
||||
|
||||
| Instruction | JSON | Description |
|
||||
|-------------|------|-------------|
|
||||
| `GetField(name)` | `{"GetField": "field"}` | Pop Map or Struct; push value of named field (or Nil) |
|
||||
| `SetField(name)` | `{"SetField": "field"}` | Pop value, update named field in Map/Struct on top of stack |
|
||||
| `GetIndex` | `"GetIndex"` | Pop index, pop container; push element at index (or Nil) |
|
||||
| `BuildMap(n)` | `{"BuildMap": 3}` | Pop 2n values (alternating key, value) and build a Map |
|
||||
| `BuildList(n)` | `{"BuildList": 4}` | Pop n values and build a List |
|
||||
| `BuildStruct { type_name, fields }` | `{"BuildStruct": {"type_name": "T", "fields": ["a","b"]}}` | Pop n values, build Struct |
|
||||
|
||||
**BuildMap protocol:** The code generator pushes keys and values interleaved: `Push(Str(k1)), Push(v1), Push(Str(k2)), Push(v2), ..., BuildMap(n)`. The `BuildMap` instruction pops `2n` values in reverse order and assembles the pairs, then reverses to restore insertion order.
|
||||
|
||||
**GetIndex on strings:** When the container is a `Str`, `GetIndex` with an integer index returns the nth character as a single-character `Str`.
|
||||
|
||||
### 3.10 Special Instructions
|
||||
|
||||
| Instruction | JSON | Description |
|
||||
|-------------|------|-------------|
|
||||
| `Activate { type_name, query }` | `{"Activate": {"type_name": "User", "query": "..."}}` | Semantic graph search; pushes `List` of results |
|
||||
| `SealedBegin` | `"SealedBegin"` | Mark start of sealed region |
|
||||
| `SealedEnd` | `"SealedEnd"` | Mark end of sealed region |
|
||||
| `Reason { query }` | `{"Reason": {"query": "..."}}` | LLM inference call; pushes `Str` response |
|
||||
| `Parallel { entries }` | `{"Parallel": {"entries": [["name", ip]]}}` | Spawn parallel sub-VMs; pushes `Map` of results |
|
||||
| `TraceBegin { label }` | `{"TraceBegin": {"label": "..."}}` | Record start time for trace region |
|
||||
| `TraceEnd { label }` | `{"TraceEnd": {"label": "..."}}` | Print elapsed ms for trace region |
|
||||
| `ContractCheck { message }` | `{"ContractCheck": {"message": "..."}}` | Assert top of stack is true; exit(1) if not |
|
||||
| `DeployFn { fn_name, route, target }` | `{"DeployFn": {"fn_name":"...", "route":"...", "target":"..."}}` | POST function to deployment API |
|
||||
| `Nop` | `"Nop"` | No operation |
|
||||
| `Halt` | `"Halt"` | Halt execution |
|
||||
|
||||
---
|
||||
|
||||
## 4. Execution Model
|
||||
|
||||
### 4.1 Execution State
|
||||
|
||||
The VM maintains the following state during execution:
|
||||
|
||||
- **Instruction pointer (IP):** Integer index into the flat instruction array.
|
||||
- **Value stack:** `Vec<Value>`, grows upward.
|
||||
- **Locals:** `HashMap<String, Value>` for the current call frame.
|
||||
- **Call stack:** `Vec<CallFrame>`, each frame holding `{ return_ip: usize, saved_locals: HashMap }`.
|
||||
- **Global state:** `HashMap<String, String>` shared across the entire program lifetime (accessible via `state_set`/`state_get` builtins).
|
||||
- **Function table:** `HashMap<String, usize>` mapping function names to entry IPs (populated by the pre-execution scan pass).
|
||||
|
||||
### 4.2 Execution Loop
|
||||
|
||||
1. Load instruction at IP.
|
||||
2. Dispatch on instruction variant.
|
||||
3. Increment IP (unless the instruction set IP explicitly via a jump or call).
|
||||
4. Repeat until `Halt` or `Return` with empty call stack.
|
||||
|
||||
### 4.3 Call Frame Protocol
|
||||
|
||||
When `Call` dispatches to a user-defined function:
|
||||
|
||||
1. Save `(ip + 1, locals.clone())` as a `CallFrame` onto the call stack.
|
||||
2. Clear locals (the new frame starts empty; parameters are passed via the stack and stored by the function's prologue).
|
||||
3. Set IP to the function's entry point.
|
||||
|
||||
When `Return` executes:
|
||||
|
||||
1. Pop the top `CallFrame`.
|
||||
2. Restore `locals` from `saved_locals`.
|
||||
3. Set IP to `return_ip`.
|
||||
4. The return value remains on the value stack.
|
||||
|
||||
### 4.4 Function Prologue Protocol
|
||||
|
||||
The codegen emits function prologues as a sequence of `StoreLocal` instructions in reverse parameter order. Because arguments are pushed left-to-right by the caller, and the stack is LIFO, storing in reverse order assigns parameters to their correct names:
|
||||
|
||||
```
|
||||
// fn f(a: Int, b: Int)
|
||||
// Caller pushes: a, b (b is on top)
|
||||
StoreLocal("b") // pops b
|
||||
StoreLocal("a") // pops a
|
||||
```
|
||||
|
||||
### 4.5 Higher-Order Function Protocol
|
||||
|
||||
The VM implements three higher-order function helpers that are resolved by name before the function table:
|
||||
|
||||
| Name | Behavior |
|
||||
|------|---------|
|
||||
| `list_map` | Pops `fn_name: Str` and `list: List`; calls `fn_name` on each element; pushes result `List` |
|
||||
| `list_filter` | Pops `fn_name: Str` and `list: List`; calls `fn_name` on each element; pushes filtered `List` |
|
||||
| `list_reduce` | Pops `fn_name: Str`, `init: Value`, and `list: List`; folds; pushes accumulated value |
|
||||
|
||||
### 4.6 Parallel Execution
|
||||
|
||||
The `Parallel` instruction spawns one thread per entry using `std::thread::spawn`. Each thread runs a sub-VM instance starting at the specified entry IP. All threads complete before the parent continues; results are collected into a `Map` keyed by entry name.
|
||||
|
||||
---
|
||||
|
||||
## 5. Builtin Dispatch
|
||||
|
||||
### 5.1 Architecture
|
||||
|
||||
The builtin dispatch system is the primary extension point of the ELVM. When a `Call` instruction is encountered:
|
||||
|
||||
1. The VM calls `builtins::dispatch(name, arity, stack, program_args, global_state)`.
|
||||
2. The dispatcher pattern-matches on `name` to identify the builtin.
|
||||
3. If matched, it pops `arity` arguments from the stack, performs the operation, and pushes the result.
|
||||
4. It returns one of: `Handled`, `HttpServe`, `Exit(code)`, or `NotBuiltin`.
|
||||
|
||||
### 5.2 Dispatch Result
|
||||
|
||||
| Variant | Meaning |
|
||||
|---------|---------|
|
||||
| `Handled` | Builtin executed successfully |
|
||||
| `HttpServe` | Builtin started an HTTP server (non-blocking from VM perspective) |
|
||||
| `Exit(code)` | Builtin called for process exit |
|
||||
| `NotBuiltin` | Name not recognized; VM should continue to user function table |
|
||||
|
||||
### 5.3 Builtin Categories
|
||||
|
||||
The dispatch table handles all builtins described in the El language specification (Section 14). All builtins are registered by name string with no external configuration. Adding a new builtin requires only adding a match arm to the dispatch function.
|
||||
|
||||
### 5.4 Engram Integration Builtins
|
||||
|
||||
The `Activate` instruction dispatches to `engram_activate_search(type_name, query)`, which performs an HTTP POST to `ENGRAM_URL/search` with a JSON body `{ "query": query, "limit": 20 }`. Results are deserialized into a `List` of `Value` instances.
|
||||
|
||||
The `Reason` instruction dispatches to `soma_reason(query)`, which performs an HTTP POST to `SOMA_URL/v1/chat/completions` with the query as the user message.
|
||||
|
||||
Both external calls are blocking and synchronous.
|
||||
|
||||
---
|
||||
|
||||
## 6. Memory Model
|
||||
|
||||
### 6.1 Locals
|
||||
|
||||
Local variables exist per call frame as a `HashMap<String, Value>`. Frame creation copies the parent's locals map (snapshot semantics). Modifications in a child frame do not affect the parent; the parent's state is restored from the saved snapshot when the child returns.
|
||||
|
||||
### 6.2 Global State
|
||||
|
||||
The `global_state` map (`HashMap<String, String>`) persists for the entire process lifetime and is accessible from any point in program execution. It is shared by all call frames and all threads in a parallel execution. Access is mediated by the `state_set`, `state_get`, and `state_delete` builtins.
|
||||
|
||||
### 6.3 Value Semantics
|
||||
|
||||
All values are cloned on push, pop, and frame save. There is no aliasing — mutating a `List` or `Map` value does not affect other bindings to the same original value.
|
||||
|
||||
### 6.4 Stack Discipline
|
||||
|
||||
The stack is unbounded. Stack underflow (pop from empty stack) returns `Nil` rather than panicking. This preserves program continuity in the face of codegen anomalies but may silently propagate type errors.
|
||||
|
||||
---
|
||||
|
||||
## 7. Error Handling
|
||||
|
||||
### 7.1 Runtime Error Policy
|
||||
|
||||
ELVM does not have a trap mechanism for arithmetic errors. Division by zero returns `Nil`. Out-of-bounds array access returns `Nil`. Missing map keys return `Nil`. Missing local variables return `Nil`. This design allows programs to use nil-checks for error detection.
|
||||
|
||||
### 7.2 ContractCheck
|
||||
|
||||
The `ContractCheck { message }` instruction provides explicit assertion. If the top of the stack is not `Bool(true)`, the VM prints the message to stderr and calls `std::process::exit(1)`.
|
||||
|
||||
### 7.3 Fatal Errors
|
||||
|
||||
The following conditions produce fatal exits:
|
||||
|
||||
- Malformed ELVM header (truncated magic or length).
|
||||
- Unsupported ELVM version.
|
||||
- JSON deserialization failure of the payload.
|
||||
|
||||
These are reported as `elvm: error: <message>` on stderr before exit.
|
||||
|
||||
---
|
||||
|
||||
## 8. Source Map Format
|
||||
|
||||
Debug builds produce a source map file (`<name>.map.json`) alongside the bytecode. The source map is a JSON array:
|
||||
|
||||
```json
|
||||
[
|
||||
{ "instruction": 0, "start": 0, "end": 5, "line": 3, "col": 0 },
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `instruction` | Bytecode instruction index |
|
||||
| `start` | Byte offset in source where the expression begins |
|
||||
| `end` | Byte offset where the expression ends |
|
||||
| `line` | 1-based source line number |
|
||||
| `col` | 0-based column number |
|
||||
|
||||
Source maps enable debuggers and error reporters to translate bytecode offsets to source positions.
|
||||
|
||||
---
|
||||
|
||||
## 9. Bootstrap Sequence
|
||||
|
||||
### 9.1 Genesis Bootstrap
|
||||
|
||||
The bootstrap sequence that produces a self-hosting El toolchain:
|
||||
|
||||
1. **Rust genesis compiler** (`engrams/el-compiler/`) reads `lexer.el`, `parser.el`, `codegen.el` and produces `el-compiler.elc` — the self-hosting compiler as an ELVM bytecode file.
|
||||
|
||||
2. **ELVM** (`bin/elvm/`) executes `el-compiler.elc`. When invoked on an El source file, it produces an `.elc` output by running the El lexer, parser, and codegen implemented in El.
|
||||
|
||||
3. **Verification pass:** The Rust genesis compiler and the self-hosted compiler are both run on the same El source file; their outputs are byte-compared. Identity confirms bootstrapping correctness.
|
||||
|
||||
4. **Production toolchain:** The `el` binary (`bin/el/`) incorporates the genesis Rust compiler for the `build-file` command and invokes `elvm` for execution. The self-hosted compiler path is used for project builds.
|
||||
|
||||
### 9.2 Invariant
|
||||
|
||||
The Rust genesis compiler and the self-hosted El compiler are structural mirrors of each other. Both implement the same lexer, parser, and codegen algorithms. The genesis compiler serves exclusively as a bootstrap vehicle; no production El bytecode is produced by it after initial bootstrap.
|
||||
|
||||
---
|
||||
|
||||
## 10. CLI Reference
|
||||
|
||||
### elvm
|
||||
|
||||
```
|
||||
elvm <file.elc> [args...]
|
||||
elvm --version
|
||||
elvm --help
|
||||
```
|
||||
|
||||
Executes a compiled El bytecode file (`.elc`). Arguments after the file path are forwarded to the program via the `args()` builtin.
|
||||
|
||||
### el build-file
|
||||
|
||||
```
|
||||
el build-file <file.el> [--target debug|release|prod] [-o output]
|
||||
```
|
||||
|
||||
Compiles a single `.el` file without a project manifest. Useful for scripts and standalone programs.
|
||||
|
||||
### el run
|
||||
|
||||
```
|
||||
el run
|
||||
```
|
||||
|
||||
Compiles the project with `debug` target and immediately executes the resulting bytecode.
|
||||
+793
-658
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user