feat: package manager, build system, native cross-compilation, plugin system
Add three new crates and extend the compiler and CLI toolchain: - el-manifest: el.toml manifest parser using serde + toml crate; supports package info, registry/path/version deps, build config with seal key sources, cross targets, and plugins; Manifest::find_manifest() walks up the directory tree - el-registry: HTTP registry client (reqwest + tokio) for packages.neurontechnologies.ai; PackageMetadata, fetch/download/publish/ search, BLAKE3 checksum verification, local cache at ~/.engram/packages/ - el-build: build orchestrator with incremental builds (BLAKE3 file hashes in .el/build-cache.json), cross-compilation target tagging, dep resolution, plugin registry with on_ast/on_typed_ast/on_bytecode hooks, test runner, fmt/check/clean commands - CrossTarget and NativeTarget enums with triple() and artifact_extension() methods; NativeTarget::Host detects compile-time platform via cfg! macros - Plugin system: CompilerPlugin trait + PluginRegistry; dynamic loading is a marked TODO with clear extension point for libloading - CLI extended with: new, add, remove, update, build --cross, run, test, check, fmt, clean, publish, search, plugin add/remove/list; old single-file commands moved to build-file/seal/unseal subcommands - Fix pre-existing debugger.rs borrow error (unwrap_or temporary lifetime) - Fix checker.rs and codegen.rs to handle TestDef/Seed/Assert Stmt variants - Add spec/language.md sections 12-14: package system, build system, plugin system, cross-compilation targets table 130 tests passing, zero warnings
This commit is contained in:
Will Anderson
+236
-1
@@ -577,7 +577,242 @@ pattern = "_"
|
||||
|
||||
---
|
||||
|
||||
## 12. Future Directions
|
||||
## 12. Package System
|
||||
|
||||
### 12.1 Project Manifest — `el.toml`
|
||||
|
||||
Every Engram project has an `el.toml` at its root. The manifest is parsed by
|
||||
the `el-manifest` crate.
|
||||
|
||||
```toml
|
||||
[package]
|
||||
name = "my-service"
|
||||
version = "0.1.0"
|
||||
description = "What this does"
|
||||
authors = ["Will Anderson <will@neurontechnologies.ai>"]
|
||||
license = "MIT"
|
||||
edition = "2026"
|
||||
|
||||
[dependencies]
|
||||
engram-http = "1.2"
|
||||
engram-auth = "0.8.1"
|
||||
some-local = { path = "../some-local" }
|
||||
|
||||
[dev-dependencies]
|
||||
el-test = "0.1"
|
||||
|
||||
[build]
|
||||
target = "prod" # debug | release | prod (default: debug)
|
||||
entry = "src/main.el" # main entry point (default: src/main.el)
|
||||
output = "dist/" # output directory (default: dist/)
|
||||
seal_key = "env:ENGRAM_SEAL_KEY" # key source for prod sealed artifacts
|
||||
|
||||
[cross]
|
||||
targets = ["x86_64-linux", "aarch64-linux", "x86_64-macos", "aarch64-macos", "wasm32"]
|
||||
|
||||
[plugins]
|
||||
el-fmt = "1.0" # code formatter plugin
|
||||
el-doc = "0.3" # documentation generator
|
||||
```
|
||||
|
||||
#### Dependency specifiers
|
||||
|
||||
| Form | Example | Meaning |
|
||||
|------|---------|---------|
|
||||
| String | `"1.2"` | Version requirement from default registry |
|
||||
| Path table | `{ path = "../lib" }` | Local path dependency |
|
||||
| Registry table | `{ version = "1.0", registry = "https://..." }` | Private registry |
|
||||
|
||||
#### Seal key sources
|
||||
|
||||
| Form | Example | Meaning |
|
||||
|------|---------|---------|
|
||||
| `env:VAR` | `env:ENGRAM_SEAL_KEY` | Read from environment variable at build time |
|
||||
| `file:path` | `file:/etc/engram/key.bin` | Read raw bytes from a file |
|
||||
| Literal | `my-secret-key` | Inline key (development only) |
|
||||
|
||||
### 12.2 Dependency Resolution
|
||||
|
||||
Dependencies are resolved by the `el-registry` crate, which talks to the
|
||||
registry at `https://packages.neurontechnologies.ai`.
|
||||
|
||||
Resolution algorithm:
|
||||
1. For each `[dependencies]` entry, fetch all available versions from the
|
||||
registry.
|
||||
2. Pick the highest version satisfying the version requirement (semver).
|
||||
3. Download the tarball and verify the BLAKE3 checksum.
|
||||
4. Cache in `~/.engram/packages/{name}/{version}/`.
|
||||
5. Path dependencies bypass the registry entirely.
|
||||
|
||||
### 12.3 Version Requirements
|
||||
|
||||
Engram uses the `semver` crate's version requirement syntax (identical to
|
||||
Cargo's):
|
||||
|
||||
| Requirement | Example | Matches |
|
||||
|-------------|---------|---------|
|
||||
| `"1.2"` | `^1.2.0` (caret) | `1.2.0`, `1.3.0`, but not `2.0.0` |
|
||||
| `">=1.0, <2.0"` | range | explicit range |
|
||||
| `"*"` | wildcard | any version |
|
||||
|
||||
---
|
||||
|
||||
## 13. Build System
|
||||
|
||||
### 13.1 Build Targets
|
||||
|
||||
| Target | Artifact | Notes |
|
||||
|--------|----------|-------|
|
||||
| `debug` | `.elc` + `.map.json` | Full debug info, source maps |
|
||||
| `release` | `.elc` | Optimized, no debug info |
|
||||
| `prod` | `.sealed` | AES-256-GCM encrypted, tamper-evident |
|
||||
|
||||
### 13.2 CLI Commands
|
||||
|
||||
```
|
||||
el new <name> scaffold a new project
|
||||
el add <pkg>[@ver] add a dependency to el.toml
|
||||
el remove <pkg> remove a dependency
|
||||
el update update all deps to latest compatible
|
||||
el build [--target prod] build (reads el.toml)
|
||||
el build --cross build for all cross targets
|
||||
el run build debug and run
|
||||
el test run tests
|
||||
el check type-check only
|
||||
el fmt format source files
|
||||
el clean clean build artifacts
|
||||
el publish publish to registry
|
||||
el search <query> search registry
|
||||
el plugin add <plugin> add a compiler plugin
|
||||
```
|
||||
|
||||
### 13.3 Incremental Builds
|
||||
|
||||
The build system tracks a BLAKE3 hash of every source file in
|
||||
`.el/build-cache.json`. On subsequent builds, only files whose hashes have
|
||||
changed (and their dependents) are recompiled. The cache is invalidated by
|
||||
`el clean`.
|
||||
|
||||
### 13.4 Cross-Compilation
|
||||
|
||||
The `[cross].targets` list specifies which native targets to produce when
|
||||
running `el build --cross`. Each cross build produces a separate artifact
|
||||
tagged with the target triple.
|
||||
|
||||
| Target name | Triple | Notes |
|
||||
|-------------|--------|-------|
|
||||
| `x86_64-linux` | `x86_64-unknown-linux-gnu` | Standard Linux 64-bit |
|
||||
| `aarch64-linux` | `aarch64-unknown-linux-gnu` | ARM64 Linux |
|
||||
| `x86_64-macos` | `x86_64-apple-darwin` | Intel Mac |
|
||||
| `aarch64-macos` | `aarch64-apple-darwin` | Apple Silicon |
|
||||
| `wasm32` | `wasm32-unknown-unknown` | WebAssembly |
|
||||
|
||||
Cross-compilation currently emits bytecode tagged with the target triple. A
|
||||
native LLVM backend (future work) will use the triple to select the correct
|
||||
code generation backend. The LLVM extension point is clearly marked in the
|
||||
`el-build` crate source.
|
||||
|
||||
### 13.5 Artifact Names
|
||||
|
||||
| Target | Cross | Artifact name |
|
||||
|--------|-------|---------------|
|
||||
| `debug` | none | `{name}.elc` |
|
||||
| `release` | none | `{name}.elc` |
|
||||
| `prod` | none | `{name}.sealed` |
|
||||
| any | `wasm32` | `{name}-wasm32.wasm` |
|
||||
| any | other | `{name}-{triple-short}.elc` |
|
||||
|
||||
---
|
||||
|
||||
## 14. Plugin System
|
||||
|
||||
### 14.1 Overview
|
||||
|
||||
Compiler plugins are Rust dynamic libraries (`.dylib` on macOS, `.so` on
|
||||
Linux) that implement the `CompilerPlugin` trait. They are loaded at compile
|
||||
time via `dlopen` (stub — full dynamic loading is a TODO) and receive hooks at
|
||||
three points in the compilation pipeline.
|
||||
|
||||
### 14.2 Plugin Trait
|
||||
|
||||
```rust
|
||||
pub trait CompilerPlugin: Send + Sync {
|
||||
fn name(&self) -> &str;
|
||||
fn version(&self) -> &str;
|
||||
|
||||
/// Called after parsing, before type checking.
|
||||
fn on_ast(&self, program: &mut Program) -> Result<(), PluginError>;
|
||||
|
||||
/// Called after type checking, before code generation.
|
||||
fn on_typed_ast(&self, program: &Program, types: &TypeEnv) -> Result<(), PluginError>;
|
||||
|
||||
/// Called after code generation, before sealing.
|
||||
fn on_bytecode(&self, bytecode: &mut Vec<u8>) -> Result<(), PluginError>;
|
||||
}
|
||||
```
|
||||
|
||||
### 14.3 Lifecycle Hooks
|
||||
|
||||
1. `on_ast` — mutate or observe the AST after parsing. Use for: AST macros,
|
||||
synthetic node injection, linting.
|
||||
2. `on_typed_ast` — observe the type-checked AST. Use for: documentation
|
||||
generation, type-aware linting.
|
||||
3. `on_bytecode` — mutate or observe the final bytecode. Use for:
|
||||
instrumentation, size analysis.
|
||||
|
||||
### 14.4 Writing a Plugin
|
||||
|
||||
```rust
|
||||
use el_build::{CompilerPlugin, PluginError};
|
||||
|
||||
pub struct MyPlugin;
|
||||
|
||||
impl CompilerPlugin for MyPlugin {
|
||||
fn name(&self) -> &str { "my-plugin" }
|
||||
fn version(&self) -> &str { "0.1.0" }
|
||||
|
||||
fn on_ast(&self, _program: &mut Program) -> Result<(), PluginError> {
|
||||
// Observe or mutate the AST
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn on_typed_ast(&self, _program: &Program, _types: &TypeEnv) -> Result<(), PluginError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn on_bytecode(&self, _bytecode: &mut Vec<u8>) -> Result<(), PluginError> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
// Required export symbol for dynamic loading:
|
||||
#[no_mangle]
|
||||
pub extern "C" fn engram_plugin_init() -> Box<dyn CompilerPlugin> {
|
||||
Box::new(MyPlugin)
|
||||
}
|
||||
```
|
||||
|
||||
### 14.5 Installing Plugins
|
||||
|
||||
Add to `[plugins]` in `el.toml`:
|
||||
|
||||
```toml
|
||||
[plugins]
|
||||
el-fmt = "1.0"
|
||||
el-doc = "0.3"
|
||||
```
|
||||
|
||||
Or use the CLI:
|
||||
```
|
||||
el plugin add el-fmt@1.0
|
||||
```
|
||||
|
||||
Plugins are looked up in the system plugin directory. The `el-registry` fetches
|
||||
and installs them like regular packages.
|
||||
|
||||
---
|
||||
|
||||
## 15. Future Directions
|
||||
|
||||
- **ML-KEM sealed artifacts** — upgrade `el-seal` to CRYSTALS-Kyber when the
|
||||
`ml-kem` crate stabilizes (drop-in: same format, new `algorithm_id`).
|
||||
|
||||
Reference in New Issue
Block a user