Bundle El runtime as installable framework
- runtime/stdlib.el: master import file for the full El standard library in correct dependency order (string→math→time→env→fs→exec→json→http→ state→thread→channel→engram→manifest); test.el excluded (dev-only) - tools/install.sh: installs El to a prefix (default /usr/local/el); copies elc binary, runtime .el files, headers, compiles libel.a from el_seed.c + el_runtime.c, generates an installed stdlib.el with absolute paths - tools/new-project.sh: scaffolds a new El project with src/main.el, build.sh (auto-discovers local elc/runtime), README.md, .gitignore; verified working end-to-end - AGENTS.md: fix elc rebuild docs — elc writes to stdout, not to its second argument; correct usage is ./dist/platform/elc elc-cli.el > elc-new.c
This commit is contained in:
Will Anderson
@@ -0,0 +1,129 @@
|
||||
# El Language — Agent Guide
|
||||
|
||||
El is a self-hosting, statically-typed language that compiles to C. This file orients agents that work on El itself or on programs written in El.
|
||||
|
||||
---
|
||||
|
||||
## What El Is
|
||||
|
||||
El compiles `.el` source → C → native binary. Every El value is `el_val_t` (int64_t). Strings are heap pointers cast through int64_t. The compiler is written in El (self-hosting).
|
||||
|
||||
**The compiler pipeline:**
|
||||
```
|
||||
elc-cli.el
|
||||
└─ imports: compiler.el
|
||||
└─ imports: lexer.el, parser.el, codegen.el, codegen-js.el
|
||||
```
|
||||
|
||||
The canonical compiler binary is `dist/platform/elc`. It was produced by running an earlier version of itself on `elc-cli.el`.
|
||||
|
||||
---
|
||||
|
||||
## The Two Layers — Know Which One You're In
|
||||
|
||||
### Layer 1: El programs (`.el` files)
|
||||
|
||||
This is where almost all work belongs. El programs are source files that get compiled by `elc`. New library functions, application logic, and language-level utilities all go here as `.el` files.
|
||||
|
||||
**Do not add C code when El can express it.** If functionality can be built from existing El primitives (string ops, `exec`, `fs_read/write`, `http_post`, etc.), write it in El.
|
||||
|
||||
### Layer 2: The C runtime (`el-compiler/runtime/el_runtime.c`)
|
||||
|
||||
This is the hand-written C layer that provides El's native built-in primitives: libcurl HTTP, pthreads server, filesystem I/O, JSON parsing, engram graph store, etc. It is **not generated** — it is maintained by hand.
|
||||
|
||||
**Only edit `el_runtime.c` when you genuinely need OS-level access** (raw sockets, GPU calls, new libcurl features). For everything else, write El.
|
||||
|
||||
When you do add a C builtin:
|
||||
1. Add the C function to `el_runtime.c`
|
||||
2. Declare it in `el_runtime.h`
|
||||
3. Add it to the `builtin_arity` table in `el-compiler/src/codegen.el` (so the compiler knows the arg count)
|
||||
4. Rebuild the elc binary (see below)
|
||||
|
||||
---
|
||||
|
||||
## Rebuilding the Compiler
|
||||
|
||||
After changing any `.el` source in `el-compiler/src/`:
|
||||
|
||||
```bash
|
||||
cd /Users/will/Development/neuron-technologies/foundation/el
|
||||
./dist/platform/elc elc-cli.el > elc-new.c
|
||||
cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \
|
||||
-o dist/platform/elc-new \
|
||||
elc-new.c el-compiler/runtime/el_runtime.c
|
||||
# Verify self-hosting:
|
||||
./dist/platform/elc-new elc-cli.el > elc-verify.c
|
||||
diff elc-new.c elc-verify.c # should be identical
|
||||
mv dist/platform/elc-new dist/platform/elc
|
||||
```
|
||||
|
||||
After changing `el_runtime.c` only (no El source changes), rebuild downstream programs but do NOT need to rebuild the compiler binary itself — the runtime is linked at the application level, not the compiler level.
|
||||
|
||||
---
|
||||
|
||||
## How El Programs Are Built
|
||||
|
||||
Each El application has a `build.sh` that:
|
||||
1. Concatenates all `.el` source files (stripping `import` lines)
|
||||
2. Runs `elc` to produce a `.c` file
|
||||
3. Runs `cc` linking against `el_runtime.c`
|
||||
|
||||
Example (cgi-studio daemon):
|
||||
```bash
|
||||
cd products/cgi-studio/el-daemon
|
||||
./build.sh
|
||||
```
|
||||
|
||||
When you add a new `.el` file to an application, add it to that application's `build.sh` concat list.
|
||||
|
||||
---
|
||||
|
||||
## Parallelism in El
|
||||
|
||||
El is single-threaded at the application level. Parallelism is achieved through subprocess fan-out:
|
||||
|
||||
```el
|
||||
// Pattern: write payloads to temp files, exec bash script with & and wait,
|
||||
// read results back from temp files.
|
||||
fn http_post_parallel(urls: [String], bodies: [String]) -> [String] {
|
||||
// ... bash fan-out via exec() ...
|
||||
}
|
||||
```
|
||||
|
||||
Use `exec()` (blocking) or `exec_bg()` (fire-and-forget) with shell scripts to run concurrent work. There is no goroutine or async/await — parallelism goes through the OS process layer.
|
||||
|
||||
---
|
||||
|
||||
## Key Files
|
||||
|
||||
| Path | What it is |
|
||||
|------|-----------|
|
||||
| `dist/platform/elc` | Canonical compiler binary (arm64 Mac) |
|
||||
| `el-compiler/src/codegen.el` | Code generator — builtin arity table lives here |
|
||||
| `el-compiler/src/lexer.el` | Lexer |
|
||||
| `el-compiler/src/parser.el` | Parser |
|
||||
| `el-compiler/runtime/el_runtime.c` | Hand-written C runtime (native builtins) |
|
||||
| `el-compiler/runtime/el_runtime.h` | Runtime header (C function declarations) |
|
||||
| `spec/language.md` | Language specification |
|
||||
| `BOOTSTRAP.md` | How to recover the compiler from scratch |
|
||||
| `elc-cli.el` | Compiler entry point |
|
||||
| `elc-combined.el` | Pre-merged single-file compiler (used during early bootstrap) |
|
||||
|
||||
---
|
||||
|
||||
## HTTP Timeout
|
||||
|
||||
The El HTTP client (libcurl) defaults to **60 seconds**. Override per-process via `EL_HTTP_TIMEOUT_MS` env var. Set it before spawning any subprocess that makes long API calls:
|
||||
|
||||
```el
|
||||
exec("EL_HTTP_TIMEOUT_MS=300000 " + SOME_BIN + " " + args + " 2>&1")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rules
|
||||
|
||||
- New library functions → write in El
|
||||
- New OS/hardware primitives → write in C and register in `codegen.el` arity table
|
||||
- Never edit `dist/platform/elc` directly — always rebuild from source
|
||||
- Never modify `el_runtime.c` to add functionality that El can express
|
||||
@@ -0,0 +1,42 @@
|
||||
// stdlib.el — El standard library master import file.
|
||||
//
|
||||
// Import this single file to get the full El runtime in the correct
|
||||
// dependency order. El programs can do:
|
||||
//
|
||||
// import "../foundation/el/runtime/stdlib.el"
|
||||
//
|
||||
// or, if El is installed via tools/install.sh:
|
||||
//
|
||||
// import "/usr/local/el/runtime/stdlib.el"
|
||||
//
|
||||
// Note: test.el is intentionally NOT included here — it is dev-only
|
||||
// and should be imported explicitly in test files only.
|
||||
//
|
||||
// Dependency order (each module may depend on earlier ones):
|
||||
// string — no deps
|
||||
// math — no deps
|
||||
// time — no deps
|
||||
// env — no deps
|
||||
// fs — no deps
|
||||
// exec — no deps
|
||||
// json — depends on string
|
||||
// http — depends on string, json
|
||||
// state — no deps
|
||||
// thread — depends on exec
|
||||
// channel — depends on thread, state
|
||||
// engram — depends on http, json, string
|
||||
// manifest — depends on fs, json, string
|
||||
|
||||
import "string.el"
|
||||
import "math.el"
|
||||
import "time.el"
|
||||
import "env.el"
|
||||
import "fs.el"
|
||||
import "exec.el"
|
||||
import "json.el"
|
||||
import "http.el"
|
||||
import "state.el"
|
||||
import "thread.el"
|
||||
import "channel.el"
|
||||
import "engram.el"
|
||||
import "manifest.el"
|
||||
Executable
+127
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env bash
|
||||
# install.sh — install El as a proper local framework tool.
|
||||
#
|
||||
# Copies the compiler binary, runtime source files, headers, and stdlib
|
||||
# from the local source tree into the install prefix. After install,
|
||||
# El programs can be built against an installed, stable copy of the runtime.
|
||||
#
|
||||
# Usage:
|
||||
# ./tools/install.sh
|
||||
# ./tools/install.sh --prefix /opt/el
|
||||
#
|
||||
# Environment:
|
||||
# EL_HOME Override install prefix (same as --prefix)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EL_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
|
||||
PREFIX="${EL_HOME:-/usr/local/el}"
|
||||
|
||||
# Parse --prefix flag
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--prefix)
|
||||
PREFIX="$2"
|
||||
shift 2
|
||||
;;
|
||||
--prefix=*)
|
||||
PREFIX="${1#--prefix=}"
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
echo "Unknown argument: $1" >&2
|
||||
echo "Usage: $0 [--prefix <path>]" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
BIN_DIR="${PREFIX}/bin"
|
||||
RUNTIME_DIR="${PREFIX}/runtime"
|
||||
INCLUDE_DIR="${PREFIX}/include"
|
||||
LIB_DIR="${PREFIX}/lib"
|
||||
|
||||
ELC_SRC="${EL_ROOT}/dist/platform/elc"
|
||||
RUNTIME_SRC="${EL_ROOT}/el-compiler/runtime"
|
||||
STDLIB_SRC="${EL_ROOT}/runtime"
|
||||
|
||||
echo "==> Installing El framework to ${PREFIX}"
|
||||
echo " bin: ${BIN_DIR}"
|
||||
echo " runtime: ${RUNTIME_DIR}"
|
||||
echo " include: ${INCLUDE_DIR}"
|
||||
echo " lib: ${LIB_DIR}"
|
||||
echo
|
||||
|
||||
# Create directories
|
||||
mkdir -p "${BIN_DIR}" "${RUNTIME_DIR}" "${INCLUDE_DIR}" "${LIB_DIR}"
|
||||
|
||||
# 1. Install elc binary
|
||||
if [[ ! -f "${ELC_SRC}" ]]; then
|
||||
echo "Error: elc binary not found at ${ELC_SRC}" >&2
|
||||
echo "Build it first or run from the el repo root." >&2
|
||||
exit 1
|
||||
fi
|
||||
install -m 755 "${ELC_SRC}" "${BIN_DIR}/elc"
|
||||
echo " installed: ${BIN_DIR}/elc"
|
||||
|
||||
# 2. Install runtime .el files
|
||||
for f in "${STDLIB_SRC}"/*.el; do
|
||||
[[ -f "$f" ]] || continue
|
||||
install -m 644 "$f" "${RUNTIME_DIR}/$(basename "$f")"
|
||||
done
|
||||
echo " installed: ${RUNTIME_DIR}/*.el ($(ls "${STDLIB_SRC}"/*.el | wc -l | tr -d ' ') files)"
|
||||
|
||||
# 3. Install headers
|
||||
for header in el_seed.h el_runtime.h; do
|
||||
src="${RUNTIME_SRC}/${header}"
|
||||
if [[ -f "${src}" ]]; then
|
||||
install -m 644 "${src}" "${INCLUDE_DIR}/${header}"
|
||||
echo " installed: ${INCLUDE_DIR}/${header}"
|
||||
fi
|
||||
done
|
||||
|
||||
# 4. Build libel.a from el_seed.c + el_runtime.c
|
||||
echo " compiling libel.a..."
|
||||
TMP_DIR="$(mktemp -d)"
|
||||
trap 'rm -rf "${TMP_DIR}"' EXIT
|
||||
|
||||
cc -std=c11 -O2 -I"${RUNTIME_SRC}" -c "${RUNTIME_SRC}/el_seed.c" -o "${TMP_DIR}/el_seed.o"
|
||||
cc -std=c11 -O2 -I"${RUNTIME_SRC}" -c "${RUNTIME_SRC}/el_runtime.c" -o "${TMP_DIR}/el_runtime.o"
|
||||
ar rcs "${LIB_DIR}/libel.a" "${TMP_DIR}/el_seed.o" "${TMP_DIR}/el_runtime.o"
|
||||
echo " installed: ${LIB_DIR}/libel.a"
|
||||
|
||||
# 5. Generate stdlib.el with absolute paths to installed runtime files
|
||||
STDLIB_OUT="${PREFIX}/stdlib.el"
|
||||
{
|
||||
echo "// stdlib.el — El standard library (installed at ${PREFIX})"
|
||||
echo "// Generated by tools/install.sh — do not edit by hand."
|
||||
echo "// Import this file to get the full El runtime:"
|
||||
echo "// import \"${PREFIX}/stdlib.el\""
|
||||
echo
|
||||
for f in "${STDLIB_SRC}"/*.el; do
|
||||
[[ -f "$f" ]] || continue
|
||||
base="$(basename "$f")"
|
||||
# Skip stdlib.el itself to avoid circular import
|
||||
[[ "${base}" == "stdlib.el" ]] && continue
|
||||
# Skip test.el — dev-only
|
||||
[[ "${base}" == "test.el" ]] && continue
|
||||
echo "import \"${RUNTIME_DIR}/${base}\""
|
||||
done
|
||||
} > "${STDLIB_OUT}"
|
||||
echo " installed: ${STDLIB_OUT}"
|
||||
|
||||
echo
|
||||
echo "==> El installed to ${PREFIX}"
|
||||
echo
|
||||
echo "Add ${BIN_DIR} to your PATH:"
|
||||
echo " export PATH=\"${BIN_DIR}:\$PATH\""
|
||||
echo
|
||||
echo "To use the stdlib in your El programs:"
|
||||
echo " import \"${PREFIX}/stdlib.el\""
|
||||
echo
|
||||
echo "To link El programs against the installed runtime:"
|
||||
echo " elc src/main.el > dist/main.c"
|
||||
echo " cc -std=c11 -O2 -I${INCLUDE_DIR} -o dist/main dist/main.c ${LIB_DIR}/libel.a -lcurl -lpthread"
|
||||
echo
|
||||
Executable
+164
@@ -0,0 +1,164 @@
|
||||
#!/usr/bin/env bash
|
||||
# new-project.sh — scaffold a new El project.
|
||||
#
|
||||
# Usage:
|
||||
# ./tools/new-project.sh <project-name>
|
||||
#
|
||||
# Creates:
|
||||
# <project-name>/
|
||||
# src/main.el — hello world entry point
|
||||
# build.sh — build script using elc + el_runtime.c
|
||||
# README.md — minimal project docs
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
echo "Usage: $0 <project-name>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NAME="$1"
|
||||
|
||||
if [[ -e "${NAME}" ]]; then
|
||||
echo "Error: '${NAME}' already exists." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Discover elc — prefer PATH, fall back to the local foundation/el tree
|
||||
ELC="elc"
|
||||
if ! command -v elc >/dev/null 2>&1; then
|
||||
# Try a sibling or parent path convention
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
LOCAL_ELC="${SCRIPT_DIR}/../dist/platform/elc"
|
||||
if [[ -x "${LOCAL_ELC}" ]]; then
|
||||
ELC="$(cd "$(dirname "${LOCAL_ELC}")" && pwd)/$(basename "${LOCAL_ELC}")"
|
||||
else
|
||||
echo "Warning: elc not found in PATH or at ${LOCAL_ELC}" >&2
|
||||
echo " The generated build.sh may need manual adjustment." >&2
|
||||
ELC="elc"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Discover el_runtime.c
|
||||
EL_RUNTIME=""
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
LOCAL_RUNTIME="${SCRIPT_DIR}/../el-compiler/runtime/el_runtime.c"
|
||||
if [[ -f "${LOCAL_RUNTIME}" ]]; then
|
||||
EL_RUNTIME="$(cd "$(dirname "${LOCAL_RUNTIME}")" && pwd)/$(basename "${LOCAL_RUNTIME}")"
|
||||
EL_INCLUDE="$(dirname "${EL_RUNTIME}")"
|
||||
else
|
||||
# Check installed location
|
||||
if [[ -f "/usr/local/el/lib/libel.a" ]]; then
|
||||
EL_RUNTIME_LINK="-L/usr/local/el/lib -lel"
|
||||
EL_INCLUDE="/usr/local/el/include"
|
||||
fi
|
||||
EL_RUNTIME="${EL_RUNTIME_LINK:-}"
|
||||
fi
|
||||
|
||||
echo "==> Scaffolding El project: ${NAME}"
|
||||
|
||||
mkdir -p "${NAME}/src" "${NAME}/dist"
|
||||
|
||||
# src/main.el
|
||||
cat > "${NAME}/src/main.el" <<'ELEOF'
|
||||
// main.el — entry point for this El program.
|
||||
|
||||
fn run() -> String {
|
||||
println("hello from El!")
|
||||
return ""
|
||||
}
|
||||
|
||||
run()
|
||||
ELEOF
|
||||
|
||||
# build.sh — adapts to whether we found elc/runtime locally or installed
|
||||
if [[ -n "${EL_RUNTIME}" && -f "${EL_RUNTIME}" ]]; then
|
||||
# Local source tree runtime
|
||||
EL_INCLUDE_DIR="$(dirname "${EL_RUNTIME}")"
|
||||
cat > "${NAME}/build.sh" <<BUILDEOF
|
||||
#!/usr/bin/env bash
|
||||
# build.sh — build ${NAME}
|
||||
set -euo pipefail
|
||||
|
||||
ELC="${ELC}"
|
||||
EL_RUNTIME="${EL_RUNTIME}"
|
||||
EL_INCLUDE="${EL_INCLUDE_DIR}"
|
||||
|
||||
mkdir -p dist
|
||||
|
||||
echo "==> Compiling El -> C"
|
||||
"\${ELC}" src/main.el > dist/main.c
|
||||
|
||||
echo "==> Compiling C -> binary"
|
||||
cc -std=c11 -O2 -I"\${EL_INCLUDE}" -o dist/${NAME} dist/main.c "\${EL_RUNTIME}" -lcurl -lpthread
|
||||
|
||||
echo "==> Built: dist/${NAME}"
|
||||
BUILDEOF
|
||||
else
|
||||
# Installed framework
|
||||
cat > "${NAME}/build.sh" <<BUILDEOF
|
||||
#!/usr/bin/env bash
|
||||
# build.sh — build ${NAME}
|
||||
# Requires El installed via tools/install.sh (elc in PATH, /usr/local/el/lib/libel.a)
|
||||
set -euo pipefail
|
||||
|
||||
ELC="\${ELC:-elc}"
|
||||
EL_PREFIX="\${EL_HOME:-/usr/local/el}"
|
||||
|
||||
mkdir -p dist
|
||||
|
||||
echo "==> Compiling El -> C"
|
||||
"\${ELC}" src/main.el > dist/main.c
|
||||
|
||||
echo "==> Compiling C -> binary"
|
||||
cc -std=c11 -O2 -I"\${EL_PREFIX}/include" -o dist/${NAME} dist/main.c -L"\${EL_PREFIX}/lib" -lel -lcurl -lpthread
|
||||
|
||||
echo "==> Built: dist/${NAME}"
|
||||
BUILDEOF
|
||||
fi
|
||||
|
||||
chmod +x "${NAME}/build.sh"
|
||||
|
||||
# README.md
|
||||
cat > "${NAME}/README.md" <<READMEEOF
|
||||
# ${NAME}
|
||||
|
||||
An El language project.
|
||||
|
||||
## Build
|
||||
|
||||
\`\`\`bash
|
||||
./build.sh
|
||||
\`\`\`
|
||||
|
||||
## Run
|
||||
|
||||
\`\`\`bash
|
||||
./dist/${NAME}
|
||||
\`\`\`
|
||||
|
||||
## Structure
|
||||
|
||||
\`\`\`
|
||||
src/main.el — entry point
|
||||
build.sh — build script
|
||||
dist/ — compiled output (gitignored)
|
||||
\`\`\`
|
||||
READMEEOF
|
||||
|
||||
# .gitignore
|
||||
cat > "${NAME}/.gitignore" <<'IGNEOF'
|
||||
dist/
|
||||
*.c
|
||||
IGNEOF
|
||||
|
||||
echo
|
||||
echo "==> Created ${NAME}/"
|
||||
echo " src/main.el"
|
||||
echo " build.sh"
|
||||
echo " README.md"
|
||||
echo " .gitignore"
|
||||
echo
|
||||
echo "To build:"
|
||||
echo " cd ${NAME} && ./build.sh"
|
||||
echo
|
||||
Reference in New Issue
Block a user