From 0676725cb79f496ea2d3673286869d67a3958281 Mon Sep 17 00:00:00 2001 From: Will Anderson Date: Sun, 3 May 2026 16:04:26 -0500 Subject: [PATCH] Bundle El runtime as installable framework MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - runtime/stdlib.el: master import file for the full El standard library in correct dependency order (string→math→time→env→fs→exec→json→http→ state→thread→channel→engram→manifest); test.el excluded (dev-only) - tools/install.sh: installs El to a prefix (default /usr/local/el); copies elc binary, runtime .el files, headers, compiles libel.a from el_seed.c + el_runtime.c, generates an installed stdlib.el with absolute paths - tools/new-project.sh: scaffolds a new El project with src/main.el, build.sh (auto-discovers local elc/runtime), README.md, .gitignore; verified working end-to-end - AGENTS.md: fix elc rebuild docs — elc writes to stdout, not to its second argument; correct usage is ./dist/platform/elc elc-cli.el > elc-new.c --- AGENTS.md | 129 ++++++++++++++++++++++++++++++++++ runtime/stdlib.el | 42 +++++++++++ tools/install.sh | 127 +++++++++++++++++++++++++++++++++ tools/new-project.sh | 164 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 462 insertions(+) create mode 100644 AGENTS.md create mode 100644 runtime/stdlib.el create mode 100755 tools/install.sh create mode 100755 tools/new-project.sh diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..1f9714d --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,129 @@ +# El Language — Agent Guide + +El is a self-hosting, statically-typed language that compiles to C. This file orients agents that work on El itself or on programs written in El. + +--- + +## What El Is + +El compiles `.el` source → C → native binary. Every El value is `el_val_t` (int64_t). Strings are heap pointers cast through int64_t. The compiler is written in El (self-hosting). + +**The compiler pipeline:** +``` +elc-cli.el + └─ imports: compiler.el + └─ imports: lexer.el, parser.el, codegen.el, codegen-js.el +``` + +The canonical compiler binary is `dist/platform/elc`. It was produced by running an earlier version of itself on `elc-cli.el`. + +--- + +## The Two Layers — Know Which One You're In + +### Layer 1: El programs (`.el` files) + +This is where almost all work belongs. El programs are source files that get compiled by `elc`. New library functions, application logic, and language-level utilities all go here as `.el` files. + +**Do not add C code when El can express it.** If functionality can be built from existing El primitives (string ops, `exec`, `fs_read/write`, `http_post`, etc.), write it in El. + +### Layer 2: The C runtime (`el-compiler/runtime/el_runtime.c`) + +This is the hand-written C layer that provides El's native built-in primitives: libcurl HTTP, pthreads server, filesystem I/O, JSON parsing, engram graph store, etc. It is **not generated** — it is maintained by hand. + +**Only edit `el_runtime.c` when you genuinely need OS-level access** (raw sockets, GPU calls, new libcurl features). For everything else, write El. + +When you do add a C builtin: +1. Add the C function to `el_runtime.c` +2. Declare it in `el_runtime.h` +3. Add it to the `builtin_arity` table in `el-compiler/src/codegen.el` (so the compiler knows the arg count) +4. Rebuild the elc binary (see below) + +--- + +## Rebuilding the Compiler + +After changing any `.el` source in `el-compiler/src/`: + +```bash +cd /Users/will/Development/neuron-technologies/foundation/el +./dist/platform/elc elc-cli.el > elc-new.c +cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \ + -o dist/platform/elc-new \ + elc-new.c el-compiler/runtime/el_runtime.c +# Verify self-hosting: +./dist/platform/elc-new elc-cli.el > elc-verify.c +diff elc-new.c elc-verify.c # should be identical +mv dist/platform/elc-new dist/platform/elc +``` + +After changing `el_runtime.c` only (no El source changes), rebuild downstream programs but do NOT need to rebuild the compiler binary itself — the runtime is linked at the application level, not the compiler level. + +--- + +## How El Programs Are Built + +Each El application has a `build.sh` that: +1. Concatenates all `.el` source files (stripping `import` lines) +2. Runs `elc` to produce a `.c` file +3. Runs `cc` linking against `el_runtime.c` + +Example (cgi-studio daemon): +```bash +cd products/cgi-studio/el-daemon +./build.sh +``` + +When you add a new `.el` file to an application, add it to that application's `build.sh` concat list. + +--- + +## Parallelism in El + +El is single-threaded at the application level. Parallelism is achieved through subprocess fan-out: + +```el +// Pattern: write payloads to temp files, exec bash script with & and wait, +// read results back from temp files. +fn http_post_parallel(urls: [String], bodies: [String]) -> [String] { + // ... bash fan-out via exec() ... +} +``` + +Use `exec()` (blocking) or `exec_bg()` (fire-and-forget) with shell scripts to run concurrent work. There is no goroutine or async/await — parallelism goes through the OS process layer. + +--- + +## Key Files + +| Path | What it is | +|------|-----------| +| `dist/platform/elc` | Canonical compiler binary (arm64 Mac) | +| `el-compiler/src/codegen.el` | Code generator — builtin arity table lives here | +| `el-compiler/src/lexer.el` | Lexer | +| `el-compiler/src/parser.el` | Parser | +| `el-compiler/runtime/el_runtime.c` | Hand-written C runtime (native builtins) | +| `el-compiler/runtime/el_runtime.h` | Runtime header (C function declarations) | +| `spec/language.md` | Language specification | +| `BOOTSTRAP.md` | How to recover the compiler from scratch | +| `elc-cli.el` | Compiler entry point | +| `elc-combined.el` | Pre-merged single-file compiler (used during early bootstrap) | + +--- + +## HTTP Timeout + +The El HTTP client (libcurl) defaults to **60 seconds**. Override per-process via `EL_HTTP_TIMEOUT_MS` env var. Set it before spawning any subprocess that makes long API calls: + +```el +exec("EL_HTTP_TIMEOUT_MS=300000 " + SOME_BIN + " " + args + " 2>&1") +``` + +--- + +## Rules + +- New library functions → write in El +- New OS/hardware primitives → write in C and register in `codegen.el` arity table +- Never edit `dist/platform/elc` directly — always rebuild from source +- Never modify `el_runtime.c` to add functionality that El can express diff --git a/runtime/stdlib.el b/runtime/stdlib.el new file mode 100644 index 0000000..53685f5 --- /dev/null +++ b/runtime/stdlib.el @@ -0,0 +1,42 @@ +// stdlib.el — El standard library master import file. +// +// Import this single file to get the full El runtime in the correct +// dependency order. El programs can do: +// +// import "../foundation/el/runtime/stdlib.el" +// +// or, if El is installed via tools/install.sh: +// +// import "/usr/local/el/runtime/stdlib.el" +// +// Note: test.el is intentionally NOT included here — it is dev-only +// and should be imported explicitly in test files only. +// +// Dependency order (each module may depend on earlier ones): +// string — no deps +// math — no deps +// time — no deps +// env — no deps +// fs — no deps +// exec — no deps +// json — depends on string +// http — depends on string, json +// state — no deps +// thread — depends on exec +// channel — depends on thread, state +// engram — depends on http, json, string +// manifest — depends on fs, json, string + +import "string.el" +import "math.el" +import "time.el" +import "env.el" +import "fs.el" +import "exec.el" +import "json.el" +import "http.el" +import "state.el" +import "thread.el" +import "channel.el" +import "engram.el" +import "manifest.el" diff --git a/tools/install.sh b/tools/install.sh new file mode 100755 index 0000000..5beea20 --- /dev/null +++ b/tools/install.sh @@ -0,0 +1,127 @@ +#!/usr/bin/env bash +# install.sh — install El as a proper local framework tool. +# +# Copies the compiler binary, runtime source files, headers, and stdlib +# from the local source tree into the install prefix. After install, +# El programs can be built against an installed, stable copy of the runtime. +# +# Usage: +# ./tools/install.sh +# ./tools/install.sh --prefix /opt/el +# +# Environment: +# EL_HOME Override install prefix (same as --prefix) + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +EL_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" + +PREFIX="${EL_HOME:-/usr/local/el}" + +# Parse --prefix flag +while [[ $# -gt 0 ]]; do + case "$1" in + --prefix) + PREFIX="$2" + shift 2 + ;; + --prefix=*) + PREFIX="${1#--prefix=}" + shift + ;; + *) + echo "Unknown argument: $1" >&2 + echo "Usage: $0 [--prefix ]" >&2 + exit 1 + ;; + esac +done + +BIN_DIR="${PREFIX}/bin" +RUNTIME_DIR="${PREFIX}/runtime" +INCLUDE_DIR="${PREFIX}/include" +LIB_DIR="${PREFIX}/lib" + +ELC_SRC="${EL_ROOT}/dist/platform/elc" +RUNTIME_SRC="${EL_ROOT}/el-compiler/runtime" +STDLIB_SRC="${EL_ROOT}/runtime" + +echo "==> Installing El framework to ${PREFIX}" +echo " bin: ${BIN_DIR}" +echo " runtime: ${RUNTIME_DIR}" +echo " include: ${INCLUDE_DIR}" +echo " lib: ${LIB_DIR}" +echo + +# Create directories +mkdir -p "${BIN_DIR}" "${RUNTIME_DIR}" "${INCLUDE_DIR}" "${LIB_DIR}" + +# 1. Install elc binary +if [[ ! -f "${ELC_SRC}" ]]; then + echo "Error: elc binary not found at ${ELC_SRC}" >&2 + echo "Build it first or run from the el repo root." >&2 + exit 1 +fi +install -m 755 "${ELC_SRC}" "${BIN_DIR}/elc" +echo " installed: ${BIN_DIR}/elc" + +# 2. Install runtime .el files +for f in "${STDLIB_SRC}"/*.el; do + [[ -f "$f" ]] || continue + install -m 644 "$f" "${RUNTIME_DIR}/$(basename "$f")" +done +echo " installed: ${RUNTIME_DIR}/*.el ($(ls "${STDLIB_SRC}"/*.el | wc -l | tr -d ' ') files)" + +# 3. Install headers +for header in el_seed.h el_runtime.h; do + src="${RUNTIME_SRC}/${header}" + if [[ -f "${src}" ]]; then + install -m 644 "${src}" "${INCLUDE_DIR}/${header}" + echo " installed: ${INCLUDE_DIR}/${header}" + fi +done + +# 4. Build libel.a from el_seed.c + el_runtime.c +echo " compiling libel.a..." +TMP_DIR="$(mktemp -d)" +trap 'rm -rf "${TMP_DIR}"' EXIT + +cc -std=c11 -O2 -I"${RUNTIME_SRC}" -c "${RUNTIME_SRC}/el_seed.c" -o "${TMP_DIR}/el_seed.o" +cc -std=c11 -O2 -I"${RUNTIME_SRC}" -c "${RUNTIME_SRC}/el_runtime.c" -o "${TMP_DIR}/el_runtime.o" +ar rcs "${LIB_DIR}/libel.a" "${TMP_DIR}/el_seed.o" "${TMP_DIR}/el_runtime.o" +echo " installed: ${LIB_DIR}/libel.a" + +# 5. Generate stdlib.el with absolute paths to installed runtime files +STDLIB_OUT="${PREFIX}/stdlib.el" +{ + echo "// stdlib.el — El standard library (installed at ${PREFIX})" + echo "// Generated by tools/install.sh — do not edit by hand." + echo "// Import this file to get the full El runtime:" + echo "// import \"${PREFIX}/stdlib.el\"" + echo + for f in "${STDLIB_SRC}"/*.el; do + [[ -f "$f" ]] || continue + base="$(basename "$f")" + # Skip stdlib.el itself to avoid circular import + [[ "${base}" == "stdlib.el" ]] && continue + # Skip test.el — dev-only + [[ "${base}" == "test.el" ]] && continue + echo "import \"${RUNTIME_DIR}/${base}\"" + done +} > "${STDLIB_OUT}" +echo " installed: ${STDLIB_OUT}" + +echo +echo "==> El installed to ${PREFIX}" +echo +echo "Add ${BIN_DIR} to your PATH:" +echo " export PATH=\"${BIN_DIR}:\$PATH\"" +echo +echo "To use the stdlib in your El programs:" +echo " import \"${PREFIX}/stdlib.el\"" +echo +echo "To link El programs against the installed runtime:" +echo " elc src/main.el > dist/main.c" +echo " cc -std=c11 -O2 -I${INCLUDE_DIR} -o dist/main dist/main.c ${LIB_DIR}/libel.a -lcurl -lpthread" +echo diff --git a/tools/new-project.sh b/tools/new-project.sh new file mode 100755 index 0000000..8f3f8d1 --- /dev/null +++ b/tools/new-project.sh @@ -0,0 +1,164 @@ +#!/usr/bin/env bash +# new-project.sh — scaffold a new El project. +# +# Usage: +# ./tools/new-project.sh +# +# Creates: +# / +# src/main.el — hello world entry point +# build.sh — build script using elc + el_runtime.c +# README.md — minimal project docs + +set -euo pipefail + +if [[ $# -lt 1 ]]; then + echo "Usage: $0 " >&2 + exit 1 +fi + +NAME="$1" + +if [[ -e "${NAME}" ]]; then + echo "Error: '${NAME}' already exists." >&2 + exit 1 +fi + +# Discover elc — prefer PATH, fall back to the local foundation/el tree +ELC="elc" +if ! command -v elc >/dev/null 2>&1; then + # Try a sibling or parent path convention + SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + LOCAL_ELC="${SCRIPT_DIR}/../dist/platform/elc" + if [[ -x "${LOCAL_ELC}" ]]; then + ELC="$(cd "$(dirname "${LOCAL_ELC}")" && pwd)/$(basename "${LOCAL_ELC}")" + else + echo "Warning: elc not found in PATH or at ${LOCAL_ELC}" >&2 + echo " The generated build.sh may need manual adjustment." >&2 + ELC="elc" + fi +fi + +# Discover el_runtime.c +EL_RUNTIME="" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LOCAL_RUNTIME="${SCRIPT_DIR}/../el-compiler/runtime/el_runtime.c" +if [[ -f "${LOCAL_RUNTIME}" ]]; then + EL_RUNTIME="$(cd "$(dirname "${LOCAL_RUNTIME}")" && pwd)/$(basename "${LOCAL_RUNTIME}")" + EL_INCLUDE="$(dirname "${EL_RUNTIME}")" +else + # Check installed location + if [[ -f "/usr/local/el/lib/libel.a" ]]; then + EL_RUNTIME_LINK="-L/usr/local/el/lib -lel" + EL_INCLUDE="/usr/local/el/include" + fi + EL_RUNTIME="${EL_RUNTIME_LINK:-}" +fi + +echo "==> Scaffolding El project: ${NAME}" + +mkdir -p "${NAME}/src" "${NAME}/dist" + +# src/main.el +cat > "${NAME}/src/main.el" <<'ELEOF' +// main.el — entry point for this El program. + +fn run() -> String { + println("hello from El!") + return "" +} + +run() +ELEOF + +# build.sh — adapts to whether we found elc/runtime locally or installed +if [[ -n "${EL_RUNTIME}" && -f "${EL_RUNTIME}" ]]; then + # Local source tree runtime + EL_INCLUDE_DIR="$(dirname "${EL_RUNTIME}")" + cat > "${NAME}/build.sh" < Compiling El -> C" +"\${ELC}" src/main.el > dist/main.c + +echo "==> Compiling C -> binary" +cc -std=c11 -O2 -I"\${EL_INCLUDE}" -o dist/${NAME} dist/main.c "\${EL_RUNTIME}" -lcurl -lpthread + +echo "==> Built: dist/${NAME}" +BUILDEOF +else + # Installed framework + cat > "${NAME}/build.sh" < Compiling El -> C" +"\${ELC}" src/main.el > dist/main.c + +echo "==> Compiling C -> binary" +cc -std=c11 -O2 -I"\${EL_PREFIX}/include" -o dist/${NAME} dist/main.c -L"\${EL_PREFIX}/lib" -lel -lcurl -lpthread + +echo "==> Built: dist/${NAME}" +BUILDEOF +fi + +chmod +x "${NAME}/build.sh" + +# README.md +cat > "${NAME}/README.md" < "${NAME}/.gitignore" <<'IGNEOF' +dist/ +*.c +IGNEOF + +echo +echo "==> Created ${NAME}/" +echo " src/main.el" +echo " build.sh" +echo " README.md" +echo " .gitignore" +echo +echo "To build:" +echo " cd ${NAME} && ./build.sh" +echo