Files
el/lang/AGENTS.md
T
bigmerge bacaf3d39c
El SDK CI - dev / build-and-test (pull_request) Failing after 4m49s
engram: reconcile M8 HNSW vindex (#109) onto current dev, restore 3 fixes the branch predated
Lands feat/reframe-region-setop (PR #109: native set-based reframe_region,
decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency
work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated
seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's
actual current HEAD, plus engram-tiered-storage's still-unique test suite.

RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/
verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/
select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two
branches; #109's copy is a strict superset (adds vindex_harvest_from_store,
used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and
engram_verify.c are also byte-identical. #109's own branch point already
carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing
left to merge into #109 for those files. The one thing engram-tiered-storage
had that #109's tree dropped: its full test suite (test_vindex.c,
test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the
interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh
harnesses) — ported over here unchanged.

WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch
forked from dev on 2026-08-14 15:40 (before restructure-adjacent history
diverged the file's merge-base for `git merge` — it presented as an add/add
conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but
it silently reverted THREE dev fixes landed on 2026-08-14/15, after the
branch point, that the PR's diff had no way to know about:

  1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of
     the query-aware propagation gate dropped the shift-and-floor rescale
     about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate
     0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped
     around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi]
     is computed) is orthogonal to WHAT it gates on and both are kept.
  2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes
     wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is
     diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global).
     PR's tree predates this and dropped all three counters + their JSON
     stats fields. Restored declarations, all 4 direct increment sites, the
     eg_wm_carry_over bll increment, and the act-stats JSON fields --
     alongside (not instead of) the PR's own P4 afferent / API-reshape
     counters already in that same struct/JSON.
  3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev
     selects the STRONGEST qualifying candidate for consolidation each call;
     PR's tree predates this and reverted to hash-slot order (arbitrary wrt
     association strength) for edge formation -- the one path that writes
     PERMANENT structure. Restored the strongest-candidate while-loop,
     keeping the PR's own genuine improvement at that site
     (engram_adj_on_edge_added incremental-index append instead of a bare
     adj_dirty=1 full-rebuild flag).

engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env
flags, /api/nodes connected-count in responses) were pure additive: dev's
side was empty, PR's side added the feature. Took PR's side whole.

VERIFIED (nsbx sandbox only, live :8742/:7770 never touched):
  - cc -std=c11 -O2, clean link against the real engram/src/server.el via
    elc, zero errors.
  - vindex_bench (built standalone, read-only harvest) against the real
    production store clone (13,671 embedded nodes, 768-dim nomic-embed-text):
    recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs
    2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s
    for the full 13,671-node set -- see the flagged risk below.
  - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned
    snapshot of the live store, 13,424 nodes / 37,656 edges) and called
    /api/activate for real: first call after boot 41.5s (pays the one-time
    HNSW build inline -- matches the standalone bench), second/third calls
    356ms/605ms, no crash, correct results, act-stats JSON (including the
    restored evict_floor/cap/bll fields) reads correctly.

KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for
this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync
builds the HNSW index synchronously, inline, on the first engram_activate()
call after every process start (or index invalidation). On the real node
count that is a ~46s blocking stall on a single-threaded server -- the first
request after every restart (or its concurrent siblings) waits the full
build. Recommend a background/incremental build (or a bounded per-call build
budget) before this ever reaches the live daemon. See PR description / final
report for the fuller writeup.
2026-08-15 16:46:44 -05:00

6.7 KiB

El Language — Agent Guide

El is a self-hosting, statically-typed language that compiles to C. This file orients agents that work on El itself or on programs written in El.


Current work in this worktree — the API reshape / decorated seam (IN PROGRESS, 2026-08-14)

This is the api-reshape worktree. The build here reshapes Neuron's external surface and how it is declared — proven on isolated dev-port clones only; live prod engram :8742 is untouched and nothing is promoted. Full framing lives in neuron/docs/architecture/06-cognitive-architecture.md (Update — 2026-08-14 deep night) and 02-components.md §5.

  • Surface collapse. The ~90 noun-organized CRUD MCP tools collapse to a few geometry opsread (the vantage-read: re-origin + salience/recency + an aperture → a bounded slice, curing the whole-self dump), write, relate, supersede (evolve/tombstone/promote, never a hard delete) — plus the agentic primitives think/attend/learn/ground/assert. The old noun is a type parameter. Implemented in tools/api-reshape/surface.el with a parity harness (parity.sh); aperture proven to bound output. Not yet: compiled into the MCP server, hot-swap, all-alias dispatch.
  • Decorated seam. @route(path,method,…) makes codegen synthesize el_route_dispatch (replacing the hand-written handle_request if-else) — proven decorate→serve on :8951. @manager/@engine/@accessor are parsed but structurally inert in the shipped compiler today; the @route codegen lives on the unmerged branch feat/el-route-decorators. Telemetry-emit and dharma-bus auto-wiring at the boundary are staged, not shipped. In-process, an @accessor reaches the engram via engram_* builtins, not http_get.

Do not edit the protected build sources while this is in flight: el-compiler/src/codegen.el, el-compiler/runtime/el_seed.c (and the archived legacy/el_runtime.c), the runtime/engram_*.c boot files, and surface.el (when present in the reshape tree) — these are owned by the build agents.


What El Is

El compiles .el source → C → native binary. Every El value is el_val_t (int64_t). Strings are heap pointers cast through int64_t. The compiler is written in El (self-hosting).

The compiler pipeline:

elc-cli.el
  └─ imports: compiler.el
       └─ imports: lexer.el, parser.el, codegen.el, codegen-js.el

The canonical compiler binary is dist/platform/elc. It was produced by running an earlier version of itself on elc-cli.el.


The Two Layers — Know Which One You're In

Layer 1: El programs (.el files)

This is where almost all work belongs. El programs are source files that get compiled by elc. New library functions, application logic, and language-level utilities all go here as .el files.

Do not add C code when El can express it. If functionality can be built from existing El primitives (string ops, exec, fs_read/write, http_post, etc.), write it in El.

Layer 2: The C seed (runtime/el_seed.c)

This is the self-contained C OS-boundary layer. It provides the __-prefixed primitives that compiled El programs call: libcurl HTTP, pthreads, filesystem I/O, arena allocation, etc. It is not generated — it is maintained by hand.

The old el_runtime.c has been archived to runtime/legacy/. The runtime is now native El (runtime/*.el). el_seed.c replaces el_runtime.c as the sole C compilation dependency.

Only edit el_seed.c when you genuinely need OS-level access (raw sockets, GPU calls, new libcurl features). For everything else, write El.

When you do add a C builtin:

  1. Add the C function to el_seed.c
  2. Declare it in el_seed.h
  3. Add it to the builtin_arity table in el-compiler/src/codegen.el (so the compiler knows the arg count)
  4. Rebuild the elc binary (see below)

Rebuilding the Compiler

After changing any .el source in el-compiler/src/:

cd /Users/will/Development/neuron-technologies/foundation/el
./dist/platform/elc elc-cli.el > elc-new.c
cc -std=c11 -I runtime -lcurl -lpthread \
   -o dist/platform/elc-new \
   elc-new.c runtime/el_seed.c
# Verify self-hosting:
./dist/platform/elc-new elc-cli.el > elc-verify.c
diff elc-new.c elc-verify.c   # should be identical
mv dist/platform/elc-new dist/platform/elc

After changing el_seed.c only (no El source changes), rebuild downstream programs but do NOT need to rebuild the compiler binary itself — the seed is linked at the application level, not the compiler level.


How El Programs Are Built

Each El application has a build.sh that:

  1. Concatenates all .el source files (stripping import lines)
  2. Runs elc to produce a .c file
  3. Runs cc linking against el_seed.c

Example (cgi-studio daemon):

cd products/cgi-studio/el-daemon
./build.sh

When you add a new .el file to an application, add it to that application's build.sh concat list.


Parallelism in El

El is single-threaded at the application level. Parallelism is achieved through subprocess fan-out:

// Pattern: write payloads to temp files, exec bash script with & and wait,
// read results back from temp files.
fn http_post_parallel(urls: [String], bodies: [String]) -> [String] {
    // ... bash fan-out via exec() ...
}

Use exec() (blocking) or exec_bg() (fire-and-forget) with shell scripts to run concurrent work. There is no goroutine or async/await — parallelism goes through the OS process layer.


Key Files

Path What it is
dist/platform/elc Canonical compiler binary (arm64 Mac)
el-compiler/src/codegen.el Code generator — builtin arity table lives here
el-compiler/src/lexer.el Lexer
el-compiler/src/parser.el Parser
runtime/el_seed.c Self-contained C OS-boundary layer (replaces el_runtime.c)
runtime/el_seed.h Seed header (C function declarations)
spec/language.md Language specification
BOOTSTRAP.md How to recover the compiler from scratch
elc-cli.el Compiler entry point
elc-combined.el Pre-merged single-file compiler (used during early bootstrap)

HTTP Timeout

The El HTTP client (libcurl) defaults to 60 seconds. Override per-process via EL_HTTP_TIMEOUT_MS env var. Set it before spawning any subprocess that makes long API calls:

exec("EL_HTTP_TIMEOUT_MS=300000 " + SOME_BIN + " " + args + " 2>&1")

Rules

  • New library functions → write in El
  • New OS/hardware primitives → write in C and register in codegen.el arity table
  • Never edit dist/platform/elc directly — always rebuild from source
  • Never modify el_seed.c to add functionality that El can express