bigmerge
bacaf3d39c
El SDK CI - dev / build-and-test (pull_request) Failing after 4m49s
Lands feat/reframe-region-setop (PR #109: native set-based reframe_region, decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's actual current HEAD, plus engram-tiered-storage's still-unique test suite. RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/ verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/ select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two branches; #109's copy is a strict superset (adds vindex_harvest_from_store, used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and engram_verify.c are also byte-identical. #109's own branch point already carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing left to merge into #109 for those files. The one thing engram-tiered-storage had that #109's tree dropped: its full test suite (test_vindex.c, test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh harnesses) — ported over here unchanged. WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch forked from dev on 2026-08-14 15:40 (before restructure-adjacent history diverged the file's merge-base for `git merge` — it presented as an add/add conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but it silently reverted THREE dev fixes landed on 2026-08-14/15, after the branch point, that the PR's diff had no way to know about: 1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of the query-aware propagation gate dropped the shift-and-floor rescale about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate 0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi] is computed) is orthogonal to WHAT it gates on and both are kept. 2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global). PR's tree predates this and dropped all three counters + their JSON stats fields. Restored declarations, all 4 direct increment sites, the eg_wm_carry_over bll increment, and the act-stats JSON fields -- alongside (not instead of) the PR's own P4 afferent / API-reshape counters already in that same struct/JSON. 3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev selects the STRONGEST qualifying candidate for consolidation each call; PR's tree predates this and reverted to hash-slot order (arbitrary wrt association strength) for edge formation -- the one path that writes PERMANENT structure. Restored the strongest-candidate while-loop, keeping the PR's own genuine improvement at that site (engram_adj_on_edge_added incremental-index append instead of a bare adj_dirty=1 full-rebuild flag). engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env flags, /api/nodes connected-count in responses) were pure additive: dev's side was empty, PR's side added the feature. Took PR's side whole. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): - cc -std=c11 -O2, clean link against the real engram/src/server.el via elc, zero errors. - vindex_bench (built standalone, read-only harvest) against the real production store clone (13,671 embedded nodes, 768-dim nomic-embed-text): recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs 2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s for the full 13,671-node set -- see the flagged risk below. - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned snapshot of the live store, 13,424 nodes / 37,656 edges) and called /api/activate for real: first call after boot 41.5s (pays the one-time HNSW build inline -- matches the standalone bench), second/third calls 356ms/605ms, no crash, correct results, act-stats JSON (including the restored evict_floor/cap/bll fields) reads correctly. KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync builds the HNSW index synchronously, inline, on the first engram_activate() call after every process start (or index invalidation). On the real node count that is a ~46s blocking stall on a single-threaded server -- the first request after every restart (or its concurrent siblings) waits the full build. Recommend a background/incremental build (or a bounded per-call build budget) before this ever reaches the live daemon. See PR description / final report for the fuller writeup.
163 lines
6.7 KiB
Markdown
163 lines
6.7 KiB
Markdown
# El Language — Agent Guide
|
|
|
|
El is a self-hosting, statically-typed language that compiles to C. This file orients agents that work on El itself or on programs written in El.
|
|
|
|
---
|
|
|
|
## Current work in this worktree — the API reshape / decorated seam (IN PROGRESS, 2026-08-14)
|
|
|
|
This is the `api-reshape` worktree. The build here reshapes Neuron's external
|
|
surface and how it is *declared* — proven on isolated dev-port clones only; **live
|
|
prod engram `:8742` is untouched and nothing is promoted.** Full framing lives in
|
|
`neuron/docs/architecture/06-cognitive-architecture.md` (Update — 2026-08-14 deep
|
|
night) and `02-components.md §5`.
|
|
|
|
- **Surface collapse.** The ~90 noun-organized CRUD MCP tools collapse to a few
|
|
**geometry ops** — `read` (the *vantage-read*: re-origin + salience/recency +
|
|
an **aperture** → a bounded slice, curing the whole-self dump), `write`,
|
|
`relate`, `supersede` (evolve/tombstone/promote, never a hard delete) — plus the
|
|
agentic primitives `think`/`attend`/`learn`/`ground`/`assert`. The old noun is a
|
|
`type` parameter. Implemented in `tools/api-reshape/surface.el` with a parity
|
|
harness (`parity.sh`); aperture proven to bound output. **Not yet:** compiled
|
|
into the MCP server, hot-swap, all-alias dispatch.
|
|
- **Decorated seam.** `@route(path,method,…)` makes codegen synthesize
|
|
`el_route_dispatch` (replacing the hand-written `handle_request` if-else) —
|
|
proven decorate→serve on `:8951`. `@manager`/`@engine`/`@accessor` are **parsed
|
|
but structurally inert** in the shipped compiler today; the `@route` codegen
|
|
lives on the **unmerged branch `feat/el-route-decorators`**. Telemetry-emit and
|
|
dharma-bus auto-wiring at the boundary are **staged, not shipped**. In-process,
|
|
an `@accessor` reaches the engram via **`engram_*` builtins**, not `http_get`.
|
|
|
|
**Do not edit** the protected build sources while this is in flight:
|
|
`el-compiler/src/codegen.el`, `el-compiler/runtime/el_seed.c` (and the archived
|
|
`legacy/el_runtime.c`), the `runtime/engram_*.c` boot files, and `surface.el`
|
|
(when present in the reshape tree) — these are owned by the build agents.
|
|
|
|
---
|
|
|
|
## What El Is
|
|
|
|
El compiles `.el` source → C → native binary. Every El value is `el_val_t` (int64_t). Strings are heap pointers cast through int64_t. The compiler is written in El (self-hosting).
|
|
|
|
**The compiler pipeline:**
|
|
```
|
|
elc-cli.el
|
|
└─ imports: compiler.el
|
|
└─ imports: lexer.el, parser.el, codegen.el, codegen-js.el
|
|
```
|
|
|
|
The canonical compiler binary is `dist/platform/elc`. It was produced by running an earlier version of itself on `elc-cli.el`.
|
|
|
|
---
|
|
|
|
## The Two Layers — Know Which One You're In
|
|
|
|
### Layer 1: El programs (`.el` files)
|
|
|
|
This is where almost all work belongs. El programs are source files that get compiled by `elc`. New library functions, application logic, and language-level utilities all go here as `.el` files.
|
|
|
|
**Do not add C code when El can express it.** If functionality can be built from existing El primitives (string ops, `exec`, `fs_read/write`, `http_post`, etc.), write it in El.
|
|
|
|
### Layer 2: The C seed (`runtime/el_seed.c`)
|
|
|
|
This is the self-contained C OS-boundary layer. It provides the `__`-prefixed primitives that compiled El programs call: libcurl HTTP, pthreads, filesystem I/O, arena allocation, etc. It is **not generated** — it is maintained by hand.
|
|
|
|
The old `el_runtime.c` has been archived to `runtime/legacy/`. The runtime is now native El (`runtime/*.el`). `el_seed.c` replaces `el_runtime.c` as the sole C compilation dependency.
|
|
|
|
**Only edit `el_seed.c` when you genuinely need OS-level access** (raw sockets, GPU calls, new libcurl features). For everything else, write El.
|
|
|
|
When you do add a C builtin:
|
|
1. Add the C function to `el_seed.c`
|
|
2. Declare it in `el_seed.h`
|
|
3. Add it to the `builtin_arity` table in `el-compiler/src/codegen.el` (so the compiler knows the arg count)
|
|
4. Rebuild the elc binary (see below)
|
|
|
|
---
|
|
|
|
## Rebuilding the Compiler
|
|
|
|
After changing any `.el` source in `el-compiler/src/`:
|
|
|
|
```bash
|
|
cd /Users/will/Development/neuron-technologies/foundation/el
|
|
./dist/platform/elc elc-cli.el > elc-new.c
|
|
cc -std=c11 -I runtime -lcurl -lpthread \
|
|
-o dist/platform/elc-new \
|
|
elc-new.c runtime/el_seed.c
|
|
# Verify self-hosting:
|
|
./dist/platform/elc-new elc-cli.el > elc-verify.c
|
|
diff elc-new.c elc-verify.c # should be identical
|
|
mv dist/platform/elc-new dist/platform/elc
|
|
```
|
|
|
|
After changing `el_seed.c` only (no El source changes), rebuild downstream programs but do NOT need to rebuild the compiler binary itself — the seed is linked at the application level, not the compiler level.
|
|
|
|
---
|
|
|
|
## How El Programs Are Built
|
|
|
|
Each El application has a `build.sh` that:
|
|
1. Concatenates all `.el` source files (stripping `import` lines)
|
|
2. Runs `elc` to produce a `.c` file
|
|
3. Runs `cc` linking against `el_seed.c`
|
|
|
|
Example (cgi-studio daemon):
|
|
```bash
|
|
cd products/cgi-studio/el-daemon
|
|
./build.sh
|
|
```
|
|
|
|
When you add a new `.el` file to an application, add it to that application's `build.sh` concat list.
|
|
|
|
---
|
|
|
|
## Parallelism in El
|
|
|
|
El is single-threaded at the application level. Parallelism is achieved through subprocess fan-out:
|
|
|
|
```el
|
|
// Pattern: write payloads to temp files, exec bash script with & and wait,
|
|
// read results back from temp files.
|
|
fn http_post_parallel(urls: [String], bodies: [String]) -> [String] {
|
|
// ... bash fan-out via exec() ...
|
|
}
|
|
```
|
|
|
|
Use `exec()` (blocking) or `exec_bg()` (fire-and-forget) with shell scripts to run concurrent work. There is no goroutine or async/await — parallelism goes through the OS process layer.
|
|
|
|
---
|
|
|
|
## Key Files
|
|
|
|
| Path | What it is |
|
|
|------|-----------|
|
|
| `dist/platform/elc` | Canonical compiler binary (arm64 Mac) |
|
|
| `el-compiler/src/codegen.el` | Code generator — builtin arity table lives here |
|
|
| `el-compiler/src/lexer.el` | Lexer |
|
|
| `el-compiler/src/parser.el` | Parser |
|
|
| `runtime/el_seed.c` | Self-contained C OS-boundary layer (replaces el_runtime.c) |
|
|
| `runtime/el_seed.h` | Seed header (C function declarations) |
|
|
| `spec/language.md` | Language specification |
|
|
| `BOOTSTRAP.md` | How to recover the compiler from scratch |
|
|
| `elc-cli.el` | Compiler entry point |
|
|
| `elc-combined.el` | Pre-merged single-file compiler (used during early bootstrap) |
|
|
|
|
---
|
|
|
|
## HTTP Timeout
|
|
|
|
The El HTTP client (libcurl) defaults to **60 seconds**. Override per-process via `EL_HTTP_TIMEOUT_MS` env var. Set it before spawning any subprocess that makes long API calls:
|
|
|
|
```el
|
|
exec("EL_HTTP_TIMEOUT_MS=300000 " + SOME_BIN + " " + args + " 2>&1")
|
|
```
|
|
|
|
---
|
|
|
|
## Rules
|
|
|
|
- New library functions → write in El
|
|
- New OS/hardware primitives → write in C and register in `codegen.el` arity table
|
|
- Never edit `dist/platform/elc` directly — always rebuild from source
|
|
- Never modify `el_seed.c` to add functionality that El can express
|