All string, I/O, math, classification, splitting, joining, counting, padding,
and URL encoding functions from el_runtime.c implemented in El using seed
primitives. No C required; compiles via the normal El pipeline.
- runtime/engram.el: thin El wrappers over all __engram_* and __generate
seed primitives (16 functions), matching the el_seed.c API exactly
- runtime/manifest.el: build manifest documenting module load order and
the cat+compile+cc command for runtime builds
- el-compiler/src/codegen.el: add 77 __-prefix seed primitive entries to
builtin_arity, covering str, fs, http, thread, exec, env, time, uuid,
math, state, html, json, and engram seeds
Migrates fs_read/write/exists/mkdir/write_bytes/list, exec/exec_bg/exec_command/exec_capture,
env/args/exit_program, state_set/get/del/keys, uuid_new/v4, and list helpers get/len from
el_runtime.c into El source as thin wrappers over seed primitives.
Introduces El's first-class threading primitives built on the seed layer's
__thread_create/__thread_join/mutex ops. parallel_map is the key deliverable:
spawns one thread per item, joins in order — replaces bash fan-out for room
dispatch and any other concurrent HTTP workload.
Thin El wrappers over seed JSON primitives (json_get, json_get_raw,
json_parse, json_stringify, json_set, json_array_len, json_array_get,
json_array_get_string) plus typed extractors (json_get_string/int/float/bool)
and pure-El builders (json_build_object, json_build_array,
json_escape_string) that require no seed call.
Thin El wrappers over seed primitives that form the public HTTP API for
El programs. Covers GET/POST/DELETE, header-map variants, binary streaming
to file, form-auth, v1/v2 server dispatch, and http_response envelope
construction. Documents two new seed primitives needed: __http_do_map and
__http_do_map_to_file (ElMap-accepting variants to avoid needing map
iteration in El).
Replace accumulate-by-concatenation loops with native_list_append + str_join.
Eliminates quadratic memory growth when processing large source files.
This is the v2 compiler state — what produced /tmp/elc-v2.
- .gitea/workflows/sdk-release.yaml: build elc from bootstrap, run tests,
publish latest release, dispatch el-sdk-updated to downstream repos
- install.sh: one-command El SDK install from Gitea release
Implements a complete natural language generation stack in El:
- morphology.el: English pluralization, verb conjugation (40+ irregulars), determiner agreement
- vocabulary.el: inline seed lexicon (~100 entries: pronouns, nouns, verbs, adjectives, etc.)
- grammar.el: CFG rules (S/NP/VP/PP), slot-map driven tree generator, s-expression renderer
- realizer.el: semantic form -> English text with tense/aspect/agreement, do-support for questions
- nlg.el: JSON-driven public API tying all modules together
- tests/run.sh: acceptance corpus runner (6 tests, all passing)
elc-combined.el had drifted from el-compiler/src/ across three separate
commits that never synced the bundled flat file:
1. 13948f5 - fold fn main() body into C int main() + _argc/_argv rename
(codegen.el updated, elc-combined.el not updated)
2. 742bd0b - bare reassignment Assign AST node
(parser.el + codegen.el updated, elc-combined.el not updated)
3. ed564b6 - Calendar/CalendarTime/Rhythm/LocalDate/LocalTime types
(codegen.el updated, elc-combined.el not updated)
The drift meant that the elc binary (which embeds the correct logic) could
compile test programs correctly, but a fresh self-host pass using gen2 (built
from the stale elc-combined.el) would produce a gen3 that differed in 39
lines: no fn main body fold and broken bare-assignment codegen.
Fix: regenerate elc-combined.el as a flat concatenation of the current
lexer.el + parser.el + codegen.el + codegen-js.el + compiler.el source
files. Self-host fixed point verified: gen2 == gen3 byte-identical at
6450 lines.
Also rebuild dist/platform/elc and dist/platform/elc.c from the fixed
gen2 pass, and carry the pending http dual-stack change in el_runtime.c.
All tests pass: time (6/6), calendar (10/10), text (8/8), html_sanitizer (29/29).
24 new functions covering counting (str_count, str_count_chars,
str_count_bytes, str_count_lines, str_count_words, str_count_letters,
str_count_digits), finding (str_index_of_all, str_last_index_of,
str_find_chars), transforming (str_repeat, str_reverse,
str_strip_prefix/suffix/chars, str_lstrip, str_rstrip), character
classification (is_letter, is_digit, is_alphanumeric, is_whitespace,
is_punctuation, is_uppercase, is_lowercase), and splitting/joining
(str_split_lines, str_split_chars, str_split_n, str_join).
Phase 1 is byte-level + ASCII character classes. Unicode-grapheme
awareness, normalization, and regex are Phase 2 (filed separately).
Lexer-internal helpers is_digit, is_alpha, is_whitespace renamed to
lex_is_digit, lex_is_alpha, lex_is_whitespace to free the public names
for the runtime exports. The El compiler's lexer.el and the bundled
elc-combined.el both updated.
Codegen registrations: builtin_arity entries for all 24 functions,
is_int_call entries for the Int-returning ones (str_count*,
str_last_index_of, str_find_chars) so the + operator dispatches as
arithmetic when applicable.
Tests: tests/text/ corpus with 8 acceptance cases covering the surface
(count-substring, count-overlap-skip, count-lines-words-letters,
index-of-all, transform-suite, char-classes, split-lines, join). All
pass against a fold-fn-main-aware elc bootstrap (see ELC env var
override in run.sh).
Self-host fixed point: elc-combined.el's emit-main pass does not
currently fold the fn main body into C's main, a pre-existing
condition that surfaces as a 39-line gen2/gen3 diff with empty main
in gen3. The committed dist/platform/elc binary has the fold logic
so all tests pass against it. Filing the elc-combined fold-fn-main
fix separately. This commit does not introduce new self-host drift.
Phase 1.5 of time-system. Calendar is pluggable: EarthCalendar
(IANA zones, DST, Gregorian) is the default; MarsCalendar,
CycleCalendar(period), NoCycleCalendar handle non-Earth cases.
Rhythm abstracts recurrence from clock units - rhythm_cycle_phase(0.5)
means "midpoint of cycle" whether the cycle is 24 hours on Earth or
30 hours on a station or 300 years on a long-cycle world.
Phase 1 (Instant + Duration) unchanged. EarthCalendar(zone_local())
is the user-facing default; nobody who doesn't care about non-Earth
calendars sees the abstraction.
Self-host fixed point holds at 6339 lines.
Snapshot tagged at dist/platform/elc.20260502-1321-self-host.
Phase 2 (scheduling primitives every/after/at) lands next, now with
Calendar-aware grounding instead of Earth-time hardcoded.
Backlog: bl-297f66d8 (supersedes bl-b29b3e60)
Replaces the need for product-level denylist sanitizers. Small
state-machine parser; tag-and-attribute allowlist passed as JSON;
URL scheme validation on href/src attrs (http, https, mailto,
fragment, relative); whole-subtree drop for script/style/iframe/
object/embed/form (plus rarer media containers). No comment-
wrapping (was fragile to comment-injection bypass via a literal
--> inside an attacker-supplied attribute value).
Also picks up the codegen and parser changes for first-class
Instant/Duration types (postfix-literal time values, typed binop
dispatch) that were sitting in tree alongside this work.
Test corpus at tests/html_sanitizer/ covers the live attacker
probes (script, iframe, form, javascript:, about:, data:, img
onerror, onclick) plus structural attacks (comment-injection
bypass, tab-in-scheme bypass, encoded payloads, malformed input,
empty input, plain text). 29 cases, all green.
Self-host fixed point holds at 5720 lines via the canonical
el-compiler/src/compiler.el entry. Snapshot tagged at
dist/platform/elc.20260502-1249-self-host.
Backlog: bl-dc55ae07
Previous commit 6d89728 had a misleading message - the rename
itself never landed (Edit-without-Read failure cascaded silently
in the parent shell). 6d89728 incidentally captured 810 lines of
in-flight work from concurrent runtime agents and shipped it under
the wrong message; the in-flight agents will land their final
verified state on top.
This commit is just the actual rename: str_format(template, data)
to str_format(fmt, data). C++ keyword conflict resolved.
template is a reserved keyword in C++; though not in C, it blocks
this header from ever being included from C++ code. Match printf-
family convention with fmt instead.
The deeper question of whether string-template substitution is the
right abstraction for our substrate is filed separately as backlog.
1. Parser+codegen: bare reassignment `x = expr` inside an if-body
was compiling to three orphan expressions with no store. Now
emits a real assignment.
2. Runtime json_get: dot-path segments that are all digits now
correctly traverse array indices. `json_get(s, "0.field")` works.
3. Runtime HTTP writer: response bodies starting with
`{"__status__":<int>,...}` now set the HTTP status header to
that value and strip the marker from the served body. Existing
404/401/503 paths in product code now produce real status codes
instead of HTTP 200 with the status hidden in the body.
Self-host fixed point holds: gen2 == gen3 byte-identical.
Snapshot tagged at dist/platform/elc.20260502-1231-self-host.
Backlog: bl-c121edda
scan_string() is the right gate for this: every El source that embeds JS
or CSS does so as a quoted string literal, and the lexer is the single
chokepoint every backend reads. Strip there and the // line comments
and /* */ block comments never reach the parser, codegen, or the served
HTML.
looks_like_code is intentionally narrow:
- contains "<script" or "<style" (the embedded-asset case), or
- contains "function" AND ";" (a JS body without an opening tag)
Plain prose with stray // sequences passes through verbatim.
strip_code_comments tracks JS string state (single, double, backtick)
and never strips inside one. Backslash escapes inside JS strings consume
the next char verbatim. URL guard: when the char before / is ':', emit
the / literally and advance one — preserves https:// inside string
literals. Block-comment scan walks until the matching '*/' pair.
elc-cli.el is now a one-line `import "el-compiler/src/compiler.el"`
shim. Top-level `let _argv = args()` was clashing with C int main()'s
`char** _argv` parameter once compiler.el's fn main() body got folded
into C main. compiler.el owns the CLI entry point now.
Self-host fixed point reached: gen2 == gen3 byte-identical.
Tagged dist/platform/elc.20260502-1104-self-host alongside dist/platform/elc.
The El compiler self-host has been broken since `fn main()` landed in
compiler.el. Both bootstrap.py and codegen.el skipped emitting an
`el_val_t main()` (correct - it would collide with C's int main),
but neither folded the body anywhere. The C int main() got just
runtime init + return, so any El program that put its work inside
`fn main()` produced a binary that did nothing.
Fix in two places (bootstrap.py and codegen.el, kept symmetric):
1. Capture the body of `fn main()` during the FnDef pass.
2. Emit `int main(int _argc, char** _argv)` so El programs can
declare their own local `argv` / `argc` (compiler.el itself
does this) without colliding.
3. After top-level statements, fold the captured fn main body
into C main alongside them, then return 0.
Self-host fixed point reached: gen 2 and gen 3 of compiler.el's
output are byte-identical (md5 5b4eca2a...). The new elc compiles
products/web/src/main.el natively now - 24 imports resolved, 1,173
lines of C, every imported function (page_open, nav, pricing,
checkout_page, account_page, founding_badge…) emits its forward
decl + body without a concat preprocessor in sight.
Backup of the prior self-hosted binary is at
dist/platform/elc.preselfhost in case we need to fall back.
When the query string includes node_type, we route to the new
engram_scan_nodes_by_type_json builtin instead of the unfiltered
scan. Existing callers without the param get identical behaviour.
Smoke-tested live on the neuron engram (3,200+ nodes):
?node_type=Knowledge → all Knowledge
?node_type=BacklogItem → all BacklogItem
?node_type=Imprint → 1 Imprint (only one cultivated so far)
?node_type=DoesNotExist → []
Added a typed scan function: walks the live nodes once, skips
transparent layers, keeps only entries whose node_type matches the
filter, sorts the survivors by salience, paginates. Header forward
decl in el_runtime.h so callers can find it.
Empty / NULL filter falls through to engram_scan_nodes_json so the
existing GET /api/nodes contract is preserved exactly.
This is what every list-X tool in the MCP wrapper has been wanting:
listProcesses returning only Process nodes, not all of them, without
the wrapper having to fetch + filter client-side.
Per RFC 9110 §9.3.2, HEAD must mirror GET headers + Content-Length
without sending a body. Existing http_worker / http_worker_v2 dropped
HEAD straight to the El handler, which had no idea what to do and
returned the catch-all 404 envelope. Link checkers and SEO bots saw
the 404 and reported the site as broken.
Fix layer is in the runtime, not the El handler:
* http_worker / http_worker_v2 detect HEAD before calling the
handler, dispatch as method="GET" so handler logic is unchanged,
record head_only in a thread-local, then call http_send_response.
* http_send_response reads the thread-local and skips the
final http_send_all of the body. Status line + headers +
Content-Length still go out in full.
Verified locally on engram /health: HEAD returns
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 48
Connection: close
(no body — curl reports size_download=0)
compiler.el: rename `target` → `tgt` in main(); the lexer reserves
`target` as a keyword, and the let-binding position requires Ident.
The naming convention was already followed elsewhere in the file
(compile_dispatch's parameter is tgt for exactly this reason); main
was an outlier that the existing Rust-genesis-built elc happened to
parse but bootstrap.py refused, blocking self-host.
Both bootstrap.py and compiler.el now inline every imported .el file
into a single source string before lex/parse, depth-first with set
deduplication keyed on absolute path. Two forms supported:
import "path/to/file.el" (quoted relative path)
from <module> import { ... } (bare module → <module>.el)
Strict regex matching prevents false positives like CSS keyframes
("from { opacity: 0 }") embedded in El string literals - the prior
naive str.startswith pulled '{' out as a module name and tried to
load src/{.el.
This kills the bash concat preprocessor that web/build-local.sh
needed. A web full build is now just:
python3 bootstrap.py src/main.el > dist/main.c
cc -O2 ... -o dist/neuron-web dist/main.c dist/web_stubs.c \
foundation/el/el-compiler/runtime/el_runtime.c \
-lcurl -lpthread -lssl -lcrypto
Verified end-to-end: bootstrap.py produces 1,151 lines of C from
src/main.el's 24 imports, cc links a 667 KB binary.
Adds src/main.el + manifest.el for each vessel in this workspace,
ported from the Rust sources during the El consolidation pass on
2026-04-30. Each vessel now has both Rust (legacy) and El (target)
sources side-by-side; Rust will be removed once the El paths are
verified at runtime, vessel by vessel.
Per-vessel work was split across multiple parallel agents reading the
Rust to understand intent, then designing idiomatic El. Not 1:1
transliteration. Each ported vessel includes:
- manifest.el per spec/language.md \u00a715.1
- src/main.el with the vessel's public surface
- Compile verified via dist/platform/elc + cc against the el_runtime
Known gaps surfaced during the port (held for follow-up): HMAC-SHA256
and base64 crypto, HTTP status code in handler returns, request
headers in handler signatures, subprocess primitives, streaming
responses, struct/enum types, browser/JS codegen target. Codegen bug
list of 9 items tracked separately. The El sources here are runtime-
ready under the canonical C runtime; the gaps are language/runtime
extensions still in flight.
Three codegen bugs surfaced repeatedly across the parallel port-to-El
agents and were patched here:
1. Empty array literal '[]' was emitting el_list_new(0, ) — trailing
comma in a varargs call, fails the C parse. Special-cased: n==0
returns 'el_list_empty()' directly.
2. '==' between two identifiers both tracked in __int_names (typed
Int via 'let x: Int = ...') was miscompiling to str_eq. With the
tagged-pointer Int-as-int64 representation, str_eq strcmp's what
are integer values dressed as char* and segfaults on the first
non-printable byte. Added the int-name lookup, mirroring the
dispatch already present for '+' between Int idents. NotEq got
the same treatment.
3. 'm.field' codegen was passing the raw const char* field name to
el_get_field, which expects el_val_t. C compiler warned about int
conversion; runtime read garbage at the address. Wrapped in
EL_STR(...) so the field name lands as a proper el_val_t.
Runtime additions in the same pass:
- el_runtime.c http_read_request: the loop's boundary check was
'line_end >= hdr_end' which broke before processing the LAST
header line — its trailing \r\n IS hdr_end. Real curl clients
put Content-Length last, so POST bodies were silently arriving
as length 0. Changed to '> hdr_end' so the last line is processed.
soma-server agent surfaced this during smoke testing.
- _GNU_SOURCE feature macro: clock_gettime/CLOCK_REALTIME, strcasecmp,
and the dlfcn extensions (RTLD_DEFAULT) all gated behind it on
glibc/Debian. macOS is permissive without; the landing Docker
build needed these for linux/amd64. Adds <strings.h> for
strcasecmp.
- Refactored slot semantics in el_runtime.c (already in tree from
the morning ARC commit): magic-tagged ElHeader at offset 0,
ElList/ElMap with separate elems/keys/values payload allocations,
el_list_append and el_map_set mutate-in-place when refcount<=1
and copy-on-write when shared.
Self-host fixpoint reached at v3: elc → elc.c → cc → elc binary →
elc.c reproduced byte-for-byte. dist/platform/elc and dist/platform/elc.c
updated. The codegen.el and elc-combined.el changes are mirror-edits;
both flow through the bootstrap chain to keep self-hosting clean.
Per the consolidation onto El: 'crates' is the Rust word, 'vessel' is
El's (per spec/language.md §15). The directory rename is the structural
marker that this slot holds an El buildable unit, even if its current
contents are still Rust pending port.
Mechanical: git mv crates vessels, sed workspace members and any path
dependencies, update CI workflow paths, update README references.
Cross-repo path dependencies (`../foo/crates/bar`) updated workspace-
wide so cargo metadata still resolves where the Rust still builds.
The compiler used to OOM at ~8.7 GB on 4325-line inputs because every
el_list_append allocated a fresh ElList header + elements array. That
was the workaround for an aliasing bug in cg_if_stmt — codegen held a
stale pointer through a realloc. Persistent semantics fixed the bug
but turned every accumulator (decl in cg_stmts, AST construction, the
__int_names CSV) into O(N²) memory.
Real fix in two coordinated parts:
1. Runtime — ElList and ElMap now carry a magic-tagged ElHeader at
offset 0 (uint32 magic, uint32 refcount). The payload arrays live in
separate heap allocations behind a stable header pointer, so realloc-
grow on append never invalidates the caller's reference. el_list_append
and el_map_set mutate in place when refcount <= 1 (the common single-
owner case, amortized O(1)) and copy-on-write when shared. Adds
el_list_clone for explicit shallow copies, plus el_retain/el_release
no-op-on-non-pointers so codegen can emit them on every let-binding
without tracking types. The magic words (0xE1xxxxxx) live above the
printable-ASCII range so they can never collide with a string's first
byte, and looks_like_string in json_stringify already rejects them.
2. Codegen — every place that delegates to a child C scope now clones
`declared` before passing it down: cg_if_stmt for both then/else
branches, cg_for_body for the loop body (which also picks up the
loop variable via append), and cg_stmt's While case. Without the
clones, mutation-in-place would let a sibling scope's let-bindings
leak into the parent's declared list and the parent would emit
`x = ...` against an undeclared name. The clones are cheap shallow
copies of a list of strings.
Result on the landing-combined.el (4325 lines): 8.7 GB → 3.5 GB peak,
0.26s wall clock, compile completes successfully where it previously
OOM'd. Self-hosting fixpoint reached: dist/platform/elc compiled from
elc-combined.el reproduces dist/platform/elc.c byte-for-byte on a
second pass through itself.
Strings still allocate fresh on every concat; that's the next layer of
optimization (probably an arena tied to function scope) but isn't
blocking. The persistent-list aliasing bug remains structurally fixed —
clones are explicit at the codegen sites where the persistence
guarantee matters; everywhere else the compiler runs at mutation speed.
Will Anderson
will.anderson