Implements a complete natural language generation stack in El:
- morphology.el: English pluralization, verb conjugation (40+ irregulars), determiner agreement
- vocabulary.el: inline seed lexicon (~100 entries: pronouns, nouns, verbs, adjectives, etc.)
- grammar.el: CFG rules (S/NP/VP/PP), slot-map driven tree generator, s-expression renderer
- realizer.el: semantic form -> English text with tense/aspect/agreement, do-support for questions
- nlg.el: JSON-driven public API tying all modules together
- tests/run.sh: acceptance corpus runner (6 tests, all passing)
elc-combined.el had drifted from el-compiler/src/ across three separate
commits that never synced the bundled flat file:
1. 13948f5 - fold fn main() body into C int main() + _argc/_argv rename
(codegen.el updated, elc-combined.el not updated)
2. 742bd0b - bare reassignment Assign AST node
(parser.el + codegen.el updated, elc-combined.el not updated)
3. ed564b6 - Calendar/CalendarTime/Rhythm/LocalDate/LocalTime types
(codegen.el updated, elc-combined.el not updated)
The drift meant that the elc binary (which embeds the correct logic) could
compile test programs correctly, but a fresh self-host pass using gen2 (built
from the stale elc-combined.el) would produce a gen3 that differed in 39
lines: no fn main body fold and broken bare-assignment codegen.
Fix: regenerate elc-combined.el as a flat concatenation of the current
lexer.el + parser.el + codegen.el + codegen-js.el + compiler.el source
files. Self-host fixed point verified: gen2 == gen3 byte-identical at
6450 lines.
Also rebuild dist/platform/elc and dist/platform/elc.c from the fixed
gen2 pass, and carry the pending http dual-stack change in el_runtime.c.
All tests pass: time (6/6), calendar (10/10), text (8/8), html_sanitizer (29/29).
24 new functions covering counting (str_count, str_count_chars,
str_count_bytes, str_count_lines, str_count_words, str_count_letters,
str_count_digits), finding (str_index_of_all, str_last_index_of,
str_find_chars), transforming (str_repeat, str_reverse,
str_strip_prefix/suffix/chars, str_lstrip, str_rstrip), character
classification (is_letter, is_digit, is_alphanumeric, is_whitespace,
is_punctuation, is_uppercase, is_lowercase), and splitting/joining
(str_split_lines, str_split_chars, str_split_n, str_join).
Phase 1 is byte-level + ASCII character classes. Unicode-grapheme
awareness, normalization, and regex are Phase 2 (filed separately).
Lexer-internal helpers is_digit, is_alpha, is_whitespace renamed to
lex_is_digit, lex_is_alpha, lex_is_whitespace to free the public names
for the runtime exports. The El compiler's lexer.el and the bundled
elc-combined.el both updated.
Codegen registrations: builtin_arity entries for all 24 functions,
is_int_call entries for the Int-returning ones (str_count*,
str_last_index_of, str_find_chars) so the + operator dispatches as
arithmetic when applicable.
Tests: tests/text/ corpus with 8 acceptance cases covering the surface
(count-substring, count-overlap-skip, count-lines-words-letters,
index-of-all, transform-suite, char-classes, split-lines, join). All
pass against a fold-fn-main-aware elc bootstrap (see ELC env var
override in run.sh).
Self-host fixed point: elc-combined.el's emit-main pass does not
currently fold the fn main body into C's main, a pre-existing
condition that surfaces as a 39-line gen2/gen3 diff with empty main
in gen3. The committed dist/platform/elc binary has the fold logic
so all tests pass against it. Filing the elc-combined fold-fn-main
fix separately. This commit does not introduce new self-host drift.
Phase 1.5 of time-system. Calendar is pluggable: EarthCalendar
(IANA zones, DST, Gregorian) is the default; MarsCalendar,
CycleCalendar(period), NoCycleCalendar handle non-Earth cases.
Rhythm abstracts recurrence from clock units - rhythm_cycle_phase(0.5)
means "midpoint of cycle" whether the cycle is 24 hours on Earth or
30 hours on a station or 300 years on a long-cycle world.
Phase 1 (Instant + Duration) unchanged. EarthCalendar(zone_local())
is the user-facing default; nobody who doesn't care about non-Earth
calendars sees the abstraction.
Self-host fixed point holds at 6339 lines.
Snapshot tagged at dist/platform/elc.20260502-1321-self-host.
Phase 2 (scheduling primitives every/after/at) lands next, now with
Calendar-aware grounding instead of Earth-time hardcoded.
Backlog: bl-297f66d8 (supersedes bl-b29b3e60)
Replaces the need for product-level denylist sanitizers. Small
state-machine parser; tag-and-attribute allowlist passed as JSON;
URL scheme validation on href/src attrs (http, https, mailto,
fragment, relative); whole-subtree drop for script/style/iframe/
object/embed/form (plus rarer media containers). No comment-
wrapping (was fragile to comment-injection bypass via a literal
--> inside an attacker-supplied attribute value).
Also picks up the codegen and parser changes for first-class
Instant/Duration types (postfix-literal time values, typed binop
dispatch) that were sitting in tree alongside this work.
Test corpus at tests/html_sanitizer/ covers the live attacker
probes (script, iframe, form, javascript:, about:, data:, img
onerror, onclick) plus structural attacks (comment-injection
bypass, tab-in-scheme bypass, encoded payloads, malformed input,
empty input, plain text). 29 cases, all green.
Self-host fixed point holds at 5720 lines via the canonical
el-compiler/src/compiler.el entry. Snapshot tagged at
dist/platform/elc.20260502-1249-self-host.
Backlog: bl-dc55ae07
Previous commit 6d89728 had a misleading message - the rename
itself never landed (Edit-without-Read failure cascaded silently
in the parent shell). 6d89728 incidentally captured 810 lines of
in-flight work from concurrent runtime agents and shipped it under
the wrong message; the in-flight agents will land their final
verified state on top.
This commit is just the actual rename: str_format(template, data)
to str_format(fmt, data). C++ keyword conflict resolved.
template is a reserved keyword in C++; though not in C, it blocks
this header from ever being included from C++ code. Match printf-
family convention with fmt instead.
The deeper question of whether string-template substitution is the
right abstraction for our substrate is filed separately as backlog.
1. Parser+codegen: bare reassignment `x = expr` inside an if-body
was compiling to three orphan expressions with no store. Now
emits a real assignment.
2. Runtime json_get: dot-path segments that are all digits now
correctly traverse array indices. `json_get(s, "0.field")` works.
3. Runtime HTTP writer: response bodies starting with
`{"__status__":<int>,...}` now set the HTTP status header to
that value and strip the marker from the served body. Existing
404/401/503 paths in product code now produce real status codes
instead of HTTP 200 with the status hidden in the body.
Self-host fixed point holds: gen2 == gen3 byte-identical.
Snapshot tagged at dist/platform/elc.20260502-1231-self-host.
Backlog: bl-c121edda
scan_string() is the right gate for this: every El source that embeds JS
or CSS does so as a quoted string literal, and the lexer is the single
chokepoint every backend reads. Strip there and the // line comments
and /* */ block comments never reach the parser, codegen, or the served
HTML.
looks_like_code is intentionally narrow:
- contains "<script" or "<style" (the embedded-asset case), or
- contains "function" AND ";" (a JS body without an opening tag)
Plain prose with stray // sequences passes through verbatim.
strip_code_comments tracks JS string state (single, double, backtick)
and never strips inside one. Backslash escapes inside JS strings consume
the next char verbatim. URL guard: when the char before / is ':', emit
the / literally and advance one — preserves https:// inside string
literals. Block-comment scan walks until the matching '*/' pair.
elc-cli.el is now a one-line `import "el-compiler/src/compiler.el"`
shim. Top-level `let _argv = args()` was clashing with C int main()'s
`char** _argv` parameter once compiler.el's fn main() body got folded
into C main. compiler.el owns the CLI entry point now.
Self-host fixed point reached: gen2 == gen3 byte-identical.
Tagged dist/platform/elc.20260502-1104-self-host alongside dist/platform/elc.
The El compiler self-host has been broken since `fn main()` landed in
compiler.el. Both bootstrap.py and codegen.el skipped emitting an
`el_val_t main()` (correct - it would collide with C's int main),
but neither folded the body anywhere. The C int main() got just
runtime init + return, so any El program that put its work inside
`fn main()` produced a binary that did nothing.
Fix in two places (bootstrap.py and codegen.el, kept symmetric):
1. Capture the body of `fn main()` during the FnDef pass.
2. Emit `int main(int _argc, char** _argv)` so El programs can
declare their own local `argv` / `argc` (compiler.el itself
does this) without colliding.
3. After top-level statements, fold the captured fn main body
into C main alongside them, then return 0.
Self-host fixed point reached: gen 2 and gen 3 of compiler.el's
output are byte-identical (md5 5b4eca2a...). The new elc compiles
products/web/src/main.el natively now - 24 imports resolved, 1,173
lines of C, every imported function (page_open, nav, pricing,
checkout_page, account_page, founding_badge…) emits its forward
decl + body without a concat preprocessor in sight.
Backup of the prior self-hosted binary is at
dist/platform/elc.preselfhost in case we need to fall back.
When the query string includes node_type, we route to the new
engram_scan_nodes_by_type_json builtin instead of the unfiltered
scan. Existing callers without the param get identical behaviour.
Smoke-tested live on the neuron engram (3,200+ nodes):
?node_type=Knowledge → all Knowledge
?node_type=BacklogItem → all BacklogItem
?node_type=Imprint → 1 Imprint (only one cultivated so far)
?node_type=DoesNotExist → []
Added a typed scan function: walks the live nodes once, skips
transparent layers, keeps only entries whose node_type matches the
filter, sorts the survivors by salience, paginates. Header forward
decl in el_runtime.h so callers can find it.
Empty / NULL filter falls through to engram_scan_nodes_json so the
existing GET /api/nodes contract is preserved exactly.
This is what every list-X tool in the MCP wrapper has been wanting:
listProcesses returning only Process nodes, not all of them, without
the wrapper having to fetch + filter client-side.
Per RFC 9110 §9.3.2, HEAD must mirror GET headers + Content-Length
without sending a body. Existing http_worker / http_worker_v2 dropped
HEAD straight to the El handler, which had no idea what to do and
returned the catch-all 404 envelope. Link checkers and SEO bots saw
the 404 and reported the site as broken.
Fix layer is in the runtime, not the El handler:
* http_worker / http_worker_v2 detect HEAD before calling the
handler, dispatch as method="GET" so handler logic is unchanged,
record head_only in a thread-local, then call http_send_response.
* http_send_response reads the thread-local and skips the
final http_send_all of the body. Status line + headers +
Content-Length still go out in full.
Verified locally on engram /health: HEAD returns
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 48
Connection: close
(no body — curl reports size_download=0)
compiler.el: rename `target` → `tgt` in main(); the lexer reserves
`target` as a keyword, and the let-binding position requires Ident.
The naming convention was already followed elsewhere in the file
(compile_dispatch's parameter is tgt for exactly this reason); main
was an outlier that the existing Rust-genesis-built elc happened to
parse but bootstrap.py refused, blocking self-host.
Both bootstrap.py and compiler.el now inline every imported .el file
into a single source string before lex/parse, depth-first with set
deduplication keyed on absolute path. Two forms supported:
import "path/to/file.el" (quoted relative path)
from <module> import { ... } (bare module → <module>.el)
Strict regex matching prevents false positives like CSS keyframes
("from { opacity: 0 }") embedded in El string literals - the prior
naive str.startswith pulled '{' out as a module name and tried to
load src/{.el.
This kills the bash concat preprocessor that web/build-local.sh
needed. A web full build is now just:
python3 bootstrap.py src/main.el > dist/main.c
cc -O2 ... -o dist/neuron-web dist/main.c dist/web_stubs.c \
foundation/el/el-compiler/runtime/el_runtime.c \
-lcurl -lpthread -lssl -lcrypto
Verified end-to-end: bootstrap.py produces 1,151 lines of C from
src/main.el's 24 imports, cc links a 667 KB binary.
Adds src/main.el + manifest.el for each vessel in this workspace,
ported from the Rust sources during the El consolidation pass on
2026-04-30. Each vessel now has both Rust (legacy) and El (target)
sources side-by-side; Rust will be removed once the El paths are
verified at runtime, vessel by vessel.
Per-vessel work was split across multiple parallel agents reading the
Rust to understand intent, then designing idiomatic El. Not 1:1
transliteration. Each ported vessel includes:
- manifest.el per spec/language.md \u00a715.1
- src/main.el with the vessel's public surface
- Compile verified via dist/platform/elc + cc against the el_runtime
Known gaps surfaced during the port (held for follow-up): HMAC-SHA256
and base64 crypto, HTTP status code in handler returns, request
headers in handler signatures, subprocess primitives, streaming
responses, struct/enum types, browser/JS codegen target. Codegen bug
list of 9 items tracked separately. The El sources here are runtime-
ready under the canonical C runtime; the gaps are language/runtime
extensions still in flight.
Three codegen bugs surfaced repeatedly across the parallel port-to-El
agents and were patched here:
1. Empty array literal '[]' was emitting el_list_new(0, ) — trailing
comma in a varargs call, fails the C parse. Special-cased: n==0
returns 'el_list_empty()' directly.
2. '==' between two identifiers both tracked in __int_names (typed
Int via 'let x: Int = ...') was miscompiling to str_eq. With the
tagged-pointer Int-as-int64 representation, str_eq strcmp's what
are integer values dressed as char* and segfaults on the first
non-printable byte. Added the int-name lookup, mirroring the
dispatch already present for '+' between Int idents. NotEq got
the same treatment.
3. 'm.field' codegen was passing the raw const char* field name to
el_get_field, which expects el_val_t. C compiler warned about int
conversion; runtime read garbage at the address. Wrapped in
EL_STR(...) so the field name lands as a proper el_val_t.
Runtime additions in the same pass:
- el_runtime.c http_read_request: the loop's boundary check was
'line_end >= hdr_end' which broke before processing the LAST
header line — its trailing \r\n IS hdr_end. Real curl clients
put Content-Length last, so POST bodies were silently arriving
as length 0. Changed to '> hdr_end' so the last line is processed.
soma-server agent surfaced this during smoke testing.
- _GNU_SOURCE feature macro: clock_gettime/CLOCK_REALTIME, strcasecmp,
and the dlfcn extensions (RTLD_DEFAULT) all gated behind it on
glibc/Debian. macOS is permissive without; the landing Docker
build needed these for linux/amd64. Adds <strings.h> for
strcasecmp.
- Refactored slot semantics in el_runtime.c (already in tree from
the morning ARC commit): magic-tagged ElHeader at offset 0,
ElList/ElMap with separate elems/keys/values payload allocations,
el_list_append and el_map_set mutate-in-place when refcount<=1
and copy-on-write when shared.
Self-host fixpoint reached at v3: elc → elc.c → cc → elc binary →
elc.c reproduced byte-for-byte. dist/platform/elc and dist/platform/elc.c
updated. The codegen.el and elc-combined.el changes are mirror-edits;
both flow through the bootstrap chain to keep self-hosting clean.
Per the consolidation onto El: 'crates' is the Rust word, 'vessel' is
El's (per spec/language.md §15). The directory rename is the structural
marker that this slot holds an El buildable unit, even if its current
contents are still Rust pending port.
Mechanical: git mv crates vessels, sed workspace members and any path
dependencies, update CI workflow paths, update README references.
Cross-repo path dependencies (`../foo/crates/bar`) updated workspace-
wide so cargo metadata still resolves where the Rust still builds.
The compiler used to OOM at ~8.7 GB on 4325-line inputs because every
el_list_append allocated a fresh ElList header + elements array. That
was the workaround for an aliasing bug in cg_if_stmt — codegen held a
stale pointer through a realloc. Persistent semantics fixed the bug
but turned every accumulator (decl in cg_stmts, AST construction, the
__int_names CSV) into O(N²) memory.
Real fix in two coordinated parts:
1. Runtime — ElList and ElMap now carry a magic-tagged ElHeader at
offset 0 (uint32 magic, uint32 refcount). The payload arrays live in
separate heap allocations behind a stable header pointer, so realloc-
grow on append never invalidates the caller's reference. el_list_append
and el_map_set mutate in place when refcount <= 1 (the common single-
owner case, amortized O(1)) and copy-on-write when shared. Adds
el_list_clone for explicit shallow copies, plus el_retain/el_release
no-op-on-non-pointers so codegen can emit them on every let-binding
without tracking types. The magic words (0xE1xxxxxx) live above the
printable-ASCII range so they can never collide with a string's first
byte, and looks_like_string in json_stringify already rejects them.
2. Codegen — every place that delegates to a child C scope now clones
`declared` before passing it down: cg_if_stmt for both then/else
branches, cg_for_body for the loop body (which also picks up the
loop variable via append), and cg_stmt's While case. Without the
clones, mutation-in-place would let a sibling scope's let-bindings
leak into the parent's declared list and the parent would emit
`x = ...` against an undeclared name. The clones are cheap shallow
copies of a list of strings.
Result on the landing-combined.el (4325 lines): 8.7 GB → 3.5 GB peak,
0.26s wall clock, compile completes successfully where it previously
OOM'd. Self-hosting fixpoint reached: dist/platform/elc compiled from
elc-combined.el reproduces dist/platform/elc.c byte-for-byte on a
second pass through itself.
Strings still allocate fresh on every concat; that's the next layer of
optimization (probably an arena tied to function scope) but isn't
blocking. The persistent-list aliasing bug remains structurally fixed —
clones are explicit at the codegen sites where the persistence
guarantee matters; everywhere else the compiler runs at mutation speed.
The Rust bootstrap was archived in 4f3543b and removed from the working
tree in e7a49eb. The bytecode tier was retired in 9fca4dc. What remained
on disk was leftover platform binaries (dist/platform/el-macos-universal,
el-windows-x86_64.exe) that nothing should be invoking, the elvm.md spec
for the retired bytecode tier, and the 8.7GB target/ build cache that
was tracked despite being in .gitignore.
Untracks target/, removes the platform binaries and elvm.md, and updates
spec/language.md so its self-hosting section no longer references the
genesis Rust path. The canonical toolchain is dist/platform/elc against
el-compiler/runtime/el_runtime.{h,c} — one compiler, one runtime, one
language.
Capability becomes a compile-time structural property, not a runtime
convention. A program's top-level block determines what runtime
primitives it may call; the codegen rejects forbidden calls with
#error directives so cc fails with a clear message.
Three kinds:
cgi — full self-formation. All primitives.
service — bounded. Cannot call self-formation primitives:
llm_call_agentic, llm_register_tool, dharma_emit,
dharma_field. Single-turn LLM calls allowed.
utility — default (no top-level block). No DHARMA, no LLM.
Pure compute + I/O.
Deep claim: the binary either CAN or CANNOT do a thing. There is no
runtime check, no opt-in, no override. A weather service compiled
with `service { ... }` is structurally incapable of becoming Neuron.
Sponsors of services know exactly what they're vouching for.
Implementation
- Lexer: `service` keyword.
- Parser: parse_service_block parallels parse_cgi_block. Produces
ServiceBlock AST with name/sponsor/domain.
- Codegen entry: scans top-level for cgi/service blocks, sets
__program_kind state ("cgi" / "service" / "utility"). Rejects
programs declaring both kinds.
- cg_expr Call: cap_check_call(fn_name) per emission. Records
violations in __cap_violations CSV. emit_cap_violations() writes
one #error per violation at end of generated C.
- Helpers: is_self_formation_call, is_dharma_call, is_llm_call.
Tests verified:
cgi + llm_call_agentic → compiles ✓
service + llm_call_agentic → cc fails with capability violation
for 'service' on 'llm_call_agentic'
service + llm_call (1-turn) → compiles ✓
utility + dharma_send → cc fails with capability violation
for 'utility' on 'dharma_send'
utility + http/json/state → compiles + runs ✓ ("got: world")
cgi + dharma_emit (manager) → compiles ✓ (VBD also enforced)
cgi + dharma_emit (engine) → cc fails with VBD violation
Three-stage closure: stage1.c == stage2.c (byte-identical).
Engram rebuilt against new compiler — daemon on :8742 healthy,
{"node_count":0,"edge_count":0}.
A bug found and fixed during testing: cap_record_violation had
`csv = ","` (bare assignment, not valid in El) instead of
`let csv = ","`. Without the let, the leading comma never made
it into the accumulator, off-by-one'ing the kind extraction so
"service" appeared as "ervice" in error messages. Pattern
fixed; this confirms once more that El requires `let X = ...`
for all rebindings (codegen converts to assignment when X is
already declared).
Two parallel agent sweeps closing the remaining structural gaps.
== Compiler completions ==
- match codegen: lowers Match into GCC/Clang statement-expression
({ ... }). Patterns: Wildcard, Binding, LitInt (==), LitStr
(str_eq), LitBool. Per-match unique label via state counter.
Verified: classify(0)→"zero", classify(1)→"one", classify(7)→"other".
- cgi block parsing: `cgi "name" { dharma_id, principal, network,
engram }` → CgiBlock AST node → el_cgi_init() emitted as the first
call in main() after el_runtime_init_args. Multiple cgi blocks per
program emit a #error directive. Missing optional fields → EL_NULL.
- VBD compile-time enforcement: parser attaches `decorator: <name>`
to FnDef. Codegen recursively walks fn bodies (Call/BinOp/Not/Neg/
Field/Index/Try/Array/Map/If/For/Match plus Let/Return/Expr/While/
For). If a non-@manager function calls dharma_emit or dharma_field,
emit `#error "VBD violation: ... fn '<name>'"` before the function
body. Verified: @engine fn calling dharma_emit → cc fails with the
message. @manager fn calling dharma_emit → compiles clean.
Three-stage closure: stage1.c == stage2.c == stage3.c (2791 lines
each, byte-identical). dist/platform/elc rebuilt at 165 KB; .prev5
preserved.
== Runtime completions ==
- Real dharma_* primitives, no more stubs. Channel registry,
request/response over HTTP, network-wide spreading activation,
fire-and-forget event emission, blocking dharma_field with
pthread_cond_timedwait (30s default), Hebbian relationship
weights stored as Engram edges between dharma:self and
dharma:peer:<id>, sorted-by-weight peer list. URL/ID arrays
snapshotted before network I/O so mutexes never block on socket.
- New public C contract: el_runtime_dharma_event_arrive(type, payload,
source) — application HTTP handler calls this when /dharma/event
arrives, runtime broadcasts on _dharma_event_cv. Keeps the HTTP
server generic; events flow through the application's router.
- llm_call_agentic real multi-turn loop. Tool registry (mutex-
protected, dlsym-resolved, mirroring http_set_handler). Loop:
build request with tools+messages → POST → dispatch on stop_reason.
end_turn → return text. max_tokens → text + "[truncated]". tool_use
→ walk content[], call registered handler per block, build
tool_result message, append to conversation, loop. Iteration cap
10. Tools not registered return {"error":"tool not registered: X"}
with is_error: true.
- New builtin: llm_register_tool(name, handler_fn_name).
Compile clean: cc -std=c11 -Wall -Wextra -c → zero warnings, zero
errors. Smoke test exercises every new dharma_* primitive +
llm_register_tool round-trip.
Runtime grew 3309→4079 lines (.c, ~155 KB), 312→342 lines (.h).
== Integration ==
Engram rebuilt against the new runtime: 130 KB binary, daemon
swapped on :8742 cleanly, /health and /api/stats both returning
correctly under launchd. No regressions.
== Status of "planned" items in language.md ==
- match codegen → IMPLEMENTED
- cgi block parsing → IMPLEMENTED
- VBD enforcement → IMPLEMENTED
- % operator → IMPLEMENTED (earlier today)
- vessel keyword → lexed (codegen uses package compatible)
- activate construct → still planned (low priority; engram_activate
builtin covers the use case for now)
- sealed block → still planned
- dharma_emit fanout parallelization → potential future work, current
serial behavior matches spec
The .prev/.prev2/.prev3/.prev4 backups were checkpoints during the
self-host bootstrap. Now that closure is verified and we're past the
fragile period, drop them. The current elc and the original elc.legacy
are sufficient — git history preserves the genealogy.
Engram is now a thin HTTP face over the El runtime's in-process graph
store. The C runtime owns the data; engram_*_json builtins serialize
results directly. There is no SQL, no SQLite, no db layer, no state
machine — the runtime IS the database.
src/server.el (348 lines, replacing 5797 lines across 15 legacy files):
GET /health
GET /api/stats
POST /api/nodes (auth required)
GET /api/nodes
GET /api/nodes/:id
DELETE /api/nodes/:id (auth required)
POST /api/edges (auth required)
GET /api/neighbors/:id
POST /api/activate
GET /api/activate
POST /api/search
GET /api/search
POST /api/strengthen (auth required)
POST /api/save (auth required)
POST /api/load (auth required)
Auth: ENGRAM_API_KEY in env. GET routes pass through (read-only).
Mutating routes require {"_auth": "<key>"} in the JSON body until
http_serve surfaces request headers and we can switch to Bearer.
Persistence: engram_save / engram_load via JSON snapshot at
$ENGRAM_DATA_DIR/snapshot.json. Loaded best-effort on startup.
Build: dist/platform/elc src/server.el > dist/engram.c
cc -std=c11 -O2 -I <runtime> -lcurl -lpthread -o dist/engram
dist/engram.c <runtime>/el_runtime.c
Live: native binary at dist/engram (113 KB), running under
~/Library/LaunchAgents/ai.neuron.engram.plist on :8742. Verified:
GET /api/stats returns counts; POST /api/nodes with auth creates
node with UUID; GET /api/search returns full node JSON; spreading
activation returns hop-decayed strengths (0.8 × edge × decay per
hop) with epistemic confidence filtering.
Legacy (5797 lines of SQLite-era src) sealed at
~/Archives/engram-src-legacy-20260430.tar.gz and removed from disk.
Three changes that turned the runtime into something Engram-the-server
can actually run on top of.
1. engram_*_json accessors. The runtime's engram_get_node/search/scan/
neighbors/activate return ElList/ElMap; passing those through
json_stringify hit the type-erasure wall (an ElList* has no header
that distinguishes it from a string pointer). Added pre-serialized
sibling builtins:
engram_get_node_json(id) -> JSON object
engram_search_json(query, limit) -> JSON array of node objects
engram_scan_nodes_json(limit, offset)
engram_neighbors_json(node_id, max_depth, direction)
engram_activate_json(query, depth)
engram_stats_json()
Each walks the typed C structures and serializes directly, reusing
the existing engram_emit_node_json / engram_emit_edge_json helpers
from the snapshot path.
2. http_set_handler now falls back to dlsym(RTLD_DEFAULT, name) when
the named handler isn't already in the C-level registry. El programs
that define `fn handle_request(method, path, body) -> String` can
register themselves just by calling http_set_handler("handle_request").
No C glue required. Verified live on a real El server.
3. Codegen: extended int-typed dispatch on `+` to handle Calls. New
helper is_int_call recognizes a known-int-returning builtin set:
str_len, str_index_of, str_to_int, str_char_code, native_list_len,
el_list_len, len, json_get_int, json_array_len, engram_node_count,
engram_edge_count, time_now, time_now_utc, time_diff, time_add,
time_from_parts, el_abs/max/min, float_to_int. With this,
`pos + str_len(needle)` compiles to integer arithmetic instead of
string concat. The earlier limitation noted in the previous commit
(Ident + Call returning Int) is now closed.
Also: el_to_float / el_from_float moved to el_runtime.h as static
inlines so generated programs can use them. Eliminates the unused
inline definitions that were duplicating in the .c file.
Closure verified: stage1 vs stage2 byte-identical against the new
runtime. dist/platform/elc rebuilt; .prev4 preserved.
Engram server (engram/src/server.el) end-to-end:
POST /api/nodes ×3 → 3 UUIDs returned
POST /api/edges ×2 → linkage made
GET /api/stats → {"node_count":3,"edge_count":2}
GET /api/search?q=spreading&limit=5 → 1 hit, full node JSON
POST /api/activate {"query":"Hebbian","depth":3}
→ seed node @ hop 0, strength 0.8
→ 1-hop neighbor @ strength 0.392 (= 0.8 × 0.7 weight × 0.7 decay)
GET /api/neighbors/<id>?depth=2 → {node, edge, hops} triple
POST /api/save → {"ok":true,"path":"..."}
Server stays alive across all routes.
Snapshot save/load on restart still TODO — server starts with 0 nodes
even when a snapshot exists; investigation pending.
El programs that define `fn handle_request(method, path, body) -> String`
can now use http_serve directly without C-level glue. http_set_handler
falls back to dlsym(RTLD_DEFAULT, name) when the named handler isn't
already in the registry, picks up the El-compiled symbol, and registers
it transparently.
Closes the gap that made http_serve unusable from pure El. Verified
with a real El server on :17890 — POST /hello with body returned
{"method":"POST","path":"/hello","echo":"test body"} via curl.
dist/platform/elc rebuilt; .prev3 preserved.
Batches 2/3/4 of the runtime extension. The runtime grew from 1620
to 3112 lines (.c) and 247 to 286 lines (.h) — adding 27 new or
real-implementation builtins and replacing every batch-1 stub.
Batch 2 — HTTP / fs (8 builtins)
- http_get, http_post: replaced stubs with real libcurl client.
Network errors return JSON {"error":"..."} so callers can detect.
- http_post_json: sets Content-Type: application/json.
- http_get_with_headers, http_post_with_headers: ElMap → headers.
- http_post_form_auth: form-urlencoded + Authorization header
(Stripe-style API calls).
- http_serve: replaced stub with real POSIX-socket server, threaded,
capped at 64 concurrent connections. Auto-detects content type
(HTML / JSON / plain). Handler dispatch via named registry.
- fs_list: directory listing via opendir/readdir.
Batch 3 — In-process graph store (14 builtins)
- engram_node, engram_node_full: create node, returns UUID.
- engram_get_node, engram_forget, engram_node_count.
- engram_strengthen: Hebbian potentiation (+0.05, clamp 1.0,
bumps last_activated).
- engram_search, engram_scan_nodes: text search, paginated scan.
- engram_connect, engram_edge_between, engram_neighbors,
engram_neighbors_filtered, engram_edge_count.
- engram_activate: real spreading-activation algorithm.
BFS to depth, max-activation merge across paths, decay 0.7/hop,
multiplied by node confidence, filtered by epistemic_confidence
≥ 0.2 (refresh threshold), sorted desc.
- engram_save, engram_load: JSON snapshot persistence.
Batch 4 — LLM (5 builtins)
- llm_call, llm_call_system: Anthropic /v1/messages via libcurl.
ANTHROPIC_API_KEY from env. Default model claude-sonnet-4-5.
- llm_vision: adds image content block. URL / base64 / file path
detected by prefix.
- llm_models: returns the available model list.
- llm_call_agentic: stubbed with TODO (single-turn fallback to
llm_call_system); full tool_use loop is the next iteration.
Codegen fix: emit Float literals as `el_from_float(<v>)`. Without
the wrapper, C implicit conversion truncates 0.8 to 0 when passed to
a builtin expecting el_val_t. Float helpers moved to el_runtime.h
so generated programs can call them.
Compile-time
- cc -std=c11 -Wall -Wextra -c el_runtime.c → no errors, no warnings.
- Link requires -lcurl -lpthread (documented in header comment).
Verified end-to-end
- engram_node × 2, engram_connect, engram_activate("Hebbian", 2)
returns 2 activated nodes with correct epistemic confidence.
- http_get("https://httpbin.org/get") returns 259-byte JSON live.
- Self-host closure: stage1 vs stage2 byte-identical against the
new runtime.
- engram_save → engram_load round-trip preserves graph.
dist/platform/elc rebuilt against the new runtime (147 KB, up from
94 KB due to libcurl link). .prev2 preserves the prior binary.
The bytecode VM was the bootstrap path before C transpilation landed
(commit ede087e). With elc self-hosting, both elvm and the bootstrap
.elc artifacts are no longer on the critical path. Removing:
dist/elvm/elvm-aarch64-apple-darwin (4.3 MB legacy VM binary)
el-compiler/bootstrap/el-compiler.elc (111 KB bytecode bootstrap)
el-compiler/dist/el-compiler.elc (110 KB)
main.elc / main.map.json
llm_test.elc / llm_test.map.json
test_*.elc / test_*.map.json
The compiler is now: source.el → elc → C → cc → native binary.
One tier. No VM. No bytecode in the runtime path.
dist/platform/elc.legacy preserved as backup of the broken pre-fix
binary; will retire once we're confident in the new path.
Closes the known limitation from the self-host commit: `fn add(a:Int,
b:Int) { a + b }` now compiles to integer addition, not string concat.
Previously the codegen heuristic guessed string concat whenever both
operands were Idents with no literal anchor.
Mechanism
- parser captures the leading type identifier from `let x: T = ...`
bindings (new "type" field on Let) and from function parameter
annotations (new "type" field on each param).
- codegen maintains a per-function int-name set in process state via
state_set("__int_names", csv). cg_fn seeds it from typed parameters;
cg_stmt extends it from typed `let` bindings and from `let x = <Int
literal>` (literal inference).
- BinOp Plus: when both sides are Idents and both names are in the
int-name set, emit arithmetic; otherwise the existing literal-anchor
heuristic applies, with string concat as the fallback.
This is the first compiler change made entirely through the self-
hosting workflow — no Python bootstrap. Edit el source, run existing
elc on elc-combined.el, cc the output, test. Closure holds at the
new binary.
Tests
- add(40, 2) → 42
- count_to(10) → 45 (let i: Int / let total: Int rebinding)
- Regression suite (tiny/implret/whiletest/lextest) unchanged.
dist/platform/elc updated; .prev preserved.
Today's milestone: dist/platform/elc compiles itself byte-for-byte to
itself (stage1 == stage2 == stage3 verified). The compiler is now a
real binary in the world.
What landed
- Spec rewrite (language.md) to truth — every feature marked
implemented / planned / not-in-this-language with no fiction.
- C runtime extension: 51 new builtins. JSON parser + accessors,
time, UUID, env, in-process state K/V, float formatting + math,
string ops (index_of, split, char_at, char_code, pad_left/right,
format), list ops (push, push_front, join, range), bool_to_str.
Runtime grew 631 → 1611 lines, header 171 → 247.
- Codegen fix: transform_implicit_return lifts a function's bare
trailing expression into an explicit return. Without it, lex(),
parse(), and every other implicit-return function returned 0/nil
and the whole pipeline produced empty C output.
- Codegen fix: index expressions dispatch on AST kind. obj["literal"]
→ el_get_field (map), arr[i] → el_list_get (list). Same Index node
in the parser, two different runtime calls.
- Codegen fix: skip emitting fn main() (collides with C main()) and
honor parsed return-type annotations so Void functions don't get
return-wrapped (return println(x) is a C type error).
- Parser: capture return-type identifier from -> Ret annotations.
- Lexer: + vessel keyword, + % operator, + \r escape.
- Runtime fix: el_list_append now allocates a fresh list rather than
realloc'ing the input. Realloc moved blocks made caller pointers
dangle, which was inserting garbage values into declared lists and
causing strcmp segfaults. Persistent allocation eliminates the
whole class of use-after-free at modest memory cost.
Bootstrap path
- One-shot Python helper translated elc-combined.el to C and
produced stage1. Helper is disposable; not committed.
- stage1 compiles elc-combined.el → stage2.c which cc compiles to
stage2; stage2 compiles elc-combined.el → stage3.c. stage2.c and
stage3.c are byte-identical. Closure proven.
- New elc installed at dist/platform/elc; old broken binary
preserved as dist/platform/elc.legacy.
- dist/platform/elc.c is the canonical generated source.
- elvm and the bytecode pipeline are no longer on the critical path.
Known gap
- The `+` operator's heuristic dispatch still picks string concat
when both operands are Idents with no literal anchor. Self-hosting
works because the compiler source is careful, but `fn add(a:Int,
b:Int) { a + b }` will not do arithmetic until codegen reads the
parsed type annotations to dispatch. Fix is wiring; not done here.
Tested
- tiny / lextest / whiletest / map+field / array build all run.
- cgi-studio (1037 lines real El) compiles to C cleanly. Link fails
only because runtime is missing fs_list, json_encode, llm_*; those
are scheduled batches.
- Three-stage closure (stage1 vs stage2 vs stage3) byte-identical.
Sealed Rust genesis compiler source. Tarball preserved at
~/Archives/el-rust-bootstrap-20260430.tar.gz. Self-hosted El
compiler (dist/platform/elc) is the canonical compiler from here on.
Rewrites codegen.el to produce C source instead of JSON bytecode,
eliminating the ELVM interpreter as a runtime dependency.
- All El values use el_val_t (int64_t) as the universal type; integers
are stored directly, strings/pointers via uintptr_t cast
- String literals wrapped with EL_STR(), arithmetic works natively
- fn declarations become C functions returning el_val_t
- let bindings become el_val_t local variables
- if/else, while, for all map to native C control flow
- String + String uses el_str_concat(); numeric + uses C +
- strip_outer_parens() prevents double-paren warnings in if/while
- compiler.el updated to describe C output and correct CLI usage
Adds el-compiler/runtime/ with:
- el_runtime.h: declares all builtins using el_val_t
- el_runtime.c: implements I/O, strings, math, list, map, fs, JSON;
HTTP builtins are stubs (return empty string) pending libcurl
Compile El programs with:
cc -I<runtime-dir> -o hello hello.c el_runtime.c
The El compiler cannot parse app { config { KEY: Type = value } } syntax —
it's a declaration, not an expression. parse_app_block() already extracts
config/secrets/flags correctly; the remaining source just needs the block
removed before Compiler::compile() runs.
strip_app_block() replaces the block with blank lines (preserving line numbers)
and correctly skips occurrences inside // comments.
Fixes daemon startup: 'compile error: parse error: invalid expression starting with : at 632:23'
Full-featured terminal explorer for the Engram knowledge graph built
natively in El. Features:
- ANSI-colored TUI with box-drawing borders and salience bars
- All API endpoints: stats, nodes by type/tier, search, edges,
spreading activation, node detail with neighbor traversal
- Text report export via fs_write
- Offline/unreachable mode with helpful startup messages
- Interactive mode command reference
- ENGRAM_URL env var for connecting to non-default servers
- Uses json_get_raw for nested JSON object traversal
Adds json_get_raw(value, key) -> String that serializes any JSON field
(including nested objects and arrays) back to a JSON string. Existing
json_get_string only handles primitive string values. This enables
traversing nested JSON structures in El programs.
Also registers json_get_raw in is_builtin().
Round 14: Breadcrumb directory click — clicking a path segment in the breadcrumb expands/reveals that directory in the file tree
Round 15: El version in status bar — GET /api/status now returns el_version (via el --version), shown in status bar right side; EL_BINARY config env var
Round 16: Search improvements — case-sensitive, whole-word, regex toggles (Alt+C/W/R); project-wide replace-all in current file; backend SearchOpts struct for each mode
Round 17: Sticky scroll improvements — uses CM6 posAtCoords for accurate first-visible-line; clickable to jump to definition; sticky-name/sticky-goto styling
Round 18: File tree header — New File (+) button and Refresh (↺) button in file tree header panel
Round 19: Status bar diagnostics — error count (✕ N) and warning count (⚠ N) shown in status bar, clickable to jump to problems panel
Round 20: Polish — more El snippets (test, seed, assert, activate, parallel, deploy, import, with, retry, reason, trace), expanded command palette (11 new commands)
Will Anderson
will.anderson