Commit Graph

90 Commits

Author SHA1 Message Date
Will Anderson cfcedff7f4 implement match statement codegen in El
Add cg_match_stmt() to lower match-as-statement to proper C if/else if/else
chains. Previously, match in statement position fell through to cg_expr() which
emitted a GCC statement-expression — fine for expression arms but wrong for the
statement form. Now matched using the same dispatch pattern as If and For in the
Expr handler of cg_stmt().

Pattern dispatch mirrors cg_match (expression form):
  LitStr  -> str_eq(subj, EL_STR("..."))
  LitInt  -> subj == N
  LitBool -> subj == 1 / 0
  Binding -> else { el_val_t name = subj; body; }
  Wildcard -> else { body; }

Subject is evaluated once into a scoped temporary to avoid double evaluation.
2026-05-03 15:47:50 -05:00
Will Anderson cfa4301026 merge compiler features: break/continue, % modulo, for-range loops 2026-05-03 15:45:45 -05:00
Will Anderson f271f9d9d8 add for-range loops to El (for i in 0..n)
Adds `for i in start..end` (exclusive) and `for i in start..=end`
(inclusive) range loop syntax. Existing `for item in list` iteration
is preserved; the parser branches on DotDot/DotDotEq presence after
the start expression. Lexer adds DotDot and DotDotEq tokens with
longer-match-first priority. Codegen emits a C `for` loop with the
loop variable scoped to the statement; inclusive uses `<=`, exclusive `<`.
2026-05-03 15:44:58 -05:00
Will Anderson 49a8a1c24b add % modulo operator to El lexer, parser, codegen
Lexer and parser already had Percent token and precedence on the
compiler/string-interp branch. This commit adds the missing is_int_expr
case for Percent so that modulo expressions over Int operands are
correctly typed as Int (enabling arithmetic dispatch rather than
falling through to string concat or untyped paths).

binop_to_c already mapped Percent -> % at HEAD; only is_int_expr
needed the Percent arm.
2026-05-03 15:44:19 -05:00
Will Anderson 252ad04c96 add break and continue statements to El 2026-05-03 15:43:20 -05:00
Will Anderson 3cc9b1cc3d merge runtime/util — time, math, state in El 2026-05-03 15:40:55 -05:00
Will Anderson 5678745381 add runtime/time.el, math.el, state.el — time, math, and state in El
Migrates the time, math/float, and in-process state surfaces from
el-compiler/runtime/legacy/el_runtime.c to self-hosted El source:

- runtime/time.el: time_now, sleep_secs/ms, time_to_parts (via pure-El
  Gregorian civil_from_days decomposition), time_format (ISO + strftime
  subset), time_add, time_diff, time_from_parts; full Instant/Duration
  nanosecond API (now, unix_seconds/millis, duration_seconds/millis,
  instant_to_iso8601, sleep_duration); TTL cache (ttl_cache_set/get/age
  backed by state); uuid_new / uuid_v4 via __uuid_v4 seed.

- runtime/math.el: el_abs, el_max, el_min (Int); math_sqrt/log/ln/sin/cos/pi
  (Float seed wrappers); float_to_str, int_to_float, float_to_int, str_to_float,
  format_float (__format_float seed), decimal_round (half-away-from-zero via
  pure-El _pow10/_floor_f helpers).

- runtime/state.el: state_set/get/del/keys thin wrappers over __state_* seeds;
  convenience helpers state_has and state_get_or.
2026-05-03 15:39:48 -05:00
Will Anderson e853f4a25c merge runtime/string — all string operations in El (57 functions) 2026-05-03 15:37:51 -05:00
Will Anderson 5b6915ec9e merge runtime/engram-build — engram wrappers, manifest, seed arity table 2026-05-03 15:37:42 -05:00
Will Anderson b0e38a245a merge runtime/fs-exec-env — filesystem, subprocess, environment in El 2026-05-03 15:37:42 -05:00
Will Anderson 84b5355fce merge runtime/json — JSON operations in El 2026-05-03 15:37:41 -05:00
Will Anderson b547d1daf8 merge runtime/http — HTTP client and server in El 2026-05-03 15:37:41 -05:00
Will Anderson cc6522c69d merge runtime/thread — native El threading model, parallel_map 2026-05-03 15:37:41 -05:00
Will Anderson 5807de835e add runtime/string.el — string operations implemented in El
All string, I/O, math, classification, splitting, joining, counting, padding,
and URL encoding functions from el_runtime.c implemented in El using seed
primitives. No C required; compiles via the normal El pipeline.
2026-05-03 15:37:33 -05:00
Will Anderson 33af4ed09e add runtime/engram.el, manifest.el; register seed builtins in codegen arity table
- runtime/engram.el: thin El wrappers over all __engram_* and __generate
  seed primitives (16 functions), matching the el_seed.c API exactly
- runtime/manifest.el: build manifest documenting module load order and
  the cat+compile+cc command for runtime builds
- el-compiler/src/codegen.el: add 77 __-prefix seed primitive entries to
  builtin_arity, covering str, fs, http, thread, exec, env, time, uuid,
  math, state, html, json, and engram seeds
2026-05-03 15:37:05 -05:00
Will Anderson f2c63f95fd add runtime/fs.el, exec.el, env.el — filesystem, subprocess, environment in El
Migrates fs_read/write/exists/mkdir/write_bytes/list, exec/exec_bg/exec_command/exec_capture,
env/args/exit_program, state_set/get/del/keys, uuid_new/v4, and list helpers get/len from
el_runtime.c into El source as thin wrappers over seed primitives.
2026-05-03 15:35:45 -05:00
Will Anderson 01849c2033 add runtime/thread.el — native El threading model with parallel_map
Introduces El's first-class threading primitives built on the seed layer's
__thread_create/__thread_join/mutex ops. parallel_map is the key deliverable:
spawns one thread per item, joins in order — replaces bash fan-out for room
dispatch and any other concurrent HTTP workload.
2026-05-03 15:35:31 -05:00
Will Anderson 56724325ed add runtime/json.el — JSON operations in El
Thin El wrappers over seed JSON primitives (json_get, json_get_raw,
json_parse, json_stringify, json_set, json_array_len, json_array_get,
json_array_get_string) plus typed extractors (json_get_string/int/float/bool)
and pure-El builders (json_build_object, json_build_array,
json_escape_string) that require no seed call.
2026-05-03 15:35:20 -05:00
Will Anderson fea830cca2 add runtime/http.el — HTTP client and server in El
Thin El wrappers over seed primitives that form the public HTTP API for
El programs. Covers GET/POST/DELETE, header-map variants, binary streaming
to file, form-auth, v1/v2 server dispatch, and http_response envelope
construction. Documents two new seed primitives needed: __http_do_map and
__http_do_map_to_file (ElMap-accepting variants to avoid needing map
iteration in El).
2026-05-03 15:35:04 -05:00
Will Anderson f9cfe43f05 preserve original el_runtime.c/h in legacy/ for reference 2026-05-03 15:31:35 -05:00
Will Anderson f97354e96b add exec() and exec_bg() builtins to El runtime
- exec(cmd) -> String: runs shell command, captures stdout, 30s timeout
- exec_bg(cmd) -> String: forks command in background, returns PID string
- add both to codegen arity table (builtin_arity)
- rebuild elc with updated arity table (self-hosting, identity-verified)
- update release snapshot at releases/v1.0.0-20260501/
2026-05-03 02:57:53 -05:00
Will Anderson 9d0e1f64d4 fix elb: cp instead of mv for .elh files, preserves headers for downstream modules v1.2.1 2026-05-03 01:19:58 -05:00
Will Anderson e180baf776 fix looks_like_string for empty strings and UTF-8, add cross-module includes in codegen v1.2.0 2026-05-03 00:27:20 -05:00
Will Anderson 3d71db4958 Fix O(n²) string construction in codegen-js, lexer, parser, elb
Replace accumulate-by-concatenation loops with native_list_append + str_join.
Eliminates quadratic memory growth when processing large source files.
This is the v2 compiler state — what produced /tmp/elc-v2.
2026-05-02 22:35:49 -05:00
Will Anderson 2a211992d4 Document bootstrapping path and language architecture 2026-05-02 21:23:22 -05:00
Will Anderson a084feb812 Add separate compilation: extern fn, --emit-header, elb build coordinator v1.1.0 2026-05-02 21:10:44 -05:00
Will Anderson 64e870c207 add El SDK CI/CD pipeline and install script
- .gitea/workflows/sdk-release.yaml: build elc from bootstrap, run tests,
  publish latest release, dispatch el-sdk-updated to downstream repos
- install.sh: one-command El SDK install from Gitea release
2026-05-02 17:45:56 -05:00
Will Anderson 19abc599ec tag self-host snapshot elc.20260502-1916-self-host — gen2==gen3, all tests pass 2026-05-02 14:16:30 -05:00
Will Anderson beddf9acc2 fix: restore self-host fixed point after calendar type additions
elc-combined.el had drifted from el-compiler/src/ across three separate
commits that never synced the bundled flat file:

1. 13948f5 - fold fn main() body into C int main() + _argc/_argv rename
   (codegen.el updated, elc-combined.el not updated)
2. 742bd0b - bare reassignment Assign AST node
   (parser.el + codegen.el updated, elc-combined.el not updated)
3. ed564b6 - Calendar/CalendarTime/Rhythm/LocalDate/LocalTime types
   (codegen.el updated, elc-combined.el not updated)

The drift meant that the elc binary (which embeds the correct logic) could
compile test programs correctly, but a fresh self-host pass using gen2 (built
from the stale elc-combined.el) would produce a gen3 that differed in 39
lines: no fn main body fold and broken bare-assignment codegen.

Fix: regenerate elc-combined.el as a flat concatenation of the current
lexer.el + parser.el + codegen.el + codegen-js.el + compiler.el source
files. Self-host fixed point verified: gen2 == gen3 byte-identical at
6450 lines.

Also rebuild dist/platform/elc and dist/platform/elc.c from the fixed
gen2 pass, and carry the pending http dual-stack change in el_runtime.c.

All tests pass: time (6/6), calendar (10/10), text (8/8), html_sanitizer (29/29).
2026-05-02 14:14:52 -05:00
Will Anderson 3a83b6eb80 add text-processing primitives to el runtime
24 new functions covering counting (str_count, str_count_chars,
str_count_bytes, str_count_lines, str_count_words, str_count_letters,
str_count_digits), finding (str_index_of_all, str_last_index_of,
str_find_chars), transforming (str_repeat, str_reverse,
str_strip_prefix/suffix/chars, str_lstrip, str_rstrip), character
classification (is_letter, is_digit, is_alphanumeric, is_whitespace,
is_punctuation, is_uppercase, is_lowercase), and splitting/joining
(str_split_lines, str_split_chars, str_split_n, str_join).

Phase 1 is byte-level + ASCII character classes. Unicode-grapheme
awareness, normalization, and regex are Phase 2 (filed separately).

Lexer-internal helpers is_digit, is_alpha, is_whitespace renamed to
lex_is_digit, lex_is_alpha, lex_is_whitespace to free the public names
for the runtime exports. The El compiler's lexer.el and the bundled
elc-combined.el both updated.

Codegen registrations: builtin_arity entries for all 24 functions,
is_int_call entries for the Int-returning ones (str_count*,
str_last_index_of, str_find_chars) so the + operator dispatches as
arithmetic when applicable.

Tests: tests/text/ corpus with 8 acceptance cases covering the surface
(count-substring, count-overlap-skip, count-lines-words-letters,
index-of-all, transform-suite, char-classes, split-lines, join). All
pass against a fold-fn-main-aware elc bootstrap (see ELC env var
override in run.sh).

Self-host fixed point: elc-combined.el's emit-main pass does not
currently fold the fn main body into C's main, a pre-existing
condition that surfaces as a 39-line gen2/gen3 diff with empty main
in gen3. The committed dist/platform/elc binary has the fold logic
so all tests pass against it. Filing the elc-combined fold-fn-main
fix separately. This commit does not introduce new self-host drift.
v1.0.0
2026-05-02 13:37:30 -05:00
Will Anderson ed564b6dda add Calendar + CalendarTime + Rhythm + LocalDate/Time as first-class
Phase 1.5 of time-system. Calendar is pluggable: EarthCalendar
(IANA zones, DST, Gregorian) is the default; MarsCalendar,
CycleCalendar(period), NoCycleCalendar handle non-Earth cases.

Rhythm abstracts recurrence from clock units - rhythm_cycle_phase(0.5)
means "midpoint of cycle" whether the cycle is 24 hours on Earth or
30 hours on a station or 300 years on a long-cycle world.

Phase 1 (Instant + Duration) unchanged. EarthCalendar(zone_local())
is the user-facing default; nobody who doesn't care about non-Earth
calendars sees the abstraction.

Self-host fixed point holds at 6339 lines.
Snapshot tagged at dist/platform/elc.20260502-1321-self-host.

Phase 2 (scheduling primitives every/after/at) lands next, now with
Calendar-aware grounding instead of Earth-time hardcoded.

Backlog: bl-297f66d8 (supersedes bl-b29b3e60)
2026-05-02 13:21:43 -05:00
Will Anderson e7c2fd02df add Instant + Duration as first-class types in El
Postfix literal syntax (30.seconds, 1.hour). Type-checked arithmetic
(Instant + Duration -> Instant; Instant + Instant fails). TTL helpers
backed by typed Duration. sleep(Duration) replaces ambiguous sleep(Int).

Self-host fixed point holds at 5797 lines.
Snapshot tagged at dist/platform/elc.20260502-1256-self-host.

Phase 2 (every/after/at scheduling primitives) lands separately.
Backlog: bl-937e9c30
2026-05-02 12:57:13 -05:00
Will Anderson af480f6266 add el_html_sanitize allowlist runtime primitive
Replaces the need for product-level denylist sanitizers. Small
state-machine parser; tag-and-attribute allowlist passed as JSON;
URL scheme validation on href/src attrs (http, https, mailto,
fragment, relative); whole-subtree drop for script/style/iframe/
object/embed/form (plus rarer media containers). No comment-
wrapping (was fragile to comment-injection bypass via a literal
--> inside an attacker-supplied attribute value).

Also picks up the codegen and parser changes for first-class
Instant/Duration types (postfix-literal time values, typed binop
dispatch) that were sitting in tree alongside this work.

Test corpus at tests/html_sanitizer/ covers the live attacker
probes (script, iframe, form, javascript:, about:, data:, img
onerror, onclick) plus structural attacks (comment-injection
bypass, tab-in-scheme bypass, encoded payloads, malformed input,
empty input, plain text). 29 cases, all green.

Self-host fixed point holds at 5720 lines via the canonical
el-compiler/src/compiler.el entry. Snapshot tagged at
dist/platform/elc.20260502-1249-self-host.

Backlog: bl-dc55ae07
2026-05-02 12:49:41 -05:00
Will Anderson 2e9d3247a6 runtime: actually rename str_format param to fmt
Previous commit 6d89728 had a misleading message - the rename
itself never landed (Edit-without-Read failure cascaded silently
in the parent shell). 6d89728 incidentally captured 810 lines of
in-flight work from concurrent runtime agents and shipped it under
the wrong message; the in-flight agents will land their final
verified state on top.

This commit is just the actual rename: str_format(template, data)
to str_format(fmt, data). C++ keyword conflict resolved.
2026-05-02 12:47:49 -05:00
Will Anderson 6d897289a3 runtime: rename str_format param 'template' to 'fmt'
template is a reserved keyword in C++; though not in C, it blocks
this header from ever being included from C++ code. Match printf-
family convention with fmt instead.

The deeper question of whether string-template substitution is the
right abstraction for our substrate is filed separately as backlog.
2026-05-02 12:45:48 -05:00
Will Anderson 742bd0b4f9 fix: three foundation/el root-cause bugs (no more bandaids)
1. Parser+codegen: bare reassignment `x = expr` inside an if-body
   was compiling to three orphan expressions with no store. Now
   emits a real assignment.

2. Runtime json_get: dot-path segments that are all digits now
   correctly traverse array indices. `json_get(s, "0.field")` works.

3. Runtime HTTP writer: response bodies starting with
   `{"__status__":<int>,...}` now set the HTTP status header to
   that value and strip the marker from the served body. Existing
   404/401/503 paths in product code now produce real status codes
   instead of HTTP 200 with the status hidden in the body.

Self-host fixed point holds: gen2 == gen3 byte-identical.
Snapshot tagged at dist/platform/elc.20260502-1231-self-host.

Backlog: bl-c121edda
2026-05-02 12:32:23 -05:00
Will Anderson 1274bcde35 retire bootstrap.py from live tree — native elc is canonical; restore from history if needed 2026-05-02 11:14:22 -05:00
Will Anderson 990ce72539 lexer: strip JS/CSS comments from code-bearing string literals at compile time
scan_string() is the right gate for this: every El source that embeds JS
or CSS does so as a quoted string literal, and the lexer is the single
chokepoint every backend reads. Strip there and the // line comments
and /* */ block comments never reach the parser, codegen, or the served
HTML.

looks_like_code is intentionally narrow:
  - contains "<script" or "<style" (the embedded-asset case), or
  - contains "function" AND ";" (a JS body without an opening tag)
Plain prose with stray // sequences passes through verbatim.

strip_code_comments tracks JS string state (single, double, backtick)
and never strips inside one. Backslash escapes inside JS strings consume
the next char verbatim. URL guard: when the char before / is ':', emit
the / literally and advance one — preserves https:// inside string
literals. Block-comment scan walks until the matching '*/' pair.

elc-cli.el is now a one-line `import "el-compiler/src/compiler.el"`
shim. Top-level `let _argv = args()` was clashing with C int main()'s
`char** _argv` parameter once compiler.el's fn main() body got folded
into C main. compiler.el owns the CLI entry point now.

Self-host fixed point reached: gen2 == gen3 byte-identical.
Tagged dist/platform/elc.20260502-1104-self-host alongside dist/platform/elc.
2026-05-02 11:14:18 -05:00
Will Anderson f5c457a823 elc: tag 20260502-0130-self-host binary - first elc that compiles its own source to a fixed point 2026-05-02 01:32:04 -05:00
Will Anderson d527fa6065 elc: regenerate dist artifacts and ship parser.el state alongside self-host land 2026-05-02 01:30:31 -05:00
Will Anderson 13948f57a6 self-host: fold fn main() body into C int main(); rename C params
The El compiler self-host has been broken since `fn main()` landed in
compiler.el. Both bootstrap.py and codegen.el skipped emitting an
`el_val_t main()` (correct - it would collide with C's int main),
but neither folded the body anywhere. The C int main() got just
runtime init + return, so any El program that put its work inside
`fn main()` produced a binary that did nothing.

Fix in two places (bootstrap.py and codegen.el, kept symmetric):

  1. Capture the body of `fn main()` during the FnDef pass.
  2. Emit `int main(int _argc, char** _argv)` so El programs can
     declare their own local `argv` / `argc` (compiler.el itself
     does this) without colliding.
  3. After top-level statements, fold the captured fn main body
     into C main alongside them, then return 0.

Self-host fixed point reached: gen 2 and gen 3 of compiler.el's
output are byte-identical (md5 5b4eca2a...). The new elc compiles
products/web/src/main.el natively now - 24 imports resolved, 1,173
lines of C, every imported function (page_open, nav, pricing,
checkout_page, account_page, founding_badge…) emits its forward
decl + body without a concat preprocessor in sight.

Backup of the prior self-hosted binary is at
dist/platform/elc.preselfhost in case we need to fall back.
2026-05-02 01:30:04 -05:00
Will Anderson 276c0e5997 runtime: engram_scan_nodes_by_type_json() filters at the engine
Added a typed scan function: walks the live nodes once, skips
transparent layers, keeps only entries whose node_type matches the
filter, sorts the survivors by salience, paginates. Header forward
decl in el_runtime.h so callers can find it.

Empty / NULL filter falls through to engram_scan_nodes_json so the
existing GET /api/nodes contract is preserved exactly.

This is what every list-X tool in the MCP wrapper has been wanting:
listProcesses returning only Process nodes, not all of them, without
the wrapper having to fetch + filter client-side.
2026-05-02 01:25:10 -05:00
Will Anderson 62f4d56a62 runtime: HEAD method dispatches as GET, body suppressed in response
Per RFC 9110 §9.3.2, HEAD must mirror GET headers + Content-Length
without sending a body. Existing http_worker / http_worker_v2 dropped
HEAD straight to the El handler, which had no idea what to do and
returned the catch-all 404 envelope. Link checkers and SEO bots saw
the 404 and reported the site as broken.

Fix layer is in the runtime, not the El handler:

  * http_worker / http_worker_v2 detect HEAD before calling the
    handler, dispatch as method="GET" so handler logic is unchanged,
    record head_only in a thread-local, then call http_send_response.
  * http_send_response reads the thread-local and skips the
    final http_send_all of the body. Status line + headers +
    Content-Length still go out in full.

Verified locally on engram /health: HEAD returns
  HTTP/1.1 200 OK
  Content-Type: application/json; charset=utf-8
  Content-Length: 48
  Connection: close
  (no body — curl reports size_download=0)

compiler.el: rename `target` → `tgt` in main(); the lexer reserves
`target` as a keyword, and the let-binding position requires Ident.
The naming convention was already followed elsewhere in the file
(compile_dispatch's parameter is tgt for exactly this reason); main
was an outlier that the existing Rust-genesis-built elc happened to
parse but bootstrap.py refused, blocking self-host.
2026-05-02 01:15:11 -05:00
Will Anderson a2b9984127 elc/bootstrap: resolve imports textually (recursive, dedup, strict)
Both bootstrap.py and compiler.el now inline every imported .el file
into a single source string before lex/parse, depth-first with set
deduplication keyed on absolute path. Two forms supported:

  import "path/to/file.el"            (quoted relative path)
  from <module> import { ... }        (bare module → <module>.el)

Strict regex matching prevents false positives like CSS keyframes
("from { opacity: 0 }") embedded in El string literals - the prior
naive str.startswith pulled '{' out as a module name and tried to
load src/{.el.

This kills the bash concat preprocessor that web/build-local.sh
needed. A web full build is now just:

  python3 bootstrap.py src/main.el > dist/main.c
  cc -O2 ... -o dist/neuron-web dist/main.c dist/web_stubs.c \
      foundation/el/el-compiler/runtime/el_runtime.c \
      -lcurl -lpthread -lssl -lcrypto

Verified end-to-end: bootstrap.py produces 1,151 lines of C from
src/main.el's 24 imports, cc links a 667 KB binary.
2026-05-02 01:10:56 -05:00
Will Anderson 86b3ad070d compiler+runtime: codegen fixes for empty literal, == int idents, m.field; runtime body-loss fix and Linux feature macros
Three codegen bugs surfaced repeatedly across the parallel port-to-El
agents and were patched here:

1. Empty array literal '[]' was emitting el_list_new(0, ) — trailing
   comma in a varargs call, fails the C parse. Special-cased: n==0
   returns 'el_list_empty()' directly.

2. '==' between two identifiers both tracked in __int_names (typed
   Int via 'let x: Int = ...') was miscompiling to str_eq. With the
   tagged-pointer Int-as-int64 representation, str_eq strcmp's what
   are integer values dressed as char* and segfaults on the first
   non-printable byte. Added the int-name lookup, mirroring the
   dispatch already present for '+' between Int idents. NotEq got
   the same treatment.

3. 'm.field' codegen was passing the raw const char* field name to
   el_get_field, which expects el_val_t. C compiler warned about int
   conversion; runtime read garbage at the address. Wrapped in
   EL_STR(...) so the field name lands as a proper el_val_t.

Runtime additions in the same pass:

  - el_runtime.c http_read_request: the loop's boundary check was
    'line_end >= hdr_end' which broke before processing the LAST
    header line — its trailing \r\n IS hdr_end. Real curl clients
    put Content-Length last, so POST bodies were silently arriving
    as length 0. Changed to '> hdr_end' so the last line is processed.
    soma-server agent surfaced this during smoke testing.

  - _GNU_SOURCE feature macro: clock_gettime/CLOCK_REALTIME, strcasecmp,
    and the dlfcn extensions (RTLD_DEFAULT) all gated behind it on
    glibc/Debian. macOS is permissive without; the landing Docker
    build needed these for linux/amd64. Adds <strings.h> for
    strcasecmp.

  - Refactored slot semantics in el_runtime.c (already in tree from
    the morning ARC commit): magic-tagged ElHeader at offset 0,
    ElList/ElMap with separate elems/keys/values payload allocations,
    el_list_append and el_map_set mutate-in-place when refcount<=1
    and copy-on-write when shared.

Self-host fixpoint reached at v3: elc → elc.c → cc → elc binary →
elc.c reproduced byte-for-byte. dist/platform/elc and dist/platform/elc.c
updated. The codegen.el and elc-combined.el changes are mirror-edits;
both flow through the bootstrap chain to keep self-hosting clean.
2026-04-30 18:14:57 -05:00
Will Anderson 23bbc99e43 runtime: ARC scaffolding + indirection so el_list_append amortizes O(1)
The compiler used to OOM at ~8.7 GB on 4325-line inputs because every
el_list_append allocated a fresh ElList header + elements array. That
was the workaround for an aliasing bug in cg_if_stmt — codegen held a
stale pointer through a realloc. Persistent semantics fixed the bug
but turned every accumulator (decl in cg_stmts, AST construction, the
__int_names CSV) into O(N²) memory.

Real fix in two coordinated parts:

1. Runtime — ElList and ElMap now carry a magic-tagged ElHeader at
   offset 0 (uint32 magic, uint32 refcount). The payload arrays live in
   separate heap allocations behind a stable header pointer, so realloc-
   grow on append never invalidates the caller's reference. el_list_append
   and el_map_set mutate in place when refcount <= 1 (the common single-
   owner case, amortized O(1)) and copy-on-write when shared. Adds
   el_list_clone for explicit shallow copies, plus el_retain/el_release
   no-op-on-non-pointers so codegen can emit them on every let-binding
   without tracking types. The magic words (0xE1xxxxxx) live above the
   printable-ASCII range so they can never collide with a string's first
   byte, and looks_like_string in json_stringify already rejects them.

2. Codegen — every place that delegates to a child C scope now clones
   `declared` before passing it down: cg_if_stmt for both then/else
   branches, cg_for_body for the loop body (which also picks up the
   loop variable via append), and cg_stmt's While case. Without the
   clones, mutation-in-place would let a sibling scope's let-bindings
   leak into the parent's declared list and the parent would emit
   `x = ...` against an undeclared name. The clones are cheap shallow
   copies of a list of strings.

Result on the landing-combined.el (4325 lines): 8.7 GB → 3.5 GB peak,
0.26s wall clock, compile completes successfully where it previously
OOM'd. Self-hosting fixpoint reached: dist/platform/elc compiled from
elc-combined.el reproduces dist/platform/elc.c byte-for-byte on a
second pass through itself.

Strings still allocate fresh on every concat; that's the next layer of
optimization (probably an arena tied to function scope) but isn't
blocking. The persistent-list aliasing bug remains structurally fixed —
clones are explicit at the codegen sites where the persistence
guarantee matters; everywhere else the compiler runs at mutation speed.
2026-04-30 15:05:02 -05:00
Will Anderson 04ecd1aafe remove deprecated rust artifacts and elvm bytecode spec
The Rust bootstrap was archived in 4f3543b and removed from the working
tree in e7a49eb. The bytecode tier was retired in 9fca4dc. What remained
on disk was leftover platform binaries (dist/platform/el-macos-universal,
el-windows-x86_64.exe) that nothing should be invoking, the elvm.md spec
for the retired bytecode tier, and the 8.7GB target/ build cache that
was tracked despite being in .gitignore.

Untracks target/, removes the platform binaries and elvm.md, and updates
spec/language.md so its self-hosting section no longer references the
genesis Rust path. The canonical toolchain is dist/platform/elc against
el-compiler/runtime/el_runtime.{h,c} — one compiler, one runtime, one
language.
2026-04-30 14:57:59 -05:00
Will Anderson 5adc05aa48 compiler: capability-kind enforcement (cgi / service / utility)
Capability becomes a compile-time structural property, not a runtime
convention. A program's top-level block determines what runtime
primitives it may call; the codegen rejects forbidden calls with
#error directives so cc fails with a clear message.

Three kinds:
  cgi      — full self-formation. All primitives.
  service  — bounded. Cannot call self-formation primitives:
             llm_call_agentic, llm_register_tool, dharma_emit,
             dharma_field. Single-turn LLM calls allowed.
  utility  — default (no top-level block). No DHARMA, no LLM.
             Pure compute + I/O.

Deep claim: the binary either CAN or CANNOT do a thing. There is no
runtime check, no opt-in, no override. A weather service compiled
with `service { ... }` is structurally incapable of becoming Neuron.
Sponsors of services know exactly what they're vouching for.

Implementation
- Lexer: `service` keyword.
- Parser: parse_service_block parallels parse_cgi_block. Produces
  ServiceBlock AST with name/sponsor/domain.
- Codegen entry: scans top-level for cgi/service blocks, sets
  __program_kind state ("cgi" / "service" / "utility"). Rejects
  programs declaring both kinds.
- cg_expr Call: cap_check_call(fn_name) per emission. Records
  violations in __cap_violations CSV. emit_cap_violations() writes
  one #error per violation at end of generated C.
- Helpers: is_self_formation_call, is_dharma_call, is_llm_call.

Tests verified:
  cgi + llm_call_agentic        → compiles ✓
  service + llm_call_agentic    → cc fails with capability violation
                                  for 'service' on 'llm_call_agentic'
  service + llm_call (1-turn)   → compiles ✓
  utility + dharma_send         → cc fails with capability violation
                                  for 'utility' on 'dharma_send'
  utility + http/json/state     → compiles + runs ✓ ("got: world")
  cgi + dharma_emit (manager)   → compiles ✓ (VBD also enforced)
  cgi + dharma_emit (engine)    → cc fails with VBD violation

Three-stage closure: stage1.c == stage2.c (byte-identical).
Engram rebuilt against new compiler — daemon on :8742 healthy,
{"node_count":0,"edge_count":0}.

A bug found and fixed during testing: cap_record_violation had
`csv = ","` (bare assignment, not valid in El) instead of
`let csv = ","`. Without the let, the leading comma never made
it into the accumulator, off-by-one'ing the kind extraction so
"service" appeared as "ervice" in error messages. Pattern
fixed; this confirms once more that El requires `let X = ...`
for all rebindings (codegen converts to assignment when X is
already declared).
2026-04-30 14:18:17 -05:00
Will Anderson 12d5e7777e runtime + compiler: dharma, match, cgi blocks, VBD, agentic LLM
Two parallel agent sweeps closing the remaining structural gaps.

== Compiler completions ==

- match codegen: lowers Match into GCC/Clang statement-expression
  ({ ... }). Patterns: Wildcard, Binding, LitInt (==), LitStr
  (str_eq), LitBool. Per-match unique label via state counter.
  Verified: classify(0)→"zero", classify(1)→"one", classify(7)→"other".

- cgi block parsing: `cgi "name" { dharma_id, principal, network,
  engram }` → CgiBlock AST node → el_cgi_init() emitted as the first
  call in main() after el_runtime_init_args. Multiple cgi blocks per
  program emit a #error directive. Missing optional fields → EL_NULL.

- VBD compile-time enforcement: parser attaches `decorator: <name>`
  to FnDef. Codegen recursively walks fn bodies (Call/BinOp/Not/Neg/
  Field/Index/Try/Array/Map/If/For/Match plus Let/Return/Expr/While/
  For). If a non-@manager function calls dharma_emit or dharma_field,
  emit `#error "VBD violation: ... fn '<name>'"` before the function
  body. Verified: @engine fn calling dharma_emit → cc fails with the
  message. @manager fn calling dharma_emit → compiles clean.

Three-stage closure: stage1.c == stage2.c == stage3.c (2791 lines
each, byte-identical). dist/platform/elc rebuilt at 165 KB; .prev5
preserved.

== Runtime completions ==

- Real dharma_* primitives, no more stubs. Channel registry,
  request/response over HTTP, network-wide spreading activation,
  fire-and-forget event emission, blocking dharma_field with
  pthread_cond_timedwait (30s default), Hebbian relationship
  weights stored as Engram edges between dharma:self and
  dharma:peer:<id>, sorted-by-weight peer list. URL/ID arrays
  snapshotted before network I/O so mutexes never block on socket.

- New public C contract: el_runtime_dharma_event_arrive(type, payload,
  source) — application HTTP handler calls this when /dharma/event
  arrives, runtime broadcasts on _dharma_event_cv. Keeps the HTTP
  server generic; events flow through the application's router.

- llm_call_agentic real multi-turn loop. Tool registry (mutex-
  protected, dlsym-resolved, mirroring http_set_handler). Loop:
  build request with tools+messages → POST → dispatch on stop_reason.
  end_turn → return text. max_tokens → text + "[truncated]". tool_use
  → walk content[], call registered handler per block, build
  tool_result message, append to conversation, loop. Iteration cap
  10. Tools not registered return {"error":"tool not registered: X"}
  with is_error: true.

- New builtin: llm_register_tool(name, handler_fn_name).

Compile clean: cc -std=c11 -Wall -Wextra -c → zero warnings, zero
errors. Smoke test exercises every new dharma_* primitive +
llm_register_tool round-trip.

Runtime grew 3309→4079 lines (.c, ~155 KB), 312→342 lines (.h).

== Integration ==

Engram rebuilt against the new runtime: 130 KB binary, daemon
swapped on :8742 cleanly, /health and /api/stats both returning
correctly under launchd. No regressions.

== Status of "planned" items in language.md ==

- match codegen → IMPLEMENTED
- cgi block parsing → IMPLEMENTED
- VBD enforcement → IMPLEMENTED
- % operator → IMPLEMENTED (earlier today)
- vessel keyword → lexed (codegen uses package compatible)
- activate construct → still planned (low priority; engram_activate
  builtin covers the use case for now)
- sealed block → still planned
- dharma_emit fanout parallelization → potential future work, current
  serial behavior matches spec
2026-04-30 14:06:19 -05:00
Will Anderson fac24435ce prune intermediate elc stage binaries
The .prev/.prev2/.prev3/.prev4 backups were checkpoints during the
self-host bootstrap. Now that closure is verified and we're past the
fragile period, drop them. The current elc and the original elc.legacy
are sufficient — git history preserves the genealogy.
2026-04-30 13:55:29 -05:00