neuron-technologies/el
3
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases 5 Wiki Activity

self-host the el compiler

Browse Source 
Today's milestone: dist/platform/elc compiles itself byte-for-byte to
itself (stage1 == stage2 == stage3 verified). The compiler is now a
real binary in the world.

What landed
- Spec rewrite (language.md) to truth — every feature marked
  implemented / planned / not-in-this-language with no fiction.
- C runtime extension: 51 new builtins. JSON parser + accessors,
  time, UUID, env, in-process state K/V, float formatting + math,
  string ops (index_of, split, char_at, char_code, pad_left/right,
  format), list ops (push, push_front, join, range), bool_to_str.
  Runtime grew 631 → 1611 lines, header 171 → 247.
- Codegen fix: transform_implicit_return lifts a function's bare
  trailing expression into an explicit return. Without it, lex(),
  parse(), and every other implicit-return function returned 0/nil
  and the whole pipeline produced empty C output.
- Codegen fix: index expressions dispatch on AST kind. obj["literal"]
  → el_get_field (map), arr[i] → el_list_get (list). Same Index node
  in the parser, two different runtime calls.
- Codegen fix: skip emitting fn main() (collides with C main()) and
  honor parsed return-type annotations so Void functions don't get
  return-wrapped (return println(x) is a C type error).
- Parser: capture return-type identifier from -> Ret annotations.
- Lexer: + vessel keyword, + % operator, + \r escape.
- Runtime fix: el_list_append now allocates a fresh list rather than
  realloc'ing the input. Realloc moved blocks made caller pointers
  dangle, which was inserting garbage values into declared lists and
  causing strcmp segfaults. Persistent allocation eliminates the
  whole class of use-after-free at modest memory cost.

Bootstrap path
- One-shot Python helper translated elc-combined.el to C and
  produced stage1. Helper is disposable; not committed.
- stage1 compiles elc-combined.el → stage2.c which cc compiles to
  stage2; stage2 compiles elc-combined.el → stage3.c. stage2.c and
  stage3.c are byte-identical. Closure proven.
- New elc installed at dist/platform/elc; old broken binary
  preserved as dist/platform/elc.legacy.
- dist/platform/elc.c is the canonical generated source.
- elvm and the bytecode pipeline are no longer on the critical path.

Known gap
- The `+` operator's heuristic dispatch still picks string concat
  when both operands are Idents with no literal anchor. Self-hosting
  works because the compiler source is careful, but `fn add(a:Int,
  b:Int) { a + b }` will not do arithmetic until codegen reads the
  parsed type annotations to dispatch. Fix is wiring; not done here.

Tested
- tiny / lextest / whiletest / map+field / array build all run.
- cgi-studio (1037 lines real El) compiles to C cleanly. Link fails
  only because runtime is missing fs_list, json_encode, llm_*; those
  are scheduled batches.
- Three-stage closure (stage1 vs stage2 vs stage3) byte-identical.
This commit is contained in:
Will Anderson
2026-04-30 13:10:29 -05:00
parent e7a49ebc34
commit 5c05ce9b99
11 changed files with 6890 additions and 954 deletions
Download Patch File Download Diff File Expand all files Collapse all files
el-compiler/src/lexer.el
+44 -31
View File
@@ -7,8 +7,8 @@
//
// Entry point: fn lex(source: String) -> [Map<String, Any>]
//
// Uses global char_buf via native_chars_init / native_char_at / native_char_len
// to avoid O(N²) cloning of the chars list.
// Uses native_string_chars to split the source into a chars list,
// then indexes it with native_list_get — avoids O(N²) string cloning.
// ── Character helpers ─────────────────────────────────────────────────────────
@@ -140,15 +140,20 @@ fn keyword_kind(word: String) -> String {
if word == "target" { return "Target" }
if word == "true" { return "Bool" }
if word == "false" { return "Bool" }
if word == "cgi" { return "Cgi" }
if word == "manager" { return "Manager" }
if word == "engine" { return "Engine" }
if word == "accessor" { return "Accessor" }
if word == "vessel" { return "Vessel" }
""
}
// ── Scan helpers ──────────────────────────────────────────────────────────────
// All scan helpers use the global char_buf (native_char_at / native_char_len).
// No chars parameter — avoids O(N²) cloning.
// All scan helpers receive the chars list and total length.
// scan_digits — advance i while char_buf[i] is a digit, return { "text": ..., "pos": i }
fn scan_digits(start: Int, total: Int) -> Map<String, Any> {
// scan_digits — advance i while chars[i] is a digit
// Returns { "text": ..., "pos": i }
fn scan_digits(chars: [String], start: Int, total: Int) -> Map<String, Any> {
let i = start
let text = ""
let running = true
@@ -156,7 +161,7 @@ fn scan_digits(start: Int, total: Int) -> Map<String, Any> {
if i >= total {
let running = false
} else {
let ch = native_char_at(i)
let ch: String = native_list_get(chars, i)
if is_digit(ch) {
let text = text + ch
let i = i + 1
@@ -168,8 +173,8 @@ fn scan_digits(start: Int, total: Int) -> Map<String, Any> {
{ "text": text, "pos": i }
}
// scan_ident — advance i while char_buf[i] is alphanumeric or underscore
fn scan_ident(start: Int, total: Int) -> Map<String, Any> {
// scan_ident — advance i while chars[i] is alphanumeric or underscore
fn scan_ident(chars: [String], start: Int, total: Int) -> Map<String, Any> {
let i = start
let text = ""
let running = true
@@ -177,7 +182,7 @@ fn scan_ident(start: Int, total: Int) -> Map<String, Any> {
if i >= total {
let running = false
} else {
let ch = native_char_at(i)
let ch: String = native_list_get(chars, i)
if is_alnum_or_underscore(ch) {
let text = text + ch
let i = i + 1
@@ -191,21 +196,20 @@ fn scan_ident(start: Int, total: Int) -> Map<String, Any> {
// scan_string — scan a quoted string literal, handling \" escapes.
// Starts AFTER the opening quote. Returns { "text": content, "pos": i_after_close }
fn scan_string(start: Int, total: Int) -> Map<String, Any> {
fn scan_string(chars: [String], start: Int, total: Int) -> Map<String, Any> {
let i = start
let text = ""
let closed = false
let running = true
while running {
if i >= total {
let running = false
} else {
let ch = native_char_at(i)
let ch: String = native_list_get(chars, i)
if ch == "\\" {
// escape: peek next char
let next_i = i + 1
if next_i < total {
let next_ch = native_char_at(next_i)
let next_ch: String = native_list_get(chars, next_i)
if next_ch == "\"" {
let text = text + "\""
let i = next_i + 1
@@ -218,12 +222,17 @@ fn scan_string(start: Int, total: Int) -> Map<String, Any> {
let text = text + "\t"
let i = next_i + 1
} else {
if next_ch == "\\" {
let text = text + "\\"
if next_ch == "r" {
let text = text + "\r"
let i = next_i + 1
} else {
let text = text + next_ch
let i = next_i + 1
if next_ch == "\\" {
let text = text + "\\"
let i = next_i + 1
} else {
let text = text + next_ch
let i = next_i + 1
}
}
}
}
@@ -233,7 +242,6 @@ fn scan_string(start: Int, total: Int) -> Map<String, Any> {
}
} else {
if ch == "\"" {
let closed = true
let i = i + 1
let running = false
} else {
@@ -249,13 +257,13 @@ fn scan_string(start: Int, total: Int) -> Map<String, Any> {
// ── Main lexer ────────────────────────────────────────────────────────────────
fn lex(source: String) -> [Map<String, Any>] {
native_chars_init(source)
let total: Int = native_char_len()
let chars: [String] = native_string_chars(source)
let total: Int = native_list_len(chars)
let tokens: [Map<String, Any>] = native_list_empty()
let i: Int = 0
while i < total {
let ch: String = native_char_at(i)
let ch: String = native_list_get(chars, i)
// Skip whitespace
if is_whitespace(ch) {
@@ -265,7 +273,7 @@ fn lex(source: String) -> [Map<String, Any>] {
if ch == "/" {
let next_i = i + 1
if next_i < total {
let next_ch: String = native_char_at(next_i)
let next_ch: String = native_list_get(chars, next_i)
if next_ch == "/" {
// skip to end of line
let i = i + 2
@@ -274,7 +282,7 @@ fn lex(source: String) -> [Map<String, Any>] {
if i >= total {
let running2 = false
} else {
let lch: String = native_char_at(i)
let lch: String = native_list_get(chars, i)
if lch == "\n" {
let running2 = false
} else {
@@ -293,7 +301,7 @@ fn lex(source: String) -> [Map<String, Any>] {
} else {
// String literal
if ch == "\"" {
let result = scan_string(i + 1, total)
let result = scan_string(chars, i + 1, total)
let str_text: String = result["text"]
let new_pos: Int = result["pos"]
let tokens = native_list_append(tokens, make_tok("Str", str_text))
@@ -301,18 +309,18 @@ fn lex(source: String) -> [Map<String, Any>] {
} else {
// Number literal
if is_digit(ch) {
let result = scan_digits(i, total)
let result = scan_digits(chars, i, total)
let num_text: String = result["text"]
let new_pos: Int = result["pos"]
// check for float (dot followed by digit)
if new_pos < total {
let dot_ch: String = native_char_at(new_pos)
let dot_ch: String = native_list_get(chars, new_pos)
if dot_ch == "." {
let after_dot = new_pos + 1
if after_dot < total {
let after_dot_ch: String = native_char_at(after_dot)
let after_dot_ch: String = native_list_get(chars, after_dot)
if is_digit(after_dot_ch) {
let frac_result = scan_digits(after_dot, total)
let frac_result = scan_digits(chars, after_dot, total)
let frac_text: String = frac_result["text"]
let frac_pos: Int = frac_result["pos"]
let tokens = native_list_append(tokens, make_tok("Float", num_text + "." + frac_text))
@@ -336,7 +344,7 @@ fn lex(source: String) -> [Map<String, Any>] {
} else {
// Identifier or keyword
if is_alpha(ch) || ch == "_" {
let result = scan_ident(i, total)
let result = scan_ident(chars, i, total)
let word: String = result["text"]
let new_pos: Int = result["pos"]
let kw = keyword_kind(word)
@@ -351,7 +359,7 @@ fn lex(source: String) -> [Map<String, Any>] {
let peek_i = i + 1
let peek_ch = ""
if peek_i < total {
let peek_ch = native_char_at(peek_i)
let peek_ch: String = native_list_get(chars, peek_i)
}
if ch == "=" {
@@ -442,6 +450,10 @@ fn lex(source: String) -> [Map<String, Any>] {
if ch == "*" {
let tokens = native_list_append(tokens, make_tok("Star", "*"))
let i = i + 1
} else {
if ch == "%" {
let tokens = native_list_append(tokens, make_tok("Percent", "%"))
let i = i + 1
} else {
if ch == "(" {
let tokens = native_list_append(tokens, make_tok("LParen", "("))
@@ -501,6 +513,7 @@ fn lex(source: String) -> [Map<String, Any>] {
}
}
}
}
}
}
}