neuron-technologies/el
2
0
Fork 0
Code Issues 1 Pull Requests 6 Actions Packages Projects Releases 5 Wiki Activity

hotfix(elc): bound token reads to Eof so malformed input errors instead of OOMing #65

Open
will.anderson wants to merge 0 commits from hotfix/elc-fixes into main
pull from: hotfix/elc-fixes
merge into: :main
Conversation 0 Commits - Files Changed -
will.anderson commented 2026-07-14 19:22:31 +00:00
Owner
Copy Link
Copy Source

Compiler hardening: bound token reads so malformed input errors instead of OOMing

Root cause

tok_kind/tok_value (parser.el:25-31) read the flat token list with no
bounds check. native_list_get out-of-range returns runtime null (0), NOT the
"Eof" sentinel. The inner parse loops — parse_block (1191), call-args
(1282), array literals (792), match arms (1104) — terminate only on their
closing delimiter or k == "Eof", and (unlike the top-level parse() loop at
1937) have no pos >= total bound. expect() (39) advances the cursor even on
mismatch, so one unclosed delimiter steps the cursor past the single trailing
Eof token. From there every == "Eof" guard is blind, parse_primary's
fallback keeps advancing +1 and appending a Nil node, and native_list_append
grows the statement list forever → ~700GB → OOM (observed compiling
neuron/sessions.el; also reproduces on fn f()->Int{ return g(1,2 }).

Fix

Single choke-point change: tok_kind returns "Eof" and tok_value returns
"" for out-of-range positions, restoring the parser-wide contract that reads
at/after the end yield Eof. Every overrun loop then terminates. expect() no
longer steps past the Eof sentinel on mismatch. A malformed program now ends
the parse (best-effort recovery) instead of exhausting memory. Low risk:
in-range behavior is unchanged; the guards are purely additive.

Verification

  • The pathology was reproduced under an RSS-watchdog (macOS ignores
    ulimit -v): unfixed elc on fn f()->Int{ return g(1,2 } blew past 1.5GB in
    ~1s and was killed (exit 137). Every run was bounded; the machine stayed
    ~5.7GB free.
  • This source change to the self-hosted compiler is not yet runtime-verified
    because that requires a bootstrap rebuild of elc, which was deliberately out
    of scope here (rebuilding the live compiler is memory-dangerous and needs
    careful review).

DO NOT MERGE without a bootstrap rebuild

This edits the self-hosted parser. It only takes effect after elc is rebuilt
from this source (bootstrap) and the new binary is validated against the full
corpus. Please review + rebuild + regression-test before merging. The live elc
binary and soul were not touched.

The companion source-typo fixes that actually unblock compilation today
(no rebuild needed) are in neuron-technologies/neuron#<see hotfix/elc-source-typos>.

## Compiler hardening: bound token reads so malformed input errors instead of OOMing ### Root cause `tok_kind`/`tok_value` (parser.el:25-31) read the flat token list with no bounds check. `native_list_get` out-of-range returns runtime null (0), NOT the `"Eof"` sentinel. The inner parse loops — `parse_block` (1191), call-args (1282), array literals (792), match arms (1104) — terminate only on their closing delimiter or `k == "Eof"`, and (unlike the top-level `parse()` loop at 1937) have no `pos >= total` bound. `expect()` (39) advances the cursor even on mismatch, so one unclosed delimiter steps the cursor past the single trailing Eof token. From there every `== "Eof"` guard is blind, `parse_primary`'s fallback keeps advancing +1 and appending a Nil node, and `native_list_append` grows the statement list forever → ~700GB → OOM (observed compiling neuron/sessions.el; also reproduces on `fn f()->Int{ return g(1,2 }`). ### Fix Single choke-point change: `tok_kind` returns `"Eof"` and `tok_value` returns `""` for out-of-range positions, restoring the parser-wide contract that reads at/after the end yield Eof. Every overrun loop then terminates. `expect()` no longer steps past the Eof sentinel on mismatch. A malformed program now ends the parse (best-effort recovery) instead of exhausting memory. Low risk: in-range behavior is unchanged; the guards are purely additive. ### Verification - The pathology was reproduced under an RSS-watchdog (macOS ignores `ulimit -v`): unfixed elc on `fn f()->Int{ return g(1,2 }` blew past 1.5GB in ~1s and was killed (exit 137). Every run was bounded; the machine stayed ~5.7GB free. - This source change to the self-hosted compiler is **not yet runtime-verified** because that requires a bootstrap rebuild of elc, which was deliberately out of scope here (rebuilding the live compiler is memory-dangerous and needs careful review). ### DO NOT MERGE without a bootstrap rebuild This edits the self-hosted parser. It only takes effect after elc is rebuilt from this source (bootstrap) and the new binary is validated against the full corpus. Please review + rebuild + regression-test before merging. The live elc binary and soul were not touched. The companion source-typo fixes that actually unblock compilation **today** (no rebuild needed) are in neuron-technologies/neuron#<see hotfix/elc-source-typos>.
will.anderson added 1 commit 2026-07-14 19:22:32 +00:00
parser: bound token reads to Eof so malformed input errors instead of OOMing
El SDK Release / build-and-release (pull_request) Failing after 16s
Details
0a0a2bcb44 
Out-of-range tok_kind/tok_value reads returned runtime null (el_list_get OOB
-> 0) rather than the Eof sentinel, so the inner parse loops (parse_block,
call-arg, array-literal, match-arm) that terminate only on their close
delimiter or k=="Eof" never saw Eof once the cursor ran past the single
trailing Eof token. On unclosed-delimiter input the parser then appended AST
nodes forever -> unbounded allocation -> ~700GB -> OOM (observed compiling
neuron/sessions.el).

Fix at the choke point: tok_kind returns "Eof" and tok_value returns "" for
out-of-range positions, restoring the parser-wide contract that reads at/after
the end yield Eof. expect() no longer steps past the Eof sentinel on mismatch.
This terminates every overrun loop simultaneously; a malformed program now
surfaces as a normal (best-effort) parse end instead of exhausting memory.

Requires a self-hosted bootstrap rebuild of elc to take effect.
Some checks are pending
El SDK Release / build-and-release (pull_request) Failing after 16s
Details
branch-policy
Required
This branch is already included in the target branch. There is nothing to merge.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin hotfix/elc-fixes:hotfix/elc-fixes
git checkout hotfix/elc-fixes
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
tim.lingo (Tim Lingo) will.anderson (Will Anderson)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/el#65
Delete Branch "hotfix/elc-fixes"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?