test framework phase 1: compile-time registry + El-side runner with per-test timing #133

2026-08-16T02:28:58Z

will.anderson commented

2026-08-16 02:28:58 +00:00

Phase 1 of the El test framework (DESIGN.md)

Replaces the hardcoded test harness main() with a compile-time registry and moves all reporting into an El-side runner.

Why

The old harness inlined direct calls into main() and counted assertions in two globals (__el_pass/__el_fail). That shape cannot report which test failed, how long any test took, or whether a test ran at all — registration paired a string to a function name by hand, so a typo reported success for a test that never executed.

It also had no timing. That is why a 3.58s test file sat in the suite unnoticed, and why the compiler's quadratic went unmeasured for months.

What changed

Compile-time registry. test { } blocks lower to a static __el_registry[] table. Discovery strictly precedes execution — the precondition for --list, filtering, sharding and per-test reporting in later phases.
Index-based accessors. El has no function pointers, so the runner works purely in indices (__el_reg_count/name/invoke/last_ns/msg/asserts). This is the entire seam between generated C and El.
Per-test results. Assertions record into per-test state, not global counters. The test is the unit of result.
Per-test wall timing, always on, taken in C on CLOCK_MONOTONIC immediately around the call, so no El call overhead lands in the measurement.
Runner in El (lang/runtime/eltest.el). Structured NDJSON events are the source of truth; human output renders from the same fields. We do not parse human output back into structure (Go's test2json mistake).

Verification

Self-hosting fixpoint holds — gen2 and gen3 byte-identical (367,901 bytes).

ok    arithmetic  (0.001ms)
FAIL  deliberate failure  (0.000ms)
      one is not two
3 tests, 2 passed, 1 failed, 4 assertions in 0.001ms

{"action":"fail","test":"deliberate failure","elapsed":0.000,"assertions":1,"failures":1,"message":"one is not two"}

6 of 11 suites now run and report per-test timing (test_compiler 82 tests, test_string 27, test_math 13, test_text 12, test_core 10, test_env 9).

The other 5 fail to COMPILE — and fail identically under the committed compiler. Pre-existing breakage (fs_list_json, state_has, assert_true undeclared), surfaced for the first time because nothing previously reported suite-level state. Not addressed here.

Deliberately not in this PR

runtime/test.el is not deleted. The end state is delete-not-shim, but 10 files (lang/tests/suite/src/*, lang/swarm/tests/*) still use its API. That is the Phase 5 migration PR.
file/line in the registry. The lexer carries no line numbers; adding them touches every token and risks the fixpoint. Phase 2.

Design update

DESIGN.md §6.5 corrected: expect allocs O(n) must fit count AND bytes and fail if either exceeds the declared curve. A quadratic accumulator's allocation count is exactly linear — only bytes reveal it (ratios 3.94/3.97/3.99 → 4.0). That is elc's own defect shape.

## Phase 1 of the El test framework (DESIGN.md) Replaces the hardcoded test harness `main()` with a **compile-time registry** and moves all reporting into an El-side runner. ### Why The old harness inlined direct calls into `main()` and counted assertions in two globals (`__el_pass`/`__el_fail`). That shape cannot report which test failed, how long any test took, or **whether a test ran at all** — registration paired a string to a function name by hand, so a typo reported success for a test that never executed. It also had no timing. That is why a 3.58s test file sat in the suite unnoticed, and why the compiler's quadratic went unmeasured for months. ### What changed - **Compile-time registry.** `test { }` blocks lower to a static `__el_registry[]` table. Discovery strictly precedes execution — the precondition for `--list`, filtering, sharding and per-test reporting in later phases. - **Index-based accessors.** El has no function pointers, so the runner works purely in indices (`__el_reg_count/name/invoke/last_ns/msg/asserts`). This is the entire seam between generated C and El. - **Per-test results.** Assertions record into per-test state, not global counters. The test is the unit of result. - **Per-test wall timing, always on**, taken in C on `CLOCK_MONOTONIC` immediately around the call, so no El call overhead lands in the measurement. - **Runner in El** (`lang/runtime/eltest.el`). Structured NDJSON events are the source of truth; human output renders from the same fields. We do not parse human output back into structure (Go's `test2json` mistake). ### Verification Self-hosting fixpoint holds — gen2 and gen3 byte-identical (367,901 bytes). ``` ok arithmetic (0.001ms) FAIL deliberate failure (0.000ms) one is not two 3 tests, 2 passed, 1 failed, 4 assertions in 0.001ms ``` ```json {"action":"fail","test":"deliberate failure","elapsed":0.000,"assertions":1,"failures":1,"message":"one is not two"} ``` 6 of 11 suites now run and report per-test timing (`test_compiler` 82 tests, `test_string` 27, `test_math` 13, `test_text` 12, `test_core` 10, `test_env` 9). **The other 5 fail to COMPILE — and fail identically under the committed compiler.** Pre-existing breakage (`fs_list_json`, `state_has`, `assert_true` undeclared), surfaced for the first time because nothing previously reported suite-level state. Not addressed here. ### Deliberately not in this PR - **`runtime/test.el` is not deleted.** The end state is delete-not-shim, but 10 files (`lang/tests/suite/src/*`, `lang/swarm/tests/*`) still use its API. That is the Phase 5 migration PR. - **file/line in the registry.** The lexer carries no line numbers; adding them touches every token and risks the fixpoint. Phase 2. ### Design update DESIGN.md §6.5 corrected: `expect allocs O(n)` must fit **count AND bytes** and fail if either exceeds the declared curve. A quadratic accumulator's allocation *count* is exactly linear — only bytes reveal it (ratios 3.94/3.97/3.99 → 4.0). That is elc's own defect shape.

will.anderson added 3 commits 2026-08-16 02:29:01 +00:00

test framework phase 1: compile-time registry + El-side runner 24fac765a6

Replace the hardcoded test harness main() with a generated static registry
and index-based accessors, and move all reporting into runtime/eltest.el.

The old harness inlined direct calls into main() and counted assertions in
two globals. That shape cannot report which test failed, how long any test
took, or whether a test ran at all -- a misspelled registration reported
success for a test that never executed.

- assertions record into per-test state instead of global counters
- registry table emitted at compile time; discovery strictly precedes
  execution, which is what later enables --list, filtering and sharding
- per-test wall timing on CLOCK_MONOTONIC, taken in C around the call
- runner in El: structured NDJSON events as source of truth, human output
  rendered from the same fields

Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile a668062e38

test framework phase 1: forward decls, void-return fix, suite migration

El SDK CI - dev / build-and-test (pull_request) Failing after 10m4s

Details

3e7ab07e82

Completes the Phase 1 runner and migrates the 11 test files onto it.

- forward-declare the registry accessors in the test preamble; they are
  defined at the end of the unit but the El runner is compiled in between
- eltest.el: explicit trailing return in the void emit_* helpers, which
  otherwise lower to 'return println(...)' and fail to compile
- test files import runtime/eltest.el explicitly, using the language's own
  textual import mechanism rather than compiler-side auto-injection
- DESIGN.md 6.5: gate on allocation COUNT AND BYTES, not count alone

Verified: self-hosting fixpoint byte-identical (gen2 == gen3). 6 of 11
suites run and report per-test timing. The other 5 fail to COMPILE, and
fail identically under the committed compiler -- pre-existing breakage
this framework makes visible for the first time.

will.anderson merged commit 5e3e69d326 into dev

2026-08-16 02:31:10 +00:00

will.anderson referenced this issue from a commit

2026-08-16 02:33:27 +00:00

runtime: math_log is base-10, not natural log

will.anderson referenced this pull request

2026-08-16 02:33:46 +00:00

runtime: math_log is base-10, not natural log #134

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#133