Engine test harness reports a permanent false green: assert counters never increment (9 test files) #116

New Issue

2026-08-06T20:36:40Z

tim.lingo commented

2026-08-06 20:36:40 +00:00

Found 2026-08-06 while adding coverage for the OpenAI-tools port (branch feat/soul-openai-tools-v2).

What is wrong

Every tests/*.el assert helper counts like this:

fn assert_eq(label: String, got: String, expected: String) -> Void {
    if str_eq(got, expected) {
        let pass_count = pass_count + 1     // <-- inside an if BLOCK
        println("  PASS: " + label)
    } else {
        let fail_count = fail_count + 1     // <-- inside an if BLOCK

El's scope rule — the one chat.el documents at every while-body mutation ('mutations inside if blocks do not escape scope') — means neither counter ever increments. The summary line every counted test file prints is 0 passed, 0 failed always, whatever happened. Verified by running the file: 32 individual PASS lines printed, summary still 0/0. 9 test files share the pattern.

Why it matters

The per-assertion PASS/FAIL lines are correct and reliable — the summary is not. Any CI step, script, or human keying off that line reads permanent green, including on a run where every assertion failed. Same class as the round-6 lesson ('a sub-assertion that can never pass is as worthless as one that can never fail'), applied to the verdict itself.

Related discovery

There was no way to run these tests at all: elc is a compiler (emits C to stdout and exits), so tests/*.el could only be read, never executed — presumably why the dead counters went unnoticed. A working runner now exists on the port branch at tests/run-el-test.sh (emits the test to C, compiles soul.c separately with main renamed away since it owns the daemon main but also defines layered_cycle et al., links the remaining modules + the repo-pinned vendor runtime, executes). It computes the verdict itself from the PASS/FAIL lines and exits non-zero on any failure or on zero assertions — proven to discriminate with a negative control (deliberately broken assertion -> 31 passed / 1 failed, exit 1).

The real fix (this issue)

The harness workaround means no one reads a false green today, but the in-file counters are still dead. Correct fix is in the test files: carry counts through a mechanism that survives El block scoping (state_set/state_get, or helpers return a value the caller accumulates at top level as an if-EXPRESSION). 9 files, mechanical. Not fixed here deliberately: it touches shared test files while a beta round is in flight.

🤖 Generated with Claude Code

**Found 2026-08-06** while adding coverage for the OpenAI-tools port (branch `feat/soul-openai-tools-v2`). ## What is wrong Every `tests/*.el` assert helper counts like this: fn assert_eq(label: String, got: String, expected: String) -> Void { if str_eq(got, expected) { let pass_count = pass_count + 1 // <-- inside an if BLOCK println(" PASS: " + label) } else { let fail_count = fail_count + 1 // <-- inside an if BLOCK El's scope rule — the one `chat.el` documents at every while-body mutation ('mutations inside if *blocks* do not escape scope') — means **neither counter ever increments**. The summary line every counted test file prints is `0 passed, 0 failed` **always**, whatever happened. Verified by running the file: 32 individual PASS lines printed, summary still 0/0. **9 test files** share the pattern. ## Why it matters The per-assertion PASS/FAIL lines are correct and reliable — the *summary* is not. Any CI step, script, or human keying off that line reads permanent green, including on a run where every assertion failed. Same class as the round-6 lesson ('a sub-assertion that can never pass is as worthless as one that can never fail'), applied to the verdict itself. ## Related discovery There was **no way to run these tests at all**: `elc` is a compiler (emits C to stdout and exits), so `tests/*.el` could only be read, never executed — presumably why the dead counters went unnoticed. A working runner now exists on the port branch at `tests/run-el-test.sh` (emits the test to C, compiles `soul.c` separately with `main` renamed away since it owns the daemon main but also defines `layered_cycle` et al., links the remaining modules + the repo-pinned vendor runtime, executes). It **computes the verdict itself** from the PASS/FAIL lines and exits non-zero on any failure or on zero assertions — proven to discriminate with a negative control (deliberately broken assertion -> 31 passed / 1 failed, exit 1). ## The real fix (this issue) The harness workaround means no one reads a false green today, but the in-file counters are still dead. Correct fix is in the test files: carry counts through a mechanism that survives El block scoping (`state_set`/`state_get`, or helpers return a value the caller accumulates at top level as an if-EXPRESSION). 9 files, mechanical. Not fixed here deliberately: it touches shared test files while a beta round is in flight. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/neuron#116