thread provenance through resolve_imports

The module question ended with a limit: textual inlining destroys file
provenance, so a duplicate-definition message could name the symbol but not the
files. Threading it exposed a bigger absence first.

TOKENS HAD NO POSITION AT ALL. A token was a flat (kind, value) pair, so NO
diagnostic in El could name a place -- every error named a symbol and never a
line. That is the prerequisite the module question was resting on.

THE CHAIN, end to end
  lexer            counts newlines; tok_append mints (kind, value, line)
  parser           stride 2 -> 3; tok_line added; FnDef carries its line
  codegen          records <fn> defines_at:<line>
  resolve_imports  publishes <file> spans <start> <end> for the combined source
  checker          maps a combined line back to file:line-within-that-file

    duplicate definition: 'helper' is defined 2 times — El has no namespacing,
    so imported modules share one global scope
        /tmp/modtest/a.el:1
        /tmp/modtest/b.el:1

PREDICTIONS AND RESULTS
  P1 15 stride sites, encapsulated in tok_kind/tok_value   TRUE, but see below
  P2 adding a line field is mechanical                     TRUE
  P3 the lexer must count newlines                         TRUE
  P4 resolve_imports can record per-file line ranges       TRUE
  P5 the message can then name both files                  TRUE
  P6 token memory grows                                    TRUE, 25.0 -> 33.9 MB (+36%)

FOUR DEFECTS, EACH FOUND BY RUNNING AND NOT BY READING

1. interp_tokens_append_all walks the token list DIRECTLY with its own copy of
   the stride. Gen1 built fine and gen2 emitted corrupt C, because the
   compiler's own source uses string interpolation. My search missed it because
   I grepped for the variable name `tokens`; it is called `dst`/`result`.
   Searching by name instead of by shape -- third time today.
2. tok_count in test_compiler.el carried the stride too. I had scoped the search
   to compiler sources and it had escaped into the tests.
3. Nested resolve_imports calls accumulated spans into shared state, so each
   republished meaningless line ranges under the parent's name. Making the
   buffer local fixed it; guarding the WRITE did not, which is what I tried
   first.
4. The first working version reported b.el:3 -- the COMBINED line against a
   filename that has no line 3. A file:line that does not match the file is
   worse than no line at all.

105/105 native, 37/37 integration, fixpoint ok, compiler self-checks clean.
This commit is contained in:
bigmerge
2026-08-17 10:07:27 -05:00
parent 1086ac9658
commit 6c975b1d50
7 changed files with 148 additions and 46 deletions
+21 -13
View File
@@ -18,7 +18,9 @@ import "../../el-compiler/src/compiler.el"
// Lexer helpers
fn tok_count(tokens: [Any]) -> Int {
native_list_len(tokens) / 2
// A token is (kind, value, line). This helper carried its own copy of the
// stride, so it escaped a search scoped to the compiler sources.
native_list_len(tokens) / 3
}
// Codegen helper: capture compile() stdout to a string
@@ -259,22 +261,28 @@ test "lex-multiline-source" {
assert tok_kind(tokens, 0) == "Let", "first token is Let"
}
test "lex-flat-stride-2-layout" {
// Verify that the flat stride-2 layout: token i has kind at index 2*i, value at 2*i+1
test "lex-flat-stride-3-layout" {
// A token is (kind, value, line): token i has kind at 3*i, value at 3*i+1,
// line at 3*i+2. Before 2026-08-17 a token carried no position at all, so
// no diagnostic in El could name a place.
let tokens: [Any] = lex("fn foo")
// tokens[0] = "Fn", tokens[1] = "fn", tokens[2] = "Ident", tokens[3] = "foo", ...
let raw_len: Int = native_list_len(tokens)
assert raw_len == 6, "fn + foo + Eof = 3 tokens = 6 raw entries"
let kind0: String = native_list_get(tokens, 0)
let val0: String = native_list_get(tokens, 1)
let kind1: String = native_list_get(tokens, 2)
let val1: String = native_list_get(tokens, 3)
assert kind0 == "Fn", "raw[0] is Fn kind"
assert val0 == "fn", "raw[1] is fn value"
assert kind1 == "Ident", "raw[2] is Ident kind"
assert val1 == "foo", "raw[3] is foo value"
assert raw_len == 9, "fn + foo + Eof = 3 tokens = 9 raw entries"
assert native_list_get(tokens, 0) == "Fn", "raw[0] is the kind"
assert native_list_get(tokens, 1) == "fn", "raw[1] is the value"
assert native_list_get(tokens, 2) == "1", "raw[2] is the line"
assert native_list_get(tokens, 3) == "Ident", "raw[3] is the next kind"
assert native_list_get(tokens, 5) == "1", "still line 1"
}
test "lexer-tracks-line-numbers" {
let tokens: [Any] = lex("fn a\nfn b\nfn c")
assert tok_line(tokens, 0) == "1", "first fn is on line 1"
assert tok_line(tokens, 2) == "2", "second fn is on line 2"
assert tok_line(tokens, 4) == "3", "third fn is on line 3"
}
// Parser tests
fn get_first_stmt_kind(src: String) -> String {