# tokens carry no position

One `Ishikawa → scientific method → Six Sigma` loop. The record below is the
commit message as written at the time, before the outcome was known to anyone
reading this file.

## Record — `6c975b1`

```
thread provenance through resolve_imports

The module question ended with a limit: textual inlining destroys file
provenance, so a duplicate-definition message could name the symbol but not the
files. Threading it exposed a bigger absence first.

TOKENS HAD NO POSITION AT ALL. A token was a flat (kind, value) pair, so NO
diagnostic in El could name a place -- every error named a symbol and never a
line. That is the prerequisite the module question was resting on.

THE CHAIN, end to end
  lexer            counts newlines; tok_append mints (kind, value, line)
  parser           stride 2 -> 3; tok_line added; FnDef carries its line
  codegen          records <fn> defines_at:<line>
  resolve_imports  publishes <file> spans <start> <end> for the combined source
  checker          maps a combined line back to file:line-within-that-file

    duplicate definition: 'helper' is defined 2 times — El has no namespacing,
    so imported modules share one global scope
        /tmp/modtest/a.el:1
        /tmp/modtest/b.el:1

PREDICTIONS AND RESULTS
  P1 15 stride sites, encapsulated in tok_kind/tok_value   TRUE, but see below
  P2 adding a line field is mechanical                     TRUE
  P3 the lexer must count newlines                         TRUE
  P4 resolve_imports can record per-file line ranges       TRUE
  P5 the message can then name both files                  TRUE
  P6 token memory grows                                    TRUE, 25.0 -> 33.9 MB (+36%)

FOUR DEFECTS, EACH FOUND BY RUNNING AND NOT BY READING

1. interp_tokens_append_all walks the token list DIRECTLY with its own copy of
   the stride. Gen1 built fine and gen2 emitted corrupt C, because the
   compiler's own source uses string interpolation. My search missed it because
   I grepped for the variable name `tokens`; it is called `dst`/`result`.
   Searching by name instead of by shape -- third time today.
2. tok_count in test_compiler.el carried the stride too. I had scoped the search
   to compiler sources and it had escaped into the tests.
3. Nested resolve_imports calls accumulated spans into shared state, so each
   republished meaningless line ranges under the parent's name. Making the
   buffer local fixed it; guarding the WRITE did not, which is what I tried
   first.
4. The first working version reported b.el:3 -- the COMBINED line against a
   filename that has no line 3. A file:line that does not match the file is
   worse than no line at all.

105/105 native, 37/37 integration, fixpoint ok, compiler self-checks clean.
```

## Record — `cb7289f`

```
thread provenance end to end: a diagnostic can finally name a place
```