The module question ended with a limit: textual inlining destroys file
provenance, so a duplicate-definition message could name the symbol but not the
files. Threading it exposed a bigger absence first.
TOKENS HAD NO POSITION AT ALL. A token was a flat (kind, value) pair, so NO
diagnostic in El could name a place -- every error named a symbol and never a
line. That is the prerequisite the module question was resting on.
THE CHAIN, end to end
lexer counts newlines; tok_append mints (kind, value, line)
parser stride 2 -> 3; tok_line added; FnDef carries its line
codegen records <fn> defines_at:<line>
resolve_imports publishes <file> spans <start> <end> for the combined source
checker maps a combined line back to file:line-within-that-file
duplicate definition: 'helper' is defined 2 times — El has no namespacing,
so imported modules share one global scope
/tmp/modtest/a.el:1
/tmp/modtest/b.el:1
PREDICTIONS AND RESULTS
P1 15 stride sites, encapsulated in tok_kind/tok_value TRUE, but see below
P2 adding a line field is mechanical TRUE
P3 the lexer must count newlines TRUE
P4 resolve_imports can record per-file line ranges TRUE
P5 the message can then name both files TRUE
P6 token memory grows TRUE, 25.0 -> 33.9 MB (+36%)
FOUR DEFECTS, EACH FOUND BY RUNNING AND NOT BY READING
1. interp_tokens_append_all walks the token list DIRECTLY with its own copy of
the stride. Gen1 built fine and gen2 emitted corrupt C, because the
compiler's own source uses string interpolation. My search missed it because
I grepped for the variable name `tokens`; it is called `dst`/`result`.
Searching by name instead of by shape -- third time today.
2. tok_count in test_compiler.el carried the stride too. I had scoped the search
to compiler sources and it had escaped into the tests.
3. Nested resolve_imports calls accumulated spans into shared state, so each
republished meaningless line ranges under the parent's name. Making the
buffer local fixed it; guarding the WRITE did not, which is what I tried
first.
4. The first working version reported b.el:3 -- the COMBINED line against a
filename that has no line 3. A file:line that does not match the file is
worse than no line at all.
105/105 native, 37/37 integration, fixpoint ok, compiler self-checks clean.
The singleton lock protected a filename, not a store. It was keyed on
$EL_SINGLETON_DIR|$TMPDIR|/tmp + /el-singleton-<program>.lock — the
program's NAME and a temp directory — and never consulted the state it
claimed to protect, while its own refusal message read "Refusing to start
a second instance against the same state."
Measured, it failed in both directions. A second engram against a
DIFFERENT data dir was refused, naming the first's pid. And
TMPDIR=/tmp/other let a second engram start against the SAME data dir
with no complaint — the two-writer data-loss condition the guard exists
to prevent, defeated by one environment variable.
Both are one error: the identity of the resource had been replaced by a
label for it.
The lock now lives inside the state it guards —
<state>/.el-singleton-<id>.lock — and the program block says what that
state is. Same directory is the same file is the same inode, so it
contends and there is no TMPDIR left in the key to change. Different
directories are different files, so they don't. Different spellings of
one directory (trailing slash, x/../x, symlink) collapse in the kernel's
own path walk, so they contend without this code comparing strings;
canonicalisation is for the message, never the decision.
`guards:` is an expression so a program can point at the resolver that
already owns its path — guards: engram_resolve_data_dir() — instead of
restating that resolver's default, which is the two-owners defect spec
18.4 exists to prevent. A `singleton:` without `guards:` is now a compile
error; emitting a name-keyed lock instead would be emitting the defect.
Kept: the flock (the kernel drops it on crash and SIGKILL, so there is
still no "delete the lock file to get unstuck" ritual — a stale file
inside a copied data dir is inert), and the holder's pid in the message.
Changed: the message is true. It says "the same state" because the lock
it failed to take is in that state, and it names the state it checked.
An unguardable state (missing, read-only) now refuses rather than
starting unguarded.
Also corrects lang/AGENTS.md's compiler rebuild line, which had gone
stale: linking el_runtime.c alone no longer resolves.
El's units of encapsulation are the function and the module. Neither can hold
a concern that belongs to the process, so each one had been expressed the only
way it could be -- as a convention: call this at every site. Conventions of
that shape do not hold. Measured here: zero process-identity guards at any
layer, 20 environment variables each with its default written inline at the
read site, 62 persist call sites, 10 per-route auth checks. One absence, four
times.
Step 0 first, because the premise was wrong. El was believed to have no
middleware or effect mechanism. It has one, and it is already load-bearing:
codegen injects engram_boundary_beat at the entry of every @manager/@accessor
fn, decorators take arguments and stack, dharma_emit from a non-@manager fn is
a #error, and the cgi block injects el_cgi_init at the head of main(). So the
correct move was not to invent a mechanism but to generalize the seam that
already existed. The real gap is narrower and is now recorded: the seam is
prologue-only and its callee is a fixed builtin.
Adds a `program` block -- the third program-level declarative block. cgi and
service declare what a program may do; program declares what it is.
program "engram" {
singleton: "engram"
env ENGRAM_BIND: String = ":8742"
env GUIDE_PORT: Int = "8771"
}
singleton takes an exclusive flock before any user statement runs and refuses a
second start, reporting the holder's pid. It is a lock rather than a pidfile so
the kernel releases it on death including SIGKILL -- no stale state, and so no
"delete the lock file to get unstuck" ritual, which would itself be a
convention. It reports the pid because "already running" is not actionable; a
pid is. That is the direct answer to a stale process surviving a pkill and
going on answering probes.
env entries resolve once at startup -- environment wins, declaration supplies
the fallback -- and validate as a whole, reporting every problem at once rather
than costing one restart per variable. config("X") for an undeclared X is
fatal, because an advisory schema is just another convention. Programs without
a program block are unaffected, so migration is per-program.
Only one keyword is added. `config` and `env` could not become keywords -- both
are real identifiers in the tree -- so the block's fields are read as
identifier token values by its own parse loop and stay usable everywhere else.
The init function is emitted at the block site and called from main() rather
than inlined into main(). The live backend is codegen_streaming, which emits in
source order and cannot hold the entry list alive until main(); this way only a
single bool has to survive.
Also fixes: config() was defined in el_runtime.c but never prototyped in
el_runtime.h, so any el program calling it failed to compile under C99.
Spec: section 18 documents what shipped. Section 9 is corrected -- it claimed
decorators had no structural meaning, which has not been true for some time.
Section 19 designs durability-as-an-epilogue-effect and route authorization
and states plainly why neither is implemented here: both land in files under
concurrent modification, and the prerequisite for both is lifting the seam
from prologue-only to prologue/epilogue.
Self-hosting fixpoint verified byte-identical.
Lands feat/reframe-region-setop (PR #109: native set-based reframe_region,
decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency
work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated
seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's
actual current HEAD, plus engram-tiered-storage's still-unique test suite.
RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/
verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/
select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two
branches; #109's copy is a strict superset (adds vindex_harvest_from_store,
used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and
engram_verify.c are also byte-identical. #109's own branch point already
carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing
left to merge into #109 for those files. The one thing engram-tiered-storage
had that #109's tree dropped: its full test suite (test_vindex.c,
test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the
interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh
harnesses) — ported over here unchanged.
WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch
forked from dev on 2026-08-14 15:40 (before restructure-adjacent history
diverged the file's merge-base for `git merge` — it presented as an add/add
conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but
it silently reverted THREE dev fixes landed on 2026-08-14/15, after the
branch point, that the PR's diff had no way to know about:
1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of
the query-aware propagation gate dropped the shift-and-floor rescale
about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate
0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped
around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi]
is computed) is orthogonal to WHAT it gates on and both are kept.
2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes
wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is
diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global).
PR's tree predates this and dropped all three counters + their JSON
stats fields. Restored declarations, all 4 direct increment sites, the
eg_wm_carry_over bll increment, and the act-stats JSON fields --
alongside (not instead of) the PR's own P4 afferent / API-reshape
counters already in that same struct/JSON.
3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev
selects the STRONGEST qualifying candidate for consolidation each call;
PR's tree predates this and reverted to hash-slot order (arbitrary wrt
association strength) for edge formation -- the one path that writes
PERMANENT structure. Restored the strongest-candidate while-loop,
keeping the PR's own genuine improvement at that site
(engram_adj_on_edge_added incremental-index append instead of a bare
adj_dirty=1 full-rebuild flag).
engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env
flags, /api/nodes connected-count in responses) were pure additive: dev's
side was empty, PR's side added the feature. Took PR's side whole.
VERIFIED (nsbx sandbox only, live :8742/:7770 never touched):
- cc -std=c11 -O2, clean link against the real engram/src/server.el via
elc, zero errors.
- vindex_bench (built standalone, read-only harvest) against the real
production store clone (13,671 embedded nodes, 768-dim nomic-embed-text):
recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs
2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s
for the full 13,671-node set -- see the flagged risk below.
- Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned
snapshot of the live store, 13,424 nodes / 37,656 edges) and called
/api/activate for real: first call after boot 41.5s (pays the one-time
HNSW build inline -- matches the standalone bench), second/third calls
356ms/605ms, no crash, correct results, act-stats JSON (including the
restored evict_floor/cap/bll fields) reads correctly.
KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for
this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync
builds the HNSW index synchronously, inline, on the first engram_activate()
call after every process start (or index invalidation). On the real node
count that is a ~46s blocking stall on a single-threaded server -- the first
request after every restart (or its concurrent siblings) waits the full
build. Recommend a background/incremental build (or a bounded per-call build
budget) before this ever reaches the live daemon. See PR description / final
report for the fuller writeup.
Ground-truth the three seams (route/telemetry+interoception/bus) with file:line
evidence. Port the tested @route codegen+parser from feat/el-route-decorators
into the worktree elc (decoration synthesizes el_route_dispatch — no hand-written
90-branch handle_request). Rebuild elc self-host; prove decorate->serve end-to-end
(route_proof.el on :8951). Rewrite surface.el as El-native decorated components:
@route + @accessor/@manager, in-process engram_* builtins (not http_get), @manager
ops emit on the real dharma_* bus (same transport as wt/swarm-ccr). Identity
keystones refused in write/relate/supersede. Gate-1 clone recipe (WAL-aside
cold-boot + ENGRAM_WAL=on) proves the FULL op set live on the clone. Boundary
auto-emit (telemetry/interoception/bus) staged as a reviewable cg_fn diff
(SEAM_STAGED.md) — needs the cognition-engram rebuild to verify link. Live :8742
untouched; no push, no cutover.
- Wire paged store into the ENGRAM SERVICE (server.el, the authoritative durable
owner): boot->engram_store_boot, persist_canonical->engram_store_checkpoint,
gated by ENGRAM_STORE.
- elc (lang/elc.c + src/parser.el + codegen.el + elc-combined.el): OOB guard in
tok_kind/tok_value + parse_block progress backstop — fixes the pre-existing
unbounded-memory fold hang on sessions.el.
- engram_prune_telemetry mirrors ISE prune to the store (store_forget) so store
live-count tracks resident and stale telemetry stays bounded.
- Deployed live 2026-08-12: engram :8742 on neuron.egm+WAL, count reconciled 11552.
Out-of-range tok_kind/tok_value reads returned runtime null (el_list_get OOB
-> 0) rather than the Eof sentinel, so the inner parse loops (parse_block,
call-arg, array-literal, match-arm) that terminate only on their close
delimiter or k=="Eof" never saw Eof once the cursor ran past the single
trailing Eof token. On unclosed-delimiter input the parser then appended AST
nodes forever -> unbounded allocation -> ~700GB -> OOM (observed compiling
neuron/sessions.el).
Fix at the choke point: tok_kind returns "Eof" and tok_value returns "" for
out-of-range positions, restoring the parser-wide contract that reads at/after
the end yield Eof. expect() no longer steps past the Eof sentinel on mismatch.
This terminates every overrun loop simultaneously; a malformed program now
surfaces as a normal (best-effort) parse end instead of exhausting memory.
Requires a self-hosted bootstrap rebuild of elc to take effect.
The --emit-header path previously called parse() which builds the entire
program AST in memory before writing the .elh file. For checkout.el (~491
lines with HTML template trees and deep BinOp string-concat chains), this
exhausted memory before the header could be written.
Fix: replace parse() + emit_header() with scan_fn_sigs_el() +
emit_header_from_sigs(). The new path tokenises the source once, then
walks the flat token list skipping over function bodies entirely — peak
memory is O(tokens) instead of O(whole-program AST).
New functions in parser.el:
- scan_type_el: reads a type annotation and returns its El source string
- scan_params_el: reads (name: Type, ...) and returns El params string
- scan_fn_sigs_el: token-level scan that collects El-style fn signatures
without building any expression AST nodes
New function in compiler.el:
- emit_header_from_sigs: writes .elh from scan_fn_sigs_el output
Self-hosting check: elc compiled with new elc, diff of outputs is
identical (zero difference).
Smoke test: elc --emit-header checkout.el produces correct three-entry
.elh (previously truncated at two entries due to mid-parse OOM).
The El lexer silently skips '#', so {#each} lexes as LBrace Ident:"each"
and {#if} lexes as LBrace If ... (using the If keyword token, not Hash).
The existing {#each} check used k2=="Hash" which was dead code.
Parser changes (parser.el):
- Add parse_raw_text_content(): collects all tokens as raw text until
</tag_name>, bypassing El expression parsing. Used for <style> and
<script> elements so CSS/JS content isn't parsed as El expressions.
- parse_html_element(): use raw-text mode for <style> and <script> tags.
- parse_html_children(): fix {#each} detection (k2=="Ident", k3=="each"
instead of dead k2=="Hash" check). Add {#if cond}...{#else}...{/if}
support generating HtmlIf AST nodes.
Codegen changes (codegen.el):
- Add cg_html_if(): generates if (cond_c) { then_c } else { else_c }
for HtmlIf nodes.
- cg_html_parts(): dispatch HtmlIf to cg_html_if.
Implement compile_test() entry point that emits a C test harness instead
of a normal program. Test blocks (previously skipped) now compile to
static functions with per-assertion pass/fail tracking. Assert statement
added to parser and codegen. Runtime extended with now_ns, fs_list_json,
json_build_object, json_build_array, json_escape_string, state_has,
state_get_or. Fix float negation codegen, float equality comparisons,
time_to_parts return type (JSON string), time_format empty-fmt, json_set
raw-value semantics, state_keys JSON array return. All 310 native tests
pass across 9 suites (core, text, string, math, env, state, json, time, fs).
Combines two orthogonal optimizations:
1. Flat token list (from beta): lex() returns [Any] with alternating kind/value
pairs instead of [Map], eliminating one ElMap per token (~3 mallocs each).
Parser updated: tok_kind(t,i) = t[2*i], tok_value(t,i) = t[2*i+1].
2. Char code dispatch (from alpha): lex() hot loop uses str_char_code -> Int
instead of str_char_at -> strdup String for all character classification.
Eliminates ~400K x 16B = 6.4MB of temporary string allocations.
scan_digits and scan_ident also updated to use str_char_code.
Result on main.el: 17.1MB -> 14.4MB peak RSS (-16%).
Self-hosting: PASS.
bigmerge
will.anderson