96 Commits

Author	SHA1	Message	Date
bigmerge	0143cc458a	ANSWER: is a grammar a convention, or a region? ... Both, at different layers -- and it is the same split as serialization: the convention is the BASIS, never the ACT. lexeme -> token `fn` means function-start because someone said so CONVENTION shape recognition given tokens, which construct is this REGION source -> structure parsing is transduction onto that basis GEOMETRY byte traversal something must read them in order IRREDUCIBLE Three things push the ACT toward region rather than convention: ambiguity (a * b needs context; a grammar resolves it with the lexer hack, a region by neighbourhood), error recovery (nearest-region is free), and precedence, which is ordering along an axis with a conventional parameter. AND THE SHOULD GATE SAYS NO TO THE OBVIOUS MOVE Every other table this session moved to data. This one stays code. The keyword set is CLOSED by the language definition -- it does not leak the way an allowlist does -- and the lexer runs before the program is understood, so a program can never declare its own keywords. Externalising it costs file I/O on every compile and buys nothing. Same verdict as is_digit in ASCII. WHAT WAS ACTUALLY WRONG: five of 46 keywords were consumed by no parser or codegen path. sealed, activate, seed, protocol, impl. Each stole an identifier from users for nothing. SECOND SILENT MISCOMPILATION OF THE DAY. Using one did not fail to parse: let seed = 42 let impl = seed + 1 compiled CLEAN -- zero cc errors -- and printed 0 instead of 44. No diagnostic at any layer. Fixed by removing the five. A DEFECT IN MY OWN MEASUREMENT, caught before it did damage: my first pass checked only parser.el and reported `test` as inert too. codegen consumes it at 4135 for --test mode, and the tree has 408 uses. Removing it would have broken every test in the suite. The measurement was re-run across all four consumers. 100/100 native + 2 new, 31/31 integration, fixpoint ok.	2026-08-17 09:43:58 -05:00
bigmerge	cbef1c1ebb	EXPERIMENT: Int return types as data — and the bug that fell out ... PREDICTIONS AND RESULTS P1 is_int_call's 35 hardcoded names move to data TRUE P2 is_int_name stays -- it is annotation propagation TRUE P3 the dispatch stays -- it is emission TRUE P4 codegen shrinks ~40 lines TRUE 4507 -> 4469 P5 the design doc's characterisation is WRONG TRUE P6 the moved data also fixes the bug it exposed TRUE P5 CORRECTS THE RECORD. el-language-design.md and geometry-vs-code.md both cite "== lowering to str_eq unless both operand names are in a hardcoded int-name set -- a literal list of variable names treated as integers" as the paradigm defect. It is not one. __int_names is populated from TYPE ANNOTATIONS (param["type"] == "Int"), which is primitive but legitimate type propagation. The actual defect was is_int_call: 35 hardcoded builtin return types, the same shape as the temporal 19. P6 IS A LIVE CORRECTNESS BUG, PRE-EXISTING, NOW FIXED let a = str_len("hello") // no annotation let b = str_len("hi") let c = a + b // -> el_str_concat(a, b) on two integers Verified identical on the pre-change compiler, so not a regression. It compiled clean, ran, and printed NOTHING where it should print 7. No error at any layer. The repair is three lines: an unannotated let takes its type from what the initialiser returns. The return types were already required for dispatch and were simply never consulted at the binding site. Moving them into data is what made the gap visible -- reading the code for eight hours did not. 98/98 native + 2 new, 31/31 integration, fixpoint ok.	2026-08-17 09:32:13 -05:00
bigmerge	e8e25a07b4	EXPERIMENT: temporal adjudication moves out; the placeholder stays ... The previous pass moved the type DATA and left the judgment inline, which I stated rather than hid. This finishes it. PREDICTIONS AND RESULTS P1 codegen can emit operand-type relations TRUE "main calls temporal:instant_plus_instant" P2 the affine rules are a small closed set as data TRUE 6 rules P3 violations still caught at build time TRUE exit=1 P4 the reporter leaves codegen TRUE 4538 -> 4507 P5 the TIME_TYPE_ERROR placeholder must STAY TRUE P5 is the boundary of this whole approach. The emitter has to emit SOMETHING for an illegal expression -- it cannot emit nothing and it cannot decide what the program meant. So the placeholder is irreducible in the same way the AST traversal was: what moved is the judgment and the wording, not the fact that something must be written. The rules are affine algebra and the set is closed because there are only two kinds of thing. An Instant is a POINT, a Duration is a DISPLACEMENT: add a displacement to a point, subtract two points for a displacement, combine displacements. Nothing else is meaningful, which is why the enumeration in temporal.rel cannot grow the way an allowlist does. A defect in my own checker, found by running it: the .rel file uses aligned columns and my awk assumed a single space, so the message came out with the rule key still prefixed. Same class as the multi-line header parse in the arity pass -- formatting assumptions that only fail when you look at the output. 98/98 native, 6/6 temporal_query.sh, fixpoint ok.	2026-08-17 09:27:42 -05:00
bigmerge	d2d89fcb60	EXPERIMENT: temporal types as data — and the pass that GREW the compiler ... This block is structurally unlike the previous four. It does not only adjudicate, it DISPATCHES: Instant + Duration must become el_instant_add_dur, LocalDate + Duration must become el_local_date_add_dur. The emitted C depends on the type answer, so it cannot move to a post-hoc query. Selecting which call to emit is an emitter's actual job. PREDICTIONS AND RESULTS P1 the block conflates dispatch with adjudication TRUE P2 adjudication can move, dispatch cannot TRUE P3 this pass shrinks codegen far less than the last TRUE, and worse: 4513 -> 4537, it GREW by 24 lines P4 the rules are affine algebra, closed by construction TRUE P5 no type propagation -- name tracking plus a hardcoded list of which builtins return which type TRUE, 19 names P3 is the honest result and it is not spun: moving 19 names into a data file cost more lines than it saved, because a generic loader is larger than the enumeration it replaces. The win is not line count. It is that adding a 20th temporal builtin is now a one-line edit to signatures.rel instead of a compiler change, and that the data is inspectable. WHY THE HEADER CANNOT SUPPLY THIS, unlike arity: el_runtime.h declares every builtin as returning el_val_t, because El has ONE type. That single type is why the whole seam is cheap and it is exactly why the C boundary cannot say that now() returns an Instant while unix_seconds() returns an Int. The El-level type is real and the boundary erases it. INCOMPLETE, and stated rather than hidden: P2 said adjudication could move to a query. It has NOT. Violations still emit TIME_TYPE_ERROR inline from the emitter. Only the type DATA moved. Moving the adjudication needs the operand types recorded as relations, which is a further pass. 98/98 native, 4/4 temporal_signatures.sh, fixpoint ok.	2026-08-17 09:25:18 -05:00
bigmerge	9cc6040df2	EXPERIMENT: derive arity from the runtime's own declarations ... codegen.el carried builtin_arity(): 344 lines, 300 entries, a hand-maintained second copy of el_runtime.h. PREDICTIONS AND RESULTS P1 the table duplicates the header TRUE 243 shared names P2 they have already drifted FALSE ZERO drift. The duplicate had been maintained correctly. P3 codegen can emit call-arity relations TRUE P4 the check becomes a query against the header TRUE P5 codegen drops to roughly baseline TRUE 4903 -> 4512, 149 BELOW the 4661 it started at P2 being false is the better result: the table was not WRONG, it was INCOMPLETE. 110 functions the runtime declares had no entry, so calling them with the wrong argument count produced no El-level diagnostic at all. Measured: the old compiler reports 0 arity errors for __http_do_map_to_file(1); the query reports "takes 5 arguments, called with 1". Deriving from the header fixes coverage AND makes drift impossible by construction. 503 signatures, versus 300 entries maintained by hand. THREE DEFECTS IN MY OWN CHECKER, each found by running it rather than reading it 1. El names and C names differ -- `println` is `__println`. 60 of 500 decls carry the prefix and codegen owns the mapping; the old table carried both keys. One rule covers all 60. 2. Multi-line declarations parsed as zero params, so the checker reported "takes 0" for a function taking 5. A diagnostic with the wrong number in it is worse than none -- the same shape as the stale caller attribution in the previous pass. 3. Fixing (2) by joining lines dropped 500 signatures to 334, because a declaration preceded by a comment no longer started its record. Comments are stripped first now. 98/98 native, 5/5 arity_query.sh, fixpoint ok.	2026-08-17 09:21:10 -05:00
bigmerge	c2d9596e76	EXPERIMENT: the capability tier becomes shipped policy plus a query ... Capability differs from prohibits_outside in one way that matters: a utility program cannot be trusted to declare its own restrictions, because it would declare none. So the policy comes from OUTSIDE the program -- it ships with the language as data, editable without a compiler release. tools/check/capabilities.rel 18 names that were string literals in codegen tools/check/capabilities.sh the query that decides PREDICTIONS AND RESULTS P1 codegen emits kind + call graph, drops the 4 name tests TRUE zero #errors P2 the 18 literals become a data file TRUE P3 the checker catches capability violations TRUE exit=1 P4 codegen drops ~76 lines TRUE 4963 -> 4881 P5 below the 4661 baseline FALSE ~+230 TWO DEFECTS THE HARNESS FOUND THAT READING WOULD NOT HAVE 1. Calls inside main became invisible. cg_fn returns early for main -- C provides its own -- so hooking the recording there left every call in main unrecorded: a blind spot exactly where a program does its work. The old cap_check_call ran from cg_expr and did see main. Moved the recording to cg_expr. 2. Caller attribution was stale. __cg_current_fn kept whatever cg_fn set last, so a violation in main was reported against the previously emitted function. The test still PASSED, because the violation was detected -- only the name was wrong, and a diagnostic naming the wrong fn is worse than none. Fixed at all three main-emission sites; the first patch missed two because the live path is codegen_streaming. 98/98 native, 7/7 + 4/4 + 5/5 integration, fixpoint ok.	2026-08-17 09:16:36 -05:00
bigmerge	c741cfe928	EXPERIMENT: prohibition becomes a query over emitted relations ... I said prohibition could not move because "a #error has no runtime". That conflated two separable things: WHEN a violation is detected (build time -- correct, and unchanged) and WHERE the rule and the checker live (the compiler -- assumed). A prohibition is a containment relation over the call graph. So codegen now records what it saw: sneaky calls raw_sql allowed calls raw_sql allowed calls @repository repository calls prohibits:raw_sql and tools/check/prohibitions.sh decides, at build time, outside the compiler. PREDICTIONS AND RESULTS P1 codegen can emit the call graph it already walks TRUE P2 the check becomes a query outside the compiler TRUE P3 all prohibition decisions leave codegen TRUE zero #errors now P4 violations still caught at build time TRUE exit=1 P5 codegen drops below the 4661 baseline FALSE 4962, +301 P5 is the finding. The TRAVERSAL is irreducible -- you must walk the AST to find calls, and those ~120 lines do not move no matter who decides. What is not irreducible is the rule (which names) or the decision (#error). Those left. I predicted the whole 223 lines would go because I had not separated walking from adjudicating. Still compiled, and measured rather than assumed: the capability-tier system (cap_check_call, is_self_formation_call, is_dharma_call, is_llm_call, cap_record_violation, emit_cap_violations) is 76 lines of the same shape -- prohibits_WITHIN rather than prohibits_outside, so the checker needs the opposite polarity to absorb it. 98/98 native, 4/4 prohibition_query.sh, 7/7 seam_binding.sh, fixpoint ok.	2026-08-17 09:11:48 -05:00
bigmerge	bc2f26ddfc	EXPERIMENT: invocation control resolves at runtime ... ISHIKAWA: why did wraps_body need compile-time knowledge? Because the wrapper called the target directly. If the wrapper calls through the seam instead, the seam can call the body itself, and a construct bound after the build decides how and whether to invoke it. PREDICTIONS AND RESULTS P1 wrap becomes runtime-bindable TRUE body x3 -> 21, never invoked -> 111 P2 codegen shrinks TRUE 5042 -> 4977 P3 cost 5-10% from an indirect call on every fn TRUE 0.36s -> 0.39s, ~8% P4 zero-param fns break on the empty struct TRUE empty struct is a GNU extension, empty init is C23. Fixed with a char field. P5 fixpoint holds TRUE PROCESS FAILURE worth recording: my first patch silently did not apply because I dropped the assert on the string replacement. The build then failed with "undeclared identifier __thunk_noargs", which I nearly attributed to the empty-struct prediction. The guard that would have caught it existed and I removed it -- the same shape as every other defect found tonight. Removed: declare_wrap, decorator_wrap, cg_wrap_target, cg_wrap_construct, params_to_call_args, and the wraps_body scanner branch. prohibits_outside is now the ONLY construct kind left at compile time, and it cannot move: a #error has no runtime.	2026-08-17 09:06:05 -05:00
bigmerge	285166c25c	EXPERIMENT: emit the wrapper unconditionally, so exit binds at runtime too ... ISHIKAWA: why did exit injection still need compile-time knowledge? Because the body-helper wrapper was only emitted when codegen already knew an exit construct existed. The wrapper being conditional was the cause, not the wrapper being necessary. PREDICTIONS AND RESULTS P1 exit becomes runtime-bindable TRUE returns 14, bound after the build P2 codegen shrinks TRUE 5094 -> 5044 P3 cost 5-15% from a call frame on every fn FALSE 0.37s -> 0.38s, ~3% P4 fixpoint holds TRUE Every fn now gets a body helper and a wrapper. It has to be unconditional: early returns must route through something for an exit construct to observe them, and codegen cannot know which fns will be bound after the binary exists. Removed with the machinery: declare_exit, decorator_exit, cg_exit_target, cg_exit_construct, and the injects_at_exit scanner branch. Two controls failed and were rewritten rather than repaired -- no-exit-construct-emits-no-wrapper asserted the optimisation this removes, so it is now inverted. The integration harness gained a seventh assertion: an exit construct declared after the build replaces the result. 99/99 native, 7/7 integration, fixpoint gen2==gen3.	2026-08-17 09:01:55 -05:00
bigmerge	28d19da7f1	strip the compile-time machinery the seam replaces ... PREDICTION: codegen.el drops below 4661, its size before any of these passes. RESULT: FALSE. 5157 -> 5096. Still +435 over baseline. injects_at_entry collapsed into the seam removed guards_at_entry collapsed into the seam removed injects_at_exit needs the body-helper wrapper STRUCTURAL wraps_body needs the closure + wrapper structural prohibits_outside a #error cannot be emitted at runtime The wrapper is not a consequence of compile-time resolution. Early returns must be routed through something no matter when the target is resolved, so exit injection was never going to collapse. I predicted it would because I had conflated "resolved late" with "emitted less". What did collapse is entry injection and refusal -- 61 lines of compiler replaced by one refusable indirection, with the capability now bindable after the binary exists. 8 tests fail, and they are exactly the 8 controls for compile-time entry injection and guards. No unrelated breakage: the controls reported precisely what moved. They assert emission of something that now happens at runtime, so they need rewriting as integration tests -- which the framework does not currently support, because runtime binding needs a built binary and an environment, not compile_capture. Verified after the strip: fixpoint gen2==gen3, observation and refusal both work through the seam with the compiler knowing nothing about either.	2026-08-17 08:43:57 -05:00
bigmerge	886626a64e	seam refusal + control tests: a runtime binding can short-circuit ... Prediction 3 was FALSE. I expected refusal to be impossible through the seam because the entry indirection discarded its return. One line: { el_val_t __s = el_seam_run(EL_STR(f), 0, 0); if (__s) return __s; } work() returns 7; bound to a refusing construct AFTER the build it returns 42. So three of the five compile-time kinds are runtime-bindable: entry injection, exit injection, and refusal. wraps_body needs invocation control and prohibits_outside is compile-time by nature. 104/104 native compiler tests pass.	2026-08-17 08:40:10 -05:00
bigmerge	35b07bade2	EXPERIMENT: resolve the crossing at execution, not at emission ... HYPOTHESIS (Will's): a compiler whose one compiled mechanism is extending the LANGUAGE — not the compiler — can compose without recompilation. ISHIKAWA — why does a construct require a recompile today? method codegen inlines the target call into the body machine the binary has no table to consult material the declaration lives in source, read at compile time measurement nothing observes what applied at runtime root cause the crossing is resolved at EMISSION, not at EXECUTION CHANGE: codegen emits one unconditional indirection per fn. Which constructs apply is read from a table that can be written AFTER the binary exists; targets resolve through dlsym against the running image. PREDICTIONS AND RESULTS P1 a construct declared after the build applies TRUE P2 an unlinked target is skipped, not fatal TRUE P3 emitting on every fn is measurably slower FALSE — 0.37s -> 0.36s with 267 indirections and no bindings. Free unused. P4 the compiler still self-hosts TRUE (see note) DEMONSTRATED: an El program with NO decorator in its source, already compiled and linked, picked up a construct declared afterwards: $ /tmp/seamrun -> 7 $ echo 'work audited entry audit_entry' > constructs.txt $ EL_CONSTRUCTS=constructs.txt /tmp/seamrun AUDIT: work applied by audited 7 P4 note: my first fixpoint test was wrong, not the code. I compared gen1 to gen2, which must differ whenever codegen's output changes. gen2 == gen3, 267 seam sites, stable. MEASURED COST, and the root cause was not where I looked 0 bindings 0.36s vs 0.37s baseline free 2 bindings, dlsym per call 2.45s 6.6x 2 bindings, resolved once 0.69s 3.5x recovered The table scan was never the cost. dlsym walks the dynamic symbol table on every call. Resolve once and cache — which is the smallest form of what salience does for memory: what is hot stays resolved. The 0.69s residual is audit_entry's own printf on two of the compiler's hottest functions, not seam overhead. CONSEQUENCE: the five compile-time declaration kinds on iteration-1 are a compile-time specialisation of something that resolves at runtime. They are not wrong, but they are not the mechanism — the mechanism is one indirection, and a kind is data.	2026-08-17 08:37:47 -05:00
bigmerge	1b324a071f	let a construct declare what may not cross it ... The other half of a boundary: not what runs when something crosses, but what may not cross at all. It was two string literals in vbd_is_restricted_name and one #error in cg_fn — one prohibition, uneditable without a compiler release. @decorator("prohibits_outside", "raw_sql") fn repository() {} fn sneaky() -> Int { raw_sql("DROP") } // #error "boundary violation: raw_sql may only be called from an // @repository fn, but 'sneaky' is not one" The recursive matcher is parameterised through a state key rather than by threading an argument through every branch of the walk — the mechanism codegen already uses for __match_counter and __if_expr_counter. Each prohibition is checked in its own turn, so the owning construct is known by construction and the diagnostic names it instead of hardcoding one rule's wording. PREDICTIONS AND RESULTS 1 the 3 duplicated uniqueness rules are textually identical TRUE 2 a declared prohibition reproduces @manager's #error TRUE 3 existing output byte-identical TRUE 4 a program can declare its own prohibition TRUE 5 fixpoint holds TRUE I misread result 2 on first pass: a @manager fn calling dharma_emit still emitted one #error, which looked like a failure. It is the CAPABILITY-tier rule at codegen.el:2578, a separate prohibition system, and it fires identically on the pre-change compiler. MEASURED DEFECTS STILL OPEN - two independent prohibition systems (VBD constructs, capability tiers); only the first is declarable - 3 uniqueness rules written 6 times, once per codegen path, kept in sync by hand and identical today 102/102 native compiler tests pass, compiler self-hosts byte-identically.	2026-08-17 08:15:27 -05:00
bigmerge	2bed8483f7	EXPERIMENT: hand the construct the body as a real closure ... ROOT CAUSE of the weaker design: "C has no closures" was taken as a fact about what is possible. It is a fact about one grammar. Every C++ lambda, every Go closure, every Rust closure compiles to a struct of captured values plus a function pointer -- which is what is emitted here. Codegen emits C; it is not written in C's syntax, and the distinction is the whole difference between a construct that can only decide whether to repeat and one that controls invocation. It would also have crippled the JS backend, which has closures natively, for a limit that applies only to the C one. PREDICTIONS AND RESULTS 1 env struct + thunk taking void* TRUE 2 fails to compile: struct redefinition FALSE -- C allows the inner declaration to shadow. Prediction wrong; C is more permissive than assumed. A different real defect surfaced instead: a wrap with no exit construct emitted `(EL_STR("f"), EL_STR(""), __r);` -- a call to an empty target -- because has_exit was reused as "needs a wrapper" and the exit line was emitted unconditionally. Fixed. 3 compiles when the target is declared in El FALSE -- and this is the root cause worth keeping: El has ONE type, el_val_t = int64_t. El's type system cannot describe a callable, so `extern fn` and the real signature cannot be made to agree in El's own vocabulary. The fix is not a cast: codegen DEFINES the wrap calling convention, so codegen emits the extern declaration. The convention is not El-expressible; it is emitted. 4 target controls invocation, 0..N times TRUE 5 existing @manager output byte-identical TRUE 6 compiler fixpoint holds TRUE 7 emitting the convention makes it compile TRUE MEASURED base(5) wrapped by a target that invokes the body twice and sums -> 10 never_runs(5) wrapped by a target that never invokes it -> 999 Neither is expressible by "decide whether to repeat". This supersedes the repeats_body experiment on experiment/repeats-body, which was built around the mistaken limit.	2026-08-17 08:08:37 -05:00
bigmerge	4f7568b07f	give a construct its after-crossing face, and let constructs compose ... §6 records 62 persist-after-mutate sites, 10 auth-per-route, and index-after-append that failed at 9 of 9 — every one an obligation at a crossing that decayed into "remember to do this afterwards." An obligation a human must remember is not an obligation, and the 9-of-9 figure is what that costs. @decorator("injects_at_exit", "persist_now") fn durable() {} The body moves into a static helper and the visible fn becomes a wrapper, so EARLY RETURNS pass through the exit injection. Emitting it only before the fall-through return would have silently missed every early return — the exact failure class this seam exists to remove. Fns with no exit construct emit byte-identically to before. Three independent constructs now compose on one fn, none known to the compiler: el_val_t mutate(el_val_t k) { { el_val_t __g = my_auth(EL_STR("mutate"), EL_STR("authenticate")); if (__g) return __g; } engram_boundary_beat(EL_STR("mutate"), EL_STR("manager")); el_val_t __r = __el_body_mutate(k); persist_now(EL_STR("mutate"), EL_STR("durable"), __r); return __r; } Guard, then entry, then body, then exit. §5.2 asked whether `hold` is one construct or two; the implementation answers one construct with two faces, selected by declared kind rather than by two mechanisms. Verified: existing output byte-identical, compiler self-hosts byte-identically, early returns pass through the exit, ordering holds under composition. 98/98 native compiler tests pass.	2026-08-17 07:56:28 -05:00
bigmerge	60737b0305	let a construct refuse, not only observe ... @authenticate (6 uses), @authorize (3), @rate_limit (3) and @validate (2) parsed, attached, and compiled to nothing. Fourteen applications that read as protection and emitted no instruction — a function decorated @authenticate compiled byte-identically to an undecorated one. The missing capability was not authentication. It was that a construct could observe a boundary but never refuse one. injects_at_entry discards the target's result; there was no form in which a construct could say no. @decorator("guards_at_entry", "my_auth") fn authenticate() {} @authenticate @authorize fn handler() -> String { ... } emits, at entry: { el_val_t __g = my_auth(EL_STR("handler"), EL_STR("authenticate")); if (__g) return __g; } { el_val_t __g = my_roles(EL_STR("handler"), EL_STR("authorize")); if (__g) return __g; } Guards precede injections because a refused call must not report a crossing, and every guard runs where the topmost injecting construct wins — refusal is not a role, so it does not follow the role convention. The compiler still knows nothing about auth. The program points the construct at its own function, which is where that decision belongs. Verified: existing @manager/@accessor output byte-identical, compiler self-hosts byte-identically, guards stack in declaration order and emit before the beat. 94/94 native compiler tests pass.	2026-08-17 07:53:10 -05:00
bigmerge	5718943f2e	let a construct declare its own meaning instead of the emitter knowing it ... codegen called fn_has_decorator for exactly three names — manager, accessor, route. Twelve others parsed, attached as {name,args}, and compiled to nothing, including four that look like protection: @authenticate (6 uses), @authorize (3), @rate_limit (3), @validate (2). The cause was not that the branches were untidy. A construct had nothing to BE, so its meaning had nowhere to live except the emitter, and every construct was therefore a compiler edit. A name -> injection table would have moved the enumeration twenty lines up without removing it. So the construct now carries its own meaning: @decorator("injects_at_entry", "engram_boundary_beat") fn audited() {} @audited fn risky_op() -> Int { ... } // gets the beat, attributed to "audited" scan_declared_decorators is a token-level pre-pass beside scan_routes, forced by streaming codegen having no whole-program AST. manager and accessor are seeded as the compiled-in core — the fixedSelf shape from substrate.go: a complete fallback exists, declaration is enrichment. This is the injection half of the seam only. The prohibition half (@manager's #error on dharma_emit) stays hardcoded, because "which calls may appear inside this boundary" is a query over program structure and there is nothing yet to ask. Verified three ways: emitted C for existing @manager/@accessor code is byte-identical to the hardcoded path; a construct with a name the compiler has never heard of injects correctly; the compiler self-hosts byte-identically. 90/90 native compiler tests pass.	2026-08-17 07:50:56 -05:00
bigmerge	dcaa77d77b	make boundary crossings attributable to the construct that caused them ... The beat reported which function crossed a boundary, never which decorator put the beat there. So the graph accumulated boundary events with no attribution, and no construct could be measured — "is this decorator earning its keep" stayed an argument instead of a traversal. engram_boundary_beat now takes the construct and carries it on the bus as {"construct":"..."}. The injection point, the beat, and the accumulation already existed; only the attribution was missing. Also pins a known defect as a test: codegen calls fn_has_decorator for exactly three names (manager, accessor, route). Twelve others parse, attach, and compile to nothing — including @authenticate (6 uses), @authorize (3), @rate_limit (3) and @validate (2), which look like protection and are not. decorator-authenticate-compiles-to-nothing asserts that @authenticate emits byte-identical C to no decorator at all, so fixing it will be a visible flip. Verified: compiler self-hosts byte-identically, 86/86 native compiler tests pass, emitted C carries the construct for both @manager and @accessor.	2026-08-17 04:53:40 -05:00
bigmerge	45325f7391	singleton: guard the state, not the program's name ... The singleton lock protected a filename, not a store. It was keyed on $EL_SINGLETON_DIR|$TMPDIR|/tmp + /el-singleton-<program>.lock — the program's NAME and a temp directory — and never consulted the state it claimed to protect, while its own refusal message read "Refusing to start a second instance against the same state." Measured, it failed in both directions. A second engram against a DIFFERENT data dir was refused, naming the first's pid. And TMPDIR=/tmp/other let a second engram start against the SAME data dir with no complaint — the two-writer data-loss condition the guard exists to prevent, defeated by one environment variable. Both are one error: the identity of the resource had been replaced by a label for it. The lock now lives inside the state it guards — <state>/.el-singleton-<id>.lock — and the program block says what that state is. Same directory is the same file is the same inode, so it contends and there is no TMPDIR left in the key to change. Different directories are different files, so they don't. Different spellings of one directory (trailing slash, x/../x, symlink) collapse in the kernel's own path walk, so they contend without this code comparing strings; canonicalisation is for the message, never the decision. `guards:` is an expression so a program can point at the resolver that already owns its path — guards: engram_resolve_data_dir() — instead of restating that resolver's default, which is the two-owners defect spec 18.4 exists to prevent. A `singleton:` without `guards:` is now a compile error; emitting a name-keyed lock instead would be emitting the defect. Kept: the flock (the kernel drops it on crash and SIGKILL, so there is still no "delete the lock file to get unstuck" ritual — a stale file inside a copied data dir is inert), and the holder's pid in the message. Changed: the message is true. It says "the same state" because the lock it failed to take is in that state, and it names the state it checked. An unguardable state (missing, read-only) now refuses rather than starting unguarded. Also corrects lang/AGENTS.md's compiler rebuild line, which had gone stale: linking el_runtime.c alone no longer resolves.	2026-08-16 16:08:40 -05:00
bigmerge	8ae163e8e5	lang: give cross-cutting concerns an owner instead of a convention ... El's units of encapsulation are the function and the module. Neither can hold a concern that belongs to the process, so each one had been expressed the only way it could be -- as a convention: call this at every site. Conventions of that shape do not hold. Measured here: zero process-identity guards at any layer, 20 environment variables each with its default written inline at the read site, 62 persist call sites, 10 per-route auth checks. One absence, four times. Step 0 first, because the premise was wrong. El was believed to have no middleware or effect mechanism. It has one, and it is already load-bearing: codegen injects engram_boundary_beat at the entry of every @manager/@accessor fn, decorators take arguments and stack, dharma_emit from a non-@manager fn is a #error, and the cgi block injects el_cgi_init at the head of main(). So the correct move was not to invent a mechanism but to generalize the seam that already existed. The real gap is narrower and is now recorded: the seam is prologue-only and its callee is a fixed builtin. Adds a `program` block -- the third program-level declarative block. cgi and service declare what a program may do; program declares what it is. program "engram" { singleton: "engram" env ENGRAM_BIND: String = ":8742" env GUIDE_PORT: Int = "8771" } singleton takes an exclusive flock before any user statement runs and refuses a second start, reporting the holder's pid. It is a lock rather than a pidfile so the kernel releases it on death including SIGKILL -- no stale state, and so no "delete the lock file to get unstuck" ritual, which would itself be a convention. It reports the pid because "already running" is not actionable; a pid is. That is the direct answer to a stale process surviving a pkill and going on answering probes. env entries resolve once at startup -- environment wins, declaration supplies the fallback -- and validate as a whole, reporting every problem at once rather than costing one restart per variable. config("X") for an undeclared X is fatal, because an advisory schema is just another convention. Programs without a program block are unaffected, so migration is per-program. Only one keyword is added. `config` and `env` could not become keywords -- both are real identifiers in the tree -- so the block's fields are read as identifier token values by its own parse loop and stay usable everywhere else. The init function is emitted at the block site and called from main() rather than inlined into main(). The live backend is codegen_streaming, which emits in source order and cannot hold the entry list alive until main(); this way only a single bool has to survive. Also fixes: config() was defined in el_runtime.c but never prototyped in el_runtime.h, so any el program calling it failed to compile under C99. Spec: section 18 documents what shipped. Section 9 is corrected -- it claimed decorators had no structural meaning, which has not been true for some time. Section 19 designs durability-as-an-epilogue-effect and route authorization and states plainly why neither is implemented here: both land in files under concurrent modification, and the prerequisite for both is lifting the seam from prologue-only to prologue/epilogue. Self-hosting fixpoint verified byte-identical.	2026-08-16 11:38:28 -05:00
bigmerge	cf060adbfd	Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile	2026-08-15 21:55:40 -05:00
bigmerge	b5a0a729e6	codegen: Bool is int-like, so Bool comparisons stop lowering to str_eq ... fn check(label: String, cond: Bool, want: Bool) -> Void { if cond == want { ... } -> if (str_eq(cond, want)) SIGSEGV } Bool has always been an integer in the value model — type_to_c maps Bool to "int", and el_runtime.h states "Bool -> el_val_t (0 = false, nonzero = true)". But Bool names were registered NOWHERE: build_int_names_for_params tracked Int and Float params, and the `let` path tracked Int and Float bindings. Neither knew about Bool. So comparing two Bools fell through to str_eq, which dereferenced 0 or 1 as a char* and segfaulted immediately. This is the third instance of one family found tonight, after el #137 (a call on either side of == poisoned the operator) and el #136 (a missing import compiled clean). All three are the same shape: something the compiler could not type, silently handled as a string. Found while writing #137's own test harness — the first version of that harness crashed on exactly this, on both the old and new compiler, which is how it surfaced. A test harness that cannot compare two Bools is a good way to notice. VERIFIED: - the harness that segfaulted on every prior compiler (exit 139, no output) now runs clean: 14 passed, 0 failed - self-hosting fixpoint byte-identical - the compiler's own generated C differs by 8 lines — only the intended registration - neuron's full soul amalgam regenerates in 424ms, exit 0, BYTE-IDENTICAL - test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12 Adds tests/runtime/operator_typing_test.el, the 15-case suite from #137, so this family is covered going forward rather than rediscovered.	2026-08-15 21:54:10 -05:00
bigmerge	b55e6bfd53	codegen: either side Int is enough for == and !=, not both ... let a: Int = 5 getint(5) == a -> str_eq(getint(5), a) SIGSEGV getint(5) == 5 -> getint(5) == 5 fine A function call whose return type codegen cannot infer poisoned the operator, and a declared Int on the other side did not save it. str_eq then read an integer as a char* and segfaulted. Only an integer LITERAL on one side forced the numeric form, which is why the bug stayed invisible: the common case happened to be safe. The check required BOTH operands to be provably Int: if is_int_expr(left) { if is_int_expr(right) { numeric } } Loosening to OR is strictly safer, not a trade: - when one side is a known Int, str_eq is ALWAYS wrong — it dereferences that integer — while numeric comparison is at worst a wrong answer on a program that was already ill-typed; - when neither side is Int nothing changes at all, so string comparison is untouched. Found by the test-framework agent while building the benchmark harness; it correctly declined to fix it mid-phase since it is a codegen semantics change. VERIFIED, because a semantics change earns more than an assertion: - 15/15 on a dedicated operator suite covering string literals, string vars, string-returning calls, mixed var/call, and != in every combination. The pre-change compiler scores 0/15 on the same file: it segfaults before printing anything. - self-hosting fixpoint byte-identical - the ONLY difference in the compiler's own generated C is the intended one: a nested if becoming two sequential ifs, in EqEq and NotEq. Nothing else moved. - neuron's full soul amalgam regenerates in 400ms, exit 0, output BYTE-IDENTICAL at 1,270,212 bytes - test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12 — 62 tests, 190 assertions, zero failures NOT fixed here, same family, flagged for a decision: Bool PARAMETERS are not tracked as int-like, so `cond == want` between two Bool params still lowers to str_eq and segfaults. Found while writing this commit's own test harness — the first version of it crashed on exactly that, on both the old and new compiler. It needs the same treatment, and it wants its own change.	2026-08-15 21:51:35 -05:00
bigmerge	906c664a65	compiler: a missing import is an error, not an empty string ... import "../../NOPE/does_not_exist.el" compiled CLEANLY — exit 0, empty stderr, and a program silently missing everything it imported. resolve_imports did `fs_read(src_path)` and used the result without checking. fs_read returns "" both for "file is empty" and "file does not exist", so a typo, a moved file, or a relative path resolved from the wrong working directory all produced a successful build of nothing. It caused a real wrong conclusion during test-framework work: a bisection run from a subdirectory where ../../runtime/ did not resolve produced ELEVEN consecutive "successful" compiles that had included no runtime at all, and the results were believed before anyone noticed. Missing dependency, confident success — the same shape as a test suite reporting pass for tests that never ran, and as a benchmark reporting 0us because the optimiser deleted the loop. fs_exists separates the two cases, so a legitimately empty file still resolves to "" and is fine. A path that does not exist now prints the resolved path and exits 1, which is what build scripts check. Verified: - bad import: exit 1 (was 0), message names the resolved path - elc-cli.el still compiles, self-hosting fixpoint byte-identical - neuron's full soul amalgam regeneration: exit 0, 405ms, output byte-identical at 1,270,212 bytes	2026-08-15 21:47:32 -05:00
Neuron	6a6b589ba0	bench: real black_box barrier + three-signal growth-curve gate ... Adds el_black_box (inline asm, +r constraint, memory clobber) and runtime/elbench.el: a growth-curve classifier that gates time AND allocation-count AND allocation-bytes, failing if any exceeds its declared curve. Refusal is a first-class verdict. The classifier REFUSES rather than classifying when the largest measurement is below the floor, or when a series is hard-flat across an 8x input range -- the shape produced when the optimiser deletes the work. Reporting O(1) there would be a confident answer with nothing behind it. Disagreeing ratios report INDETERMINATE rather than a guess. Deviation from DESIGN.md 6.2, stated in the source: uses consecutive ratios on a mandated geometric sweep rather than least-squares over candidate curves. Ratios are directly interpretable on a doubling sweep and need no floating point; the cost is weaker O(n) vs O(n log n) separation, reported as an ambiguous band rather than guessed. Documents the counter scope limit: engram_*.c and libcurl malloc are NOT tracked, so a flat curve over engram/HTTP-dominated work is not evidence of anything. 13 tests prove the classifier against real measured series from fitprobe.el -- including that an accumulator's allocation COUNT is linear while its bytes are quadratic, and that el #132's pure-CPU shape reads FLAT on both allocation signals and is caught only by time.	2026-08-15 21:45:13 -05:00
Neuron	3e7ab07e82	test framework phase 1: forward decls, void-return fix, suite migration ... Completes the Phase 1 runner and migrates the 11 test files onto it. - forward-declare the registry accessors in the test preamble; they are defined at the end of the unit but the El runner is compiled in between - eltest.el: explicit trailing return in the void emit_* helpers, which otherwise lower to 'return println(...)' and fail to compile - test files import runtime/eltest.el explicitly, using the language's own textual import mechanism rather than compiler-side auto-injection - DESIGN.md 6.5: gate on allocation COUNT AND BYTES, not count alone Verified: self-hosting fixpoint byte-identical (gen2 == gen3). 6 of 11 suites run and report per-test timing. The other 5 fail to COMPILE, and fail identically under the committed compiler -- pre-existing breakage this framework makes visible for the first time.	2026-08-15 21:28:30 -05:00
bigmerge	a668062e38	Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile	2026-08-15 21:24:03 -05:00
Neuron	24fac765a6	test framework phase 1: compile-time registry + El-side runner ... Replace the hardcoded test harness main() with a generated static registry and index-based accessors, and move all reporting into runtime/eltest.el. The old harness inlined direct calls into main() and counted assertions in two globals. That shape cannot report which test failed, how long any test took, or whether a test ran at all -- a misspelled registration reported success for a test that never executed. - assertions record into per-test state instead of global counters - registry table emitted at compile time; discovery strictly precedes execution, which is what later enables --list, filtering and sharding - per-test wall timing on CLOCK_MONOTONIC, taken in C around the call - runner in El: structured NDJSON events as source of truth, human output rendered from the same fields	2026-08-15 21:24:03 -05:00
bigmerge	37bcf7eb74	runtime: allocation accounting — the deterministic signal for complexity gating ... Implements the three primitives the test-framework design (DESIGN.md §6.5) requires for gating on growth curves: el_alloc_count, el_alloc_bytes, el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per the project's C-builtin recipe. WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build failure, which means the signal cannot flake. Wall-clock needs warmup, statistics, and a quiet machine; on shared CI it is unusable as a gate. Allocation counts are perfectly deterministic — same input, same number, every machine, every run. Fit them against n and a complexity regression becomes a build failure with zero noise. All four runtime string allocators (el_strdup, el_strbuf, and their _persist variants) funnel every allocation the language performs, so instrumenting there counts everything. WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate. Measured with two El programs, one allocating once per item, one rebuilding its accumulator each iteration: n linear allocs / bytes quadratic allocs / bytes 100 100 / 290 100 / 5,150 200 200 / 690 200 / 20,300 400 400 / 1,490 400 / 80,600 800 800 / 3,090 800 / 321,200 The quadratic program's allocation COUNT is exactly linear — identical to the healthy one. Counting allocations alone would have missed it completely. Bytes catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 -> converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0. That shape — count linear, per-allocation size growing — is the classic accidental quadratic, and it is exactly elc's defect: quadratic allocation VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across both; RSS and wall-clock were just the two ways it surfaced. el_peak_rss is exported for context and is explicitly NOT a gating signal — it is perturbed by allocator internals, the page cache, and the OS. Gate on the deterministic numbers; report the physical one. Counters are unsynchronised by design: this is measurement, and a lock would change the thing being measured. Exact on the single-threaded compile path, approximate under threads.	2026-08-15 21:21:42 -05:00
bigmerge	e917b3d439	store: make the buffer pool sense its own state and correct from it ... Follow-on to the edge write barrier. That fix removed the full-store walk; this one makes the pool able to notice if anything like it happens again. WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill twice on 2026-08-15 and said nothing. From outside it was indistinguishable from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each costing a deploy or a rollback. The whole time, hits/misses/evictions were already being counted in PgCache, and the struct comment read: /* stats (introspection only — never affect semantics) */ That comment was the bug. Self-measurement treated as decoration is why the pool could not correct itself and why no one outside could see what it was doing. A system that cannot read its own state cannot correct, and neither can anyone watching it. - pc_adapt_budget(): the loop, closed. Over a sliding window, evictions running at a large fraction of accesses WHILE reuse is real means the working set exceeds the budget — so grow it, geometrically, bounded by a LIVE re-read of physical memory. Evictions alone are not pressure (a scan evicts and never returns); evictions with reuse are. An explicit ENGRAM_POOL_FRAMES still wins — an operator override must not be silently overruled. - Budget derived, not declared. A constant cannot be right: 16 GiB of frames is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM at startup" is a guess about the future — it cannot know the store grew or the machine changed. Hence the live re-read. - pc_report(): ONE structured emission carrying the entire sensed state, through emit_log — El's existing telemetry, already exporting to OTLP. Deliberately not a function per stat, and deliberately not a bespoke /api/pool endpoint: both make observability something hand-written per noun instead of the uniform mechanism every component already has. - engram_pool_stats_json(): the same state readable live, wired through the normal builtin path (codegen arity + el_seed wrapper), so the pool can be observed in real time rather than reconstructed afterward from a stack sample. Verified: with the exact configuration that took production down (ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete — and NO pressure event fires, because the barrier removed the walk that caused it. The controller is defense in depth; the barrier is the fix.	2026-08-15 20:44:23 -05:00
bigmerge	4e24d7d3f1	runtime: engram_edges_json — read edges without a whole-graph file round trip ... /api/graph/edges answered a read query by calling engram_save() to serialize the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in one line, and both bit production on 2026-08-15: 1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram server's CANONICAL store. A READ route overwriting the persistence owner's canonical file. This defect had been fixed once (export moved to a scratch path); it came back when the hand-written dispatch block was replaced by @route dispatch and the unfixed copy is the one that survived the merge. 2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole graph, per request, to return a bounded slice. Calling it tonight overwrote the canonical snapshot and immediately preceded an engram crash loop. engram_edges_json(limit, offset) is the builtin that route's own TODO asked for ("Future: add an engram_edges_json() builtin and drop the file round trip entirely"). It walks g->edges directly and emits every persisted field. limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell over, and an unbounded default would preserve the failure mode under a new name. Callers page explicitly. Registered in codegen.el's builtin_arity (both plain and __ spellings) and wrapped in el_seed.c per the project's C-builtin recipe.	2026-08-15 20:10:48 -05:00
bigmerge	09dade0613	Merge remote-tracking branch 'origin/pr/103' into HEAD ... # Conflicts: # lang/AGENTS.md # lang/runtime/el_runtime.h	2026-08-15 18:26:57 -05:00
will.anderson	ee39aa5f17	Merge pull request 'ingest: unify transduce_prose/transduce_structured into one transduce()' (#117) from feat/transduce-unify into dev	2026-08-15 22:36:12 +00:00
bigmerge	e29fe4fd0b	ingest: unify transduce_prose/transduce_structured into one transduce() ... transduce() is now THE single mechanism: one function, no content-type branch inside it. It never asks whether `source` is prose, JSON, or raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally: split on "\n\n" as a universal boundary-marker check, and if that finds no boundary, fall back to fixed 4096-char windows. Same node/edge wiring (root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets a heading/section_of link) regardless of what's inside a chunk. Dedup is the existing find_existing_by_content path via merge_manifold, applied uniformly. The old transduce_structured JSON dataset/records/feature-node interpretation is deleted outright, not just unused — a JSON file now gets chunked and deduped like anything else, with no pre-computed structure. All five ingest_* entry points still exist unchanged in name and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one transduce() (ingest_stream builds its own turn-nodes directly and never called either old function, so it's untouched). This unlocks raw/opaque content (audio, or anything else with no natural text/JSON shape) without any DSP, LLM call, or external API: transduce() chunks it exactly like it chunks anything else. There is zero semantic understanding of audio (or any payload) claimed or built here — any meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time. Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk) because El strings are NUL-unsafe under strlen-based ops and fs_read()'s result silently truncates at the first embedded NUL, which is routine in real binary/audio bytes. ingest_file compares fs_read()'s string length against a real fs_size() stat() count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim, no invention), joined with the same "\n\n" marker transduce()'s boundary scan already looks for. This is a mechanical fidelity fix, not interpretation of content — transduce() never learns a fallback happened. Registered both builtins' arity in codegen.el; did not rebuild the elc compiler binary itself (unrelated, pre-existing gap: self-hosting elc via el_seed.c fails on this worktree independent of this change, reproduced with codegen.el reverted) — the existing elc binary compiles calls to unregistered builtins via its already-existing arity=-1 passthrough, confirmed by an actual clean `elc ingest.el` + `cc` build against the modified el_runtime.c. INGEST_KIND keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a content-type flag; the redundant "structured" value (an alias for "file" that hinted the now-deleted JSON branch) is removed. ingest_dir drops its file-extension filter for the same reason: transduce() takes anything now. Verification: local manifold construction confirmed correct against a real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte real prefix slice) — exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes. Compiles clean via the real elc + the modified el_runtime.c/engram_*.c (built and booted an actual sandbox engram off this exact source with `nsbx create --branch`). NOT verified this session, disclosed rather than papered over: end-to-end server-confirmed persistence (a real before/after /api/stats delta, and a fetched node by id) for the audio, prose, and JSON-fixture cases. Every local nsbx sandbox engram tried tonight (two stock pre-#109 binaries hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW binary built from current dev) took minutes-to indefinitely long on the final /api/load-merge write's embedding step and hit the client's 60s HTTP timeout before responding, even for a 5-node write. This is confirmed as real (if slow) forward progress, not a hang: the sandbox's WAL file was observed growing steadily across every attempt. The code's own pre-existing HONESTY GATE correctly refused to report success in every case, returning "load-merge failed: ..." with a "nothing below this manifold was confirmed persisted by the server" note instead — exactly as designed. This is an environment/infrastructure limitation, not a defect introduced by this change: the engram server binary itself is untouched by this commit.	2026-08-15 17:28:19 -05:00
bigmerge	bacaf3d39c	engram: reconcile M8 HNSW vindex (#109) onto current dev, restore 3 fixes the branch predated ... Lands feat/reframe-region-setop (PR #109: native set-based reframe_region, decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's actual current HEAD, plus engram-tiered-storage's still-unique test suite. RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/ verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/ select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two branches; #109's copy is a strict superset (adds vindex_harvest_from_store, used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and engram_verify.c are also byte-identical. #109's own branch point already carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing left to merge into #109 for those files. The one thing engram-tiered-storage had that #109's tree dropped: its full test suite (test_vindex.c, test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh harnesses) — ported over here unchanged. WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch forked from dev on 2026-08-14 15:40 (before restructure-adjacent history diverged the file's merge-base for `git merge` — it presented as an add/add conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but it silently reverted THREE dev fixes landed on 2026-08-14/15, after the branch point, that the PR's diff had no way to know about: 1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of the query-aware propagation gate dropped the shift-and-floor rescale about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate 0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi] is computed) is orthogonal to WHAT it gates on and both are kept. 2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global). PR's tree predates this and dropped all three counters + their JSON stats fields. Restored declarations, all 4 direct increment sites, the eg_wm_carry_over bll increment, and the act-stats JSON fields -- alongside (not instead of) the PR's own P4 afferent / API-reshape counters already in that same struct/JSON. 3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev selects the STRONGEST qualifying candidate for consolidation each call; PR's tree predates this and reverted to hash-slot order (arbitrary wrt association strength) for edge formation -- the one path that writes PERMANENT structure. Restored the strongest-candidate while-loop, keeping the PR's own genuine improvement at that site (engram_adj_on_edge_added incremental-index append instead of a bare adj_dirty=1 full-rebuild flag). engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env flags, /api/nodes connected-count in responses) were pure additive: dev's side was empty, PR's side added the feature. Took PR's side whole. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): - cc -std=c11 -O2, clean link against the real engram/src/server.el via elc, zero errors. - vindex_bench (built standalone, read-only harvest) against the real production store clone (13,671 embedded nodes, 768-dim nomic-embed-text): recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs 2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s for the full 13,671-node set -- see the flagged risk below. - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned snapshot of the live store, 13,424 nodes / 37,656 edges) and called /api/activate for real: first call after boot 41.5s (pays the one-time HNSW build inline -- matches the standalone bench), second/third calls 356ms/605ms, no crash, correct results, act-stats JSON (including the restored evict_floor/cap/bll fields) reads correctly. KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync builds the HNSW index synchronously, inline, on the first engram_activate() call after every process start (or index invalidation). On the real node count that is a ~46s blocking stall on a single-threaded server -- the first request after every restart (or its concurrent siblings) waits the full build. Recommend a background/incremental build (or a bounded per-call build budget) before this ever reaches the live daemon. See PR description / final report for the fuller writeup.	2026-08-15 16:46:44 -05:00
will.anderson	69870ac883	Merge pull request 'fix(codegen): emit the declared cgi identity — it was searched for in a list that cannot contain it' (#89) from fix/cgi-identity-emission-clean into dev	2026-08-15 19:58:16 +00:00
will.anderson	09ae14a970	Merge pull request 'fix: float arithmetic codegen (segfault/garbage) and math_log aliasing' (#104) from worktree-agent-a456e0cf8cd2ee361 into dev	2026-08-15 19:54:46 +00:00
bigmerge	2f832c8def	Fix float arithmetic codegen and math_log aliasing ... Float + previously fell through to string concat (segfault); -, *, /, % operated on raw IEEE-754 bit patterns as integers (garbage results). Floats are now tracked via a __float_names typed-binding set (parallel to the existing int-tracking scheme) and arithmetic is emitted as real C double ops. Also fixes math_log, which was wrongly aliased to natural log (duplicating math_ln) — now uses log10 — and adds the missing <math.h> include. Rebuilt elc binary included.	2026-08-15 14:24:14 -05:00
bigmerge	1010185978	Add op_assert grounded-envelope primitive and purview-bounded mutation wrappers ... Adds engram_assert_json — a grounded "assertion envelope" primitive for a realizer/op_assert seam (per backlog bl-53/#57) — plus purview-scoped mutation wrappers engram_node_full_in/engram_connect_in, which refuse non-default purviews rather than silently mutating the live store. Threads through el_seed.c/h wrappers and the codegen.el arity table per the project's existing C-builtin recipe. Also rewrites lang/AGENTS.md build docs with verified (2026-08-15) findings that el_seed.c does not compile standalone.	2026-08-15 14:24:11 -05:00
bigmerge	01826421c4	seam: implement decorated-fn boundary auto-emit; prove on clone ... Will waived diff review -> build it for real. Add engram_boundary_beat() to the runtime (afferent counter++ + engram_chrono_tick + engram_strengthen(self-anchor) + dharma_emit) and two act-stats counters (aff_boundary_ops, dharma_emits). codegen cg_fn injects ONE engram_boundary_beat(op) at the entry of every @manager/@accessor fn (fn_has_decorator, so it fires under @route @manager too) — a decorated op self-reports with ZERO hand-written instrumentation. Rebuilt elc self-host + the cognition engram in the worktree; ran it as the clone daemon on :8900. Proof (/api/boundary-proof, @manager, empty body, 5x): aff_boundary_ops 0->5, dharma_emits 0->5, self activation_count 1510->1513, chrono stamp advanced. Brought in feat/cognitive-architecture engram runtime+server for the build. strengthen = activation bump (not content/edge write) -> identity protection intact. Live :8742 untouched; no push, no cutover.	2026-08-14 21:20:18 -05:00
bigmerge	d4f401de1c	reshape: decorator-as-seam — port @route codegen, prove decorate->serve, rewrite surface as decorated El ... Ground-truth the three seams (route/telemetry+interoception/bus) with file:line evidence. Port the tested @route codegen+parser from feat/el-route-decorators into the worktree elc (decoration synthesizes el_route_dispatch — no hand-written 90-branch handle_request). Rebuild elc self-host; prove decorate->serve end-to-end (route_proof.el on :8951). Rewrite surface.el as El-native decorated components: @route + @accessor/@manager, in-process engram_* builtins (not http_get), @manager ops emit on the real dharma_* bus (same transport as wt/swarm-ccr). Identity keystones refused in write/relate/supersede. Gate-1 clone recipe (WAL-aside cold-boot + ENGRAM_WAL=on) proves the FULL op set live on the clone. Boundary auto-emit (telemetry/interoception/bus) staged as a reviewable cg_fn diff (SEAM_STAGED.md) — needs the cognition-engram rebuild to verify link. Live :8742 untouched; no push, no cutover.	2026-08-14 21:01:27 -05:00
will.anderson	7aa847e32a	Merge origin/dev into engram-tiered-storage ... Resolve 3 conflicts: - lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork; single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh). - lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated artifact folder, not a source path; a release is a git tag, not a folder). - lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL. Nothing in dev's build consumes the deprecated fork or releases/ folder.	2026-08-12 15:23:02 -05:00
will.anderson	bb64a236ed	engram tiered storage: engram-service wiring + elc fold-hang fix + prune-store mirror ... - Wire paged store into the ENGRAM SERVICE (server.el, the authoritative durable owner): boot->engram_store_boot, persist_canonical->engram_store_checkpoint, gated by ENGRAM_STORE. - elc (lang/elc.c + src/parser.el + codegen.el + elc-combined.el): OOB guard in tok_kind/tok_value + parse_block progress backstop — fixes the pre-existing unbounded-memory fold hang on sessions.el. - engram_prune_telemetry mirrors ISE prune to the store (store_forget) so store live-count tracks resident and stale telemetry stays bounded. - Deployed live 2026-08-12: engram :8742 on neuron.egm+WAL, count reconciled 11552.	2026-08-12 14:14:20 -05:00
will.anderson	0a72fced28	engram: WAL persistence + integrity hardening + single canonical runtime ... Establish lang/runtime/ as the ONE canonical el runtime (from the active runtime that carries hebb/emb persistence + the new WAL); repoint the el CI publish, engram build, elb default, and in-repo build scripts to it; delete the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh drift guard. Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian edge weights — learned co-activation was wiped on every restart. Publishing from canonical ships the stranded 'learning that cannot outlive the process' fix. WAL storage engine + integrity fixes (DELETE->tombstone + store-layer protection, safe data-dir default) ride in behind ENGRAM_WAL (default off = byte-identical to today). Verified: engram elb per-module build clean, WAL gate 66/66, native smoke ok, drift-guard green.	2026-08-11 21:31:37 -05:00
Neuron	866c75e5e2	fix(codegen): emit the declared cgi identity — it was searched for in a list that cannot contain it ... A cgi block is a top-level declaration, so codegen_streaming classifies it via is_top_level_decl and releases it. The identity emission then searched toplevel_exec_stmts for that same block. Declarations are excluded from that list by construction, so the search could never succeed. A probe printed what it actually saw for a program whose first statement is a cgi block: [Let, Expr]. It emitted nothing, silently, with no diagnostic on any channel. The code documented its own assumption — 'Since cgi blocks are rare and small, they end up in toplevel_exec_stmts' — and that assumption was false. Capture the declared values before the release and emit from them. The search is deleted rather than repaired, so the failure mode is removed rather than relocated. Proven discriminating (old fails, new passes): minimal cgi program, old -> 0 el_cgi_init minimal cgi program, fixed -> el_cgi_init with all four declared values neuron soul, fixed -> principal present in the compiled binary (0 before), boots in 2s, interface 110 routes in / 110 out Consequence: a binary now carries its declared identity as a compiled constant, which is what the identity protocol requires. Whether the runtime surfaces it to state_get("soul_principal") is unverified and separate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 13:48:27 -05:00
will.anderson	23552ed40a	make the el-compiler runtime compile again ... The loopback/API-key hardening carried in this file since 2026-07-15 called el_http_request_authorized and el_http_send_401 from http_worker with no forward declarations, so the calls were implicit and the later static definitions conflicted. The file did not build. Two prototypes fix it. Worth naming the pattern: uncommitted work is invisible to every check that would have caught this. Three weeks of desktop security hardening was neither committed nor compiling, and nothing reported either fact.	2026-08-08 08:45:12 -05:00
will.anderson	6838e5cbff	port the \uXXXX UTF-8 decode fix to the el-compiler runtime copy ... Same defect as the release runtime: \uXXXX was skipped and a literal '?' emitted, destroying every non-ASCII character in JSON entering the runtime. Two copies of one parser bug is how this class of fault survives a fix, so it lands in both. NOTE: this file also carries pre-existing uncommitted work from 2026-07-15/16 that this commit preserves rather than authors - loopback bind hardening (EL_HTTP_BIND_HOST) and per-install API-key auth (EL_HTTP_AUTH_KEY) for the shipped desktop build, plus goal-bias and node-json changes. It had been sitting in the working tree for three weeks. Committing it because uncommitted work is work that does not survive, which is the same durability lesson as yesterday's Hebbian write-back finding. It needs review on its own terms - see the backlog item for reconciling the two runtime copies.	2026-08-08 08:44:51 -05:00
will.anderson	43636aed99	runtime: pair fs_read length hint with its buffer in BOTH runtimes — kill response truncation for good ... The binary-safe fs_read length (_tl_fs_read_len) was consumed by the HTTP response path for ANY body, even when a handler wrapped a smaller file into a larger reply. Content-Length then lied AND the send stopped short: the safety-contact (988) routes returned 178 of 208/218 bytes, cut mid-'set_at' — unparseable JSON. The desktop app read that as failure. On Windows the shipped brain is an OLD build without even the per-handler workaround, so EVERY reply truncated: the app can't read confirmations and refuses the new user. Durable fix: pair the length hint with the exact buffer pointer it describes (_tl_fs_read_buf). Apply the raw byte count ONLY when the response IS that buffer (binary file serving stays correct); every wrapped/enveloped/derived body is measured with strlen. Reset both at request start and in fs_read / json_get_raw. This also closes the stale-hint heap over-read (a length larger than a later body would read past it out the socket) that a plain max() leaves open — so this class of bug dies on every platform, not just where a handler happened to be patched. Applied identically to the mainline runtime (lang/el-compiler/runtime) AND the frozen release runtime (lang/releases/v1.0.0-20260501) the desktop souls compile against — the release copy still carried the raw leak, which is why the Windows brain kept truncating. Same proven approach as PR #78 (Tim Lingo), extended to cover the release runtime and rebased onto current main. Both runtimes: gcc -fsyntax-only clean.	2026-07-22 15:04:25 -05:00
will.anderson	1e49560f1f	Merge remote-tracking branch 'origin/feat/engram-semantic-search' into hotfix/stage-elc-engram-integration ... # Conflicts: # lang/el-compiler/runtime/el_runtime.c	2026-07-15 09:33:05 -05:00
will.anderson	e8f0b5a9de	Merge remote-tracking branch 'origin/fix/engram-lexical-tokenized-search' into hotfix/stage-elc-engram-integration	2026-07-15 09:28:44 -05:00