let a: Int = 5
getint(5) == a -> str_eq(getint(5), a) SIGSEGV
getint(5) == 5 -> getint(5) == 5 fine
A function call whose return type codegen cannot infer poisoned the operator,
and a declared Int on the other side did not save it. str_eq then read an
integer as a char* and segfaulted. Only an integer LITERAL on one side forced
the numeric form, which is why the bug stayed invisible: the common case
happened to be safe.
The check required BOTH operands to be provably Int:
if is_int_expr(left) { if is_int_expr(right) { numeric } }
Loosening to OR is strictly safer, not a trade:
- when one side is a known Int, str_eq is ALWAYS wrong — it dereferences
that integer — while numeric comparison is at worst a wrong answer on a
program that was already ill-typed;
- when neither side is Int nothing changes at all, so string comparison is
untouched.
Found by the test-framework agent while building the benchmark harness; it
correctly declined to fix it mid-phase since it is a codegen semantics change.
VERIFIED, because a semantics change earns more than an assertion:
- 15/15 on a dedicated operator suite covering string literals, string vars,
string-returning calls, mixed var/call, and != in every combination. The
pre-change compiler scores 0/15 on the same file: it segfaults before
printing anything.
- self-hosting fixpoint byte-identical
- the ONLY difference in the compiler's own generated C is the intended one:
a nested if becoming two sequential ifs, in EqEq and NotEq. Nothing else
moved.
- neuron's full soul amalgam regenerates in 400ms, exit 0, output
BYTE-IDENTICAL at 1,270,212 bytes
- test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12 —
62 tests, 190 assertions, zero failures
NOT fixed here, same family, flagged for a decision: Bool PARAMETERS are not
tracked as int-like, so `cond == want` between two Bool params still lowers to
str_eq and segfaults. Found while writing this commit's own test harness — the
first version of it crashed on exactly that, on both the old and new compiler.
It needs the same treatment, and it wants its own change.
import "../../NOPE/does_not_exist.el"
compiled CLEANLY — exit 0, empty stderr, and a program silently missing
everything it imported.
resolve_imports did `fs_read(src_path)` and used the result without checking.
fs_read returns "" both for "file is empty" and "file does not exist", so a
typo, a moved file, or a relative path resolved from the wrong working
directory all produced a successful build of nothing.
It caused a real wrong conclusion during test-framework work: a bisection run
from a subdirectory where ../../runtime/ did not resolve produced ELEVEN
consecutive "successful" compiles that had included no runtime at all, and the
results were believed before anyone noticed.
Missing dependency, confident success — the same shape as a test suite
reporting pass for tests that never ran, and as a benchmark reporting 0us
because the optimiser deleted the loop.
fs_exists separates the two cases, so a legitimately empty file still resolves
to "" and is fine. A path that does not exist now prints the resolved path and
exits 1, which is what build scripts check.
Verified:
- bad import: exit 1 (was 0), message names the resolved path
- elc-cli.el still compiles, self-hosting fixpoint byte-identical
- neuron's full soul amalgam regeneration: exit 0, 405ms, output
byte-identical at 1,270,212 bytes
el #131 instrumented the four string allocators, which meant list- and map-heavy
code reported ZERO allocations — a benchmark over lists would have been fitted
against a flat line and passed anything. Caught during framework work: a
"linear" specimen read 0 allocs until it was rewritten to allocate strings.
A gate is only as good as its blind spots are small, and a signal that silently
reads zero is worse than no signal: it produces a confident pass.
Now counted at every container allocation — ElList and ElMap bodies, their
backing arrays, the copy-on-write clones, and the realloc growth path.
Verified on an append loop (n = 100..800):
allocs 7, 8, 9, 10 +1 per doubling = O(log n) reallocations
bytes 2048, 4096, 8192, 16384 exactly 2x per doubling = O(n)
Both curves are what correct amortized growth should look like, and both read
zero before this change.
Known remaining scope, stated rather than left implicit: these counters cover
the runtime's own allocations. They do not see malloc inside engram_*.c or
libcurl, which is correct — the gate is for El-level complexity, not for
third-party memory behaviour.
el_val_t math_log(el_val_t f) { return el_from_float(log(el_to_float(f))); }
el_val_t math_ln(el_val_t f) { return el_from_float(log(el_to_float(f))); }
Both were natural log, so math_log and math_ln were the same function.
log10(100) returned 4.605 instead of 2.
Three sources already agreed it should be base-10 and were being contradicted
by this one line:
- runtime/math.el:55 "// math_log — base-10 logarithm."
- el_seed.c:1278 __log_f -> log10() (the path math.el actually calls)
- tests/native/test_math.el:133 asserts log10(100) == 2
FOUND BY THE NEW TEST FRAMEWORK ON ITS FIRST RUN (el #133). The assertion had
been sitting in the suite the whole time; nothing could report it. The old
harness printed "N passed, M failed" with no per-test detail, and half the
suites were not compiling at all — so a failing assertion in a suite nobody
could run was indistinguishable from no failure.
That is the entire argument for the framework, demonstrated on day one: this is
not a bug the framework introduced, it is a bug the framework made VISIBLE.
Verified: tests/native/test_math.el goes 12/13 -> 13/13, math-log passing.
Completes the Phase 1 runner and migrates the 11 test files onto it.
- forward-declare the registry accessors in the test preamble; they are
defined at the end of the unit but the El runner is compiled in between
- eltest.el: explicit trailing return in the void emit_* helpers, which
otherwise lower to 'return println(...)' and fail to compile
- test files import runtime/eltest.el explicitly, using the language's own
textual import mechanism rather than compiler-side auto-injection
- DESIGN.md 6.5: gate on allocation COUNT AND BYTES, not count alone
Verified: self-hosting fixpoint byte-identical (gen2 == gen3). 6 of 11
suites run and report per-test timing. The other 5 fail to COMPILE, and
fail identically under the committed compiler -- pre-existing breakage
this framework makes visible for the first time.
THE BUG. str_char_code() and str_slice() each called strlen() on every
invocation. The lexer walks source one character at a time, so every character
access rescanned the whole remaining input: O(n) per character over n
characters = O(n^2).
el_val_t str_char_code(el_val_t s, el_val_t i) {
...
int64_t n = (int64_t)strlen(str); // <- O(n), every call
if (idx < 0 || idx >= n) return 0;
return str[idx];
}
HOW IT WAS FOUND. Not by reading code — by sampling the running process, which
is the same method that resolved tonight's engram outage after four wrong
theories. A geometric sweep of synthetic sources showed wall-clock rising 3.0x,
3.0x, 4.0x, 4.14x per doubling (converging on 4x = quadratic), and a stack
sample put 779 of 779 samples inside lex(), every one bottoming out in
_platform_strlen via str_char_code and str_slice.
THE FIX. Remember the length instead of recomputing it. The subtlety is
INVALIDATION: El strings are arena-allocated, so a freed pointer can be reused
for a different string at the same address, and a naive pointer-keyed cache
would hand back a stale length and read past the end of the new string —
trading a performance bug for a memory-safety one. So entries carry a
generation, a hit requires pointer AND generation to match, and every path that
frees or mutates a runtime string bumps the generation: el_arena_pop,
seed_request_end, __str_set_char. Stale entries cannot be believed; they miss
and recompute.
MEASURED, same host, same inputs:
n(fns) before after
512 0.10s 0.01s
1024 0.37s 0.02s
2048 1.51s 0.03s 50x
the compiler's own 422 KB source concatenated (DESIGN.md's 3.58s case):
3.55s -> 0.03s 118x
The speedup GROWS with input size, which is the signature of removing a
complexity class rather than a constant factor. After the fix each doubling
adds ~0.01s: linear.
CORRECTNESS, verified rather than assumed:
- byte-identical output on every sweep input (n = 128..2048)
- byte-identical output on the 422 KB compiler concatenation
- byte-identical output on tests/runtime/string_test.el
- self-hosting fixpoint byte-identical
- new tests/runtime/str_cache_test.el: 17 assertions covering bounds, empty
strings, negative indices, slice clamping, distinct strings not sharing a
cached length, 1000 interleaved strings forcing cache-slot collisions, and
a grown string not reporting its old length. All pass.
This is the defect that made dist/soul.c a committed artifact: elc could not run
in CI because it needed 24 GB+ and minutes. It needs neither now.
Replace the hardcoded test harness main() with a generated static registry
and index-based accessors, and move all reporting into runtime/eltest.el.
The old harness inlined direct calls into main() and counted assertions in
two globals. That shape cannot report which test failed, how long any test
took, or whether a test ran at all -- a misspelled registration reported
success for a test that never executed.
- assertions record into per-test state instead of global counters
- registry table emitted at compile time; discovery strictly precedes
execution, which is what later enables --list, filtering and sharding
- per-test wall timing on CLOCK_MONOTONIC, taken in C around the call
- runner in El: structured NDJSON events as source of truth, human output
rendered from the same fields
Implements the three primitives the test-framework design (DESIGN.md §6.5)
requires for gating on growth curves: el_alloc_count, el_alloc_bytes,
el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per
the project's C-builtin recipe.
WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build
failure, which means the signal cannot flake. Wall-clock needs warmup,
statistics, and a quiet machine; on shared CI it is unusable as a gate.
Allocation counts are perfectly deterministic — same input, same number, every
machine, every run. Fit them against n and a complexity regression becomes a
build failure with zero noise.
All four runtime string allocators (el_strdup, el_strbuf, and their _persist
variants) funnel every allocation the language performs, so instrumenting there
counts everything.
WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate.
Measured with two El programs, one allocating once per item, one rebuilding its
accumulator each iteration:
n linear allocs / bytes quadratic allocs / bytes
100 100 / 290 100 / 5,150
200 200 / 690 200 / 20,300
400 400 / 1,490 400 / 80,600
800 800 / 3,090 800 / 321,200
The quadratic program's allocation COUNT is exactly linear — identical to the
healthy one. Counting allocations alone would have missed it completely. Bytes
catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 ->
converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0.
That shape — count linear, per-allocation size growing — is the classic
accidental quadratic, and it is exactly elc's defect: quadratic allocation
VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt
one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across
both; RSS and wall-clock were just the two ways it surfaced.
el_peak_rss is exported for context and is explicitly NOT a gating signal — it
is perturbed by allocator internals, the page cache, and the OS. Gate on the
deterministic numbers; report the physical one.
Counters are unsynchronised by design: this is measurement, and a lock would
change the thing being measured. Exact on the single-threaded compile path,
approximate under threads.
The guard I added minutes ago checked swap availability as a level
(avail < total/8 -> report zero available). That is the wrong signal, and the
same host proved it twice within minutes:
47.65 / 48.00 GiB swap used, 2047 swapouts/s -> genuinely thrashing
26.67 / 28.00 GiB swap used, 0 swapouts/s -> healthy, 15.6 GiB free
Both are ~97% "used". macOS grows swap files on demand and trims them lazily,
so the level says almost nothing about now — it is a high-water mark. The level
check calls the second state an emergency and starves the pool for no reason,
which is its own failure mode: a guard that fires on healthy machines gets
disabled, and then guards nothing.
What separates the two is whether pages are moving. So sample the swapout
counter across calls and judge the delta:
- > 200 pages/s (~3 MiB/s) sustained outward paging => report zero available;
callers refuse to grow and pc_relieve_pressure hands frames back.
- The first call primes the baseline and reports no pressure. One sample
cannot have a rate, and inferring one from a single reading is exactly the
mistake this commit removes.
Measured thresholds, not guessed: idle sat at 0/s, recovery burst hit 24,845/s
while the compressor drained (transient, correctly not a growth decision since
growth is only evaluated on eviction passes), and real thrash held ~2000/s.
200/s sits clearly above noise and far below either.
The compressor-footprint subtraction stays: that RAM is genuinely spoken for
regardless of paging rate.
The adaptive budget I added an hour ago could only grow, and grew toward a
share of TOTAL ram (80%, ~38 GiB on a 48 GB host). That is a memory leak with
extra steps: total never shrinks when other processes need memory, so the pool
had no way to notice it was starving the machine it runs on. Deployed briefly;
caught as memory pressure on the host.
A control loop with only one direction is not a control loop.
- pc_available_ram(): free + inactive + purgeable via host_statistics64 on
Darwin, MemAvailable on Linux. Availability is the quantity that moves when
the machine is under pressure; total is not. Returns 0 when it cannot be
read, and callers then refuse to grow — a cache is never worth swapping the
host, so unknown means no.
- Growth is bounded by availability minus a free-memory floor (2 GiB default,
ENGRAM_POOL_FREE_FLOOR_MB), not by total. The share-of-total ceiling stays
as a second bound and drops 80% -> 50%.
- pc_relieve_pressure(): the missing direction. On every eviction pass, if
available memory is under the floor, hand back ~25% of held frames; the
resident set follows on the next pass so the memory is actually returned
rather than merely re-labelled. Counted as adapt_shrinks alongside
adapt_grows so both directions are visible in the same report.
- pc_default_cap() also clamps the STARTING budget to what is spare right
now, so a cold boot on a loaded machine does not open at a size the host
cannot afford.
Verified on a 48 GB host: engram boots in ~30s, RSS settles at 2.22 GiB (the
store's actual size, resident, not creeping), 0.0% CPU, 13,439 nodes / 37,670
edges, embeddings complete. Guard reports 9.71 GiB available against a 2.00 GiB
floor — 7.71 GiB of headroom it is permitted to use and no more.
Follow-on to the edge write barrier. That fix removed the full-store walk;
this one makes the pool able to notice if anything like it happens again.
WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill
twice on 2026-08-15 and said nothing. From outside it was indistinguishable
from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories
got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each
costing a deploy or a rollback. The whole time, hits/misses/evictions were
already being counted in PgCache, and the struct comment read:
/* stats (introspection only — never affect semantics) */
That comment was the bug. Self-measurement treated as decoration is why the
pool could not correct itself and why no one outside could see what it was
doing. A system that cannot read its own state cannot correct, and neither can
anyone watching it.
- pc_adapt_budget(): the loop, closed. Over a sliding window, evictions
running at a large fraction of accesses WHILE reuse is real means the
working set exceeds the budget — so grow it, geometrically, bounded by a
LIVE re-read of physical memory. Evictions alone are not pressure (a scan
evicts and never returns); evictions with reuse are. An explicit
ENGRAM_POOL_FRAMES still wins — an operator override must not be silently
overruled.
- Budget derived, not declared. A constant cannot be right: 16 GiB of frames
is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM
at startup" is a guess about the future — it cannot know the store grew or
the machine changed. Hence the live re-read.
- pc_report(): ONE structured emission carrying the entire sensed state,
through emit_log — El's existing telemetry, already exporting to OTLP.
Deliberately not a function per stat, and deliberately not a bespoke
/api/pool endpoint: both make observability something hand-written per noun
instead of the uniform mechanism every component already has.
- engram_pool_stats_json(): the same state readable live, wired through the
normal builtin path (codegen arity + el_seed wrapper), so the pool can be
observed in real time rather than reconstructed afterward from a stack
sample.
Verified: with the exact configuration that took production down
(ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots
clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete —
and NO pressure event fires, because the barrier removed the walk that caused
it. The controller is defense in depth; the barrier is the fix.
Checkpointing pushes the ENTIRE resident graph through store_put_node and
store_put_edge (see engram_store_checkpoint). Nodes were cheap: a durable-hash
compare skipped unchanged records with zero page I/O. Edges had no barrier at
all — struct comment at PgCache.barrier_on even says "node durable-hash
barrier" — so every edge was rewritten on every checkpoint, and each rewrite
runs the idempotency probe max_page_lsn_for_id -> btree lookup -> page_read.
Edges outnumber nodes ~3:1 here (37,663 vs 13,436), so routine checkpointing
degenerated into a FULL-STORE WALK in id order: random page access across the
whole 2 GiB store, repeated, overwhelmingly to rediscover nothing had changed.
LRU is worst-case under exactly that pattern — it evicts the page it is about
to want — so once the page cache was smaller than the store, the walk collapsed
into thrashing: 100% CPU, flat RSS, no forward progress, port never bound.
That took the live engram down twice on 2026-08-15.
The walk is the defect. Sizing the cache to survive it treats the symptom.
Changes:
- dh_edge_hash(): edge counterpart of dh_node_hash, with a kind discriminator
byte so an edge can never collide with a node of the same id in the shared
map. created_at/updated_at/last_fired are excluded deliberately: last_fired
is touched by activation without changing what the edge IS, and folding it
in would defeat the barrier on precisely the hot edges that most need it.
- store_put_edge(): barrier check + dh_set on success, mirroring
store_put_node exactly.
- store_scan_edges(): seed the barrier map from on-disk truth at load, so the
FIRST post-boot checkpoint already skips unchanged edges. store_scan_nodes
already did this and its comment says why; edges were simply never done.
Verified: with the exact configuration that killed production
(ENGRAM_POOL_FRAMES=65536 -> 1 GiB cache against a 2 GiB store), the engram now
boots clean and serves — LISTENING, 13,436 nodes / 37,663 edges, embeddings
complete, 0.0% CPU, RSS 1.14 GiB (cache resting at its budget rather than
thrashing against it). Same small cache, same store, no walk.
/api/graph/edges answered a read query by calling engram_save() to serialize
the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in
one line, and both bit production on 2026-08-15:
1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram
server's CANONICAL store. A READ route overwriting the persistence
owner's canonical file. This defect had been fixed once (export moved to
a scratch path); it came back when the hand-written dispatch block was
replaced by @route dispatch and the unfixed copy is the one that
survived the merge.
2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole
graph, per request, to return a bounded slice.
Calling it tonight overwrote the canonical snapshot and immediately preceded
an engram crash loop.
engram_edges_json(limit, offset) is the builtin that route's own TODO asked
for ("Future: add an engram_edges_json() builtin and drop the file round trip
entirely"). It walks g->edges directly and emits every persisted field.
limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell
over, and an unbounded default would preserve the failure mode under a new
name. Callers page explicitly.
Registered in codegen.el's builtin_arity (both plain and __ spellings) and
wrapped in el_seed.c per the project's C-builtin recipe.
neuron's soul calls engram_recall_json (neuron-api.el:618, memory.el:80) and
cgi_principal (studio.el:72). Both existed in the runtime neuron vendored
(v1.0.0-20260501) and were absent here, so the soul could not link against
current el at all.
The dangerous part is what the obvious "fix" would have done. These look like
redundant wrappers over one impl:
engram_search_json(q, limit) -> eg_search_json_impl(q, limit, 0) LEXICAL
engram_recall_json(q, limit) -> eg_search_json_impl(q, limit, 1) SEMANTIC
They are not interchangeable, and the split is documented at neuron-api.el:613:
search stays LEXICAL because ~40 internal call sites pass a KEY and seven of
them DELETE every record returned. Point those at a semantic matcher and they
delete fuzzy matches. Conversely, pointing recall at search silently downgrades
the mind's entire retrieval surface from semantic to lexical — no error, just
permanently worse recall.
Implemented over engram_activate(), which in this runtime already IS the
semantic path the old with_legs=1 branch built by hand (embeds the query via
eg_embed_fetch, scores by cosine, then spreads activation one hop). Output
shape matches engram_search_json — a flat array via engram_emit_node_json —
because callers parse search's shape, not activate's envelope.
Verified: neuron's soul now compiles and links against current el, boots, and
serves /health with layers initialized.
NOTE for follow-up: current el also ships engram_retrieve_geometric_json, a
structure-first retrieval that appears to be the intended successor to recall.
Repointing the two recall call sites at it may well be the right end state and
would remove the two-wrapper shape entirely — but that is a behavioral change
that must be measured against neuron/tools/retrieval-eval/'s gold set, not
assumed. This commit preserves existing behavior exactly; it does not decide
that question.
The committed elc binary could not be refreshed from its own source. Rebuilding
failed with three implicit-declaration errors: el_mem_check, stdout_to_file,
stdout_restore. The compiler's own source calls all three (compiler.el:472,479,574
and codegen.el:4248) and two are registered in codegen.el's builtin_arity table —
but none were defined in this runtime.
They were found intact in ui/examples/native-hello-ios/NativeHello/el_runtime.c,
a divergent private copy of this runtime that still carried them. Ported verbatim.
Consequence of them being missing: the canonical elc binary was frozen. Source
gained @route dispatch codegen (emit_route_dispatch, codegen.el:3948) and the
@manager boundary-beat seam, but no rebuilt binary could carry them, so
neuron's soul — whose routes.el now calls the compiler-synthesized
el_route_dispatch — could not be built at all.
Verified after the fix:
- elc rebuilds from current source, clean.
- Self-hosting fixpoint byte-identical (stage3 == stage2).
- The rebuilt elc emits el_route_dispatch (2 occurrences in the soul amalgam,
previously 0) and injects engram_boundary_beat at @manager boundaries,
i.e. the decorator seam is live rather than inert.
el_mem_check is itself the compiler's memory guard (ELC_MAX_MEM_MB, default
512MB, self-terminates before the OS OOM-killer fires) — so the runtime was
missing the very guard that would have surfaced the compiler's memory blowup
as a clean error instead of a 27GB host-killer.
Three real bugs, all found by actually running the thing rather than reading it.
1. query_param never URL-decoded. A GET of /api/search?q=neural%20network
searched for the literal string "neural%20network" and returned []. Every
multi-word search against the live engram has been silently returning empty
results — not an error, an empty result, which is why it went unnoticed.
Affects every GET route that reads query params, not just search.
2. query_param matched key names unanchored. str_index_of(qs, "q=") matches
inside "faq=", so "?faq=X&q=Y" returned X for key "q". Verified live before
the fix. Now searches for "&key=" against "&"+querystring so a match can
only land on a real parameter boundary.
3. el_request_start/el_request_end were defined in BOTH el_seed.c and
el_runtime.c, so linking the two objects together — which is exactly what
the product build does — failed with duplicate symbols. el_seed.c's own
comment already says these moved there ("formerly defined in el_runtime.c.
Now self-contained in el_seed.c"); the el_runtime.c copies were left behind
during that move. Removed them, kept declarations since http_worker calls
them. Also added the three missing prototypes (engram_op_assert_json,
engram_node_full_in, engram_connect_in) that el_seed.c wraps but never
declared, which made it fail to compile standalone under C99+.
Verified: engram builds and links clean from canonical source; before/after
comparison on a copy of the real store shows "neural network" returning a real
match where the live build returns [], and "?faq=WRONG&q=MetaColloc" now
resolving to MetaColloc. Live engram on :8742 was never touched.
#116 shipped the ggml strategy at 0.9933 id-recall against the CPU oracle
while the hand-rolled Metal kernel it replaced scored 0.9997 — a ~150x worse
error margin. That was not an inherent property of ggml. It was a usage bug in
this file, and this commit fixes it.
ggml-metal has two F32xF32 matmul kernels and picks between them purely on
ne11, the number of B rows, which for us is the query-batch size:
ne11 <= 8 -> kernel_mul_mv_ext_f32_f32_* / kernel_mul_mv_f32_f32_*,
templated <float, float> — genuine F32.
ne11 > 8 -> kernel_mul_mm_f32_f32, templated
<half, half4x4, simdgroup_half8x8, half, half2x4, ...> —
BOTH operands narrowed to F16, despite F32 tensors on both
sides.
The old code issued one ggml_mul_mat with ne11 = nq (300 in the benchmark),
landing squarely on the F16 path. The file's own header comment asserted the
opposite ("computes in F32 on the Metal backend"); that claim was wrong and is
replaced with the measurement.
Fix: emit ceil(nq/8) mul_mats over ne11<=8 ggml_view_2d slices of one query
tensor, all expanded into ONE graph and one ggml_backend_graph_compute, so the
node matrix is still uploaded and shared exactly once. EL_GGML_MULMAT_CHUNK
overrides the 8; setting it >= nq reproduces the old behaviour exactly, which
is also how the before/after below was measured in a single binary.
Measured, real store snapshot, 13415 live embedded nodes, dim=768, 300 real
queries, vs the CPU double-accumulated oracle (vindex_bench, offline copy of
the store — no live service touched):
id-recall same-rank |Δdist| max mean
old (ne11=300) 0.9933 6.80e-05 1.43e-05
new (ne11<=8) 0.9987 4.77e-07 9.30e-08
hand-rolled 0.9997 3.58e-07 7.55e-08
~145x better max error, ~154x better mean — now the same order of magnitude as
the hand-rolled kernel rather than 150x off it.
The cost is real and is documented rather than buried. Median of 15 reps of
the whole batch_multi() call, three runs: 13.2-14.4ms unchunked, 19.9-20.2ms
chunked, 17.7-18.0ms hand-rolled. Correctness costs ~+6.7ms per 300-query
batch and leaves ggml ~12% behind the hand-rolled kernel instead of ~35%
ahead. It cannot be recovered inside ggml: an fp32 matmul on Metal must
re-stream the node matrix once per <=8 queries, and ggml's Metal backend ships
no fp32 TILED matmul, so "fast" and "fp32" are genuinely exclusive there.
Two things that did NOT work, recorded so nobody retries them:
- ggml_mul_mat_set_prec(t, GGML_PREC_F32) does nothing here. Error was
bit-identical with and without it (1.038e-05 either way) — ggml-metal has
no F32-accumulating mul_mm kernel to switch to. ne11 is the only lever.
- The ACCEL/BLAS device looked excellent in an isolated compute-only probe
(3.4-4.0ms, mean |Δdot| 1.5e-08) but is dominated on BOTH axes end-to-end
(0.191 ms/query at 0.9973 recall vs 0.125-0.142 at 0.9987), because the
probe was not competing for the same CPU cores the real call path is. It
stays reachable via EL_GGML_DEVICE as a no-Metal fallback, labelled as
measured-and-rejected, not as a recommendation.
Also corrected: the ~7.8s "cold start" blamed on this file is not this file
re-initialising per call — init was already cached. It is Apple's shader cache
missing on ggml's embedded metallib (~650 kernels), keyed on the library and
shared across processes: the first load on a machine reports
"loaded in 7.670 sec", the next run of a *different* binary reports 0.009 sec.
Once per machine per ggml version, not once per process, and not ours to fix.
Warm ggml init is 44-53ms vs 36-117ms for the hand-rolled strategy.
Loading only libggml-metal.so instead of every plugin in the directory is kept
for tidiness, and explicitly documented as NOT a speedup: 44.7-52.4ms against
46.9-58.9ms, the same number inside noise.
The -2.0 sentinel contract is unchanged and re-verified at batch sizes that
straddle the chunk boundary (1,7,8,9,16,17,33), plus NULL rows, dim
mismatches, zero-norm rows, and an all-invalid population. Notably the old
ne11=300 path fails that same check at a 2e-6 cosine tolerance with 2299
mismatches, which is an independent confirmation of the defect.
runtime/channel.el has always called __channel_new/__channel_send/
__channel_recv/__channel_try_recv/__channel_close, but these were only ever
implemented in the pre-restructure lang/el-compiler/runtime/el_runtime.c.
When the canonical runtime was consolidated onto the release copy
(lang/runtime/el_runtime.c) and el_seed.c became the sole C dependency,
the channel implementation was never carried forward — __mutex_new made the
move, __channel_* did not. Any El program using Go-style channels currently
fails to link on dev.
Ported the working buffered-MPMC-channel implementation (mutex+condvar+
circular buffer, bounded and unbounded modes) from the old el_runtime.c
verbatim, adapted only to el_seed.c's arena API (seed_arena_track in place
of el_arena_track). Declared in el_seed.h alongside the existing mutex
primitives.
transduce() is now THE single mechanism: one function, no content-type
branch inside it. It never asks whether `source` is prose, JSON, or
raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally:
split on "\n\n" as a universal boundary-marker check, and if that finds
no boundary, fall back to fixed 4096-char windows. Same node/edge wiring
(root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets
a heading/section_of link) regardless of what's inside a chunk. Dedup is
the existing find_existing_by_content path via merge_manifold, applied
uniformly. The old transduce_structured JSON dataset/records/feature-node
interpretation is deleted outright, not just unused — a JSON file now
gets chunked and deduped like anything else, with no pre-computed
structure. All five ingest_* entry points still exist unchanged in name
and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one
transduce() (ingest_stream builds its own turn-nodes directly and never
called either old function, so it's untouched).
This unlocks raw/opaque content (audio, or anything else with no natural
text/JSON shape) without any DSP, LLM call, or external API: transduce()
chunks it exactly like it chunks anything else. There is zero semantic
understanding of audio (or any payload) claimed or built here — any
meaning is expected to emerge later from Neuron's own existing mechanisms
(embedding, spreading activation, dedup) acting on this real geometry
over time.
Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk)
because El strings are NUL-unsafe under strlen-based ops and fs_read()'s
result silently truncates at the first embedded NUL, which is routine in
real binary/audio bytes. ingest_file compares fs_read()'s string length
against a real fs_size() stat() count; on mismatch it rebuilds the
payload as base64-encoded fixed 3072-byte windows read directly off disk
(binary-safe in C, verbatim, no invention), joined with the same "\n\n"
marker transduce()'s boundary scan already looks for. This is a
mechanical fidelity fix, not interpretation of content — transduce()
never learns a fallback happened. Registered both builtins' arity in
codegen.el; did not rebuild the elc compiler binary itself (unrelated,
pre-existing gap: self-hosting elc via el_seed.c fails on this worktree
independent of this change, reproduced with codegen.el reverted) — the
existing elc binary compiles calls to unregistered builtins via its
already-existing arity=-1 passthrough, confirmed by an actual clean
`elc ingest.el` + `cc` build against the modified el_runtime.c.
INGEST_KIND keeps existing only as an acquisition-mechanism selector
(dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a
content-type flag; the redundant "structured" value (an alias for "file"
that hinted the now-deleted JSON branch) is removed. ingest_dir drops its
file-extension filter for the same reason: transduce() takes anything now.
Verification: local manifold construction confirmed correct against a
real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte
real prefix slice) — exact expected node/edge counts both times
(101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching
ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64
content confirmed decoding back to the actual WAV header bytes. Compiles
clean via the real elc + the modified el_runtime.c/engram_*.c (built and
booted an actual sandbox engram off this exact source with `nsbx create
--branch`).
NOT verified this session, disclosed rather than papered over: end-to-end
server-confirmed persistence (a real before/after /api/stats delta, and a
fetched node by id) for the audio, prose, and JSON-fixture cases. Every
local nsbx sandbox engram tried tonight (two stock pre-#109 binaries
hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW
binary built from current dev) took minutes-to indefinitely long on the
final /api/load-merge write's embedding step and hit the client's 60s
HTTP timeout before responding, even for a 5-node write. This is
confirmed as real (if slow) forward progress, not a hang: the sandbox's
WAL file was observed growing steadily across every attempt. The code's
own pre-existing HONESTY GATE correctly refused to report success in
every case, returning "load-merge failed: ..." with a
"nothing below this manifold was confirmed persisted by the server" note
instead — exactly as designed. This is an environment/infrastructure
limitation, not a defect introduced by this change: the engram server
binary itself is untouched by this commit.
Reconciles PR #105 ("fix: engram search latency — pin embed model, cache
query embeddings, bound activate BFS") with dev's ACTUAL current
engram_activate, rather than the ancient pre-restructure snapshot #105 was
built against.
WHY THIS NEEDED RECONCILIATION, NOT A DIRECT PORT: #105's single commit
(1dc49b1) modifies `lang/el-compiler/runtime/el_runtime.c` — a path that does
not exist on dev (dev has `lang/runtime/el_runtime.c`; the restructure that
renamed it happened after #105's branch point, which traces to a July 22
merge-base, weeks before the M8/M8.1/qgate/fan-effect/adjacency-index work
this file has grown since). #105's own engram_activate is consequently the
PRE-restructure version: no adjacency index (O(E) full edge scan per hop),
no query-aware qgate, no ACT-R fan effect, no eg_edge_eff_weight, and no
awareness of dev's cosq/e_eff embedding-blend semantic layer — it built a
parallel `g_qcache`/`engram_embed_raw` mechanism from scratch against code
that no longer exists at that path. A raw merge/cherry-pick was not possible
and would have been wrong even if it were: taking #105's tree wholesale would
have thrown away everything dev grew in the meantime (qgate, fan effect,
adjacency index, and this session's own M8 HNSW vindex integration).
RECONCILIATION: kept dev's cosq/e_eff mechanism as the semantic layer
entirely intact (unchanged by this commit) and ported #105's three genuinely
additive wins on TOP of it, at their equivalent sites in the CURRENT
eg_embed_fetch/engram_activate:
1. keep_alive:-1 on the Ollama embed request body (eg_embed_fetch) — pins
the embed model resident so a larger generation model loading under
unified-memory pressure can't evict it and force a cold reload on the
next search (#105 measured ~2.2s cold vs ~0.02-0.05s warm).
2. Query-embedding cache upgraded from dev's single-slot (`_eg_qcache_text`,
only ever remembered the LAST query) to a direct-mapped, FNV-1a-keyed,
1024-slot cache (reusing the existing engram_id_hash) — so the
curiosity loop's rotating phrases actually hit the cache instead of
evicting each other every call. Same "pointer owned by the cache, not
freed by caller" contract as before, just per-slot instead of global.
3. Beam cap on the layer-1 spreading-activation BFS (new
engram_activate_beam(), tunable via ENGRAM_ACTIVATE_BEAM, default 128).
The FIFO frontier is processed in hop-level batches (entries sharing
.hops are provably contiguous — see the code comment); when a level
exceeds the beam width, only the top-`beam` by activation actually
EXPAND. Every node in an oversized level still gets reached[]/best_bg[]
recorded (that happens at enqueue time, one level up) and appears in
the reported/promoted set — the cap bounds associative SPREAD width
only, never recall of what was already found. Kept as a genuine
additional bound even though the adjacency index + qgate + fan effect
already mitigate #105's original "hub-node explosion" failure mode for
a different reason: those prune WHICH targets matter; this bounds
worst-case width regardless.
Everything else in dev's engram_activate — cosq/e_eff, the qgate rescale,
the fan effect, eg_edge_eff_weight, the M8 HNSW vindex seed discovery from
the #109 reconciliation earlier this session — is untouched.
VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): cc -std=c11
-O2 clean build; booted in an isolated sandbox against a real cloned
production snapshot (13,424 nodes / 37,656 edges); ran 5 activate() calls
across rotating queries at depth 3, including the same query issued twice
non-consecutively (2nd hit landed at 476ms vs the 1st at 483ms — consistent
with a cache hit once Ollama's own warm-model latency is accounted for; no
crash, correct varied result counts (367-2610 nodes) each call; act-stats
JSON read correctly throughout.
Built on top of the M8/#109 reconciliation (bacaf3d, merged to dev as
#109) — dev's current HEAD at the time of this commit.
runtime/el_seed.c does not compile standalone via the exact command
tools/install.sh uses (`cc -std=c11 -O2 -I runtime -c runtime/el_seed.c`):
51 of its __-prefixed wrapper functions (http serving, JSON access, key-val
state, URL/HTML escaping, and the whole engram_* node/edge/layer/search
surface) call unprefixed counterparts that are implemented in el_runtime.c,
not in el_seed.c itself, and el_seed.c never declared them -- a toolchain
that treats an implicit function declaration as a hard error under C11
fails the compile outright.
install.sh already compiles el_seed.c and el_runtime.c as separate objects
and archives both into libel.a, so the symbols are always present at link
time; el_seed.c alone was just missing the prototypes.
A plain `#include "el_runtime.h"` was tried first and rejected: it redefines
el_to_float/el_from_float, which el_seed.h already provides -- a real
compile error, not a style preference. Added narrow prototypes instead,
copied verbatim from el_runtime.h, for exactly the 51 symbols el_seed.c's
wrappers reference and nothing else.
Verified clean:
- `cc -std=c11 -O2 -I runtime -c runtime/el_seed.c` (install.sh's exact
per-file compile) -- 0 errors, 0 warnings, even with -ferror-limit=0.
- full `tools/install.sh` run -- compiles both objects and archives them
into libel.a successfully.
Separately (not fixed here, out of scope): AGENTS.md's documented compiler
self-rebuild command links elc-new.c against el_seed.c, but elc-new.c's own
generated `#include "el_runtime.h"` line and 3 undeclared symbols
(el_mem_check, stdout_to_file, stdout_restore -- present in neither
el_runtime.c nor el_seed.c) mean that command fails regardless of which
runtime file it's linked against; and install.sh's libel.a only archives
el_seed.o + el_runtime.o, so any program that calls into the engram_*
surface fails to link against it (el_runtime.c's engram_* wrappers need
engram_store.c/engram_geometry.c/engram_reason.c/engram_cognition.c/
engram_vindex.c, none of which install.sh compiles in). Both are real,
pre-existing, and independent of this fix -- worth their own look.
Stop hand-rolling GPU kernels for batch cosine similarity — use ggml (the
MIT-licensed compute library underneath llama.cpp, installed standalone via
Homebrew) as the preferred backend, without ripping out PR #114's
carefully-verified hand-rolled Metal shader.
Structure: one stable public adapter (eg_cosine_batch.h, zero #ifdef at call
sites) backed by three selectable concrete Strategies behind an internal
vtable (eg_cosine_batch_strategy.h) chosen by a Factory (eg_cosine_batch.c):
- eg_cosine_batch_strategy_ggml.c — NEW. ggml + dynamically-loaded Metal
backend plugin (ggml_backend_load_all_from_path
+ ggml_mul_mat for the batched dot
product), gather/scatter around the
-2.0 sentinel contract.
- eg_cosine_batch_strategy_metal_hand.m — PR #114's original hand-rolled
Metal shader bridge, preserved
almost verbatim, now one strategy
among several rather than the only
option. eg_cosine_batch.metal kept
byte-identical to the original.
- eg_cosine_batch_strategy_cpu.c — universal always-false fallback
(direct descendant of PR #114's
eg_metal_cosine_stub.c).
Selection: EL_COSINE_BATCH_STRATEGY=ggml|metal|cpu|auto (default: ggml first,
then hand-rolled Metal, then CPU — first available wins), plus back-compat
EL_METAL_COSINE=0 to disable every GPU-backed strategy. build_vindex_bench.sh
compiles all three strategies on Darwin, CPU-fallback-only elsewhere.
vindex_bench.c now reports BRUTE-GGML and BRUTE-METAL side by side against
the same CPU oracle, on the same dataset, in one run (real numbers vs. real
store snapshot in the PR body).
Lands feat/reframe-region-setop (PR #109: native set-based reframe_region,
decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency
work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated
seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's
actual current HEAD, plus engram-tiered-storage's still-unique test suite.
RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/
verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/
select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two
branches; #109's copy is a strict superset (adds vindex_harvest_from_store,
used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and
engram_verify.c are also byte-identical. #109's own branch point already
carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing
left to merge into #109 for those files. The one thing engram-tiered-storage
had that #109's tree dropped: its full test suite (test_vindex.c,
test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the
interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh
harnesses) — ported over here unchanged.
WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch
forked from dev on 2026-08-14 15:40 (before restructure-adjacent history
diverged the file's merge-base for `git merge` — it presented as an add/add
conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but
it silently reverted THREE dev fixes landed on 2026-08-14/15, after the
branch point, that the PR's diff had no way to know about:
1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of
the query-aware propagation gate dropped the shift-and-floor rescale
about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate
0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped
around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi]
is computed) is orthogonal to WHAT it gates on and both are kept.
2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes
wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is
diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global).
PR's tree predates this and dropped all three counters + their JSON
stats fields. Restored declarations, all 4 direct increment sites, the
eg_wm_carry_over bll increment, and the act-stats JSON fields --
alongside (not instead of) the PR's own P4 afferent / API-reshape
counters already in that same struct/JSON.
3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev
selects the STRONGEST qualifying candidate for consolidation each call;
PR's tree predates this and reverted to hash-slot order (arbitrary wrt
association strength) for edge formation -- the one path that writes
PERMANENT structure. Restored the strongest-candidate while-loop,
keeping the PR's own genuine improvement at that site
(engram_adj_on_edge_added incremental-index append instead of a bare
adj_dirty=1 full-rebuild flag).
engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env
flags, /api/nodes connected-count in responses) were pure additive: dev's
side was empty, PR's side added the feature. Took PR's side whole.
VERIFIED (nsbx sandbox only, live :8742/:7770 never touched):
- cc -std=c11 -O2, clean link against the real engram/src/server.el via
elc, zero errors.
- vindex_bench (built standalone, read-only harvest) against the real
production store clone (13,671 embedded nodes, 768-dim nomic-embed-text):
recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs
2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s
for the full 13,671-node set -- see the flagged risk below.
- Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned
snapshot of the live store, 13,424 nodes / 37,656 edges) and called
/api/activate for real: first call after boot 41.5s (pays the one-time
HNSW build inline -- matches the standalone bench), second/third calls
356ms/605ms, no crash, correct results, act-stats JSON (including the
restored evict_floor/cap/bll fields) reads correctly.
KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for
this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync
builds the HNSW index synchronously, inline, on the first engram_activate()
call after every process start (or index invalidation). On the real node
count that is a ~46s blocking stall on a single-threaded server -- the first
request after every restart (or its concurrent siblings) waits the full
build. Recommend a background/incremental build (or a bounded per-call build
budget) before this ever reaches the live daemon. See PR description / final
report for the fuller writeup.
Adds an O(1) "seen" bitmap so lazily-embedded older nodes get picked up
incrementally instead of only on a full rebuild (embed-gap #20).
Replaces engram_activate's O(N*D) cosine prescan with a lazy-memoized
cosine cache (eg_cosq_at), proven bit-identical to the old path.
Extracts a clean vindex_harvest_from_store primitive (read-only vector
harvest, careful malloc/ownership/error-path handling) reused by both
index-build and the new vindex_bench.c — a read-only proof harness
comparing brute-force vs HNSW recall/latency on both the real store and
synthetic data.
.nsbx-env intentionally excluded — local sandbox config (ports, paths,
dev-only placeholder key), not checked in.
Cancellation-token control channel checked at every step boundary lets a
coordinator PAUSE/RESUME/REDIRECT/KILL a running worker mid-task instead of
waiting for the whole (possibly wrong) plan to finish. Bounded purviews
mean no half-committed state to unwind on interrupt. Includes a proof
harness (proof.el, run.sh) comparing a broken non-interruptible worker
against the new one under identical kill/redirect/pause timing.
Distinct from the already-preserved swarm-ccr orchestrator (fan-out/
converge dispatch): this is single-worker interruptibility, a
complementary mechanism, not a duplicate.
Float + previously fell through to string concat (segfault); -, *, /, %
operated on raw IEEE-754 bit patterns as integers (garbage results). Floats
are now tracked via a __float_names typed-binding set (parallel to the
existing int-tracking scheme) and arithmetic is emitted as real C double
ops.
Also fixes math_log, which was wrongly aliased to natural log (duplicating
math_ln) — now uses log10 — and adds the missing <math.h> include. Rebuilt
elc binary included.
Adds engram_assert_json — a grounded "assertion envelope" primitive for a
realizer/op_assert seam (per backlog bl-53/#57) — plus purview-scoped
mutation wrappers engram_node_full_in/engram_connect_in, which refuse
non-default purviews rather than silently mutating the live store. Threads
through el_seed.c/h wrappers and the codegen.el arity table per the
project's existing C-builtin recipe.
Also rewrites lang/AGENTS.md build docs with verified (2026-08-15) findings
that el_seed.c does not compile standalone.
The link-formation scan walked candidate slots ascending and stopped at
ENGRAM_HEBB_LINK_PER_CALL (2). Slot index is a hash of the node id pair, so
whenever more than two candidates cleared LINK_MIN in the same call, the two
consolidated were the two with the lowest hash and a stronger association
waited - indefinitely, since the scan restarts from slot 0 every call while
the leader decays at ENGRAM_HEBB_DECAY.
Measured 08-13..08-15: hebb_cand_max peaked at 0.4963, 3.3x LINK_MIN, during
a ~14h stretch of continuous qualification at the 2/call cap.
Same defect the 2026-08-02 review named and fixed for breakthrough weights
(index order is not a cognitive criterion), never carried across to the one
path that writes permanent structure - and there is no pruning path, so
growth is one-way. Selection pressure matters most where the result is
irreversible.
No-op when <=2 candidates qualify; picks the best when more do.
New hard invariant (Will): only the orchestrator mutates global engram state;
workers are read-only against the full engram + write only their own local
geometry. This is an AUTHORITY gate (capability), not a health gate — a worker
is STRUCTURALLY UNABLE to mutate global engram state regardless of engram health.
- containment.el: scope tokens now carry a caps set. Orchestrator token holds
engram:write + dharma:emit (@manager-only, the VBD rule that only the manager
mutates global state); worker token holds ONLY engram:read. Rule 4:
containment_check_engram_write / _dharma_emit reject any caller lacking the
capability — same scope-token mechanism as the live Rule-2 denial.
- swarm.el: swarm_engram_write is the ONLY engram write path, gated by Rule 4;
a worker token is denied before any HTTP is issued (no mutation). The curated
merge (commit=1) is the sole writer: the orchestrator commits approved
geometry via its write-capable token. Workers' full-engram READ stays intact.
- reshape_surface.el: compose op_write (json_escape_string) for the commit path.
- harness: Rule-4 suite proven — worker engram-write DENIED by capability, no
node created, violation journalled; orchestrator passes the gate as sole
writer. 24/24 green on the :8901 clone with real cognition.
Authority gate holds independent of daemon write-health (proven with daemon
both alive and, earlier, crashed). Prod :8742 untouched.
Will waived diff review -> build it for real. Add engram_boundary_beat() to the
runtime (afferent counter++ + engram_chrono_tick + engram_strengthen(self-anchor)
+ dharma_emit) and two act-stats counters (aff_boundary_ops, dharma_emits).
codegen cg_fn injects ONE engram_boundary_beat(op) at the entry of every
@manager/@accessor fn (fn_has_decorator, so it fires under @route @manager too) —
a decorated op self-reports with ZERO hand-written instrumentation. Rebuilt elc
self-host + the cognition engram in the worktree; ran it as the clone daemon on
:8900. Proof (/api/boundary-proof, @manager, empty body, 5x): aff_boundary_ops
0->5, dharma_emits 0->5, self activation_count 1510->1513, chrono stamp advanced.
Brought in feat/cognitive-architecture engram runtime+server for the build.
strengthen = activation bump (not content/edge write) -> identity protection
intact. Live :8742 untouched; no push, no cutover.
Binds the api-reshape surface at wt/api-reshape@d4f401d (op_think/read/attend/
learn, verified against engram.cognition-20260814) into the swarm:
- reshape_surface.el composes the reshape's proven read/cognition primitives
verbatim (write ops omitted — they need the gate-1 write-healthy clone).
- primitive_binding.el: bound_think -> op_think over the worker's NODE-ID
anchor (ctx.input); attend/learn bound behind SWARM_WRITE_HEALTHY.
- cognize blueprint derives the vote verdict from the REAL gradient's n_support
(json_get_int) — per-anchor diversity (6/16/87 support) drives a genuine vote.
- build.sh now defines HAVE_CURL. CRITICAL FIX: without it every http_* was a
'{"error":"not built with HAVE_CURL"}' stub, so prior 'live engram'
retrieval was a false positive (matched the ref string, not real content).
With HAVE_CURL the swarm genuinely hits /api/think on the :8901 clone.
harness_real_cognition.el: 17/17 GREEN with seam=decorated — 8 native-thread
workers each a REAL think (768-dim gradient) over its CCR-scoped node-id anchor,
@manager reduce+vote convergence, all 3 containment rules incl. live Rule-2
denial, afferent telemetry (8 real think signals), durable work-tracking. Reads
only — daemon stays healthy; writes stay gated on the gate-1 clone. Prod :8742
untouched.
Ground-truth the three seams (route/telemetry+interoception/bus) with file:line
evidence. Port the tested @route codegen+parser from feat/el-route-decorators
into the worktree elc (decoration synthesizes el_route_dispatch — no hand-written
90-branch handle_request). Rebuild elc self-host; prove decorate->serve end-to-end
(route_proof.el on :8951). Rewrite surface.el as El-native decorated components:
@route + @accessor/@manager, in-process engram_* builtins (not http_get), @manager
ops emit on the real dharma_* bus (same transport as wt/swarm-ccr). Identity
keystones refused in write/relate/supersede. Gate-1 clone recipe (WAL-aside
cold-boot + ENGRAM_WAL=on) proves the FULL op set live on the clone. Boundary
auto-emit (telemetry/interoception/bus) staged as a reviewable cg_fn diff
(SEAM_STAGED.md) — needs the cognition-engram rebuild to verify link. Live :8742
untouched; no push, no cutover.
- primitive_seam.el: SWARM_PRIMITIVE_SEAM selects stub (default, hermetic) vs
decorated (reshape's dharma-bus primitives). Every seam call is an afferent
signal; telemetry (seam_mode + afferent tick) rides the vertical result path.
- primitive_binding.el: THE ONE FLIP POINT — bound_think/attend/learn today fall
back to the stub; when the reshape's decorated primitives land, flip one line
each and set SWARM_PRIMITIVE_SEAM=decorated. No other change anywhere.
- swarm.el: default blueprint routes think through the seam; the @manager
aggregates afferent counters from worker results (containment-safe, no shared
bus register) and journals a swarm.telemetry record; telemetry in the return.
- harness_local_swarm.el: 17/17 GREEN on :8901 with the stub — 8 native-thread
workers at concurrency 4, reduce+vote convergence, CCR scoping+non-leak, all
three containment rules (incl. live Rule-2 denial), durable work-tracking,
afferent telemetry observed. Runs identically under seam=decorated today
(binding fallback), proving the flip path executes.
Engram writes stay opt-in (durable journal is the substrate); daemon healthy.
- primitive_attend retrieves over HTTP (POST /api/search) when ENGRAM_URL is
set — the location-independent worker model — falling back to the in-process
store otherwise. Proven against the isolated :8901 clone: CCR compiled a
bounded context from REAL mind content (VBD/intellectual-dna).
- gate the engram work-tracking mirror behind SWARM_MIRROR=1; the durable
substrate is always the JSONL journal, so a swarm never depends on the mind
to track its work. (Repeated POST /api/nodes mirror writes were observed to
crash the isolated daemon — a daemon-side write-path robustness issue;
retrieval POST /api/search is solid. Prod :8742 never touched.)
- integ_engram: CCR real-retrieval + full swarm completion against live clone.
- vote/merge/reduce/collect convergence proven end-to-end; failure threshold
aborts a swarm below min_success_ratio (integer per-mille) and completes
when failures are within tolerance, with worker.failed + swarm.aborted
tracked durably.
- worked around three El runtime/codegen semantics surfaced during the build:
json_set inserts RAW (use json_set_str for string values); json_set cannot
update an existing key (vote tallies via list rescanning); json_array_get
keeps quotes (use json_array_get_string). Also: float division is unreliable
(swarm uses integer math), and a let-rebind in a deeply nested if/else does
not propagate outward (accumulators kept at one block level).
test_convergence: 8/8; test_swarm: 12/12.
bigmerge
will.anderson