engram_ground_json resolved each seed to a REGION, wrote the grounded-by
edge between the two regions' HUBS, and then echoed those hubs back in the
"claim"/"evidence" fields as if they were the caller's input:
const char* cid = C->hub_id ? C->hub_id : EL_CSTR(claim);
const char* eid = E->hub_id ? E->hub_id : EL_CSTR(evidence);
cog_ground_edge(g_engram_store, cid, eid, grounding, fw);
Three consequences, all measured against a clone of the live store:
1. The edge landed on a node the caller never named. Grounding 3b9ced5d
against 6edf8c79 wrote an edge on the hubs of their regions instead.
2. When both seeds resolve into the same region the support is circular
and scores near 1.0 for structural reasons, not evidential ones. Four
probe nodes written together landed in one region, and every grounding
among them returned 0.93-0.99 as if it were evidence. Two independent
agents hit this and reported 0.885 / 0.909 self-groundings as confident.
3. The echo concealed both: the response was indistinguishable from a
successful grounding of the ids that were passed in.
The region is HOW a claim is evaluated; it is not WHAT the claim is about.
So the edge now attaches to the requested ids, and the resolved hubs are
reported separately as claim_region / evidence_region.
Degeneracy is broader than hub == hub. Three circular shapes, all
previously invisible:
same-region both seeds resolve to one region
claim-region-is-evidence the evidence IS the hub of the claim's own
neighbourhood — measured at 0.98883
evidence-region-is-claim the mirror case
Each sets grounding to 0 and writes no edge. Circular support is not
support, and a grounding that is degenerate by construction must not
enter the graph as though it were evidence.
Verified:
6edf8c79 -> 6edf8c79 degenerate=same-region g=0 written=false
6edf8c79 -> d0406dfd degenerate=same-region g=0 written=false
ebc1413e -> 64cc96ef degenerate=false g=0.774563 written=true
64cc96ef -> ebc1413e degenerate=false g=0.802896 written=true
Legitimate grounding across distinct regions is unchanged and still
writes; only circular support is refused.
This is the same class as #142 and #146 — a value that looked like an
answer with nothing behind it — except here it was also writing that
non-answer into the canonical store.
engram_think_json built a NEUTRAL stance on every call — cog_stance_init
with a NULL id, all axis_gain 1.0, bias_dir NULL, reliability 0.5 — and
never loaded the stance the correspondence-beat had been persisting.
That mattered because the faculty enters engram_think ONLY through the
stance: axis_gain[k] warps the per-axis extents and bias_dir seeds the
steering direction. cog_stance_init stores the faculty NAME and nothing
reads it. So with a neutral stance, reason/abduce/induce/plan/analogize
were byte-identical output under different labels, and confidence was
pinned to 0.5 because GeoGradient.confidence IS stance->reliability.
The machinery already existed and only this call site ignored it.
engram_correspondence_beat_json resumes via cog_stance_from_node and
persists via cog_stance_to_node under "stance-<faculty>-<hub>". Every
beat's calibration was written and then thrown away on the next read.
Same defect as the NULL anchor fixed in #142, one line below: a neutral
argument collapsing a capability to a constant.
Resume the same id the beat writes, so learning compounds across beats and
cold boot. Fall back to neutral only when no stance exists — a genuine
uninformed prior rather than a discarded informed one.
Also emit stance_resumed, so confidence 0.5 from a learned-but-unreliable
stance is distinguishable from confidence 0.5 from "no stance exists".
That reporting gap is what let the neutral stance hide.
Verified against a clone of the production store (13,627 nodes):
before beat, no stance stance_resumed=false confidence=0.5
beat on a NON-keystone brier 0.00458568 -> 0.00329654
reduction 28.11%, n_trials 6000,
reliability 0.930726, stance_written=true
after beat stance_resumed=true confidence=0.930726
Confidence now equals the learned reliability instead of the uninformed
prior. The keystone self-anchor correctly stays at 0.5 — calibration is
deliberately refused on protected identity regions, and that refusal is
now visible as resumed=true with confidence unchanged, rather than being
indistinguishable from the bug.
STILL OPEN: with no learned bias_dir the faculties remain identical in
direction. What distinguishes abduce from induce geometrically is a
design decision about how Neuron thinks, not a plumbing defect, and is
deliberately left to Will.
#141 let signal enter as geometry and it worked, but it was placed at the
CONSUMER and said so in its own commit message. This is the correction.
Three defects, all of them placement:
1. It sat in the engram. Ingest is a LANGUAGE concern — every el program
touching any modality needs it, and the engram is merely one el program
that happens to hold a graph. The geometry surface is now defined in
el_runtime.c immediately ABOVE the engram section and depends on nothing
inside it. Delete the entire engram and geometry still enters el.
2. It marshalled the vector as a hex STRING, because el had no first-class
geometry value — which reintroduced text as the TRANSPORT medium one layer
below the problem being fixed. Geometry is now an el value: a magic-tagged
heap object carried in el_val_t, same discipline as List/Map. Hex survives
only as an adapter at the edge, which is all an encoding should ever be.
3. It needed an arbitrary `dim <= 8192` bound purely to size an allocation
from a caller's CLAIM about a string's length. A value carries its own
width, so the width is derived and never asserted. The bound is gone, not
raised — there is nothing left to validate.
Language surface, none of it engram-prefixed: geometry_new / _dim / _is /
_get / _set / _norm / _free, geometry_from_f32le_hex + geometry_to_f32le_hex
as the wire adapters, realizer_register(modality, fn_name), realizer_has, and
transduce(signal, modality) -> Geometry.
REALIZERS ARE DECLARABLE IN EL. This is the part that makes the move real
rather than nominal: registration resolves a name with dlsym against the
running binary, the identical mechanism http_set_handler already relies on,
because every el `fn name(...)` compiles to a global C symbol with that exact
name. So an ordinary el function IS a realizer and a new modality needs no
runtime patch. Verified end to end in lang/examples/transduce.el: an el-defined
tone_realizer is registered by name, transduce dispatches to it, and the
signal demonstrably reaches it (distinct signals produce distinct geometry).
A modality with no realizer transduces to NOTHING. There is deliberately no
built-in realizer, not even for text — silently embedding a description of a
signal and calling that perception is the exact defect this ends.
engram/src/server.el is migrated: POST /api/nodes decodes "emb" hex exactly
once, at the edge, into a Geometry, and everything below that line moves
geometry. The wire is unchanged because production clients speak it. "dim" is
now an ASSERTION about the vector, not the source of its width; disagreement
is a rejected ingest, not a silent reinterpretation.
#141's engram_node_set_emb becomes a DEPRECATED WRAPPER over
geometry_from_f32le_hex + node_attach_geometry — kept only because the runtime
ships as an SDK asset and a downstream binary may link the symbol. Its exact
contract, negative cases included, is preserved and re-verified.
ingest.el's `fn transduce` is renamed transduce_manifold. Mechanically it had
to yield the name (duplicate C symbol, a hard compile error, measured). But it
was never signal->geometry: it chunks already-extracted content into a node+edge
manifold, one layer up, and had taken the name belonging to the primitive
underneath it. Behaviour unchanged.
PROPERTIES FROM #141 PRESERVED, each re-measured on a scratch engram (:8971,
never prod :8742):
- off-dimension vectors stored but NOT indexed — the HNSW build loop still
filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is
durable and addressable without perturbing the 768-dim canonical index
- geometry makes a node ineligible for embed_backfill: after backfill the
64-dim voice node was still 64-dim while the text control acquired 768
- the create response reports whether geometry landed, and the node document
always emits emb_dim and embedded
Read-back with control and negatives, all verified against a PID-confirmed
fresh binary: geometry node emb_dim=64 embedded=true / emb_set=1; text-only
control emb_dim=0 embedded=false / emb_set=0; malformed hex, ragged length,
and dim-disagreement each emb_set=0.
Two compiler landmines found by reading the generated C rather than trusting a
successful build, both documented at their sites: elc lowers `a == b` to
str_eq unless both operand NAMES are in the per-function int-name set (which
does NOT propagate into nested if-expression blocks — the first cut would have
strcmp'd two integers as pointers on the first geometry-bearing request), and
`+` lowers to string concat when either operand is a user-defined call.
The crash (SIGTRAP in engram_activate -> eg_vindex_sync -> vindex_insert ->
_realloc) had three read paths mutating five process-global statics.
engram_activate, eg_knn_for_node (whose own comment says "No writes.") and
engram_geo_reify_run_json all called eg_vindex_sync, which frees the index,
reallocs the seen-map and inserts — on a read.
Three moves, in decreasing order of how much they dissolve:
1. Misfiled scratch is not shared state. visited/visit_epoch/visited_cap
were never owned by the index; they are one traversal's local, hoisted
into struct VIndex as an allocation optimisation. They want neither a
lock nor a capability nor a pool — just to go back in the call frame.
Two concurrent READS stomped each other purely because of this.
2. const IS the capability. Once the scratch leaves the struct, search
reads and nothing else, so vindex_search takes a const VIndex*. That is
exactly what a capability-pointer ABI would have bought — a read path
physically cannot call vindex_insert, enforced by the compiler on every
future caller — for one qualifier instead of an ABI swept across
hundreds of builtins.
3. What survives is publication, not ownership. HNSW insert is NOT an
append: it rewires the neighbour links of already-existing elements and
reallocs elems[], so the store's append-only property does not transfer
to the index derived from it. eg_vindex_sync therefore splits into
eg_vindex_maintain (exclusive, sole mutator) and eg_vindex_view (shared,
returns const VIndex*). A read path may demand that a current snapshot
exist — a request to the owner, not a mutation by the reader.
Write-side owner: eg_vindex_note_embedded hooks the embedding-ASSIGNMENT
sites rather than the append sites, because a node with no embedding cannot
be in a vector index — embedding assignment is the event that owns index
membership. One O(log n) insert, no O(node_count) presence scan. This also
retires the "STALENESS (honest tradeoff)" note where a lazily-embedded
older node stayed invisible to route_nearest/autoconnect until a full
rebuild (the embed-gap #20 shape).
Evidence. The existing harness conflated two hazards, which is why fixing
half of it read as failure. Split into four:
single (3000 vec, ASan+UBSan) clean -> clean
readers (4 readers, no writer, TSan) RACE -> clean
unsynchronized (writer+reader, bare) race -> race, expected forever
published (owner + 4 readers) n/a -> clean, 3000/3000 landed
RESULT: PASS. recall@10 = 0.9365 at ef_search=128 (gate >= 0.90);
determinism byte-identical across two independent builds.
The unsynchronized half is now permanently expected to race, deliberately:
it is the executable proof that the boundary must live above the data
structure, not inside it.
fb32d15's guard is KEPT, correcting this design's own section 5. Measured,
it guards TWO structures and only one was converted here: g->nodes/g->edges
are realloc'd in place (el_runtime.c:7618,7629) and engram_activate_inner's
embed-backfill writes n->emb through exactly such a borrowed pointer.
Deleting the guard reintroduces a measured 11171->9579 edge loss. Its
comment is narrowed to the RAM graph and the deletion precondition named.
That corrects the ordering claim too: the residual is not one ABI that
dissolves everything at once, it is a PROPERTY applied per structure.
Residues evaporate in the order the property is applied, and a residue
whose structure has not been converted must be left standing.
Promotes the two throwaway sanitizer harnesses used to diagnose the
2026-08-16 soul crash into engram/test/ so the bug cannot silently regress.
The harness has two halves and the PAIR is the point — it is what localises
the defect to concurrency rather than to HNSW logic:
single 3000 clustered vectors, one thread, ASan+UBSan. The CONTROL.
Must always be clean. During diagnosis this cleared all 13,820
real dim-768 vectors from the live store, which DISPROVED an
inspection-derived hypothesis about an out-of-bounds
reverse-link write at engram_vindex.c:340.
concurrent writer + reader on one shared index, TSan. Currently reports a
race at engram_vindex.c:195 (visited_reset) reached from both
vindex_search and vindex_insert, because VIndex still owns its
visited[]/visit_epoch scratch — so even two concurrent READS
corrupt each other's traversal.
Verified: half 1 passes, half 2 reproduces the race.
Gated on EXPECT_RACE, default 1, so the concurrent half documents the known
defect without failing the suite today. When the visited set moves to a
per-query checkout pool (hnswlib VisitedListPool style — NOT thread_local,
since http_worker is a thread per connection and a __thread buffer would leak
~55KB per connection), flip EXPECT_RACE=0 and it becomes a real gate.
The soul daemon had two engram callers and only one of them locked.
soul.el:729 starts the HTTP server via http_serve_async (spawning
http_worker threads); soul.el:731 then runs awareness_run() on the MAIN
thread. awareness.el's perceive() -> engram_activate_json() ->
engram_activate() -> eg_vindex_sync() -> vindex_insert() mutates the same
g->nodes/g->edges and the process-global _eg_vindex HNSW index that the
workers touch. g_engram_req_lock existed to serialize exactly this, but it
was only ever taken inside http_worker: engram_req_lock/engram_req_unlock
appear in ZERO .el sources, so the awareness loop ran lock-free beside the
workers on every tick (SOUL_TICK_MS=1000).
Result was a crash-loop under launchd KeepAlive: five crashes in ~4 minutes
on 2026-08-16 with varying faulting frames -- search_layer<-vindex_insert
<-eg_vindex_sync, engram_activate, abort, and one inside xzm_realloc's own
freelist. Varying sites plus a fault in allocator metadata means heap
corruption. The SIGSEGV address 0x65646f4e6d617267 is little-endian ASCII
"gramNode": string bytes dereferenced as an Elem vector pointer.
Diagnosed by bisection rather than inspection:
- Replaying all 13,820 real dim-768 vectors harvested from the live store
through the index single-threaded under ASan is 100% clean, which rules
out an HNSW logic/bounds bug.
- Two threads on one index trip ThreadSanitizer immediately at
engram_vindex.c:195 (visited_reset), reached from both vindex_search and
vindex_insert. VIndex keeps a SHARED visited-epoch scratch buffer, so
even two concurrent READS corrupt each other's traversal and walk bogus
element indices.
So this is purely a concurrency defect, not an HNSW logic error. (An
inspection-derived hypothesis about an out-of-bounds reverse-link write at
engram_vindex.c:340 was disproved by the single-threaded run.)
Fix: a thread-local ownership depth (_eg_req_depth) lets engram entry points
self-guard. engram_activate() becomes a wrapper over engram_activate_inner()
that acquires g_engram_req_lock when called with depth 0 (the awareness
thread) and passes through when depth > 0 (nested inside an http_worker that
already holds it), so the non-recursive mutex cannot self-deadlock. The depth
is a plain counter, never a recursive-mutex count, preserving
engram_self_reify_beat_json's contract of genuinely releasing the lock
mid-beat.
engram_think_json passed NULL as the anchor. NULL is not "no opinion":
engram_think re-origins at `anchor ? anchor : region->centroid`, so NULL
means "read from the centroid" — and the centroid is the one point where
the gradient is zero by construction. r = x - centroid = 0, so every axis
projection is 0, grad is 0, and direction takes the "at rest" branch at
engram_cognition.c:137.
Measured consequence: EVERY faculty returned an identical null result,
differing only in its label —
{"direction":[0,0,0,0,0,0,0,0],"spread":0,"magnitude":1,"confidence":0.5}
magnitude 1 is membership evaluated at the centroid, spread 0 is its
distance to itself, confidence 0.5 is the stance fallback. The geometry was
never at fault: /api/drift computes real values (centroid_sep 0.104,
core_disp 0.045) over the very same 87 members. Neuron could not think
because the read was always taken from the region's own centre.
The seeds choose WHICH region; they must also supply the VANTAGE. Anchor at
the first resolvable embedded seed — the same seed eg_geo_build_desc infers
dim from, so the two can never disagree. One seed still yields a real
gradient because the descriptor expands to that seed's neighbourhood, so
the seed's position is distinct from the neighbourhood centroid.
The vector is COPIED, never borrowed: g->nodes is realloc'd in place on
append, so a borrowed EngramNode* dangles across any concurrent write.
Verified against a clone of the production store (13,616 nodes / 37,865
edges):
self anchor n_support 87 magnitude 0.00282 spread 18.79
values hub n_support 28 magnitude 0.00318 spread 17.72
with distinct unit direction vectors. Previously both returned the zero
vector with magnitude 1 and spread 0.
STILL OPEN, now isolated by this fix: all five faculties return identical
numbers and confidence stays 0.5, because cog_stance_init is passed NULL
for the stance and the faculty enters the computation only through the
stance's axis_gain[] and bias_dir. The faculty label is inert until a
stance is loaded — which is what learn()'s correspondence-beat calibrates.
Same shape as this bug: a neutral parameter collapsing a capability to a
constant.
No ingest path could carry a vector. engram_node/_full/_layered take text
only, and a node acquired an embedding solely via engram_embed_backfill
DERIVING one from n->content. That made text the mandatory entry medium:
any non-text modality had to be described in prose first, so the geometry
we then reasoned over was the geometry OF THE DESCRIPTION, not of the
signal. Measured: POST /api/nodes accepted an "emb" field, returned 200
with a fresh id, and stored nothing — emb_dim=None, embedded=false.
engram_node_set_emb attaches a vector to an existing node. Off-dimension
vectors are stored but not indexed (the HNSW build loop already filters on
emb_dim), so modality geometry is durable and addressable without
perturbing the canonical index. Setting emb also makes the node ineligible
for embed_backfill, so a realizer's vector is never overwritten by a
text-derived one.
Two reporting fixes ride along, because both are how the drop stayed
invisible: the create response now reports emb_set instead of being
success-shaped regardless, and the node document now always emits emb_dim
and embedded — without which a genuine ingest drop and a mere reporting
gap are indistinguishable.
Verified live: voice node emb_dim=64 embedded=true; text control emb_dim=0
embedded=false; malformed hex, length mismatch and dim<=0 all reject.
KNOWN PLACEMENT DEFECT: this is at the consumer. Ingest is a language
concern, not an engram feature — every el program touching any modality
needs it. The vector also marshals as a hex STRING because el has no
first-class geometry value, which reintroduces text as the transport
medium one layer below the problem being fixed. The durable shape is
geometry as an el value plus declarable realizers, after which the engram
stops having an ingest concept at all. Landing this as the verified probe
that proves the path.
char* result = el_strdup_persist(e ? e->value : ""); // never freed
pthread_mutex_unlock(&_state_mu);
char* copy = el_strdup(result); // arena-tracked
return el_wrap_str(copy);
Two copies were made. `result` existed only as the source for `copy` — never
returned, never freed — and el_strdup_persist bypasses the arena BY DESIGN
("state_set, engram internals"), so arena-pop could never reclaim it. Every
state_get leaked its full value string, permanently.
MEASURED: 200,000 state_get calls against a 64-byte value.
before 15 MB peak RSS growth (~75 bytes/call — the value plus overhead)
after 0 MB
IMPACT. The soul's awareness loop has 68 state_get call sites and ticks every
200ms. Live measurement before the fix: RSS climbing 112 MB per 20s, about
19 GB/hour, in awareness_run -> one_cycle -> perceive, while node_count stayed
flat at ~13,479 — growth with no data behind it. It drove the host from 20 GB
free to 4.3 GB in roughly an hour.
WHY NOW, since the code is old: the soul used to restart constantly (no
write-through, divergent graph, 2.11 GB). Stabilising it (neuron #162) let it
stay up long enough to accumulate. The fix did not cause this leak; it removed
the crashes that were hiding it. Same pattern as the test framework surfacing
math_log — the defect was always there, something finally made it visible.
Found by Ishikawa rather than by reading the nearest code: method (arena
push/pop IS correctly paired per tick), material (node count flat, so not data
growth), environment (19 GB/hr / 18,000 ticks = ~1.1 MB per tick, so per-tick
not one-shot), machine (an allocator that bypasses the arena) — which is where
the evidence pointed.
el_strdup tracks into the thread-local arena, which touches no shared state, so
taking the single copy under _state_mu is safe and removes the temporary
entirely.
Verified: self-hosting fixpoint byte-identical; state round-trip correct for
hit, miss, and overwrite.
Adds tests/native/test_lexer_scaling.el, the regression gate for el #132.
Both directions are proven on LIVE workloads, not synthetic series:
healthy per-character scan 1821 3251 6007 10422 us -> O(n) PASS
rescan-from-zero (the #132 shape) 922 3667 13524 44792 -> O(n^2) FAIL
A gate only proven to pass is decoration. The quadratic specimen exists so
the gate is proven to FIRE.
Also fixes elb_spread_ok to judge the ASYMPTOTIC TAIL (last three ratios)
rather than the whole sweep. Measured on a genuinely linear scan the ratios
ran 3.37 2.92 1.76 1.65 -- the head looks quadratic because it is cold
cache, the tail is the truth. Whole-sweep spread rejected correct data. A
complexity bound is an asymptotic claim and must be judged asymptotically.
That fix came from the classifier refusing to rubber-stamp my own bad
measurement: it reported INDETERMINATE on an unwarmed sweep rather than
passing it. Warmup is now taken and discarded at every sweep point.
Reverts the == workarounds in test_elbench.el now that el #137 has landed;
the natural form generates no str_eq and all 13 fitter tests stay green.
The workaround remains -- the Plus arm is still open.
fn check(label: String, cond: Bool, want: Bool) -> Void {
if cond == want { ... } -> if (str_eq(cond, want)) SIGSEGV
}
Bool has always been an integer in the value model — type_to_c maps Bool to
"int", and el_runtime.h states "Bool -> el_val_t (0 = false, nonzero = true)".
But Bool names were registered NOWHERE: build_int_names_for_params tracked Int
and Float params, and the `let` path tracked Int and Float bindings. Neither
knew about Bool.
So comparing two Bools fell through to str_eq, which dereferenced 0 or 1 as a
char* and segfaulted immediately.
This is the third instance of one family found tonight, after el #137 (a call
on either side of == poisoned the operator) and el #136 (a missing import
compiled clean). All three are the same shape: something the compiler could not
type, silently handled as a string.
Found while writing #137's own test harness — the first version of that harness
crashed on exactly this, on both the old and new compiler, which is how it
surfaced. A test harness that cannot compare two Bools is a good way to notice.
VERIFIED:
- the harness that segfaulted on every prior compiler (exit 139, no output)
now runs clean: 14 passed, 0 failed
- self-hosting fixpoint byte-identical
- the compiler's own generated C differs by 8 lines — only the intended
registration
- neuron's full soul amalgam regenerates in 424ms, exit 0, BYTE-IDENTICAL
- test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12
Adds tests/runtime/operator_typing_test.el, the 15-case suite from #137, so
this family is covered going forward rather than rediscovered.
let a: Int = 5
getint(5) == a -> str_eq(getint(5), a) SIGSEGV
getint(5) == 5 -> getint(5) == 5 fine
A function call whose return type codegen cannot infer poisoned the operator,
and a declared Int on the other side did not save it. str_eq then read an
integer as a char* and segfaulted. Only an integer LITERAL on one side forced
the numeric form, which is why the bug stayed invisible: the common case
happened to be safe.
The check required BOTH operands to be provably Int:
if is_int_expr(left) { if is_int_expr(right) { numeric } }
Loosening to OR is strictly safer, not a trade:
- when one side is a known Int, str_eq is ALWAYS wrong — it dereferences
that integer — while numeric comparison is at worst a wrong answer on a
program that was already ill-typed;
- when neither side is Int nothing changes at all, so string comparison is
untouched.
Found by the test-framework agent while building the benchmark harness; it
correctly declined to fix it mid-phase since it is a codegen semantics change.
VERIFIED, because a semantics change earns more than an assertion:
- 15/15 on a dedicated operator suite covering string literals, string vars,
string-returning calls, mixed var/call, and != in every combination. The
pre-change compiler scores 0/15 on the same file: it segfaults before
printing anything.
- self-hosting fixpoint byte-identical
- the ONLY difference in the compiler's own generated C is the intended one:
a nested if becoming two sequential ifs, in EqEq and NotEq. Nothing else
moved.
- neuron's full soul amalgam regenerates in 400ms, exit 0, output
BYTE-IDENTICAL at 1,270,212 bytes
- test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12 —
62 tests, 190 assertions, zero failures
NOT fixed here, same family, flagged for a decision: Bool PARAMETERS are not
tracked as int-like, so `cond == want` between two Bool params still lowers to
str_eq and segfaults. Found while writing this commit's own test harness — the
first version of it crashed on exactly that, on both the old and new compiler.
It needs the same treatment, and it wants its own change.
import "../../NOPE/does_not_exist.el"
compiled CLEANLY — exit 0, empty stderr, and a program silently missing
everything it imported.
resolve_imports did `fs_read(src_path)` and used the result without checking.
fs_read returns "" both for "file is empty" and "file does not exist", so a
typo, a moved file, or a relative path resolved from the wrong working
directory all produced a successful build of nothing.
It caused a real wrong conclusion during test-framework work: a bisection run
from a subdirectory where ../../runtime/ did not resolve produced ELEVEN
consecutive "successful" compiles that had included no runtime at all, and the
results were believed before anyone noticed.
Missing dependency, confident success — the same shape as a test suite
reporting pass for tests that never ran, and as a benchmark reporting 0us
because the optimiser deleted the loop.
fs_exists separates the two cases, so a legitimately empty file still resolves
to "" and is fine. A path that does not exist now prints the resolved path and
exits 1, which is what build scripts check.
Verified:
- bad import: exit 1 (was 0), message names the resolved path
- elc-cli.el still compiles, self-hosting fixpoint byte-identical
- neuron's full soul amalgam regeneration: exit 0, 405ms, output
byte-identical at 1,270,212 bytes
Adds el_black_box (inline asm, +r constraint, memory clobber) and
runtime/elbench.el: a growth-curve classifier that gates time AND
allocation-count AND allocation-bytes, failing if any exceeds its
declared curve.
Refusal is a first-class verdict. The classifier REFUSES rather than
classifying when the largest measurement is below the floor, or when a
series is hard-flat across an 8x input range -- the shape produced when
the optimiser deletes the work. Reporting O(1) there would be a
confident answer with nothing behind it. Disagreeing ratios report
INDETERMINATE rather than a guess.
Deviation from DESIGN.md 6.2, stated in the source: uses consecutive
ratios on a mandated geometric sweep rather than least-squares over
candidate curves. Ratios are directly interpretable on a doubling sweep
and need no floating point; the cost is weaker O(n) vs O(n log n)
separation, reported as an ambiguous band rather than guessed.
Documents the counter scope limit: engram_*.c and libcurl malloc are
NOT tracked, so a flat curve over engram/HTTP-dominated work is not
evidence of anything.
13 tests prove the classifier against real measured series from
fitprobe.el -- including that an accumulator's allocation COUNT is
linear while its bytes are quadratic, and that el #132's pure-CPU shape
reads FLAT on both allocation signals and is caught only by time.
el #131 instrumented the four string allocators, which meant list- and map-heavy
code reported ZERO allocations — a benchmark over lists would have been fitted
against a flat line and passed anything. Caught during framework work: a
"linear" specimen read 0 allocs until it was rewritten to allocate strings.
A gate is only as good as its blind spots are small, and a signal that silently
reads zero is worse than no signal: it produces a confident pass.
Now counted at every container allocation — ElList and ElMap bodies, their
backing arrays, the copy-on-write clones, and the realloc growth path.
Verified on an append loop (n = 100..800):
allocs 7, 8, 9, 10 +1 per doubling = O(log n) reallocations
bytes 2048, 4096, 8192, 16384 exactly 2x per doubling = O(n)
Both curves are what correct amortized growth should look like, and both read
zero before this change.
Known remaining scope, stated rather than left implicit: these counters cover
the runtime's own allocations. They do not see malloc inside engram_*.c or
libcurl, which is correct — the gate is for El-level complexity, not for
third-party memory behaviour.
el #132's quadratic (strlen per character in str_char_code/str_slice) is
pure CPU and allocates NOTHING. Measured on three controlled specimens:
specimen allocs bytes time
linear 2.00 -> O(n) 2.16 -> O(n) 2.05 -> O(n)
accum 2.00 -> O(n) 3.99 -> O(n2) noisy
compute FLAT FLAT 3.96 -> O(n2)
'compute' is #132's shape. A gate fitting only allocation count and bytes
classifies it FLAT and passes -- it would not have caught the defect it
was created for. The gate now fits time AND count AND bytes, failing if
any exceeds its declared curve.
Also: black_box is mandatory and consuming the result is NOT sufficient.
The first 'compute' reported 0us at every n while returning a correct n2 --
clang closed the loop to a multiply. Only an opaque call restored the curve.
Adds lang/tests/bench/fitprobe.el as the fitter's known-good/known-bad set,
so the classifier is provable without depending on a real bug existing.
Marks DESIGN.md 1.3 stale: test_compiler 3.58s -> 0.03s (119x).
el_val_t math_log(el_val_t f) { return el_from_float(log(el_to_float(f))); }
el_val_t math_ln(el_val_t f) { return el_from_float(log(el_to_float(f))); }
Both were natural log, so math_log and math_ln were the same function.
log10(100) returned 4.605 instead of 2.
Three sources already agreed it should be base-10 and were being contradicted
by this one line:
- runtime/math.el:55 "// math_log — base-10 logarithm."
- el_seed.c:1278 __log_f -> log10() (the path math.el actually calls)
- tests/native/test_math.el:133 asserts log10(100) == 2
FOUND BY THE NEW TEST FRAMEWORK ON ITS FIRST RUN (el #133). The assertion had
been sitting in the suite the whole time; nothing could report it. The old
harness printed "N passed, M failed" with no per-test detail, and half the
suites were not compiling at all — so a failing assertion in a suite nobody
could run was indistinguishable from no failure.
That is the entire argument for the framework, demonstrated on day one: this is
not a bug the framework introduced, it is a bug the framework made VISIBLE.
Verified: tests/native/test_math.el goes 12/13 -> 13/13, math-log passing.
Completes the Phase 1 runner and migrates the 11 test files onto it.
- forward-declare the registry accessors in the test preamble; they are
defined at the end of the unit but the El runner is compiled in between
- eltest.el: explicit trailing return in the void emit_* helpers, which
otherwise lower to 'return println(...)' and fail to compile
- test files import runtime/eltest.el explicitly, using the language's own
textual import mechanism rather than compiler-side auto-injection
- DESIGN.md 6.5: gate on allocation COUNT AND BYTES, not count alone
Verified: self-hosting fixpoint byte-identical (gen2 == gen3). 6 of 11
suites run and report per-test timing. The other 5 fail to COMPILE, and
fail identically under the committed compiler -- pre-existing breakage
this framework makes visible for the first time.
THE BUG. str_char_code() and str_slice() each called strlen() on every
invocation. The lexer walks source one character at a time, so every character
access rescanned the whole remaining input: O(n) per character over n
characters = O(n^2).
el_val_t str_char_code(el_val_t s, el_val_t i) {
...
int64_t n = (int64_t)strlen(str); // <- O(n), every call
if (idx < 0 || idx >= n) return 0;
return str[idx];
}
HOW IT WAS FOUND. Not by reading code — by sampling the running process, which
is the same method that resolved tonight's engram outage after four wrong
theories. A geometric sweep of synthetic sources showed wall-clock rising 3.0x,
3.0x, 4.0x, 4.14x per doubling (converging on 4x = quadratic), and a stack
sample put 779 of 779 samples inside lex(), every one bottoming out in
_platform_strlen via str_char_code and str_slice.
THE FIX. Remember the length instead of recomputing it. The subtlety is
INVALIDATION: El strings are arena-allocated, so a freed pointer can be reused
for a different string at the same address, and a naive pointer-keyed cache
would hand back a stale length and read past the end of the new string —
trading a performance bug for a memory-safety one. So entries carry a
generation, a hit requires pointer AND generation to match, and every path that
frees or mutates a runtime string bumps the generation: el_arena_pop,
seed_request_end, __str_set_char. Stale entries cannot be believed; they miss
and recompute.
MEASURED, same host, same inputs:
n(fns) before after
512 0.10s 0.01s
1024 0.37s 0.02s
2048 1.51s 0.03s 50x
the compiler's own 422 KB source concatenated (DESIGN.md's 3.58s case):
3.55s -> 0.03s 118x
The speedup GROWS with input size, which is the signature of removing a
complexity class rather than a constant factor. After the fix each doubling
adds ~0.01s: linear.
CORRECTNESS, verified rather than assumed:
- byte-identical output on every sweep input (n = 128..2048)
- byte-identical output on the 422 KB compiler concatenation
- byte-identical output on tests/runtime/string_test.el
- self-hosting fixpoint byte-identical
- new tests/runtime/str_cache_test.el: 17 assertions covering bounds, empty
strings, negative indices, slice clamping, distinct strings not sharing a
cached length, 1000 interleaved strings forcing cache-slot collisions, and
a grown string not reporting its old length. All pass.
This is the defect that made dist/soul.c a committed artifact: elc could not run
in CI because it needed 24 GB+ and minutes. It needs neither now.
Replace the hardcoded test harness main() with a generated static registry
and index-based accessors, and move all reporting into runtime/eltest.el.
The old harness inlined direct calls into main() and counted assertions in
two globals. That shape cannot report which test failed, how long any test
took, or whether a test ran at all -- a misspelled registration reported
success for a test that never executed.
- assertions record into per-test state instead of global counters
- registry table emitted at compile time; discovery strictly precedes
execution, which is what later enables --list, filtering and sharding
- per-test wall timing on CLOCK_MONOTONIC, taken in C around the call
- runner in El: structured NDJSON events as source of truth, human output
rendered from the same fields
Implements the three primitives the test-framework design (DESIGN.md §6.5)
requires for gating on growth curves: el_alloc_count, el_alloc_bytes,
el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per
the project's C-builtin recipe.
WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build
failure, which means the signal cannot flake. Wall-clock needs warmup,
statistics, and a quiet machine; on shared CI it is unusable as a gate.
Allocation counts are perfectly deterministic — same input, same number, every
machine, every run. Fit them against n and a complexity regression becomes a
build failure with zero noise.
All four runtime string allocators (el_strdup, el_strbuf, and their _persist
variants) funnel every allocation the language performs, so instrumenting there
counts everything.
WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate.
Measured with two El programs, one allocating once per item, one rebuilding its
accumulator each iteration:
n linear allocs / bytes quadratic allocs / bytes
100 100 / 290 100 / 5,150
200 200 / 690 200 / 20,300
400 400 / 1,490 400 / 80,600
800 800 / 3,090 800 / 321,200
The quadratic program's allocation COUNT is exactly linear — identical to the
healthy one. Counting allocations alone would have missed it completely. Bytes
catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 ->
converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0.
That shape — count linear, per-allocation size growing — is the classic
accidental quadratic, and it is exactly elc's defect: quadratic allocation
VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt
one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across
both; RSS and wall-clock were just the two ways it surfaced.
el_peak_rss is exported for context and is explicitly NOT a gating signal — it
is perturbed by allocator internals, the page cache, and the OS. Gate on the
deterministic numbers; report the physical one.
Counters are unsynchronised by design: this is measurement, and a lock would
change the thing being measured. Exact on the single-threaded compile path,
approximate under threads.
The guard I added minutes ago checked swap availability as a level
(avail < total/8 -> report zero available). That is the wrong signal, and the
same host proved it twice within minutes:
47.65 / 48.00 GiB swap used, 2047 swapouts/s -> genuinely thrashing
26.67 / 28.00 GiB swap used, 0 swapouts/s -> healthy, 15.6 GiB free
Both are ~97% "used". macOS grows swap files on demand and trims them lazily,
so the level says almost nothing about now — it is a high-water mark. The level
check calls the second state an emergency and starves the pool for no reason,
which is its own failure mode: a guard that fires on healthy machines gets
disabled, and then guards nothing.
What separates the two is whether pages are moving. So sample the swapout
counter across calls and judge the delta:
- > 200 pages/s (~3 MiB/s) sustained outward paging => report zero available;
callers refuse to grow and pc_relieve_pressure hands frames back.
- The first call primes the baseline and reports no pressure. One sample
cannot have a rate, and inferring one from a single reading is exactly the
mistake this commit removes.
Measured thresholds, not guessed: idle sat at 0/s, recovery burst hit 24,845/s
while the compressor drained (transient, correctly not a growth decision since
growth is only evaluated on eviction passes), and real thrash held ~2000/s.
200/s sits clearly above noise and far below either.
The compressor-footprint subtraction stays: that RAM is genuinely spoken for
regardless of paging rate.
The adaptive budget I added an hour ago could only grow, and grew toward a
share of TOTAL ram (80%, ~38 GiB on a 48 GB host). That is a memory leak with
extra steps: total never shrinks when other processes need memory, so the pool
had no way to notice it was starving the machine it runs on. Deployed briefly;
caught as memory pressure on the host.
A control loop with only one direction is not a control loop.
- pc_available_ram(): free + inactive + purgeable via host_statistics64 on
Darwin, MemAvailable on Linux. Availability is the quantity that moves when
the machine is under pressure; total is not. Returns 0 when it cannot be
read, and callers then refuse to grow — a cache is never worth swapping the
host, so unknown means no.
- Growth is bounded by availability minus a free-memory floor (2 GiB default,
ENGRAM_POOL_FREE_FLOOR_MB), not by total. The share-of-total ceiling stays
as a second bound and drops 80% -> 50%.
- pc_relieve_pressure(): the missing direction. On every eviction pass, if
available memory is under the floor, hand back ~25% of held frames; the
resident set follows on the next pass so the memory is actually returned
rather than merely re-labelled. Counted as adapt_shrinks alongside
adapt_grows so both directions are visible in the same report.
- pc_default_cap() also clamps the STARTING budget to what is spare right
now, so a cold boot on a loaded machine does not open at a size the host
cannot afford.
Verified on a 48 GB host: engram boots in ~30s, RSS settles at 2.22 GiB (the
store's actual size, resident, not creeping), 0.0% CPU, 13,439 nodes / 37,670
edges, embeddings complete. Guard reports 9.71 GiB available against a 2.00 GiB
floor — 7.71 GiB of headroom it is permitted to use and no more.
Follow-on to the edge write barrier. That fix removed the full-store walk;
this one makes the pool able to notice if anything like it happens again.
WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill
twice on 2026-08-15 and said nothing. From outside it was indistinguishable
from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories
got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each
costing a deploy or a rollback. The whole time, hits/misses/evictions were
already being counted in PgCache, and the struct comment read:
/* stats (introspection only — never affect semantics) */
That comment was the bug. Self-measurement treated as decoration is why the
pool could not correct itself and why no one outside could see what it was
doing. A system that cannot read its own state cannot correct, and neither can
anyone watching it.
- pc_adapt_budget(): the loop, closed. Over a sliding window, evictions
running at a large fraction of accesses WHILE reuse is real means the
working set exceeds the budget — so grow it, geometrically, bounded by a
LIVE re-read of physical memory. Evictions alone are not pressure (a scan
evicts and never returns); evictions with reuse are. An explicit
ENGRAM_POOL_FRAMES still wins — an operator override must not be silently
overruled.
- Budget derived, not declared. A constant cannot be right: 16 GiB of frames
is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM
at startup" is a guess about the future — it cannot know the store grew or
the machine changed. Hence the live re-read.
- pc_report(): ONE structured emission carrying the entire sensed state,
through emit_log — El's existing telemetry, already exporting to OTLP.
Deliberately not a function per stat, and deliberately not a bespoke
/api/pool endpoint: both make observability something hand-written per noun
instead of the uniform mechanism every component already has.
- engram_pool_stats_json(): the same state readable live, wired through the
normal builtin path (codegen arity + el_seed wrapper), so the pool can be
observed in real time rather than reconstructed afterward from a stack
sample.
Verified: with the exact configuration that took production down
(ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots
clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete —
and NO pressure event fires, because the barrier removed the walk that caused
it. The controller is defense in depth; the barrier is the fix.
Checkpointing pushes the ENTIRE resident graph through store_put_node and
store_put_edge (see engram_store_checkpoint). Nodes were cheap: a durable-hash
compare skipped unchanged records with zero page I/O. Edges had no barrier at
all — struct comment at PgCache.barrier_on even says "node durable-hash
barrier" — so every edge was rewritten on every checkpoint, and each rewrite
runs the idempotency probe max_page_lsn_for_id -> btree lookup -> page_read.
Edges outnumber nodes ~3:1 here (37,663 vs 13,436), so routine checkpointing
degenerated into a FULL-STORE WALK in id order: random page access across the
whole 2 GiB store, repeated, overwhelmingly to rediscover nothing had changed.
LRU is worst-case under exactly that pattern — it evicts the page it is about
to want — so once the page cache was smaller than the store, the walk collapsed
into thrashing: 100% CPU, flat RSS, no forward progress, port never bound.
That took the live engram down twice on 2026-08-15.
The walk is the defect. Sizing the cache to survive it treats the symptom.
Changes:
- dh_edge_hash(): edge counterpart of dh_node_hash, with a kind discriminator
byte so an edge can never collide with a node of the same id in the shared
map. created_at/updated_at/last_fired are excluded deliberately: last_fired
is touched by activation without changing what the edge IS, and folding it
in would defeat the barrier on precisely the hot edges that most need it.
- store_put_edge(): barrier check + dh_set on success, mirroring
store_put_node exactly.
- store_scan_edges(): seed the barrier map from on-disk truth at load, so the
FIRST post-boot checkpoint already skips unchanged edges. store_scan_nodes
already did this and its comment says why; edges were simply never done.
Verified: with the exact configuration that killed production
(ENGRAM_POOL_FRAMES=65536 -> 1 GiB cache against a 2 GiB store), the engram now
boots clean and serves — LISTENING, 13,436 nodes / 37,663 edges, embeddings
complete, 0.0% CPU, RSS 1.14 GiB (cache resting at its budget rather than
thrashing against it). Same small cache, same store, no walk.
/api/graph/edges answered a read query by calling engram_save() to serialize
the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in
one line, and both bit production on 2026-08-15:
1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram
server's CANONICAL store. A READ route overwriting the persistence
owner's canonical file. This defect had been fixed once (export moved to
a scratch path); it came back when the hand-written dispatch block was
replaced by @route dispatch and the unfixed copy is the one that
survived the merge.
2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole
graph, per request, to return a bounded slice.
Calling it tonight overwrote the canonical snapshot and immediately preceded
an engram crash loop.
engram_edges_json(limit, offset) is the builtin that route's own TODO asked
for ("Future: add an engram_edges_json() builtin and drop the file round trip
entirely"). It walks g->edges directly and emits every persisted field.
limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell
over, and an unbounded default would preserve the failure mode under a new
name. Callers page explicitly.
Registered in codegen.el's builtin_arity (both plain and __ spellings) and
wrapped in el_seed.c per the project's C-builtin recipe.
will.anderson
Neuron