There was no SIGPIPE handling anywhere in this runtime: no signal
disposition, no MSG_NOSIGNAL, no SO_NOSIGPIPE, and send() called with bare
flags. The default disposition of SIGPIPE is to TERMINATE THE PROCESS, so
any client that hangs up mid-response takes the whole engram with it.
MEASURED, and it is not hypothetical. Production has restarted 254 times
since 2026-08-13T19:37 at a flat ~10 minute cadence:
17:05:18 17:15:29 17:25:38 17:35:50 17:46:00 17:56:10 18:06:22 18:16:30
Intervals of 10m09s-10m12s, not 10m00s. That excess is the whole story:
ai.neuron.engram-tick has StartInterval 600, and engram-tick.sh:13 calls
curl -s -m10 -X POST .../api/tick
The beat does not finish within 10s over 13,634 nodes, so curl waits its
full timeout and closes. The engram then writes the tick response to a dead
socket, takes SIGPIPE, and dies. launchd KeepAlive restarts it, so the
failure presents as a mysterious restart rather than a crash — and
~/.neuron/logs/engram.log records nothing but "[http] listening on" 254
times, with no exit reason. launchctl list confirms the last exit as -13.
Root cause is one level out: consolidation had no owner, so an external
ticker was created to poke it, and the ticker is what kills it. The fix
here does not address that; it makes the process survivable while it is
addressed.
Two layers, because neither alone is portable:
- SO_NOSIGPIPE per accepted socket (Darwin/BSD) and MSG_NOSIGNAL per send
(Linux), so the signal is never raised for socket writes at all.
- A process-wide SIG_IGN backstop, installed once and idempotent, for
platforms and paths with neither. With the signal ignored, send()
returns -1/EPIPE and the existing error path closes the connection.
Also retries send() on EINTR, which the previous loop treated as fatal.
This is an exemption in the sense of lang/spec §8: the write never checked
whether the peer was still there, and the consequence of not checking was
fatal rather than merely wrong.
Three nodes in the live graph carry labels truncated to exactly 80 bytes
ending in a lone 0xE2 — the first byte of an em-dash, cut mid-sequence.
jb_emit_escaped copied every byte >= 0x20 through verbatim, so those three
nodes made the ENTIRE /api/nodes/list response undecodable and no strict
parser could read the graph at all.
production binary 25,929,607 bytes INVALID at byte 89260
this build 26,338,389 bytes VALID, parses to 13,630 nodes
The damage was NOT written by this runtime. No 80-byte truncation exists
here (the only label truncation is engram_first_n_chars at 60), and the
content of those nodes is 2572 and 2746 bytes. Some other producer wrote
them. That is exactly why fixing a writer could not have fixed this: the
store already holds the damage, and it accepts data from importers, other
producers and older binaries.
So the fix goes where the promise is made. A serializer that emits JSON
owes valid UTF-8 whatever it is handed. jb_emit_escaped now validates each
multi-byte sequence before emitting any of it and substitutes U+FFFD for a
bad lead byte, a missing or malformed continuation, an overlong encoding, a
UTF-16 surrogate, or a codepoint above U+10FFFF. Invalid bytes are REPLACED
rather than dropped, so the damage stays visible in the output instead of
being silently papered over. Well-formed input is byte-identical to before.
Second, preventive and explicitly NOT the cause of the above:
engram_first_n_chars truncated by BYTES despite its name, so content with a
multi-byte character crossing byte 60 would produce a half codepoint in the
label. It now uses el_utf8_safe_len, which returns the largest byte length
<= max that does not split a codepoint. Bounded by bytes, not codepoints,
so existing labels never grow — they only stop splitting.
el_utf8_safe_len lives beside str_count_chars rather than in the engram
because the rest of el's string layer is already codepoint-aware
(str_count_chars counts codepoints, str_reverse walks codepoint lengths).
Byte truncation was the outlier and the concern is a string concern.
Note on the investigation: I first "fixed" the truncator and wrote a test
that passed on the UNPATCHED build too, because route_create_node passes
label = content when no label is supplied, so engram_first_n_chars is never
reached over HTTP. The test proved nothing. The real cause was only found
by decoding the actual failing bytes out of the live response.
engram_ground_json resolved each seed to a REGION, wrote the grounded-by
edge between the two regions' HUBS, and then echoed those hubs back in the
"claim"/"evidence" fields as if they were the caller's input:
const char* cid = C->hub_id ? C->hub_id : EL_CSTR(claim);
const char* eid = E->hub_id ? E->hub_id : EL_CSTR(evidence);
cog_ground_edge(g_engram_store, cid, eid, grounding, fw);
Three consequences, all measured against a clone of the live store:
1. The edge landed on a node the caller never named. Grounding 3b9ced5d
against 6edf8c79 wrote an edge on the hubs of their regions instead.
2. When both seeds resolve into the same region the support is circular
and scores near 1.0 for structural reasons, not evidential ones. Four
probe nodes written together landed in one region, and every grounding
among them returned 0.93-0.99 as if it were evidence. Two independent
agents hit this and reported 0.885 / 0.909 self-groundings as confident.
3. The echo concealed both: the response was indistinguishable from a
successful grounding of the ids that were passed in.
The region is HOW a claim is evaluated; it is not WHAT the claim is about.
So the edge now attaches to the requested ids, and the resolved hubs are
reported separately as claim_region / evidence_region.
Degeneracy is broader than hub == hub. Three circular shapes, all
previously invisible:
same-region both seeds resolve to one region
claim-region-is-evidence the evidence IS the hub of the claim's own
neighbourhood — measured at 0.98883
evidence-region-is-claim the mirror case
Each sets grounding to 0 and writes no edge. Circular support is not
support, and a grounding that is degenerate by construction must not
enter the graph as though it were evidence.
Verified:
6edf8c79 -> 6edf8c79 degenerate=same-region g=0 written=false
6edf8c79 -> d0406dfd degenerate=same-region g=0 written=false
ebc1413e -> 64cc96ef degenerate=false g=0.774563 written=true
64cc96ef -> ebc1413e degenerate=false g=0.802896 written=true
Legitimate grounding across distinct regions is unchanged and still
writes; only circular support is refused.
This is the same class as #142 and #146 — a value that looked like an
answer with nothing behind it — except here it was also writing that
non-answer into the canonical store.
engram_think_json built a NEUTRAL stance on every call — cog_stance_init
with a NULL id, all axis_gain 1.0, bias_dir NULL, reliability 0.5 — and
never loaded the stance the correspondence-beat had been persisting.
That mattered because the faculty enters engram_think ONLY through the
stance: axis_gain[k] warps the per-axis extents and bias_dir seeds the
steering direction. cog_stance_init stores the faculty NAME and nothing
reads it. So with a neutral stance, reason/abduce/induce/plan/analogize
were byte-identical output under different labels, and confidence was
pinned to 0.5 because GeoGradient.confidence IS stance->reliability.
The machinery already existed and only this call site ignored it.
engram_correspondence_beat_json resumes via cog_stance_from_node and
persists via cog_stance_to_node under "stance-<faculty>-<hub>". Every
beat's calibration was written and then thrown away on the next read.
Same defect as the NULL anchor fixed in #142, one line below: a neutral
argument collapsing a capability to a constant.
Resume the same id the beat writes, so learning compounds across beats and
cold boot. Fall back to neutral only when no stance exists — a genuine
uninformed prior rather than a discarded informed one.
Also emit stance_resumed, so confidence 0.5 from a learned-but-unreliable
stance is distinguishable from confidence 0.5 from "no stance exists".
That reporting gap is what let the neutral stance hide.
Verified against a clone of the production store (13,627 nodes):
before beat, no stance stance_resumed=false confidence=0.5
beat on a NON-keystone brier 0.00458568 -> 0.00329654
reduction 28.11%, n_trials 6000,
reliability 0.930726, stance_written=true
after beat stance_resumed=true confidence=0.930726
Confidence now equals the learned reliability instead of the uninformed
prior. The keystone self-anchor correctly stays at 0.5 — calibration is
deliberately refused on protected identity regions, and that refusal is
now visible as resumed=true with confidence unchanged, rather than being
indistinguishable from the bug.
STILL OPEN: with no learned bias_dir the faculties remain identical in
direction. What distinguishes abduce from induce geometrically is a
design decision about how Neuron thinks, not a plumbing defect, and is
deliberately left to Will.
The binary was stamped before dev advanced (vindex publication landed in
el_runtime.c and engram_vindex.c). Rebuilt against the merged runtime so the
committed compiler matches the runtime it ships beside. Fixpoint re-verified
byte-identical; test_compiler 82/82; engram/src/server.el still compiles and
still emits its 18 config declarations.
Migrates engram to the `program` block. 18 configuration variables that each
carried their default inline at the point of use now declare it in one place,
and engram declares itself a singleton.
The read sites lose their defaults entirely: `let v = env("X")` followed by
`if str_eq(v,"") { "default" } else { v }` collapses to `config("X")`. The
guide_env_or(key, dflt) helper is deleted -- its whole job was supplying a
per-site default, which is the thing being removed.
Fixes ENGRAM_DATA_DIR, which was the clearest instance of the defect. It was
read at six sites. Five were dead: `let dir_raw = env("ENGRAM_DATA_DIR")`
immediately shadowed on the next line by `engram_resolve_data_dir()`. The sixth
was live and defaulted to /tmp/engram, contradicting the canonical resolver's
$HOME/.neuron/engram -- and its consumer is the pre-destructive reseed backup,
so with ENGRAM_DATA_DIR unset the safety copy was written to ephemeral storage
while the store it protected lived elsewhere. All six now go through
engram_resolve_data_dir().
ENGRAM_DATA_DIR is deliberately NOT declared in the program block, and the
source says why: engram_resolve_data_dir() already owns it, and a second
declaration would give it two owners that can disagree -- recreating the exact
defect being removed here. A variable belongs in the block when the block would
be its only owner. HOME stays a raw env() read; it is an environment fact, not
configuration.
singleton: "engram" matters more than it looks. Today a second engram whose
bind() fails merely returns from http_serve -- after it has already replayed
the WAL and written boot-time backup files -- and then exits 0, indistinguishable
from a clean run. That is how two instances came to share one data dir. Verified
that the second instance now refuses before any side effect: with instance 1
holding the lock (lsof pid, shell pid, and lock file contents all agreeing at
5946), the second start named that pid, exited 1, and left the data directory
untouched.
Verified by bijection on the generated C: 18 config() reads, 18 declarations,
no read without a declaration and no declaration without a read. Three bad Int
values are reported in a single run rather than costing one restart each.
ENGRAM_API_KEY keeps its permissive empty default, which disables auth -- that
is pre-existing behaviour and changing it is out of scope. The source marks
making it `required` as the obvious hardening follow-up.
server.el declares a `program` block, which the previously committed elc cannot
parse. Without this the tree is internally inconsistent: source in the repo that
the compiler in the repo rejects.
This is the documented re-stamp from BOOTSTRAP.md / AGENTS.md, and its
precondition is met -- the self-hosting fixpoint was verified byte-identical
(stage3 output == stage2 output) both before installing and again with the
installed binary. tests/native/test_compiler.el passes 82/82 against it.
Two pre-existing failures are unchanged and are NOT from this work, confirmed
by rebuilding them against the original runtime: test_env's
"state_keys returns JSON array" fails identically before and after, and
test_json/test_state fail to link on symbols (json_build_array, state_has) that
were never prototyped -- the same class of gap as config(), which this branch
fixed because it blocked the build.
El's units of encapsulation are the function and the module. Neither can hold
a concern that belongs to the process, so each one had been expressed the only
way it could be -- as a convention: call this at every site. Conventions of
that shape do not hold. Measured here: zero process-identity guards at any
layer, 20 environment variables each with its default written inline at the
read site, 62 persist call sites, 10 per-route auth checks. One absence, four
times.
Step 0 first, because the premise was wrong. El was believed to have no
middleware or effect mechanism. It has one, and it is already load-bearing:
codegen injects engram_boundary_beat at the entry of every @manager/@accessor
fn, decorators take arguments and stack, dharma_emit from a non-@manager fn is
a #error, and the cgi block injects el_cgi_init at the head of main(). So the
correct move was not to invent a mechanism but to generalize the seam that
already existed. The real gap is narrower and is now recorded: the seam is
prologue-only and its callee is a fixed builtin.
Adds a `program` block -- the third program-level declarative block. cgi and
service declare what a program may do; program declares what it is.
program "engram" {
singleton: "engram"
env ENGRAM_BIND: String = ":8742"
env GUIDE_PORT: Int = "8771"
}
singleton takes an exclusive flock before any user statement runs and refuses a
second start, reporting the holder's pid. It is a lock rather than a pidfile so
the kernel releases it on death including SIGKILL -- no stale state, and so no
"delete the lock file to get unstuck" ritual, which would itself be a
convention. It reports the pid because "already running" is not actionable; a
pid is. That is the direct answer to a stale process surviving a pkill and
going on answering probes.
env entries resolve once at startup -- environment wins, declaration supplies
the fallback -- and validate as a whole, reporting every problem at once rather
than costing one restart per variable. config("X") for an undeclared X is
fatal, because an advisory schema is just another convention. Programs without
a program block are unaffected, so migration is per-program.
Only one keyword is added. `config` and `env` could not become keywords -- both
are real identifiers in the tree -- so the block's fields are read as
identifier token values by its own parse loop and stay usable everywhere else.
The init function is emitted at the block site and called from main() rather
than inlined into main(). The live backend is codegen_streaming, which emits in
source order and cannot hold the entry list alive until main(); this way only a
single bool has to survive.
Also fixes: config() was defined in el_runtime.c but never prototyped in
el_runtime.h, so any el program calling it failed to compile under C99.
Spec: section 18 documents what shipped. Section 9 is corrected -- it claimed
decorators had no structural meaning, which has not been true for some time.
Section 19 designs durability-as-an-epilogue-effect and route authorization
and states plainly why neither is implemented here: both land in files under
concurrent modification, and the prerequisite for both is lifting the seam
from prologue-only to prologue/epilogue.
Self-hosting fixpoint verified byte-identical.
#141 let signal enter as geometry and it worked, but it was placed at the
CONSUMER and said so in its own commit message. This is the correction.
Three defects, all of them placement:
1. It sat in the engram. Ingest is a LANGUAGE concern — every el program
touching any modality needs it, and the engram is merely one el program
that happens to hold a graph. The geometry surface is now defined in
el_runtime.c immediately ABOVE the engram section and depends on nothing
inside it. Delete the entire engram and geometry still enters el.
2. It marshalled the vector as a hex STRING, because el had no first-class
geometry value — which reintroduced text as the TRANSPORT medium one layer
below the problem being fixed. Geometry is now an el value: a magic-tagged
heap object carried in el_val_t, same discipline as List/Map. Hex survives
only as an adapter at the edge, which is all an encoding should ever be.
3. It needed an arbitrary `dim <= 8192` bound purely to size an allocation
from a caller's CLAIM about a string's length. A value carries its own
width, so the width is derived and never asserted. The bound is gone, not
raised — there is nothing left to validate.
Language surface, none of it engram-prefixed: geometry_new / _dim / _is /
_get / _set / _norm / _free, geometry_from_f32le_hex + geometry_to_f32le_hex
as the wire adapters, realizer_register(modality, fn_name), realizer_has, and
transduce(signal, modality) -> Geometry.
REALIZERS ARE DECLARABLE IN EL. This is the part that makes the move real
rather than nominal: registration resolves a name with dlsym against the
running binary, the identical mechanism http_set_handler already relies on,
because every el `fn name(...)` compiles to a global C symbol with that exact
name. So an ordinary el function IS a realizer and a new modality needs no
runtime patch. Verified end to end in lang/examples/transduce.el: an el-defined
tone_realizer is registered by name, transduce dispatches to it, and the
signal demonstrably reaches it (distinct signals produce distinct geometry).
A modality with no realizer transduces to NOTHING. There is deliberately no
built-in realizer, not even for text — silently embedding a description of a
signal and calling that perception is the exact defect this ends.
engram/src/server.el is migrated: POST /api/nodes decodes "emb" hex exactly
once, at the edge, into a Geometry, and everything below that line moves
geometry. The wire is unchanged because production clients speak it. "dim" is
now an ASSERTION about the vector, not the source of its width; disagreement
is a rejected ingest, not a silent reinterpretation.
#141's engram_node_set_emb becomes a DEPRECATED WRAPPER over
geometry_from_f32le_hex + node_attach_geometry — kept only because the runtime
ships as an SDK asset and a downstream binary may link the symbol. Its exact
contract, negative cases included, is preserved and re-verified.
ingest.el's `fn transduce` is renamed transduce_manifold. Mechanically it had
to yield the name (duplicate C symbol, a hard compile error, measured). But it
was never signal->geometry: it chunks already-extracted content into a node+edge
manifold, one layer up, and had taken the name belonging to the primitive
underneath it. Behaviour unchanged.
PROPERTIES FROM #141 PRESERVED, each re-measured on a scratch engram (:8971,
never prod :8742):
- off-dimension vectors stored but NOT indexed — the HNSW build loop still
filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is
durable and addressable without perturbing the 768-dim canonical index
- geometry makes a node ineligible for embed_backfill: after backfill the
64-dim voice node was still 64-dim while the text control acquired 768
- the create response reports whether geometry landed, and the node document
always emits emb_dim and embedded
Read-back with control and negatives, all verified against a PID-confirmed
fresh binary: geometry node emb_dim=64 embedded=true / emb_set=1; text-only
control emb_dim=0 embedded=false / emb_set=0; malformed hex, ragged length,
and dim-disagreement each emb_set=0.
Two compiler landmines found by reading the generated C rather than trusting a
successful build, both documented at their sites: elc lowers `a == b` to
str_eq unless both operand NAMES are in the per-function int-name set (which
does NOT propagate into nested if-expression blocks — the first cut would have
strcmp'd two integers as pointers on the first geometry-bearing request), and
`+` lowers to string concat when either operand is a user-defined call.
The crash (SIGTRAP in engram_activate -> eg_vindex_sync -> vindex_insert ->
_realloc) had three read paths mutating five process-global statics.
engram_activate, eg_knn_for_node (whose own comment says "No writes.") and
engram_geo_reify_run_json all called eg_vindex_sync, which frees the index,
reallocs the seen-map and inserts — on a read.
Three moves, in decreasing order of how much they dissolve:
1. Misfiled scratch is not shared state. visited/visit_epoch/visited_cap
were never owned by the index; they are one traversal's local, hoisted
into struct VIndex as an allocation optimisation. They want neither a
lock nor a capability nor a pool — just to go back in the call frame.
Two concurrent READS stomped each other purely because of this.
2. const IS the capability. Once the scratch leaves the struct, search
reads and nothing else, so vindex_search takes a const VIndex*. That is
exactly what a capability-pointer ABI would have bought — a read path
physically cannot call vindex_insert, enforced by the compiler on every
future caller — for one qualifier instead of an ABI swept across
hundreds of builtins.
3. What survives is publication, not ownership. HNSW insert is NOT an
append: it rewires the neighbour links of already-existing elements and
reallocs elems[], so the store's append-only property does not transfer
to the index derived from it. eg_vindex_sync therefore splits into
eg_vindex_maintain (exclusive, sole mutator) and eg_vindex_view (shared,
returns const VIndex*). A read path may demand that a current snapshot
exist — a request to the owner, not a mutation by the reader.
Write-side owner: eg_vindex_note_embedded hooks the embedding-ASSIGNMENT
sites rather than the append sites, because a node with no embedding cannot
be in a vector index — embedding assignment is the event that owns index
membership. One O(log n) insert, no O(node_count) presence scan. This also
retires the "STALENESS (honest tradeoff)" note where a lazily-embedded
older node stayed invisible to route_nearest/autoconnect until a full
rebuild (the embed-gap #20 shape).
Evidence. The existing harness conflated two hazards, which is why fixing
half of it read as failure. Split into four:
single (3000 vec, ASan+UBSan) clean -> clean
readers (4 readers, no writer, TSan) RACE -> clean
unsynchronized (writer+reader, bare) race -> race, expected forever
published (owner + 4 readers) n/a -> clean, 3000/3000 landed
RESULT: PASS. recall@10 = 0.9365 at ef_search=128 (gate >= 0.90);
determinism byte-identical across two independent builds.
The unsynchronized half is now permanently expected to race, deliberately:
it is the executable proof that the boundary must live above the data
structure, not inside it.
fb32d15's guard is KEPT, correcting this design's own section 5. Measured,
it guards TWO structures and only one was converted here: g->nodes/g->edges
are realloc'd in place (el_runtime.c:7618,7629) and engram_activate_inner's
embed-backfill writes n->emb through exactly such a borrowed pointer.
Deleting the guard reintroduces a measured 11171->9579 edge loss. Its
comment is narrowed to the RAM graph and the deletion precondition named.
That corrects the ordering claim too: the residual is not one ABI that
dissolves everything at once, it is a PROPERTY applied per structure.
Residues evaporate in the order the property is applied, and a residue
whose structure has not been converted must be left standing.
Promotes the two throwaway sanitizer harnesses used to diagnose the
2026-08-16 soul crash into engram/test/ so the bug cannot silently regress.
The harness has two halves and the PAIR is the point — it is what localises
the defect to concurrency rather than to HNSW logic:
single 3000 clustered vectors, one thread, ASan+UBSan. The CONTROL.
Must always be clean. During diagnosis this cleared all 13,820
real dim-768 vectors from the live store, which DISPROVED an
inspection-derived hypothesis about an out-of-bounds
reverse-link write at engram_vindex.c:340.
concurrent writer + reader on one shared index, TSan. Currently reports a
race at engram_vindex.c:195 (visited_reset) reached from both
vindex_search and vindex_insert, because VIndex still owns its
visited[]/visit_epoch scratch — so even two concurrent READS
corrupt each other's traversal.
Verified: half 1 passes, half 2 reproduces the race.
Gated on EXPECT_RACE, default 1, so the concurrent half documents the known
defect without failing the suite today. When the visited set moves to a
per-query checkout pool (hnswlib VisitedListPool style — NOT thread_local,
since http_worker is a thread per connection and a __thread buffer would leak
~55KB per connection), flip EXPECT_RACE=0 and it becomes a real gate.
The soul daemon had two engram callers and only one of them locked.
soul.el:729 starts the HTTP server via http_serve_async (spawning
http_worker threads); soul.el:731 then runs awareness_run() on the MAIN
thread. awareness.el's perceive() -> engram_activate_json() ->
engram_activate() -> eg_vindex_sync() -> vindex_insert() mutates the same
g->nodes/g->edges and the process-global _eg_vindex HNSW index that the
workers touch. g_engram_req_lock existed to serialize exactly this, but it
was only ever taken inside http_worker: engram_req_lock/engram_req_unlock
appear in ZERO .el sources, so the awareness loop ran lock-free beside the
workers on every tick (SOUL_TICK_MS=1000).
Result was a crash-loop under launchd KeepAlive: five crashes in ~4 minutes
on 2026-08-16 with varying faulting frames -- search_layer<-vindex_insert
<-eg_vindex_sync, engram_activate, abort, and one inside xzm_realloc's own
freelist. Varying sites plus a fault in allocator metadata means heap
corruption. The SIGSEGV address 0x65646f4e6d617267 is little-endian ASCII
"gramNode": string bytes dereferenced as an Elem vector pointer.
Diagnosed by bisection rather than inspection:
- Replaying all 13,820 real dim-768 vectors harvested from the live store
through the index single-threaded under ASan is 100% clean, which rules
out an HNSW logic/bounds bug.
- Two threads on one index trip ThreadSanitizer immediately at
engram_vindex.c:195 (visited_reset), reached from both vindex_search and
vindex_insert. VIndex keeps a SHARED visited-epoch scratch buffer, so
even two concurrent READS corrupt each other's traversal and walk bogus
element indices.
So this is purely a concurrency defect, not an HNSW logic error. (An
inspection-derived hypothesis about an out-of-bounds reverse-link write at
engram_vindex.c:340 was disproved by the single-threaded run.)
Fix: a thread-local ownership depth (_eg_req_depth) lets engram entry points
self-guard. engram_activate() becomes a wrapper over engram_activate_inner()
that acquires g_engram_req_lock when called with depth 0 (the awareness
thread) and passes through when depth > 0 (nested inside an http_worker that
already holds it), so the non-recursive mutex cannot self-deadlock. The depth
is a plain counter, never a recursive-mutex count, preserving
engram_self_reify_beat_json's contract of genuinely releasing the lock
mid-beat.
engram_think_json passed NULL as the anchor. NULL is not "no opinion":
engram_think re-origins at `anchor ? anchor : region->centroid`, so NULL
means "read from the centroid" — and the centroid is the one point where
the gradient is zero by construction. r = x - centroid = 0, so every axis
projection is 0, grad is 0, and direction takes the "at rest" branch at
engram_cognition.c:137.
Measured consequence: EVERY faculty returned an identical null result,
differing only in its label —
{"direction":[0,0,0,0,0,0,0,0],"spread":0,"magnitude":1,"confidence":0.5}
magnitude 1 is membership evaluated at the centroid, spread 0 is its
distance to itself, confidence 0.5 is the stance fallback. The geometry was
never at fault: /api/drift computes real values (centroid_sep 0.104,
core_disp 0.045) over the very same 87 members. Neuron could not think
because the read was always taken from the region's own centre.
The seeds choose WHICH region; they must also supply the VANTAGE. Anchor at
the first resolvable embedded seed — the same seed eg_geo_build_desc infers
dim from, so the two can never disagree. One seed still yields a real
gradient because the descriptor expands to that seed's neighbourhood, so
the seed's position is distinct from the neighbourhood centroid.
The vector is COPIED, never borrowed: g->nodes is realloc'd in place on
append, so a borrowed EngramNode* dangles across any concurrent write.
Verified against a clone of the production store (13,616 nodes / 37,865
edges):
self anchor n_support 87 magnitude 0.00282 spread 18.79
values hub n_support 28 magnitude 0.00318 spread 17.72
with distinct unit direction vectors. Previously both returned the zero
vector with magnitude 1 and spread 0.
STILL OPEN, now isolated by this fix: all five faculties return identical
numbers and confidence stays 0.5, because cog_stance_init is passed NULL
for the stance and the faculty enters the computation only through the
stance's axis_gain[] and bias_dir. The faculty label is inert until a
stance is loaded — which is what learn()'s correspondence-beat calibrates.
Same shape as this bug: a neutral parameter collapsing a capability to a
constant.
No ingest path could carry a vector. engram_node/_full/_layered take text
only, and a node acquired an embedding solely via engram_embed_backfill
DERIVING one from n->content. That made text the mandatory entry medium:
any non-text modality had to be described in prose first, so the geometry
we then reasoned over was the geometry OF THE DESCRIPTION, not of the
signal. Measured: POST /api/nodes accepted an "emb" field, returned 200
with a fresh id, and stored nothing — emb_dim=None, embedded=false.
engram_node_set_emb attaches a vector to an existing node. Off-dimension
vectors are stored but not indexed (the HNSW build loop already filters on
emb_dim), so modality geometry is durable and addressable without
perturbing the canonical index. Setting emb also makes the node ineligible
for embed_backfill, so a realizer's vector is never overwritten by a
text-derived one.
Two reporting fixes ride along, because both are how the drop stayed
invisible: the create response now reports emb_set instead of being
success-shaped regardless, and the node document now always emits emb_dim
and embedded — without which a genuine ingest drop and a mere reporting
gap are indistinguishable.
Verified live: voice node emb_dim=64 embedded=true; text control emb_dim=0
embedded=false; malformed hex, length mismatch and dim<=0 all reject.
KNOWN PLACEMENT DEFECT: this is at the consumer. Ingest is a language
concern, not an engram feature — every el program touching any modality
needs it. The vector also marshals as a hex STRING because el has no
first-class geometry value, which reintroduces text as the transport
medium one layer below the problem being fixed. The durable shape is
geometry as an el value plus declarable realizers, after which the engram
stops having an ingest concept at all. Landing this as the verified probe
that proves the path.
char* result = el_strdup_persist(e ? e->value : ""); // never freed
pthread_mutex_unlock(&_state_mu);
char* copy = el_strdup(result); // arena-tracked
return el_wrap_str(copy);
Two copies were made. `result` existed only as the source for `copy` — never
returned, never freed — and el_strdup_persist bypasses the arena BY DESIGN
("state_set, engram internals"), so arena-pop could never reclaim it. Every
state_get leaked its full value string, permanently.
MEASURED: 200,000 state_get calls against a 64-byte value.
before 15 MB peak RSS growth (~75 bytes/call — the value plus overhead)
after 0 MB
IMPACT. The soul's awareness loop has 68 state_get call sites and ticks every
200ms. Live measurement before the fix: RSS climbing 112 MB per 20s, about
19 GB/hour, in awareness_run -> one_cycle -> perceive, while node_count stayed
flat at ~13,479 — growth with no data behind it. It drove the host from 20 GB
free to 4.3 GB in roughly an hour.
WHY NOW, since the code is old: the soul used to restart constantly (no
write-through, divergent graph, 2.11 GB). Stabilising it (neuron #162) let it
stay up long enough to accumulate. The fix did not cause this leak; it removed
the crashes that were hiding it. Same pattern as the test framework surfacing
math_log — the defect was always there, something finally made it visible.
Found by Ishikawa rather than by reading the nearest code: method (arena
push/pop IS correctly paired per tick), material (node count flat, so not data
growth), environment (19 GB/hr / 18,000 ticks = ~1.1 MB per tick, so per-tick
not one-shot), machine (an allocator that bypasses the arena) — which is where
the evidence pointed.
el_strdup tracks into the thread-local arena, which touches no shared state, so
taking the single copy under _state_mu is safe and removes the temporary
entirely.
Verified: self-hosting fixpoint byte-identical; state round-trip correct for
hit, miss, and overwrite.
Adds tests/native/test_lexer_scaling.el, the regression gate for el #132.
Both directions are proven on LIVE workloads, not synthetic series:
healthy per-character scan 1821 3251 6007 10422 us -> O(n) PASS
rescan-from-zero (the #132 shape) 922 3667 13524 44792 -> O(n^2) FAIL
A gate only proven to pass is decoration. The quadratic specimen exists so
the gate is proven to FIRE.
Also fixes elb_spread_ok to judge the ASYMPTOTIC TAIL (last three ratios)
rather than the whole sweep. Measured on a genuinely linear scan the ratios
ran 3.37 2.92 1.76 1.65 -- the head looks quadratic because it is cold
cache, the tail is the truth. Whole-sweep spread rejected correct data. A
complexity bound is an asymptotic claim and must be judged asymptotically.
That fix came from the classifier refusing to rubber-stamp my own bad
measurement: it reported INDETERMINATE on an unwarmed sweep rather than
passing it. Warmup is now taken and discarded at every sweep point.
Reverts the == workarounds in test_elbench.el now that el #137 has landed;
the natural form generates no str_eq and all 13 fitter tests stay green.
The workaround remains -- the Plus arm is still open.
fn check(label: String, cond: Bool, want: Bool) -> Void {
if cond == want { ... } -> if (str_eq(cond, want)) SIGSEGV
}
Bool has always been an integer in the value model — type_to_c maps Bool to
"int", and el_runtime.h states "Bool -> el_val_t (0 = false, nonzero = true)".
But Bool names were registered NOWHERE: build_int_names_for_params tracked Int
and Float params, and the `let` path tracked Int and Float bindings. Neither
knew about Bool.
So comparing two Bools fell through to str_eq, which dereferenced 0 or 1 as a
char* and segfaulted immediately.
This is the third instance of one family found tonight, after el #137 (a call
on either side of == poisoned the operator) and el #136 (a missing import
compiled clean). All three are the same shape: something the compiler could not
type, silently handled as a string.
Found while writing #137's own test harness — the first version of that harness
crashed on exactly this, on both the old and new compiler, which is how it
surfaced. A test harness that cannot compare two Bools is a good way to notice.
VERIFIED:
- the harness that segfaulted on every prior compiler (exit 139, no output)
now runs clean: 14 passed, 0 failed
- self-hosting fixpoint byte-identical
- the compiler's own generated C differs by 8 lines — only the intended
registration
- neuron's full soul amalgam regenerates in 424ms, exit 0, BYTE-IDENTICAL
- test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12
Adds tests/runtime/operator_typing_test.el, the 15-case suite from #137, so
this family is covered going forward rather than rediscovered.
let a: Int = 5
getint(5) == a -> str_eq(getint(5), a) SIGSEGV
getint(5) == 5 -> getint(5) == 5 fine
A function call whose return type codegen cannot infer poisoned the operator,
and a declared Int on the other side did not save it. str_eq then read an
integer as a char* and segfaulted. Only an integer LITERAL on one side forced
the numeric form, which is why the bug stayed invisible: the common case
happened to be safe.
The check required BOTH operands to be provably Int:
if is_int_expr(left) { if is_int_expr(right) { numeric } }
Loosening to OR is strictly safer, not a trade:
- when one side is a known Int, str_eq is ALWAYS wrong — it dereferences
that integer — while numeric comparison is at worst a wrong answer on a
program that was already ill-typed;
- when neither side is Int nothing changes at all, so string comparison is
untouched.
Found by the test-framework agent while building the benchmark harness; it
correctly declined to fix it mid-phase since it is a codegen semantics change.
VERIFIED, because a semantics change earns more than an assertion:
- 15/15 on a dedicated operator suite covering string literals, string vars,
string-returning calls, mixed var/call, and != in every combination. The
pre-change compiler scores 0/15 on the same file: it segfaults before
printing anything.
- self-hosting fixpoint byte-identical
- the ONLY difference in the compiler's own generated C is the intended one:
a nested if becoming two sequential ifs, in EqEq and NotEq. Nothing else
moved.
- neuron's full soul amalgam regenerates in 400ms, exit 0, output
BYTE-IDENTICAL at 1,270,212 bytes
- test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12 —
62 tests, 190 assertions, zero failures
NOT fixed here, same family, flagged for a decision: Bool PARAMETERS are not
tracked as int-like, so `cond == want` between two Bool params still lowers to
str_eq and segfaults. Found while writing this commit's own test harness — the
first version of it crashed on exactly that, on both the old and new compiler.
It needs the same treatment, and it wants its own change.
import "../../NOPE/does_not_exist.el"
compiled CLEANLY — exit 0, empty stderr, and a program silently missing
everything it imported.
resolve_imports did `fs_read(src_path)` and used the result without checking.
fs_read returns "" both for "file is empty" and "file does not exist", so a
typo, a moved file, or a relative path resolved from the wrong working
directory all produced a successful build of nothing.
It caused a real wrong conclusion during test-framework work: a bisection run
from a subdirectory where ../../runtime/ did not resolve produced ELEVEN
consecutive "successful" compiles that had included no runtime at all, and the
results were believed before anyone noticed.
Missing dependency, confident success — the same shape as a test suite
reporting pass for tests that never ran, and as a benchmark reporting 0us
because the optimiser deleted the loop.
fs_exists separates the two cases, so a legitimately empty file still resolves
to "" and is fine. A path that does not exist now prints the resolved path and
exits 1, which is what build scripts check.
Verified:
- bad import: exit 1 (was 0), message names the resolved path
- elc-cli.el still compiles, self-hosting fixpoint byte-identical
- neuron's full soul amalgam regeneration: exit 0, 405ms, output
byte-identical at 1,270,212 bytes
Adds el_black_box (inline asm, +r constraint, memory clobber) and
runtime/elbench.el: a growth-curve classifier that gates time AND
allocation-count AND allocation-bytes, failing if any exceeds its
declared curve.
Refusal is a first-class verdict. The classifier REFUSES rather than
classifying when the largest measurement is below the floor, or when a
series is hard-flat across an 8x input range -- the shape produced when
the optimiser deletes the work. Reporting O(1) there would be a
confident answer with nothing behind it. Disagreeing ratios report
INDETERMINATE rather than a guess.
Deviation from DESIGN.md 6.2, stated in the source: uses consecutive
ratios on a mandated geometric sweep rather than least-squares over
candidate curves. Ratios are directly interpretable on a doubling sweep
and need no floating point; the cost is weaker O(n) vs O(n log n)
separation, reported as an ambiguous band rather than guessed.
Documents the counter scope limit: engram_*.c and libcurl malloc are
NOT tracked, so a flat curve over engram/HTTP-dominated work is not
evidence of anything.
13 tests prove the classifier against real measured series from
fitprobe.el -- including that an accumulator's allocation COUNT is
linear while its bytes are quadratic, and that el #132's pure-CPU shape
reads FLAT on both allocation signals and is caught only by time.
el #131 instrumented the four string allocators, which meant list- and map-heavy
code reported ZERO allocations — a benchmark over lists would have been fitted
against a flat line and passed anything. Caught during framework work: a
"linear" specimen read 0 allocs until it was rewritten to allocate strings.
A gate is only as good as its blind spots are small, and a signal that silently
reads zero is worse than no signal: it produces a confident pass.
Now counted at every container allocation — ElList and ElMap bodies, their
backing arrays, the copy-on-write clones, and the realloc growth path.
Verified on an append loop (n = 100..800):
allocs 7, 8, 9, 10 +1 per doubling = O(log n) reallocations
bytes 2048, 4096, 8192, 16384 exactly 2x per doubling = O(n)
Both curves are what correct amortized growth should look like, and both read
zero before this change.
Known remaining scope, stated rather than left implicit: these counters cover
the runtime's own allocations. They do not see malloc inside engram_*.c or
libcurl, which is correct — the gate is for El-level complexity, not for
third-party memory behaviour.
el #132's quadratic (strlen per character in str_char_code/str_slice) is
pure CPU and allocates NOTHING. Measured on three controlled specimens:
specimen allocs bytes time
linear 2.00 -> O(n) 2.16 -> O(n) 2.05 -> O(n)
accum 2.00 -> O(n) 3.99 -> O(n2) noisy
compute FLAT FLAT 3.96 -> O(n2)
'compute' is #132's shape. A gate fitting only allocation count and bytes
classifies it FLAT and passes -- it would not have caught the defect it
was created for. The gate now fits time AND count AND bytes, failing if
any exceeds its declared curve.
Also: black_box is mandatory and consuming the result is NOT sufficient.
The first 'compute' reported 0us at every n while returning a correct n2 --
clang closed the loop to a multiply. Only an opaque call restored the curve.
Adds lang/tests/bench/fitprobe.el as the fitter's known-good/known-bad set,
so the classifier is provable without depending on a real bug existing.
Marks DESIGN.md 1.3 stale: test_compiler 3.58s -> 0.03s (119x).
el_val_t math_log(el_val_t f) { return el_from_float(log(el_to_float(f))); }
el_val_t math_ln(el_val_t f) { return el_from_float(log(el_to_float(f))); }
Both were natural log, so math_log and math_ln were the same function.
log10(100) returned 4.605 instead of 2.
Three sources already agreed it should be base-10 and were being contradicted
by this one line:
- runtime/math.el:55 "// math_log — base-10 logarithm."
- el_seed.c:1278 __log_f -> log10() (the path math.el actually calls)
- tests/native/test_math.el:133 asserts log10(100) == 2
FOUND BY THE NEW TEST FRAMEWORK ON ITS FIRST RUN (el #133). The assertion had
been sitting in the suite the whole time; nothing could report it. The old
harness printed "N passed, M failed" with no per-test detail, and half the
suites were not compiling at all — so a failing assertion in a suite nobody
could run was indistinguishable from no failure.
That is the entire argument for the framework, demonstrated on day one: this is
not a bug the framework introduced, it is a bug the framework made VISIBLE.
Verified: tests/native/test_math.el goes 12/13 -> 13/13, math-log passing.
Completes the Phase 1 runner and migrates the 11 test files onto it.
- forward-declare the registry accessors in the test preamble; they are
defined at the end of the unit but the El runner is compiled in between
- eltest.el: explicit trailing return in the void emit_* helpers, which
otherwise lower to 'return println(...)' and fail to compile
- test files import runtime/eltest.el explicitly, using the language's own
textual import mechanism rather than compiler-side auto-injection
- DESIGN.md 6.5: gate on allocation COUNT AND BYTES, not count alone
Verified: self-hosting fixpoint byte-identical (gen2 == gen3). 6 of 11
suites run and report per-test timing. The other 5 fail to COMPILE, and
fail identically under the committed compiler -- pre-existing breakage
this framework makes visible for the first time.
THE BUG. str_char_code() and str_slice() each called strlen() on every
invocation. The lexer walks source one character at a time, so every character
access rescanned the whole remaining input: O(n) per character over n
characters = O(n^2).
el_val_t str_char_code(el_val_t s, el_val_t i) {
...
int64_t n = (int64_t)strlen(str); // <- O(n), every call
if (idx < 0 || idx >= n) return 0;
return str[idx];
}
HOW IT WAS FOUND. Not by reading code — by sampling the running process, which
is the same method that resolved tonight's engram outage after four wrong
theories. A geometric sweep of synthetic sources showed wall-clock rising 3.0x,
3.0x, 4.0x, 4.14x per doubling (converging on 4x = quadratic), and a stack
sample put 779 of 779 samples inside lex(), every one bottoming out in
_platform_strlen via str_char_code and str_slice.
THE FIX. Remember the length instead of recomputing it. The subtlety is
INVALIDATION: El strings are arena-allocated, so a freed pointer can be reused
for a different string at the same address, and a naive pointer-keyed cache
would hand back a stale length and read past the end of the new string —
trading a performance bug for a memory-safety one. So entries carry a
generation, a hit requires pointer AND generation to match, and every path that
frees or mutates a runtime string bumps the generation: el_arena_pop,
seed_request_end, __str_set_char. Stale entries cannot be believed; they miss
and recompute.
MEASURED, same host, same inputs:
n(fns) before after
512 0.10s 0.01s
1024 0.37s 0.02s
2048 1.51s 0.03s 50x
the compiler's own 422 KB source concatenated (DESIGN.md's 3.58s case):
3.55s -> 0.03s 118x
The speedup GROWS with input size, which is the signature of removing a
complexity class rather than a constant factor. After the fix each doubling
adds ~0.01s: linear.
CORRECTNESS, verified rather than assumed:
- byte-identical output on every sweep input (n = 128..2048)
- byte-identical output on the 422 KB compiler concatenation
- byte-identical output on tests/runtime/string_test.el
- self-hosting fixpoint byte-identical
- new tests/runtime/str_cache_test.el: 17 assertions covering bounds, empty
strings, negative indices, slice clamping, distinct strings not sharing a
cached length, 1000 interleaved strings forcing cache-slot collisions, and
a grown string not reporting its old length. All pass.
This is the defect that made dist/soul.c a committed artifact: elc could not run
in CI because it needed 24 GB+ and minutes. It needs neither now.
Replace the hardcoded test harness main() with a generated static registry
and index-based accessors, and move all reporting into runtime/eltest.el.
The old harness inlined direct calls into main() and counted assertions in
two globals. That shape cannot report which test failed, how long any test
took, or whether a test ran at all -- a misspelled registration reported
success for a test that never executed.
- assertions record into per-test state instead of global counters
- registry table emitted at compile time; discovery strictly precedes
execution, which is what later enables --list, filtering and sharding
- per-test wall timing on CLOCK_MONOTONIC, taken in C around the call
- runner in El: structured NDJSON events as source of truth, human output
rendered from the same fields
Implements the three primitives the test-framework design (DESIGN.md §6.5)
requires for gating on growth curves: el_alloc_count, el_alloc_bytes,
el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per
the project's C-builtin recipe.
WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build
failure, which means the signal cannot flake. Wall-clock needs warmup,
statistics, and a quiet machine; on shared CI it is unusable as a gate.
Allocation counts are perfectly deterministic — same input, same number, every
machine, every run. Fit them against n and a complexity regression becomes a
build failure with zero noise.
All four runtime string allocators (el_strdup, el_strbuf, and their _persist
variants) funnel every allocation the language performs, so instrumenting there
counts everything.
WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate.
Measured with two El programs, one allocating once per item, one rebuilding its
accumulator each iteration:
n linear allocs / bytes quadratic allocs / bytes
100 100 / 290 100 / 5,150
200 200 / 690 200 / 20,300
400 400 / 1,490 400 / 80,600
800 800 / 3,090 800 / 321,200
The quadratic program's allocation COUNT is exactly linear — identical to the
healthy one. Counting allocations alone would have missed it completely. Bytes
catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 ->
converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0.
That shape — count linear, per-allocation size growing — is the classic
accidental quadratic, and it is exactly elc's defect: quadratic allocation
VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt
one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across
both; RSS and wall-clock were just the two ways it surfaced.
el_peak_rss is exported for context and is explicitly NOT a gating signal — it
is perturbed by allocator internals, the page cache, and the OS. Gate on the
deterministic numbers; report the physical one.
Counters are unsynchronised by design: this is measurement, and a lock would
change the thing being measured. Exact on the single-threaded compile path,
approximate under threads.
Neuron
will.anderson