lang/AGENTS.md said the collapse was 'not yet compiled into the MCP server'.
Verified against the live tool surface: it is exactly the nine ops. Noted that
think's faculty parameter and ground's minted edge are both documented as the
wrong shape.
The line references were correct but silently implied the code was on dev.
It is on design/correspondence-and-censorship (a8845e1). On dev,
co_registration is still at engram_geometry.h:79 with its original comment
and still unread by anything.
The docs described a mind made of subsystems — a grounding subsystem, a wonder
manifest, a dreamer on a beat, faculties as arguments to one call. Each of those
is a supervisor invented for something that should be a property of the
substrate, and two of the documents carrying them are load-bearing for a build
agent: cognitive-architecture.design.md says "a build agent executes from this
doc", and tools/api-reshape/README.md marks the refuted shapes PROVEN on a live
clone.
Corrections carried, per lang/spec/correspondence-and-censorship.md (PR #149)
and lang/spec/runtime-ownership.md:
- Grounding is not a subsystem — it IS the edge weight. grounded-by as a
relation type should not exist; grounding is a property of a relation, not a
relation between nodes. Never computed on demand.
- Faculties are operations, not parameters. reason changes the estimate, induce
changes the parameters, abduce changes the structure — a write, which
GeoGradient cannot express. A write is not a parameter of a read.
- Wonder is the boundary, not a manifest. Curiosity is wonder crystallized at a
nucleation site: one thing at two phases. Removed wonder from the operator
table in AGENTS.md.
- Consolidation is ambient, not scheduled. A brain has no cron job. The presence
of a ticker is the diagnostic.
- co_registration is deprecated — it averaged a per-edge property into a region
scalar, so opposing sites cancelled. GeoEdge.discord replaces it. Nothing new
may read it.
- In an immutable substrate, any mechanism that refuses a write is either
redundant with immutability or an epistemic constraint misfiled as a
protective one.
The two design docs are marked superseded-in-part with the refutation at the
point each claim is made, not rewritten. Preserving what was argued down is the
point of an immutable record.
Also measured and corrected while verifying the above: engram/README.md
documented a Rust engram-core crate on sled with "flat cosine scan until scale
demands HNSW" — there is no Rust in engram/ and HNSW is the index; lang/releases/
no longer exists, so both README.md and AGENTS.md pointed at a deleted path for
the authored runtime; language.md listed the engram_* and http_* runtimes as
stubs. Added language.md §20 for geometry-as-a-value, realizers and transduce
(#144), which had landed with no spec coverage.
Documentation only. No .c, .h, or .el file is touched.
engram_scan_nodes_emb_json has existed as a builtin with NO ROUTE. The
embeddings — the actual positions every distance, angle, membership and
grounding is computed from — were unreadable from outside the process.
That is not a missing convenience. It means every claim about the
coordinate frame was unfalsifiable from the API: whether the space is
isotropic, where the centering offset sits, what the origin is, whether a
node carries geometry at all. You cannot verify a coordinate system you
cannot see, and a system whose frame cannot be checked is exactly the
shape this codebase spent 2026-08-16 removing everywhere else.
GET /api/nodes/emb?limit=&offset=. Read-only, paged, no writes.
Measured consequence of having it: the value manifold and the love
component manifold were both decomposed, null-controlled against random
node sets drawn from the same graph, and several published claims were
retracted because the geometry contradicted them. None of that was
possible before this route existed.
lang/AGENTS.md:71-77 gives four steps for adding a C builtin and ends at
'confirm the self-host fixpoint is byte-identical'. No step asks for a test.
The only 'verify' in the file is that fixpoint, which proves the COMPILER
REPRODUCES ITSELF and says nothing about whether the builtin works — so the
recipe reads as complete while having checked nothing about the thing just
added.
Measured on 2026-08-16: engram_node_set_emb, engram_curiosity_json and
dream_set_handler were all added in a single session with zero tests, by an
agent following this recipe. Separately a UTF-8 fix was written and tested
and THE TEST PASSED ON THE UNPATCHED BUILD — the real defect was elsewhere,
and only building the pre-fix binary exposed it. Without a negative control
that fix would have merged as verified.
Adds step 5 with the two failure shapes actually encountered: a test that
never exercises the change (a route default bypassed the code under test),
and an induction that loses a race (curl --max-time left BOTH builds alive;
only SO_LINGER 0, a real RST, reproduced it). Plus the port-binding check,
because a stale instance answering has silently produced false results here
more than once and pkill -f does not reliably match argv './engram'.
Documentation only. Does not touch the (a) split-the-C / (b) close-the-
compiler-gap question, which is a separate decision.
There was no SIGPIPE handling anywhere in this runtime: no signal
disposition, no MSG_NOSIGNAL, no SO_NOSIGPIPE, and send() called with bare
flags. The default disposition of SIGPIPE is to TERMINATE THE PROCESS, so
any client that hangs up mid-response takes the whole engram with it.
MEASURED, and it is not hypothetical. Production has restarted 254 times
since 2026-08-13T19:37 at a flat ~10 minute cadence:
17:05:18 17:15:29 17:25:38 17:35:50 17:46:00 17:56:10 18:06:22 18:16:30
Intervals of 10m09s-10m12s, not 10m00s. That excess is the whole story:
ai.neuron.engram-tick has StartInterval 600, and engram-tick.sh:13 calls
curl -s -m10 -X POST .../api/tick
The beat does not finish within 10s over 13,634 nodes, so curl waits its
full timeout and closes. The engram then writes the tick response to a dead
socket, takes SIGPIPE, and dies. launchd KeepAlive restarts it, so the
failure presents as a mysterious restart rather than a crash — and
~/.neuron/logs/engram.log records nothing but "[http] listening on" 254
times, with no exit reason. launchctl list confirms the last exit as -13.
Root cause is one level out: consolidation had no owner, so an external
ticker was created to poke it, and the ticker is what kills it. The fix
here does not address that; it makes the process survivable while it is
addressed.
Two layers, because neither alone is portable:
- SO_NOSIGPIPE per accepted socket (Darwin/BSD) and MSG_NOSIGNAL per send
(Linux), so the signal is never raised for socket writes at all.
- A process-wide SIG_IGN backstop, installed once and idempotent, for
platforms and paths with neither. With the signal ignored, send()
returns -1/EPIPE and the existing error path closes the connection.
Also retries send() on EINTR, which the previous loop treated as fatal.
This is an exemption in the sense of lang/spec §8: the write never checked
whether the peer was still there, and the consequence of not checking was
fatal rather than merely wrong.
A relation that keeps holding up strengthens; one that stops corresponding
decays. That is not analogous to grounding, it IS grounding — so it belongs on
the edge, not in a subsystem beside it. The graph was already the grounding
structure; this stops modelling it as something else.
Deleted, not refactored:
- cog_ground_edge and the `grounded-by` relation type. A grounded-by edge
models grounding as a relation BETWEEN nodes when it is a property OF a
relation. #147 fixed which endpoints that edge landed on and left the wrong
idea intact. Measured on the live store: the old path scored two nodes with
ZERO edges between them at 0.925237 and wrote an edge for it.
- ground() writing. It was a read that wrote — the eg_vindex_sync defect.
Three identical calls produced three writes to the same edge id.
- keystone_write_blocked. Its measured cost was 0.00% brier reduction over
n_trials 0 on the keystone: the loop never ran, so the self was never
calibrated and never falsifiable. Nothing replaces it — non-circularity of
the reference frame is temporal, not a permission.
- a graph predicate for "evidence downstream of itself", built and then
withdrawn. Reachability from the self region covers 89.2% of the live graph
(10,580 of 11,861 nodes), so any topological predicate marks nearly all
evidence tainted and degenerates into the total block censorship began as.
The vector, carried in a GRD1 block on the edge's own metadata:
factual, relational, associative (the existing hebb), polarity (SIGNED — near
zero is "no support", negative is "actively contradicts"; `inhibitory` is that
distinction crushed to one bit), provenance class, and a timestamp. Confidence,
recency, staleness and volatility are DERIVED at read and never serialized.
Decay is one model, not two: cog_decay_factor is the single implementation and
engram_temporal_decay now delegates to it — proven bit-identical over 24
(age, reinforcement) points.
Values reference: thirteen regions, aggregate MIN, binding value named. Measured
— the 13 have pairwise centroid cosine min 0.1525 / mean 0.5199 / max 0.9278, so
they demonstrably are not one region, and a mean would let agreement with twelve
mask a violation of the thirteenth.
Supersession versions the whole vector jointly, gated by consequence and
salience with no epsilon anywhere: floor crossings and sign changes only.
Polarity flips and provenance-class changes are inherently significant and
bypass the salience gate.
Also fixed: the frame contract. Descriptors are built over L2-normalized member
embeddings; think() and the grounding path were fitting RAW vectors against them.
Measured on the self region, same data, same 106 members:
magnitude 0.00283443 -> 0.536134, spread 18.7565 -> 0.930163.
Every fit score sat three decimal places below the 0.5 floors that gate on them.
assert() gates on both floors and computes still_held instead of returning a
hardcoded `true` — the old build reported still_held for a node that does not
exist.
Three nodes in the live graph carry labels truncated to exactly 80 bytes
ending in a lone 0xE2 — the first byte of an em-dash, cut mid-sequence.
jb_emit_escaped copied every byte >= 0x20 through verbatim, so those three
nodes made the ENTIRE /api/nodes/list response undecodable and no strict
parser could read the graph at all.
production binary 25,929,607 bytes INVALID at byte 89260
this build 26,338,389 bytes VALID, parses to 13,630 nodes
The damage was NOT written by this runtime. No 80-byte truncation exists
here (the only label truncation is engram_first_n_chars at 60), and the
content of those nodes is 2572 and 2746 bytes. Some other producer wrote
them. That is exactly why fixing a writer could not have fixed this: the
store already holds the damage, and it accepts data from importers, other
producers and older binaries.
So the fix goes where the promise is made. A serializer that emits JSON
owes valid UTF-8 whatever it is handed. jb_emit_escaped now validates each
multi-byte sequence before emitting any of it and substitutes U+FFFD for a
bad lead byte, a missing or malformed continuation, an overlong encoding, a
UTF-16 surrogate, or a codepoint above U+10FFFF. Invalid bytes are REPLACED
rather than dropped, so the damage stays visible in the output instead of
being silently papered over. Well-formed input is byte-identical to before.
Second, preventive and explicitly NOT the cause of the above:
engram_first_n_chars truncated by BYTES despite its name, so content with a
multi-byte character crossing byte 60 would produce a half codepoint in the
label. It now uses el_utf8_safe_len, which returns the largest byte length
<= max that does not split a codepoint. Bounded by bytes, not codepoints,
so existing labels never grow — they only stop splitting.
el_utf8_safe_len lives beside str_count_chars rather than in the engram
because the rest of el's string layer is already codepoint-aware
(str_count_chars counts codepoints, str_reverse walks codepoint lengths).
Byte truncation was the outlier and the concern is a string concern.
Note on the investigation: I first "fixed" the truncator and wrote a test
that passed on the UNPATCHED build too, because route_create_node passes
label = content when no label is supplied, so engram_first_n_chars is never
reached over HTTP. The test proved nothing. The real cause was only found
by decoding the actual failing bytes out of the live response.
engram_ground_json resolved each seed to a REGION, wrote the grounded-by
edge between the two regions' HUBS, and then echoed those hubs back in the
"claim"/"evidence" fields as if they were the caller's input:
const char* cid = C->hub_id ? C->hub_id : EL_CSTR(claim);
const char* eid = E->hub_id ? E->hub_id : EL_CSTR(evidence);
cog_ground_edge(g_engram_store, cid, eid, grounding, fw);
Three consequences, all measured against a clone of the live store:
1. The edge landed on a node the caller never named. Grounding 3b9ced5d
against 6edf8c79 wrote an edge on the hubs of their regions instead.
2. When both seeds resolve into the same region the support is circular
and scores near 1.0 for structural reasons, not evidential ones. Four
probe nodes written together landed in one region, and every grounding
among them returned 0.93-0.99 as if it were evidence. Two independent
agents hit this and reported 0.885 / 0.909 self-groundings as confident.
3. The echo concealed both: the response was indistinguishable from a
successful grounding of the ids that were passed in.
The region is HOW a claim is evaluated; it is not WHAT the claim is about.
So the edge now attaches to the requested ids, and the resolved hubs are
reported separately as claim_region / evidence_region.
Degeneracy is broader than hub == hub. Three circular shapes, all
previously invisible:
same-region both seeds resolve to one region
claim-region-is-evidence the evidence IS the hub of the claim's own
neighbourhood — measured at 0.98883
evidence-region-is-claim the mirror case
Each sets grounding to 0 and writes no edge. Circular support is not
support, and a grounding that is degenerate by construction must not
enter the graph as though it were evidence.
Verified:
6edf8c79 -> 6edf8c79 degenerate=same-region g=0 written=false
6edf8c79 -> d0406dfd degenerate=same-region g=0 written=false
ebc1413e -> 64cc96ef degenerate=false g=0.774563 written=true
64cc96ef -> ebc1413e degenerate=false g=0.802896 written=true
Legitimate grounding across distinct regions is unchanged and still
writes; only circular support is refused.
This is the same class as #142 and #146 — a value that looked like an
answer with nothing behind it — except here it was also writing that
non-answer into the canonical store.
engram_think_json built a NEUTRAL stance on every call — cog_stance_init
with a NULL id, all axis_gain 1.0, bias_dir NULL, reliability 0.5 — and
never loaded the stance the correspondence-beat had been persisting.
That mattered because the faculty enters engram_think ONLY through the
stance: axis_gain[k] warps the per-axis extents and bias_dir seeds the
steering direction. cog_stance_init stores the faculty NAME and nothing
reads it. So with a neutral stance, reason/abduce/induce/plan/analogize
were byte-identical output under different labels, and confidence was
pinned to 0.5 because GeoGradient.confidence IS stance->reliability.
The machinery already existed and only this call site ignored it.
engram_correspondence_beat_json resumes via cog_stance_from_node and
persists via cog_stance_to_node under "stance-<faculty>-<hub>". Every
beat's calibration was written and then thrown away on the next read.
Same defect as the NULL anchor fixed in #142, one line below: a neutral
argument collapsing a capability to a constant.
Resume the same id the beat writes, so learning compounds across beats and
cold boot. Fall back to neutral only when no stance exists — a genuine
uninformed prior rather than a discarded informed one.
Also emit stance_resumed, so confidence 0.5 from a learned-but-unreliable
stance is distinguishable from confidence 0.5 from "no stance exists".
That reporting gap is what let the neutral stance hide.
Verified against a clone of the production store (13,627 nodes):
before beat, no stance stance_resumed=false confidence=0.5
beat on a NON-keystone brier 0.00458568 -> 0.00329654
reduction 28.11%, n_trials 6000,
reliability 0.930726, stance_written=true
after beat stance_resumed=true confidence=0.930726
Confidence now equals the learned reliability instead of the uninformed
prior. The keystone self-anchor correctly stays at 0.5 — calibration is
deliberately refused on protected identity regions, and that refusal is
now visible as resumed=true with confidence unchanged, rather than being
indistinguishable from the bug.
STILL OPEN: with no learned bias_dir the faculties remain identical in
direction. What distinguishes abduce from induce geometrically is a
design decision about how Neuron thinks, not a plumbing defect, and is
deliberately left to Will.
The binary was stamped before dev advanced (vindex publication landed in
el_runtime.c and engram_vindex.c). Rebuilt against the merged runtime so the
committed compiler matches the runtime it ships beside. Fixpoint re-verified
byte-identical; test_compiler 82/82; engram/src/server.el still compiles and
still emits its 18 config declarations.
Migrates engram to the `program` block. 18 configuration variables that each
carried their default inline at the point of use now declare it in one place,
and engram declares itself a singleton.
The read sites lose their defaults entirely: `let v = env("X")` followed by
`if str_eq(v,"") { "default" } else { v }` collapses to `config("X")`. The
guide_env_or(key, dflt) helper is deleted -- its whole job was supplying a
per-site default, which is the thing being removed.
Fixes ENGRAM_DATA_DIR, which was the clearest instance of the defect. It was
read at six sites. Five were dead: `let dir_raw = env("ENGRAM_DATA_DIR")`
immediately shadowed on the next line by `engram_resolve_data_dir()`. The sixth
was live and defaulted to /tmp/engram, contradicting the canonical resolver's
$HOME/.neuron/engram -- and its consumer is the pre-destructive reseed backup,
so with ENGRAM_DATA_DIR unset the safety copy was written to ephemeral storage
while the store it protected lived elsewhere. All six now go through
engram_resolve_data_dir().
ENGRAM_DATA_DIR is deliberately NOT declared in the program block, and the
source says why: engram_resolve_data_dir() already owns it, and a second
declaration would give it two owners that can disagree -- recreating the exact
defect being removed here. A variable belongs in the block when the block would
be its only owner. HOME stays a raw env() read; it is an environment fact, not
configuration.
singleton: "engram" matters more than it looks. Today a second engram whose
bind() fails merely returns from http_serve -- after it has already replayed
the WAL and written boot-time backup files -- and then exits 0, indistinguishable
from a clean run. That is how two instances came to share one data dir. Verified
that the second instance now refuses before any side effect: with instance 1
holding the lock (lsof pid, shell pid, and lock file contents all agreeing at
5946), the second start named that pid, exited 1, and left the data directory
untouched.
Verified by bijection on the generated C: 18 config() reads, 18 declarations,
no read without a declaration and no declaration without a read. Three bad Int
values are reported in a single run rather than costing one restart each.
ENGRAM_API_KEY keeps its permissive empty default, which disables auth -- that
is pre-existing behaviour and changing it is out of scope. The source marks
making it `required` as the obvious hardening follow-up.
server.el declares a `program` block, which the previously committed elc cannot
parse. Without this the tree is internally inconsistent: source in the repo that
the compiler in the repo rejects.
This is the documented re-stamp from BOOTSTRAP.md / AGENTS.md, and its
precondition is met -- the self-hosting fixpoint was verified byte-identical
(stage3 output == stage2 output) both before installing and again with the
installed binary. tests/native/test_compiler.el passes 82/82 against it.
Two pre-existing failures are unchanged and are NOT from this work, confirmed
by rebuilding them against the original runtime: test_env's
"state_keys returns JSON array" fails identically before and after, and
test_json/test_state fail to link on symbols (json_build_array, state_has) that
were never prototyped -- the same class of gap as config(), which this branch
fixed because it blocked the build.
El's units of encapsulation are the function and the module. Neither can hold
a concern that belongs to the process, so each one had been expressed the only
way it could be -- as a convention: call this at every site. Conventions of
that shape do not hold. Measured here: zero process-identity guards at any
layer, 20 environment variables each with its default written inline at the
read site, 62 persist call sites, 10 per-route auth checks. One absence, four
times.
Step 0 first, because the premise was wrong. El was believed to have no
middleware or effect mechanism. It has one, and it is already load-bearing:
codegen injects engram_boundary_beat at the entry of every @manager/@accessor
fn, decorators take arguments and stack, dharma_emit from a non-@manager fn is
a #error, and the cgi block injects el_cgi_init at the head of main(). So the
correct move was not to invent a mechanism but to generalize the seam that
already existed. The real gap is narrower and is now recorded: the seam is
prologue-only and its callee is a fixed builtin.
Adds a `program` block -- the third program-level declarative block. cgi and
service declare what a program may do; program declares what it is.
program "engram" {
singleton: "engram"
env ENGRAM_BIND: String = ":8742"
env GUIDE_PORT: Int = "8771"
}
singleton takes an exclusive flock before any user statement runs and refuses a
second start, reporting the holder's pid. It is a lock rather than a pidfile so
the kernel releases it on death including SIGKILL -- no stale state, and so no
"delete the lock file to get unstuck" ritual, which would itself be a
convention. It reports the pid because "already running" is not actionable; a
pid is. That is the direct answer to a stale process surviving a pkill and
going on answering probes.
env entries resolve once at startup -- environment wins, declaration supplies
the fallback -- and validate as a whole, reporting every problem at once rather
than costing one restart per variable. config("X") for an undeclared X is
fatal, because an advisory schema is just another convention. Programs without
a program block are unaffected, so migration is per-program.
Only one keyword is added. `config` and `env` could not become keywords -- both
are real identifiers in the tree -- so the block's fields are read as
identifier token values by its own parse loop and stay usable everywhere else.
The init function is emitted at the block site and called from main() rather
than inlined into main(). The live backend is codegen_streaming, which emits in
source order and cannot hold the entry list alive until main(); this way only a
single bool has to survive.
Also fixes: config() was defined in el_runtime.c but never prototyped in
el_runtime.h, so any el program calling it failed to compile under C99.
Spec: section 18 documents what shipped. Section 9 is corrected -- it claimed
decorators had no structural meaning, which has not been true for some time.
Section 19 designs durability-as-an-epilogue-effect and route authorization
and states plainly why neither is implemented here: both land in files under
concurrent modification, and the prerequisite for both is lifting the seam
from prologue-only to prologue/epilogue.
Self-hosting fixpoint verified byte-identical.
#141 let signal enter as geometry and it worked, but it was placed at the
CONSUMER and said so in its own commit message. This is the correction.
Three defects, all of them placement:
1. It sat in the engram. Ingest is a LANGUAGE concern — every el program
touching any modality needs it, and the engram is merely one el program
that happens to hold a graph. The geometry surface is now defined in
el_runtime.c immediately ABOVE the engram section and depends on nothing
inside it. Delete the entire engram and geometry still enters el.
2. It marshalled the vector as a hex STRING, because el had no first-class
geometry value — which reintroduced text as the TRANSPORT medium one layer
below the problem being fixed. Geometry is now an el value: a magic-tagged
heap object carried in el_val_t, same discipline as List/Map. Hex survives
only as an adapter at the edge, which is all an encoding should ever be.
3. It needed an arbitrary `dim <= 8192` bound purely to size an allocation
from a caller's CLAIM about a string's length. A value carries its own
width, so the width is derived and never asserted. The bound is gone, not
raised — there is nothing left to validate.
Language surface, none of it engram-prefixed: geometry_new / _dim / _is /
_get / _set / _norm / _free, geometry_from_f32le_hex + geometry_to_f32le_hex
as the wire adapters, realizer_register(modality, fn_name), realizer_has, and
transduce(signal, modality) -> Geometry.
REALIZERS ARE DECLARABLE IN EL. This is the part that makes the move real
rather than nominal: registration resolves a name with dlsym against the
running binary, the identical mechanism http_set_handler already relies on,
because every el `fn name(...)` compiles to a global C symbol with that exact
name. So an ordinary el function IS a realizer and a new modality needs no
runtime patch. Verified end to end in lang/examples/transduce.el: an el-defined
tone_realizer is registered by name, transduce dispatches to it, and the
signal demonstrably reaches it (distinct signals produce distinct geometry).
A modality with no realizer transduces to NOTHING. There is deliberately no
built-in realizer, not even for text — silently embedding a description of a
signal and calling that perception is the exact defect this ends.
engram/src/server.el is migrated: POST /api/nodes decodes "emb" hex exactly
once, at the edge, into a Geometry, and everything below that line moves
geometry. The wire is unchanged because production clients speak it. "dim" is
now an ASSERTION about the vector, not the source of its width; disagreement
is a rejected ingest, not a silent reinterpretation.
#141's engram_node_set_emb becomes a DEPRECATED WRAPPER over
geometry_from_f32le_hex + node_attach_geometry — kept only because the runtime
ships as an SDK asset and a downstream binary may link the symbol. Its exact
contract, negative cases included, is preserved and re-verified.
ingest.el's `fn transduce` is renamed transduce_manifold. Mechanically it had
to yield the name (duplicate C symbol, a hard compile error, measured). But it
was never signal->geometry: it chunks already-extracted content into a node+edge
manifold, one layer up, and had taken the name belonging to the primitive
underneath it. Behaviour unchanged.
PROPERTIES FROM #141 PRESERVED, each re-measured on a scratch engram (:8971,
never prod :8742):
- off-dimension vectors stored but NOT indexed — the HNSW build loop still
filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is
durable and addressable without perturbing the 768-dim canonical index
- geometry makes a node ineligible for embed_backfill: after backfill the
64-dim voice node was still 64-dim while the text control acquired 768
- the create response reports whether geometry landed, and the node document
always emits emb_dim and embedded
Read-back with control and negatives, all verified against a PID-confirmed
fresh binary: geometry node emb_dim=64 embedded=true / emb_set=1; text-only
control emb_dim=0 embedded=false / emb_set=0; malformed hex, ragged length,
and dim-disagreement each emb_set=0.
Two compiler landmines found by reading the generated C rather than trusting a
successful build, both documented at their sites: elc lowers `a == b` to
str_eq unless both operand NAMES are in the per-function int-name set (which
does NOT propagate into nested if-expression blocks — the first cut would have
strcmp'd two integers as pointers on the first geometry-bearing request), and
`+` lowers to string concat when either operand is a user-defined call.
The crash (SIGTRAP in engram_activate -> eg_vindex_sync -> vindex_insert ->
_realloc) had three read paths mutating five process-global statics.
engram_activate, eg_knn_for_node (whose own comment says "No writes.") and
engram_geo_reify_run_json all called eg_vindex_sync, which frees the index,
reallocs the seen-map and inserts — on a read.
Three moves, in decreasing order of how much they dissolve:
1. Misfiled scratch is not shared state. visited/visit_epoch/visited_cap
were never owned by the index; they are one traversal's local, hoisted
into struct VIndex as an allocation optimisation. They want neither a
lock nor a capability nor a pool — just to go back in the call frame.
Two concurrent READS stomped each other purely because of this.
2. const IS the capability. Once the scratch leaves the struct, search
reads and nothing else, so vindex_search takes a const VIndex*. That is
exactly what a capability-pointer ABI would have bought — a read path
physically cannot call vindex_insert, enforced by the compiler on every
future caller — for one qualifier instead of an ABI swept across
hundreds of builtins.
3. What survives is publication, not ownership. HNSW insert is NOT an
append: it rewires the neighbour links of already-existing elements and
reallocs elems[], so the store's append-only property does not transfer
to the index derived from it. eg_vindex_sync therefore splits into
eg_vindex_maintain (exclusive, sole mutator) and eg_vindex_view (shared,
returns const VIndex*). A read path may demand that a current snapshot
exist — a request to the owner, not a mutation by the reader.
Write-side owner: eg_vindex_note_embedded hooks the embedding-ASSIGNMENT
sites rather than the append sites, because a node with no embedding cannot
be in a vector index — embedding assignment is the event that owns index
membership. One O(log n) insert, no O(node_count) presence scan. This also
retires the "STALENESS (honest tradeoff)" note where a lazily-embedded
older node stayed invisible to route_nearest/autoconnect until a full
rebuild (the embed-gap #20 shape).
Evidence. The existing harness conflated two hazards, which is why fixing
half of it read as failure. Split into four:
single (3000 vec, ASan+UBSan) clean -> clean
readers (4 readers, no writer, TSan) RACE -> clean
unsynchronized (writer+reader, bare) race -> race, expected forever
published (owner + 4 readers) n/a -> clean, 3000/3000 landed
RESULT: PASS. recall@10 = 0.9365 at ef_search=128 (gate >= 0.90);
determinism byte-identical across two independent builds.
The unsynchronized half is now permanently expected to race, deliberately:
it is the executable proof that the boundary must live above the data
structure, not inside it.
fb32d15's guard is KEPT, correcting this design's own section 5. Measured,
it guards TWO structures and only one was converted here: g->nodes/g->edges
are realloc'd in place (el_runtime.c:7618,7629) and engram_activate_inner's
embed-backfill writes n->emb through exactly such a borrowed pointer.
Deleting the guard reintroduces a measured 11171->9579 edge loss. Its
comment is narrowed to the RAM graph and the deletion precondition named.
That corrects the ordering claim too: the residual is not one ABI that
dissolves everything at once, it is a PROPERTY applied per structure.
Residues evaporate in the order the property is applied, and a residue
whose structure has not been converted must be left standing.
Promotes the two throwaway sanitizer harnesses used to diagnose the
2026-08-16 soul crash into engram/test/ so the bug cannot silently regress.
The harness has two halves and the PAIR is the point — it is what localises
the defect to concurrency rather than to HNSW logic:
single 3000 clustered vectors, one thread, ASan+UBSan. The CONTROL.
Must always be clean. During diagnosis this cleared all 13,820
real dim-768 vectors from the live store, which DISPROVED an
inspection-derived hypothesis about an out-of-bounds
reverse-link write at engram_vindex.c:340.
concurrent writer + reader on one shared index, TSan. Currently reports a
race at engram_vindex.c:195 (visited_reset) reached from both
vindex_search and vindex_insert, because VIndex still owns its
visited[]/visit_epoch scratch — so even two concurrent READS
corrupt each other's traversal.
Verified: half 1 passes, half 2 reproduces the race.
Gated on EXPECT_RACE, default 1, so the concurrent half documents the known
defect without failing the suite today. When the visited set moves to a
per-query checkout pool (hnswlib VisitedListPool style — NOT thread_local,
since http_worker is a thread per connection and a __thread buffer would leak
~55KB per connection), flip EXPECT_RACE=0 and it becomes a real gate.
The soul daemon had two engram callers and only one of them locked.
soul.el:729 starts the HTTP server via http_serve_async (spawning
http_worker threads); soul.el:731 then runs awareness_run() on the MAIN
thread. awareness.el's perceive() -> engram_activate_json() ->
engram_activate() -> eg_vindex_sync() -> vindex_insert() mutates the same
g->nodes/g->edges and the process-global _eg_vindex HNSW index that the
workers touch. g_engram_req_lock existed to serialize exactly this, but it
was only ever taken inside http_worker: engram_req_lock/engram_req_unlock
appear in ZERO .el sources, so the awareness loop ran lock-free beside the
workers on every tick (SOUL_TICK_MS=1000).
Result was a crash-loop under launchd KeepAlive: five crashes in ~4 minutes
on 2026-08-16 with varying faulting frames -- search_layer<-vindex_insert
<-eg_vindex_sync, engram_activate, abort, and one inside xzm_realloc's own
freelist. Varying sites plus a fault in allocator metadata means heap
corruption. The SIGSEGV address 0x65646f4e6d617267 is little-endian ASCII
"gramNode": string bytes dereferenced as an Elem vector pointer.
Diagnosed by bisection rather than inspection:
- Replaying all 13,820 real dim-768 vectors harvested from the live store
through the index single-threaded under ASan is 100% clean, which rules
out an HNSW logic/bounds bug.
- Two threads on one index trip ThreadSanitizer immediately at
engram_vindex.c:195 (visited_reset), reached from both vindex_search and
vindex_insert. VIndex keeps a SHARED visited-epoch scratch buffer, so
even two concurrent READS corrupt each other's traversal and walk bogus
element indices.
So this is purely a concurrency defect, not an HNSW logic error. (An
inspection-derived hypothesis about an out-of-bounds reverse-link write at
engram_vindex.c:340 was disproved by the single-threaded run.)
Fix: a thread-local ownership depth (_eg_req_depth) lets engram entry points
self-guard. engram_activate() becomes a wrapper over engram_activate_inner()
that acquires g_engram_req_lock when called with depth 0 (the awareness
thread) and passes through when depth > 0 (nested inside an http_worker that
already holds it), so the non-recursive mutex cannot self-deadlock. The depth
is a plain counter, never a recursive-mutex count, preserving
engram_self_reify_beat_json's contract of genuinely releasing the lock
mid-beat.
engram_think_json passed NULL as the anchor. NULL is not "no opinion":
engram_think re-origins at `anchor ? anchor : region->centroid`, so NULL
means "read from the centroid" — and the centroid is the one point where
the gradient is zero by construction. r = x - centroid = 0, so every axis
projection is 0, grad is 0, and direction takes the "at rest" branch at
engram_cognition.c:137.
Measured consequence: EVERY faculty returned an identical null result,
differing only in its label —
{"direction":[0,0,0,0,0,0,0,0],"spread":0,"magnitude":1,"confidence":0.5}
magnitude 1 is membership evaluated at the centroid, spread 0 is its
distance to itself, confidence 0.5 is the stance fallback. The geometry was
never at fault: /api/drift computes real values (centroid_sep 0.104,
core_disp 0.045) over the very same 87 members. Neuron could not think
because the read was always taken from the region's own centre.
The seeds choose WHICH region; they must also supply the VANTAGE. Anchor at
the first resolvable embedded seed — the same seed eg_geo_build_desc infers
dim from, so the two can never disagree. One seed still yields a real
gradient because the descriptor expands to that seed's neighbourhood, so
the seed's position is distinct from the neighbourhood centroid.
The vector is COPIED, never borrowed: g->nodes is realloc'd in place on
append, so a borrowed EngramNode* dangles across any concurrent write.
Verified against a clone of the production store (13,616 nodes / 37,865
edges):
self anchor n_support 87 magnitude 0.00282 spread 18.79
values hub n_support 28 magnitude 0.00318 spread 17.72
with distinct unit direction vectors. Previously both returned the zero
vector with magnitude 1 and spread 0.
STILL OPEN, now isolated by this fix: all five faculties return identical
numbers and confidence stays 0.5, because cog_stance_init is passed NULL
for the stance and the faculty enters the computation only through the
stance's axis_gain[] and bias_dir. The faculty label is inert until a
stance is loaded — which is what learn()'s correspondence-beat calibrates.
Same shape as this bug: a neutral parameter collapsing a capability to a
constant.
No ingest path could carry a vector. engram_node/_full/_layered take text
only, and a node acquired an embedding solely via engram_embed_backfill
DERIVING one from n->content. That made text the mandatory entry medium:
any non-text modality had to be described in prose first, so the geometry
we then reasoned over was the geometry OF THE DESCRIPTION, not of the
signal. Measured: POST /api/nodes accepted an "emb" field, returned 200
with a fresh id, and stored nothing — emb_dim=None, embedded=false.
engram_node_set_emb attaches a vector to an existing node. Off-dimension
vectors are stored but not indexed (the HNSW build loop already filters on
emb_dim), so modality geometry is durable and addressable without
perturbing the canonical index. Setting emb also makes the node ineligible
for embed_backfill, so a realizer's vector is never overwritten by a
text-derived one.
Two reporting fixes ride along, because both are how the drop stayed
invisible: the create response now reports emb_set instead of being
success-shaped regardless, and the node document now always emits emb_dim
and embedded — without which a genuine ingest drop and a mere reporting
gap are indistinguishable.
Verified live: voice node emb_dim=64 embedded=true; text control emb_dim=0
embedded=false; malformed hex, length mismatch and dim<=0 all reject.
KNOWN PLACEMENT DEFECT: this is at the consumer. Ingest is a language
concern, not an engram feature — every el program touching any modality
needs it. The vector also marshals as a hex STRING because el has no
first-class geometry value, which reintroduces text as the transport
medium one layer below the problem being fixed. The durable shape is
geometry as an el value plus declarable realizers, after which the engram
stops having an ingest concept at all. Landing this as the verified probe
that proves the path.
char* result = el_strdup_persist(e ? e->value : ""); // never freed
pthread_mutex_unlock(&_state_mu);
char* copy = el_strdup(result); // arena-tracked
return el_wrap_str(copy);
Two copies were made. `result` existed only as the source for `copy` — never
returned, never freed — and el_strdup_persist bypasses the arena BY DESIGN
("state_set, engram internals"), so arena-pop could never reclaim it. Every
state_get leaked its full value string, permanently.
MEASURED: 200,000 state_get calls against a 64-byte value.
before 15 MB peak RSS growth (~75 bytes/call — the value plus overhead)
after 0 MB
IMPACT. The soul's awareness loop has 68 state_get call sites and ticks every
200ms. Live measurement before the fix: RSS climbing 112 MB per 20s, about
19 GB/hour, in awareness_run -> one_cycle -> perceive, while node_count stayed
flat at ~13,479 — growth with no data behind it. It drove the host from 20 GB
free to 4.3 GB in roughly an hour.
WHY NOW, since the code is old: the soul used to restart constantly (no
write-through, divergent graph, 2.11 GB). Stabilising it (neuron #162) let it
stay up long enough to accumulate. The fix did not cause this leak; it removed
the crashes that were hiding it. Same pattern as the test framework surfacing
math_log — the defect was always there, something finally made it visible.
Found by Ishikawa rather than by reading the nearest code: method (arena
push/pop IS correctly paired per tick), material (node count flat, so not data
growth), environment (19 GB/hr / 18,000 ticks = ~1.1 MB per tick, so per-tick
not one-shot), machine (an allocator that bypasses the arena) — which is where
the evidence pointed.
el_strdup tracks into the thread-local arena, which touches no shared state, so
taking the single copy under _state_mu is safe and removes the temporary
entirely.
Verified: self-hosting fixpoint byte-identical; state round-trip correct for
hit, miss, and overwrite.
Adds tests/native/test_lexer_scaling.el, the regression gate for el #132.
Both directions are proven on LIVE workloads, not synthetic series:
healthy per-character scan 1821 3251 6007 10422 us -> O(n) PASS
rescan-from-zero (the #132 shape) 922 3667 13524 44792 -> O(n^2) FAIL
A gate only proven to pass is decoration. The quadratic specimen exists so
the gate is proven to FIRE.
Also fixes elb_spread_ok to judge the ASYMPTOTIC TAIL (last three ratios)
rather than the whole sweep. Measured on a genuinely linear scan the ratios
ran 3.37 2.92 1.76 1.65 -- the head looks quadratic because it is cold
cache, the tail is the truth. Whole-sweep spread rejected correct data. A
complexity bound is an asymptotic claim and must be judged asymptotically.
That fix came from the classifier refusing to rubber-stamp my own bad
measurement: it reported INDETERMINATE on an unwarmed sweep rather than
passing it. Warmup is now taken and discarded at every sweep point.
Reverts the == workarounds in test_elbench.el now that el #137 has landed;
the natural form generates no str_eq and all 13 fitter tests stay green.
The workaround remains -- the Plus arm is still open.
fn check(label: String, cond: Bool, want: Bool) -> Void {
if cond == want { ... } -> if (str_eq(cond, want)) SIGSEGV
}
Bool has always been an integer in the value model — type_to_c maps Bool to
"int", and el_runtime.h states "Bool -> el_val_t (0 = false, nonzero = true)".
But Bool names were registered NOWHERE: build_int_names_for_params tracked Int
and Float params, and the `let` path tracked Int and Float bindings. Neither
knew about Bool.
So comparing two Bools fell through to str_eq, which dereferenced 0 or 1 as a
char* and segfaulted immediately.
This is the third instance of one family found tonight, after el #137 (a call
on either side of == poisoned the operator) and el #136 (a missing import
compiled clean). All three are the same shape: something the compiler could not
type, silently handled as a string.
Found while writing #137's own test harness — the first version of that harness
crashed on exactly this, on both the old and new compiler, which is how it
surfaced. A test harness that cannot compare two Bools is a good way to notice.
VERIFIED:
- the harness that segfaulted on every prior compiler (exit 139, no output)
now runs clean: 14 passed, 0 failed
- self-hosting fixpoint byte-identical
- the compiler's own generated C differs by 8 lines — only the intended
registration
- neuron's full soul amalgam regenerates in 424ms, exit 0, BYTE-IDENTICAL
- test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12
Adds tests/runtime/operator_typing_test.el, the 15-case suite from #137, so
this family is covered going forward rather than rediscovered.
let a: Int = 5
getint(5) == a -> str_eq(getint(5), a) SIGSEGV
getint(5) == 5 -> getint(5) == 5 fine
A function call whose return type codegen cannot infer poisoned the operator,
and a declared Int on the other side did not save it. str_eq then read an
integer as a char* and segfaulted. Only an integer LITERAL on one side forced
the numeric form, which is why the bug stayed invisible: the common case
happened to be safe.
The check required BOTH operands to be provably Int:
if is_int_expr(left) { if is_int_expr(right) { numeric } }
Loosening to OR is strictly safer, not a trade:
- when one side is a known Int, str_eq is ALWAYS wrong — it dereferences
that integer — while numeric comparison is at worst a wrong answer on a
program that was already ill-typed;
- when neither side is Int nothing changes at all, so string comparison is
untouched.
Found by the test-framework agent while building the benchmark harness; it
correctly declined to fix it mid-phase since it is a codegen semantics change.
VERIFIED, because a semantics change earns more than an assertion:
- 15/15 on a dedicated operator suite covering string literals, string vars,
string-returning calls, mixed var/call, and != in every combination. The
pre-change compiler scores 0/15 on the same file: it segfaults before
printing anything.
- self-hosting fixpoint byte-identical
- the ONLY difference in the compiler's own generated C is the intended one:
a nested if becoming two sequential ifs, in EqEq and NotEq. Nothing else
moved.
- neuron's full soul amalgam regenerates in 400ms, exit 0, output
BYTE-IDENTICAL at 1,270,212 bytes
- test_math 13/13, test_string 27/27, test_core 10/10, test_text 12/12 —
62 tests, 190 assertions, zero failures
NOT fixed here, same family, flagged for a decision: Bool PARAMETERS are not
tracked as int-like, so `cond == want` between two Bool params still lowers to
str_eq and segfaults. Found while writing this commit's own test harness — the
first version of it crashed on exactly that, on both the old and new compiler.
It needs the same treatment, and it wants its own change.
import "../../NOPE/does_not_exist.el"
compiled CLEANLY — exit 0, empty stderr, and a program silently missing
everything it imported.
resolve_imports did `fs_read(src_path)` and used the result without checking.
fs_read returns "" both for "file is empty" and "file does not exist", so a
typo, a moved file, or a relative path resolved from the wrong working
directory all produced a successful build of nothing.
It caused a real wrong conclusion during test-framework work: a bisection run
from a subdirectory where ../../runtime/ did not resolve produced ELEVEN
consecutive "successful" compiles that had included no runtime at all, and the
results were believed before anyone noticed.
Missing dependency, confident success — the same shape as a test suite
reporting pass for tests that never ran, and as a benchmark reporting 0us
because the optimiser deleted the loop.
fs_exists separates the two cases, so a legitimately empty file still resolves
to "" and is fine. A path that does not exist now prints the resolved path and
exits 1, which is what build scripts check.
Verified:
- bad import: exit 1 (was 0), message names the resolved path
- elc-cli.el still compiles, self-hosting fixpoint byte-identical
- neuron's full soul amalgam regeneration: exit 0, 405ms, output
byte-identical at 1,270,212 bytes
Adds el_black_box (inline asm, +r constraint, memory clobber) and
runtime/elbench.el: a growth-curve classifier that gates time AND
allocation-count AND allocation-bytes, failing if any exceeds its
declared curve.
Refusal is a first-class verdict. The classifier REFUSES rather than
classifying when the largest measurement is below the floor, or when a
series is hard-flat across an 8x input range -- the shape produced when
the optimiser deletes the work. Reporting O(1) there would be a
confident answer with nothing behind it. Disagreeing ratios report
INDETERMINATE rather than a guess.
Deviation from DESIGN.md 6.2, stated in the source: uses consecutive
ratios on a mandated geometric sweep rather than least-squares over
candidate curves. Ratios are directly interpretable on a doubling sweep
and need no floating point; the cost is weaker O(n) vs O(n log n)
separation, reported as an ambiguous band rather than guessed.
Documents the counter scope limit: engram_*.c and libcurl malloc are
NOT tracked, so a flat curve over engram/HTTP-dominated work is not
evidence of anything.
13 tests prove the classifier against real measured series from
fitprobe.el -- including that an accumulator's allocation COUNT is
linear while its bytes are quadratic, and that el #132's pure-CPU shape
reads FLAT on both allocation signals and is caught only by time.
el #131 instrumented the four string allocators, which meant list- and map-heavy
code reported ZERO allocations — a benchmark over lists would have been fitted
against a flat line and passed anything. Caught during framework work: a
"linear" specimen read 0 allocs until it was rewritten to allocate strings.
A gate is only as good as its blind spots are small, and a signal that silently
reads zero is worse than no signal: it produces a confident pass.
Now counted at every container allocation — ElList and ElMap bodies, their
backing arrays, the copy-on-write clones, and the realloc growth path.
Verified on an append loop (n = 100..800):
allocs 7, 8, 9, 10 +1 per doubling = O(log n) reallocations
bytes 2048, 4096, 8192, 16384 exactly 2x per doubling = O(n)
Both curves are what correct amortized growth should look like, and both read
zero before this change.
Known remaining scope, stated rather than left implicit: these counters cover
the runtime's own allocations. They do not see malloc inside engram_*.c or
libcurl, which is correct — the gate is for El-level complexity, not for
third-party memory behaviour.
el #132's quadratic (strlen per character in str_char_code/str_slice) is
pure CPU and allocates NOTHING. Measured on three controlled specimens:
specimen allocs bytes time
linear 2.00 -> O(n) 2.16 -> O(n) 2.05 -> O(n)
accum 2.00 -> O(n) 3.99 -> O(n2) noisy
compute FLAT FLAT 3.96 -> O(n2)
'compute' is #132's shape. A gate fitting only allocation count and bytes
classifies it FLAT and passes -- it would not have caught the defect it
was created for. The gate now fits time AND count AND bytes, failing if
any exceeds its declared curve.
Also: black_box is mandatory and consuming the result is NOT sufficient.
The first 'compute' reported 0us at every n while returning a correct n2 --
clang closed the loop to a multiply. Only an opaque call restored the curve.
Adds lang/tests/bench/fitprobe.el as the fitter's known-good/known-bad set,
so the classifier is provable without depending on a real bug existing.
Marks DESIGN.md 1.3 stale: test_compiler 3.58s -> 0.03s (119x).
will.anderson
bigmerge