26af149aa1f4f0b2941ac2863e9720fa18dfec00
32 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bigmerge
|
8ae163e8e5 |
lang: give cross-cutting concerns an owner instead of a convention
El's units of encapsulation are the function and the module. Neither can hold
a concern that belongs to the process, so each one had been expressed the only
way it could be -- as a convention: call this at every site. Conventions of
that shape do not hold. Measured here: zero process-identity guards at any
layer, 20 environment variables each with its default written inline at the
read site, 62 persist call sites, 10 per-route auth checks. One absence, four
times.
Step 0 first, because the premise was wrong. El was believed to have no
middleware or effect mechanism. It has one, and it is already load-bearing:
codegen injects engram_boundary_beat at the entry of every @manager/@accessor
fn, decorators take arguments and stack, dharma_emit from a non-@manager fn is
a #error, and the cgi block injects el_cgi_init at the head of main(). So the
correct move was not to invent a mechanism but to generalize the seam that
already existed. The real gap is narrower and is now recorded: the seam is
prologue-only and its callee is a fixed builtin.
Adds a `program` block -- the third program-level declarative block. cgi and
service declare what a program may do; program declares what it is.
program "engram" {
singleton: "engram"
env ENGRAM_BIND: String = ":8742"
env GUIDE_PORT: Int = "8771"
}
singleton takes an exclusive flock before any user statement runs and refuses a
second start, reporting the holder's pid. It is a lock rather than a pidfile so
the kernel releases it on death including SIGKILL -- no stale state, and so no
"delete the lock file to get unstuck" ritual, which would itself be a
convention. It reports the pid because "already running" is not actionable; a
pid is. That is the direct answer to a stale process surviving a pkill and
going on answering probes.
env entries resolve once at startup -- environment wins, declaration supplies
the fallback -- and validate as a whole, reporting every problem at once rather
than costing one restart per variable. config("X") for an undeclared X is
fatal, because an advisory schema is just another convention. Programs without
a program block are unaffected, so migration is per-program.
Only one keyword is added. `config` and `env` could not become keywords -- both
are real identifiers in the tree -- so the block's fields are read as
identifier token values by its own parse loop and stay usable everywhere else.
The init function is emitted at the block site and called from main() rather
than inlined into main(). The live backend is codegen_streaming, which emits in
source order and cannot hold the entry list alive until main(); this way only a
single bool has to survive.
Also fixes: config() was defined in el_runtime.c but never prototyped in
el_runtime.h, so any el program calling it failed to compile under C99.
Spec: section 18 documents what shipped. Section 9 is corrected -- it claimed
decorators had no structural meaning, which has not been true for some time.
Section 19 designs durability-as-an-epilogue-effect and route authorization
and states plainly why neither is implemented here: both land in files under
concurrent modification, and the prerequisite for both is lifting the seam
from prologue-only to prologue/epilogue.
Self-hosting fixpoint verified byte-identical.
|
||
|
Neuron
|
8e9d88fc01 |
runtime: publish the vector index instead of guarding it
El SDK CI - dev / build-and-test (pull_request) Failing after 10m59s
The crash (SIGTRAP in engram_activate -> eg_vindex_sync -> vindex_insert -> _realloc) had three read paths mutating five process-global statics. engram_activate, eg_knn_for_node (whose own comment says "No writes.") and engram_geo_reify_run_json all called eg_vindex_sync, which frees the index, reallocs the seen-map and inserts — on a read. Three moves, in decreasing order of how much they dissolve: 1. Misfiled scratch is not shared state. visited/visit_epoch/visited_cap were never owned by the index; they are one traversal's local, hoisted into struct VIndex as an allocation optimisation. They want neither a lock nor a capability nor a pool — just to go back in the call frame. Two concurrent READS stomped each other purely because of this. 2. const IS the capability. Once the scratch leaves the struct, search reads and nothing else, so vindex_search takes a const VIndex*. That is exactly what a capability-pointer ABI would have bought — a read path physically cannot call vindex_insert, enforced by the compiler on every future caller — for one qualifier instead of an ABI swept across hundreds of builtins. 3. What survives is publication, not ownership. HNSW insert is NOT an append: it rewires the neighbour links of already-existing elements and reallocs elems[], so the store's append-only property does not transfer to the index derived from it. eg_vindex_sync therefore splits into eg_vindex_maintain (exclusive, sole mutator) and eg_vindex_view (shared, returns const VIndex*). A read path may demand that a current snapshot exist — a request to the owner, not a mutation by the reader. Write-side owner: eg_vindex_note_embedded hooks the embedding-ASSIGNMENT sites rather than the append sites, because a node with no embedding cannot be in a vector index — embedding assignment is the event that owns index membership. One O(log n) insert, no O(node_count) presence scan. This also retires the "STALENESS (honest tradeoff)" note where a lazily-embedded older node stayed invisible to route_nearest/autoconnect until a full rebuild (the embed-gap #20 shape). Evidence. The existing harness conflated two hazards, which is why fixing half of it read as failure. Split into four: single (3000 vec, ASan+UBSan) clean -> clean readers (4 readers, no writer, TSan) RACE -> clean unsynchronized (writer+reader, bare) race -> race, expected forever published (owner + 4 readers) n/a -> clean, 3000/3000 landed RESULT: PASS. recall@10 = 0.9365 at ef_search=128 (gate >= 0.90); determinism byte-identical across two independent builds. The unsynchronized half is now permanently expected to race, deliberately: it is the executable proof that the boundary must live above the data structure, not inside it. fb32d15's guard is KEPT, correcting this design's own section 5. Measured, it guards TWO structures and only one was converted here: g->nodes/g->edges are realloc'd in place (el_runtime.c:7618,7629) and engram_activate_inner's embed-backfill writes n->emb through exactly such a borrowed pointer. Deleting the guard reintroduces a measured 11171->9579 edge loss. Its comment is narrowed to the RAM graph and the deletion precondition named. That corrects the ordering claim too: the residual is not one ABI that dissolves everything at once, it is a PROPERTY applied per structure. Residues evaporate in the order the property is applied, and a residue whose structure has not been converted must be left standing. |
||
|
bigmerge
|
bdc1f99fb9 |
runtime: guard engram activation against the unsynchronized awareness thread
The soul daemon had two engram callers and only one of them locked.
soul.el:729 starts the HTTP server via http_serve_async (spawning
http_worker threads); soul.el:731 then runs awareness_run() on the MAIN
thread. awareness.el's perceive() -> engram_activate_json() ->
engram_activate() -> eg_vindex_sync() -> vindex_insert() mutates the same
g->nodes/g->edges and the process-global _eg_vindex HNSW index that the
workers touch. g_engram_req_lock existed to serialize exactly this, but it
was only ever taken inside http_worker: engram_req_lock/engram_req_unlock
appear in ZERO .el sources, so the awareness loop ran lock-free beside the
workers on every tick (SOUL_TICK_MS=1000).
Result was a crash-loop under launchd KeepAlive: five crashes in ~4 minutes
on 2026-08-16 with varying faulting frames -- search_layer<-vindex_insert
<-eg_vindex_sync, engram_activate, abort, and one inside xzm_realloc's own
freelist. Varying sites plus a fault in allocator metadata means heap
corruption. The SIGSEGV address 0x65646f4e6d617267 is little-endian ASCII
"gramNode": string bytes dereferenced as an Elem vector pointer.
Diagnosed by bisection rather than inspection:
- Replaying all 13,820 real dim-768 vectors harvested from the live store
through the index single-threaded under ASan is 100% clean, which rules
out an HNSW logic/bounds bug.
- Two threads on one index trip ThreadSanitizer immediately at
engram_vindex.c:195 (visited_reset), reached from both vindex_search and
vindex_insert. VIndex keeps a SHARED visited-epoch scratch buffer, so
even two concurrent READS corrupt each other's traversal and walk bogus
element indices.
So this is purely a concurrency defect, not an HNSW logic error. (An
inspection-derived hypothesis about an out-of-bounds reverse-link write at
engram_vindex.c:340 was disproved by the single-threaded run.)
Fix: a thread-local ownership depth (_eg_req_depth) lets engram entry points
self-guard. engram_activate() becomes a wrapper over engram_activate_inner()
that acquires g_engram_req_lock when called with depth 0 (the awareness
thread) and passes through when depth > 0 (nested inside an http_worker that
already holds it), so the non-recursive mutex cannot self-deadlock. The depth
is a plain counter, never a recursive-mutex count, preserving
engram_self_reify_beat_json's contract of genuinely releasing the lock
mid-beat.
|
||
|
Neuron
|
ded6ca546f |
runtime: anchor the think read, so Neuron can think at all
El SDK CI - dev / build-and-test (pull_request) Failing after 13m24s
engram_think_json passed NULL as the anchor. NULL is not "no opinion":
engram_think re-origins at `anchor ? anchor : region->centroid`, so NULL
means "read from the centroid" — and the centroid is the one point where
the gradient is zero by construction. r = x - centroid = 0, so every axis
projection is 0, grad is 0, and direction takes the "at rest" branch at
engram_cognition.c:137.
Measured consequence: EVERY faculty returned an identical null result,
differing only in its label —
{"direction":[0,0,0,0,0,0,0,0],"spread":0,"magnitude":1,"confidence":0.5}
magnitude 1 is membership evaluated at the centroid, spread 0 is its
distance to itself, confidence 0.5 is the stance fallback. The geometry was
never at fault: /api/drift computes real values (centroid_sep 0.104,
core_disp 0.045) over the very same 87 members. Neuron could not think
because the read was always taken from the region's own centre.
The seeds choose WHICH region; they must also supply the VANTAGE. Anchor at
the first resolvable embedded seed — the same seed eg_geo_build_desc infers
dim from, so the two can never disagree. One seed still yields a real
gradient because the descriptor expands to that seed's neighbourhood, so
the seed's position is distinct from the neighbourhood centroid.
The vector is COPIED, never borrowed: g->nodes is realloc'd in place on
append, so a borrowed EngramNode* dangles across any concurrent write.
Verified against a clone of the production store (13,616 nodes / 37,865
edges):
self anchor n_support 87 magnitude 0.00282 spread 18.79
values hub n_support 28 magnitude 0.00318 spread 17.72
with distinct unit direction vectors. Previously both returned the zero
vector with magnitude 1 and spread 0.
STILL OPEN, now isolated by this fix: all five faculties return identical
numbers and confidence stays 0.5, because cog_stance_init is passed NULL
for the stance and the faculty enters the computation only through the
stance's axis_gain[] and bias_dir. The faculty label is inert until a
stance is loaded — which is what learn()'s correspondence-beat calibrates.
Same shape as this bug: a neutral parameter collapsing a capability to a
constant.
|
||
|
Neuron
|
c79033b749 |
runtime: let signal enter as geometry, not as prose about signal
El SDK CI - dev / build-and-test (pull_request) Failing after 10m55s
No ingest path could carry a vector. engram_node/_full/_layered take text only, and a node acquired an embedding solely via engram_embed_backfill DERIVING one from n->content. That made text the mandatory entry medium: any non-text modality had to be described in prose first, so the geometry we then reasoned over was the geometry OF THE DESCRIPTION, not of the signal. Measured: POST /api/nodes accepted an "emb" field, returned 200 with a fresh id, and stored nothing — emb_dim=None, embedded=false. engram_node_set_emb attaches a vector to an existing node. Off-dimension vectors are stored but not indexed (the HNSW build loop already filters on emb_dim), so modality geometry is durable and addressable without perturbing the canonical index. Setting emb also makes the node ineligible for embed_backfill, so a realizer's vector is never overwritten by a text-derived one. Two reporting fixes ride along, because both are how the drop stayed invisible: the create response now reports emb_set instead of being success-shaped regardless, and the node document now always emits emb_dim and embedded — without which a genuine ingest drop and a mere reporting gap are indistinguishable. Verified live: voice node emb_dim=64 embedded=true; text control emb_dim=0 embedded=false; malformed hex, length mismatch and dim<=0 all reject. KNOWN PLACEMENT DEFECT: this is at the consumer. Ingest is a language concern, not an engram feature — every el program touching any modality needs it. The vector also marshals as a hex STRING because el has no first-class geometry value, which reintroduces text as the transport medium one layer below the problem being fixed. The durable shape is geometry as an el value plus declarable realizers, after which the engram stops having an ingest concept at all. Landing this as the verified probe that proves the path. |
||
|
bigmerge
|
9c07970943 |
runtime: state_get leaked its value on every call
El SDK CI - dev / build-and-test (pull_request) Failing after 14m26s
char* result = el_strdup_persist(e ? e->value : ""); // never freed
pthread_mutex_unlock(&_state_mu);
char* copy = el_strdup(result); // arena-tracked
return el_wrap_str(copy);
Two copies were made. `result` existed only as the source for `copy` — never
returned, never freed — and el_strdup_persist bypasses the arena BY DESIGN
("state_set, engram internals"), so arena-pop could never reclaim it. Every
state_get leaked its full value string, permanently.
MEASURED: 200,000 state_get calls against a 64-byte value.
before 15 MB peak RSS growth (~75 bytes/call — the value plus overhead)
after 0 MB
IMPACT. The soul's awareness loop has 68 state_get call sites and ticks every
200ms. Live measurement before the fix: RSS climbing 112 MB per 20s, about
19 GB/hour, in awareness_run -> one_cycle -> perceive, while node_count stayed
flat at ~13,479 — growth with no data behind it. It drove the host from 20 GB
free to 4.3 GB in roughly an hour.
WHY NOW, since the code is old: the soul used to restart constantly (no
write-through, divergent graph, 2.11 GB). Stabilising it (neuron #162) let it
stay up long enough to accumulate. The fix did not cause this leak; it removed
the crashes that were hiding it. Same pattern as the test framework surfacing
math_log — the defect was always there, something finally made it visible.
Found by Ishikawa rather than by reading the nearest code: method (arena
push/pop IS correctly paired per tick), material (node count flat, so not data
growth), environment (19 GB/hr / 18,000 ticks = ~1.1 MB per tick, so per-tick
not one-shot), machine (an allocator that bypasses the arena) — which is where
the evidence pointed.
el_strdup tracks into the thread-local arena, which touches no shared state, so
taking the single copy under _state_mu is safe and removes the temporary
entirely.
Verified: self-hosting fixpoint byte-identical; state round-trip correct for
hit, miss, and overwrite.
|
||
|
Neuron
|
6a6b589ba0 |
bench: real black_box barrier + three-signal growth-curve gate
Adds el_black_box (inline asm, +r constraint, memory clobber) and runtime/elbench.el: a growth-curve classifier that gates time AND allocation-count AND allocation-bytes, failing if any exceeds its declared curve. Refusal is a first-class verdict. The classifier REFUSES rather than classifying when the largest measurement is below the floor, or when a series is hard-flat across an 8x input range -- the shape produced when the optimiser deletes the work. Reporting O(1) there would be a confident answer with nothing behind it. Disagreeing ratios report INDETERMINATE rather than a guess. Deviation from DESIGN.md 6.2, stated in the source: uses consecutive ratios on a mandated geometric sweep rather than least-squares over candidate curves. Ratios are directly interpretable on a doubling sweep and need no floating point; the cost is weaker O(n) vs O(n log n) separation, reported as an ambiguous band rather than guessed. Documents the counter scope limit: engram_*.c and libcurl malloc are NOT tracked, so a flat curve over engram/HTTP-dominated work is not evidence of anything. 13 tests prove the classifier against real measured series from fitprobe.el -- including that an accumulator's allocation COUNT is linear while its bytes are quadratic, and that el #132's pure-CPU shape reads FLAT on both allocation signals and is caught only by time. |
||
|
bigmerge
|
a8908908df |
runtime: count container allocations too, not just strings
El SDK CI - dev / build-and-test (pull_request) Failing after 12m11s
el #131 instrumented the four string allocators, which meant list- and map-heavy code reported ZERO allocations — a benchmark over lists would have been fitted against a flat line and passed anything. Caught during framework work: a "linear" specimen read 0 allocs until it was rewritten to allocate strings. A gate is only as good as its blind spots are small, and a signal that silently reads zero is worse than no signal: it produces a confident pass. Now counted at every container allocation — ElList and ElMap bodies, their backing arrays, the copy-on-write clones, and the realloc growth path. Verified on an append loop (n = 100..800): allocs 7, 8, 9, 10 +1 per doubling = O(log n) reallocations bytes 2048, 4096, 8192, 16384 exactly 2x per doubling = O(n) Both curves are what correct amortized growth should look like, and both read zero before this change. Known remaining scope, stated rather than left implicit: these counters cover the runtime's own allocations. They do not see malloc inside engram_*.c or libcurl, which is correct — the gate is for El-level complexity, not for third-party memory behaviour. |
||
|
bigmerge
|
edafd8cce8 |
runtime: math_log is base-10, not natural log
El SDK CI - dev / build-and-test (pull_request) Failing after 10m8s
el_val_t math_log(el_val_t f) { return el_from_float(log(el_to_float(f))); }
el_val_t math_ln(el_val_t f) { return el_from_float(log(el_to_float(f))); }
Both were natural log, so math_log and math_ln were the same function.
log10(100) returned 4.605 instead of 2.
Three sources already agreed it should be base-10 and were being contradicted
by this one line:
- runtime/math.el:55 "// math_log — base-10 logarithm."
- el_seed.c:1278 __log_f -> log10() (the path math.el actually calls)
- tests/native/test_math.el:133 asserts log10(100) == 2
FOUND BY THE NEW TEST FRAMEWORK ON ITS FIRST RUN (el #133). The assertion had
been sitting in the suite the whole time; nothing could report it. The old
harness printed "N passed, M failed" with no per-test detail, and half the
suites were not compiling at all — so a failing assertion in a suite nobody
could run was indistinguishable from no failure.
That is the entire argument for the framework, demonstrated on day one: this is
not a bug the framework introduced, it is a bug the framework made VISIBLE.
Verified: tests/native/test_math.el goes 12/13 -> 13/13, math-log passing.
|
||
|
bigmerge
|
d231b7e5e7 |
compiler: fix the quadratic — strlen() on every character access
El SDK CI - dev / build-and-test (pull_request) Failing after 10m21s
THE BUG. str_char_code() and str_slice() each called strlen() on every
invocation. The lexer walks source one character at a time, so every character
access rescanned the whole remaining input: O(n) per character over n
characters = O(n^2).
el_val_t str_char_code(el_val_t s, el_val_t i) {
...
int64_t n = (int64_t)strlen(str); // <- O(n), every call
if (idx < 0 || idx >= n) return 0;
return str[idx];
}
HOW IT WAS FOUND. Not by reading code — by sampling the running process, which
is the same method that resolved tonight's engram outage after four wrong
theories. A geometric sweep of synthetic sources showed wall-clock rising 3.0x,
3.0x, 4.0x, 4.14x per doubling (converging on 4x = quadratic), and a stack
sample put 779 of 779 samples inside lex(), every one bottoming out in
_platform_strlen via str_char_code and str_slice.
THE FIX. Remember the length instead of recomputing it. The subtlety is
INVALIDATION: El strings are arena-allocated, so a freed pointer can be reused
for a different string at the same address, and a naive pointer-keyed cache
would hand back a stale length and read past the end of the new string —
trading a performance bug for a memory-safety one. So entries carry a
generation, a hit requires pointer AND generation to match, and every path that
frees or mutates a runtime string bumps the generation: el_arena_pop,
seed_request_end, __str_set_char. Stale entries cannot be believed; they miss
and recompute.
MEASURED, same host, same inputs:
n(fns) before after
512 0.10s 0.01s
1024 0.37s 0.02s
2048 1.51s 0.03s 50x
the compiler's own 422 KB source concatenated (DESIGN.md's 3.58s case):
3.55s -> 0.03s 118x
The speedup GROWS with input size, which is the signature of removing a
complexity class rather than a constant factor. After the fix each doubling
adds ~0.01s: linear.
CORRECTNESS, verified rather than assumed:
- byte-identical output on every sweep input (n = 128..2048)
- byte-identical output on the 422 KB compiler concatenation
- byte-identical output on tests/runtime/string_test.el
- self-hosting fixpoint byte-identical
- new tests/runtime/str_cache_test.el: 17 assertions covering bounds, empty
strings, negative indices, slice clamping, distinct strings not sharing a
cached length, 1000 interleaved strings forcing cache-slot collisions, and
a grown string not reporting its old length. All pass.
This is the defect that made dist/soul.c a committed artifact: elc could not run
in CI because it needed 24 GB+ and minutes. It needs neither now.
|
||
|
bigmerge
|
37bcf7eb74 |
runtime: allocation accounting — the deterministic signal for complexity gating
El SDK CI - dev / build-and-test (pull_request) Failing after 12m7s
Implements the three primitives the test-framework design (DESIGN.md §6.5)
requires for gating on growth curves: el_alloc_count, el_alloc_bytes,
el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per
the project's C-builtin recipe.
WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build
failure, which means the signal cannot flake. Wall-clock needs warmup,
statistics, and a quiet machine; on shared CI it is unusable as a gate.
Allocation counts are perfectly deterministic — same input, same number, every
machine, every run. Fit them against n and a complexity regression becomes a
build failure with zero noise.
All four runtime string allocators (el_strdup, el_strbuf, and their _persist
variants) funnel every allocation the language performs, so instrumenting there
counts everything.
WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate.
Measured with two El programs, one allocating once per item, one rebuilding its
accumulator each iteration:
n linear allocs / bytes quadratic allocs / bytes
100 100 / 290 100 / 5,150
200 200 / 690 200 / 20,300
400 400 / 1,490 400 / 80,600
800 800 / 3,090 800 / 321,200
The quadratic program's allocation COUNT is exactly linear — identical to the
healthy one. Counting allocations alone would have missed it completely. Bytes
catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 ->
converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0.
That shape — count linear, per-allocation size growing — is the classic
accidental quadratic, and it is exactly elc's defect: quadratic allocation
VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt
one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across
both; RSS and wall-clock were just the two ways it surfaced.
el_peak_rss is exported for context and is explicitly NOT a gating signal — it
is perturbed by allocator internals, the page cache, and the OS. Gate on the
deterministic numbers; report the physical one.
Counters are unsynchronised by design: this is measurement, and a lock would
change the thing being measured. Exact on the single-threaded compile path,
approximate under threads.
|
||
|
bigmerge
|
e917b3d439 |
store: make the buffer pool sense its own state and correct from it
El SDK CI - dev / build-and-test (pull_request) Failing after 14m35s
Follow-on to the edge write barrier. That fix removed the full-store walk;
this one makes the pool able to notice if anything like it happens again.
WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill
twice on 2026-08-15 and said nothing. From outside it was indistinguishable
from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories
got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each
costing a deploy or a rollback. The whole time, hits/misses/evictions were
already being counted in PgCache, and the struct comment read:
/* stats (introspection only — never affect semantics) */
That comment was the bug. Self-measurement treated as decoration is why the
pool could not correct itself and why no one outside could see what it was
doing. A system that cannot read its own state cannot correct, and neither can
anyone watching it.
- pc_adapt_budget(): the loop, closed. Over a sliding window, evictions
running at a large fraction of accesses WHILE reuse is real means the
working set exceeds the budget — so grow it, geometrically, bounded by a
LIVE re-read of physical memory. Evictions alone are not pressure (a scan
evicts and never returns); evictions with reuse are. An explicit
ENGRAM_POOL_FRAMES still wins — an operator override must not be silently
overruled.
- Budget derived, not declared. A constant cannot be right: 16 GiB of frames
is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM
at startup" is a guess about the future — it cannot know the store grew or
the machine changed. Hence the live re-read.
- pc_report(): ONE structured emission carrying the entire sensed state,
through emit_log — El's existing telemetry, already exporting to OTLP.
Deliberately not a function per stat, and deliberately not a bespoke
/api/pool endpoint: both make observability something hand-written per noun
instead of the uniform mechanism every component already has.
- engram_pool_stats_json(): the same state readable live, wired through the
normal builtin path (codegen arity + el_seed wrapper), so the pool can be
observed in real time rather than reconstructed afterward from a stack
sample.
Verified: with the exact configuration that took production down
(ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots
clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete —
and NO pressure event fires, because the barrier removed the walk that caused
it. The controller is defense in depth; the barrier is the fix.
|
||
|
bigmerge
|
4e24d7d3f1 |
runtime: engram_edges_json — read edges without a whole-graph file round trip
El SDK CI - dev / build-and-test (pull_request) Failing after 13m4s
/api/graph/edges answered a read query by calling engram_save() to serialize
the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in
one line, and both bit production on 2026-08-15:
1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram
server's CANONICAL store. A READ route overwriting the persistence
owner's canonical file. This defect had been fixed once (export moved to
a scratch path); it came back when the hand-written dispatch block was
replaced by @route dispatch and the unfixed copy is the one that
survived the merge.
2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole
graph, per request, to return a bounded slice.
Calling it tonight overwrote the canonical snapshot and immediately preceded
an engram crash loop.
engram_edges_json(limit, offset) is the builtin that route's own TODO asked
for ("Future: add an engram_edges_json() builtin and drop the file round trip
entirely"). It walks g->edges directly and emits every persisted field.
limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell
over, and an unbounded default would preserve the failure mode under a new
name. Callers page explicitly.
Registered in codegen.el's builtin_arity (both plain and __ spellings) and
wrapped in el_seed.c per the project's C-builtin recipe.
|
||
|
bigmerge
|
7351fb0a8d |
runtime: restore engram_recall_json + cgi_* accessors
El SDK CI - dev / build-and-test (pull_request) Failing after 10m24s
neuron's soul calls engram_recall_json (neuron-api.el:618, memory.el:80) and
cgi_principal (studio.el:72). Both existed in the runtime neuron vendored
(v1.0.0-20260501) and were absent here, so the soul could not link against
current el at all.
The dangerous part is what the obvious "fix" would have done. These look like
redundant wrappers over one impl:
engram_search_json(q, limit) -> eg_search_json_impl(q, limit, 0) LEXICAL
engram_recall_json(q, limit) -> eg_search_json_impl(q, limit, 1) SEMANTIC
They are not interchangeable, and the split is documented at neuron-api.el:613:
search stays LEXICAL because ~40 internal call sites pass a KEY and seven of
them DELETE every record returned. Point those at a semantic matcher and they
delete fuzzy matches. Conversely, pointing recall at search silently downgrades
the mind's entire retrieval surface from semantic to lexical — no error, just
permanently worse recall.
Implemented over engram_activate(), which in this runtime already IS the
semantic path the old with_legs=1 branch built by hand (embeds the query via
eg_embed_fetch, scores by cosine, then spreads activation one hop). Output
shape matches engram_search_json — a flat array via engram_emit_node_json —
because callers parse search's shape, not activate's envelope.
Verified: neuron's soul now compiles and links against current el, boots, and
serves /health with layers initialized.
NOTE for follow-up: current el also ships engram_retrieve_geometric_json, a
structure-first retrieval that appears to be the intended successor to recall.
Repointing the two recall call sites at it may well be the right end state and
would remove the two-wrapper shape entirely — but that is a behavioral change
that must be measured against neuron/tools/retrieval-eval/'s gold set, not
assumed. This commit preserves existing behavior exactly; it does not decide
that question.
|
||
|
bigmerge
|
598915cc61 |
runtime: restore the three builtins that made elc unrebuildable
El SDK CI - dev / build-and-test (pull_request) Failing after 3m52s
The committed elc binary could not be refreshed from its own source. Rebuilding
failed with three implicit-declaration errors: el_mem_check, stdout_to_file,
stdout_restore. The compiler's own source calls all three (compiler.el:472,479,574
and codegen.el:4248) and two are registered in codegen.el's builtin_arity table —
but none were defined in this runtime.
They were found intact in ui/examples/native-hello-ios/NativeHello/el_runtime.c,
a divergent private copy of this runtime that still carried them. Ported verbatim.
Consequence of them being missing: the canonical elc binary was frozen. Source
gained @route dispatch codegen (emit_route_dispatch, codegen.el:3948) and the
@manager boundary-beat seam, but no rebuilt binary could carry them, so
neuron's soul — whose routes.el now calls the compiler-synthesized
el_route_dispatch — could not be built at all.
Verified after the fix:
- elc rebuilds from current source, clean.
- Self-hosting fixpoint byte-identical (stage3 == stage2).
- The rebuilt elc emits el_route_dispatch (2 occurrences in the soul amalgam,
previously 0) and injects engram_boundary_beat at @manager boundaries,
i.e. the decorator seam is live rather than inert.
el_mem_check is itself the compiler's memory guard (ELC_MAX_MEM_MB, default
512MB, self-terminates before the OS OOM-killer fires) — so the runtime was
missing the very guard that would have surfaced the compiler's memory blowup
as a clean error instead of a 27GB host-killer.
|
||
|
bigmerge
|
40eb48e92f |
engram: fix silently-wrong query params, and make el_seed.o + el_runtime.o link
El SDK CI - dev / build-and-test (pull_request) Failing after 14m49s
Three real bugs, all found by actually running the thing rather than reading it.
1. query_param never URL-decoded. A GET of /api/search?q=neural%20network
searched for the literal string "neural%20network" and returned []. Every
multi-word search against the live engram has been silently returning empty
results — not an error, an empty result, which is why it went unnoticed.
Affects every GET route that reads query params, not just search.
2. query_param matched key names unanchored. str_index_of(qs, "q=") matches
inside "faq=", so "?faq=X&q=Y" returned X for key "q". Verified live before
the fix. Now searches for "&key=" against "&"+querystring so a match can
only land on a real parameter boundary.
3. el_request_start/el_request_end were defined in BOTH el_seed.c and
el_runtime.c, so linking the two objects together — which is exactly what
the product build does — failed with duplicate symbols. el_seed.c's own
comment already says these moved there ("formerly defined in el_runtime.c.
Now self-contained in el_seed.c"); the el_runtime.c copies were left behind
during that move. Removed them, kept declarations since http_worker calls
them. Also added the three missing prototypes (engram_op_assert_json,
engram_node_full_in, engram_connect_in) that el_seed.c wraps but never
declared, which made it fail to compile standalone under C99+.
Verified: engram builds and links clean from canonical source; before/after
comparison on a copy of the real store shows "neural network" returning a real
match where the live build returns [], and "?faq=WRONG&q=MetaColloc" now
resolving to MetaColloc. Live engram on :8742 was never touched.
|
||
|
bigmerge
|
09dade0613 |
Merge remote-tracking branch 'origin/pr/103' into HEAD
El SDK CI - dev / build-and-test (pull_request) Failing after 3m56s
# Conflicts: # lang/AGENTS.md # lang/runtime/el_runtime.h |
||
|
bigmerge
|
9883aa7564 |
Merge remote-tracking branch 'origin/wt/swarm-ccr' into merge-swarm-ccr-v2
El SDK CI - dev / build-and-test (pull_request) Failing after 4m13s
# Conflicts: # lang/runtime/el_runtime.c # lang/runtime/el_runtime.h |
||
 will.anderson
|
ee39aa5f17 | Merge pull request 'ingest: unify transduce_prose/transduce_structured into one transduce()' (#117) from feat/transduce-unify into dev | ||
|
bigmerge
|
e29fe4fd0b |
ingest: unify transduce_prose/transduce_structured into one transduce()
El SDK CI - dev / build-and-test (pull_request) Failing after 3m42s
transduce() is now THE single mechanism: one function, no content-type branch inside it. It never asks whether `source` is prose, JSON, or raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally: split on "\n\n" as a universal boundary-marker check, and if that finds no boundary, fall back to fixed 4096-char windows. Same node/edge wiring (root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets a heading/section_of link) regardless of what's inside a chunk. Dedup is the existing find_existing_by_content path via merge_manifold, applied uniformly. The old transduce_structured JSON dataset/records/feature-node interpretation is deleted outright, not just unused — a JSON file now gets chunked and deduped like anything else, with no pre-computed structure. All five ingest_* entry points still exist unchanged in name and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one transduce() (ingest_stream builds its own turn-nodes directly and never called either old function, so it's untouched). This unlocks raw/opaque content (audio, or anything else with no natural text/JSON shape) without any DSP, LLM call, or external API: transduce() chunks it exactly like it chunks anything else. There is zero semantic understanding of audio (or any payload) claimed or built here — any meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time. Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk) because El strings are NUL-unsafe under strlen-based ops and fs_read()'s result silently truncates at the first embedded NUL, which is routine in real binary/audio bytes. ingest_file compares fs_read()'s string length against a real fs_size() stat() count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim, no invention), joined with the same "\n\n" marker transduce()'s boundary scan already looks for. This is a mechanical fidelity fix, not interpretation of content — transduce() never learns a fallback happened. Registered both builtins' arity in codegen.el; did not rebuild the elc compiler binary itself (unrelated, pre-existing gap: self-hosting elc via el_seed.c fails on this worktree independent of this change, reproduced with codegen.el reverted) — the existing elc binary compiles calls to unregistered builtins via its already-existing arity=-1 passthrough, confirmed by an actual clean `elc ingest.el` + `cc` build against the modified el_runtime.c. INGEST_KIND keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a content-type flag; the redundant "structured" value (an alias for "file" that hinted the now-deleted JSON branch) is removed. ingest_dir drops its file-extension filter for the same reason: transduce() takes anything now. Verification: local manifold construction confirmed correct against a real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte real prefix slice) — exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes. Compiles clean via the real elc + the modified el_runtime.c/engram_*.c (built and booted an actual sandbox engram off this exact source with `nsbx create --branch`). NOT verified this session, disclosed rather than papered over: end-to-end server-confirmed persistence (a real before/after /api/stats delta, and a fetched node by id) for the audio, prose, and JSON-fixture cases. Every local nsbx sandbox engram tried tonight (two stock pre-#109 binaries hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW binary built from current dev) took minutes-to indefinitely long on the final /api/load-merge write's embedding step and hit the client's 60s HTTP timeout before responding, even for a 5-node write. This is confirmed as real (if slow) forward progress, not a hang: the sandbox's WAL file was observed growing steadily across every attempt. The code's own pre-existing HONESTY GATE correctly refused to report success in every case, returning "load-merge failed: ..." with a "nothing below this manifold was confirmed persisted by the server" note instead — exactly as designed. This is an environment/infrastructure limitation, not a defect introduced by this change: the engram server binary itself is untouched by this commit. |
||
|
bigmerge
|
1bb1edc851 |
engram: port PR #105's 3 genuine wins onto dev's existing cosq/e_eff semantic layer
El SDK CI - dev / build-and-test (pull_request) Failing after 4m45s
Reconciles PR #105 ("fix: engram search latency — pin embed model, cache query embeddings, bound activate BFS") with dev's ACTUAL current engram_activate, rather than the ancient pre-restructure snapshot #105 was built against. WHY THIS NEEDED RECONCILIATION, NOT A DIRECT PORT: #105's single commit ( |
||
|
bigmerge
|
bacaf3d39c |
engram: reconcile M8 HNSW vindex (#109) onto current dev, restore 3 fixes the branch predated
El SDK CI - dev / build-and-test (pull_request) Failing after 4m49s
Lands feat/reframe-region-setop (PR #109: native set-based reframe_region, decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's actual current HEAD, plus engram-tiered-storage's still-unique test suite. RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/ verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/ select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two branches; #109's copy is a strict superset (adds vindex_harvest_from_store, used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and engram_verify.c are also byte-identical. #109's own branch point already carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing left to merge into #109 for those files. The one thing engram-tiered-storage had that #109's tree dropped: its full test suite (test_vindex.c, test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh harnesses) — ported over here unchanged. WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch forked from dev on 2026-08-14 15:40 (before restructure-adjacent history diverged the file's merge-base for `git merge` — it presented as an add/add conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but it silently reverted THREE dev fixes landed on 2026-08-14/15, after the branch point, that the PR's diff had no way to know about: 1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of the query-aware propagation gate dropped the shift-and-floor rescale about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate 0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi] is computed) is orthogonal to WHAT it gates on and both are kept. 2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global). PR's tree predates this and dropped all three counters + their JSON stats fields. Restored declarations, all 4 direct increment sites, the eg_wm_carry_over bll increment, and the act-stats JSON fields -- alongside (not instead of) the PR's own P4 afferent / API-reshape counters already in that same struct/JSON. 3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev selects the STRONGEST qualifying candidate for consolidation each call; PR's tree predates this and reverted to hash-slot order (arbitrary wrt association strength) for edge formation -- the one path that writes PERMANENT structure. Restored the strongest-candidate while-loop, keeping the PR's own genuine improvement at that site (engram_adj_on_edge_added incremental-index append instead of a bare adj_dirty=1 full-rebuild flag). engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env flags, /api/nodes connected-count in responses) were pure additive: dev's side was empty, PR's side added the feature. Took PR's side whole. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): - cc -std=c11 -O2, clean link against the real engram/src/server.el via elc, zero errors. - vindex_bench (built standalone, read-only harvest) against the real production store clone (13,671 embedded nodes, 768-dim nomic-embed-text): recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs 2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s for the full 13,671-node set -- see the flagged risk below. - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned snapshot of the live store, 13,424 nodes / 37,656 edges) and called /api/activate for real: first call after boot 41.5s (pays the one-time HNSW build inline -- matches the standalone bench), second/third calls 356ms/605ms, no crash, correct results, act-stats JSON (including the restored evict_floor/cap/bll fields) reads correctly. KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync builds the HNSW index synchronously, inline, on the first engram_activate() call after every process start (or index invalidation). On the real node count that is a ~46s blocking stall on a single-threaded server -- the first request after every restart (or its concurrent siblings) waits the full build. Recommend a background/incremental build (or a bounded per-call build budget) before this ever reaches the live daemon. See PR description / final report for the fuller writeup. |
||
|
will.anderson
|
e34ebd4b3d |
Merge pull request 'nsbx + cognitive architecture design + engram self-review series' (#113) from feat/neuron-sandbox into dev
El SDK CI - dev / build-and-test (push) Failing after 4m2s
|
||
|
will.anderson
|
09ae14a970 |
Merge pull request 'fix: float arithmetic codegen (segfault/garbage) and math_log aliasing' (#104) from worktree-agent-a456e0cf8cd2ee361 into dev
El SDK CI - dev / build-and-test (push) Failing after 3m57s
|
||
|
will.anderson
|
fdf0d6cb64 |
Merge pull request 'nsbx: one-command dev onboarding (branch + worktree + isolated engram)' (#99) from feat/nsbx-dev-env into dev
El SDK CI - dev / build-and-test (push) Failing after 13m56s
|
||
|
bigmerge
|
08cbcef5d9 |
engram: fix lazy-embed index gap (#20) and make activate's cosine scan lazy; extract vindex harvest primitive with a bench/oracle harness
El SDK CI - dev / build-and-test (pull_request) Successful in 6m42s
Adds an O(1) "seen" bitmap so lazily-embedded older nodes get picked up incrementally instead of only on a full rebuild (embed-gap #20). Replaces engram_activate's O(N*D) cosine prescan with a lazy-memoized cosine cache (eg_cosq_at), proven bit-identical to the old path. Extracts a clean vindex_harvest_from_store primitive (read-only vector harvest, careful malloc/ownership/error-path handling) reused by both index-build and the new vindex_bench.c — a read-only proof harness comparing brute-force vs HNSW recall/latency on both the real store and synthetic data. .nsbx-env intentionally excluded — local sandbox config (ports, paths, dev-only placeholder key), not checked in. |
||
|
bigmerge
|
01826421c4 |
seam: implement decorated-fn boundary auto-emit; prove on clone
Will waived diff review -> build it for real. Add engram_boundary_beat() to the runtime (afferent counter++ + engram_chrono_tick + engram_strengthen(self-anchor) + dharma_emit) and two act-stats counters (aff_boundary_ops, dharma_emits). codegen cg_fn injects ONE engram_boundary_beat(op) at the entry of every @manager/@accessor fn (fn_has_decorator, so it fires under @route @manager too) — a decorated op self-reports with ZERO hand-written instrumentation. Rebuilt elc self-host + the cognition engram in the worktree; ran it as the clone daemon on :8900. Proof (/api/boundary-proof, @manager, empty body, 5x): aff_boundary_ops 0->5, dharma_emits 0->5, self activation_count 1510->1513, chrono stamp advanced. Brought in feat/cognitive-architecture engram runtime+server for the build. strengthen = activation bump (not content/edge write) -> identity protection intact. Live :8742 untouched; no push, no cutover. |
||
|
will.anderson
|
7aa847e32a |
Merge origin/dev into engram-tiered-storage
El SDK CI - dev / build-and-test (pull_request) Failing after 10m51s
Resolve 3 conflicts: - lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork; single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh). - lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated artifact folder, not a source path; a release is a git tag, not a folder). - lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL. Nothing in dev's build consumes the deprecated fork or releases/ folder. |
||
|
will.anderson
|
bb64a236ed |
engram tiered storage: engram-service wiring + elc fold-hang fix + prune-store mirror
- Wire paged store into the ENGRAM SERVICE (server.el, the authoritative durable owner): boot->engram_store_boot, persist_canonical->engram_store_checkpoint, gated by ENGRAM_STORE. - elc (lang/elc.c + src/parser.el + codegen.el + elc-combined.el): OOB guard in tok_kind/tok_value + parse_block progress backstop — fixes the pre-existing unbounded-memory fold hang on sessions.el. - engram_prune_telemetry mirrors ISE prune to the store (store_forget) so store live-count tracks resident and stale telemetry stays bounded. - Deployed live 2026-08-12: engram :8742 on neuron.egm+WAL, count reconciled 11552. |
||
|
will.anderson
|
9a0266cbf9 |
engram tiered storage M3.5: persist activation field updates (pre-flip gate)
Flag-on checkpoint now full-walks the resident graph: store_put_node (WM weight, activation_count, last_activated, wm_anchor) + store_put_edge (hebb, last_fired) for every node/edge, then engram_checkpoint. Uses store_put_edge (idempotent upsert) not store_hebb_batch, because activation FORMS new hebbian-associate edges that bypass the create hook and delta-only hebb_batch can't create them. Store-on boot now applies the same WM-halving + floor + cap transforms as engram_load. This is the hebb-survives-restart fix. Gate: reboot from neuron.egm with snapshot.json deleted -> edge hebb + activation_count survive unchanged, WM weight survives with identical boot transform; negative control proves persist is load-bearing (hebb->0 without it). M1 33/33 + M2 36/36 + M3 parity PASS, ASan/UBSan clean, flag-off untouched. Engine unchanged (boundary held). |
||
|
will.anderson
|
a72145b44e |
engram tiered storage M3: wire store behind ENGRAM_STORE (default off) + .egm rename
Caller-side shim in el_runtime.c maps EngramNode/Edge <-> StoreNode/Edge; engine keeps zero soul deps (libengram boundary, design §10). Flag off = today's JSON path byte-for-byte (proven: no neuron.egm created, graph identical). Flag on = engram_open (import snapshot.json once into neuron.egm, else WAL-replay) + resident load; node/edge create + forget dual-write via guarded hooks. Files renamed engram.store->neuron.egm, engram.wal->neuron.wal. Gate: M3 parity PASS (graph on==off byte-exact modulo ordering; snapshot round-trip; reboot-from-egm with snapshot.json deleted; activation set+sequence identical; ASan/UBSan clean). M1 33/33 + M2 36/36 green post-rename. Known gap (pre-flip): in-place hebb/WM/activation_count updates during activation are not yet persisted to the store (create/connect/forget are). Must close before live flip so learned edges survive restart. |
||
|
will.anderson
|
0a72fced28 |
engram: WAL persistence + integrity hardening + single canonical runtime
El SDK CI - dev / build-and-test (pull_request) Failing after 13m17s
Establish lang/runtime/ as the ONE canonical el runtime (from the active runtime that carries hebb/emb persistence + the new WAL); repoint the el CI publish, engram build, elb default, and in-repo build scripts to it; delete the el-compiler/runtime + lang/releases/ forks; add scripts/check-single-runtime.sh drift guard. Fixes a live prod bug: the el CI published el-runtime-c/-h from the LAGGING el-compiler fork (0 hebb refs), so the shipped soul never persisted Hebbian edge weights — learned co-activation was wiped on every restart. Publishing from canonical ships the stranded 'learning that cannot outlive the process' fix. WAL storage engine + integrity fixes (DELETE->tombstone + store-layer protection, safe data-dir default) ride in behind ENGRAM_WAL (default off = byte-identical to today). Verified: engram elb per-module build clean, WAL gate 66/66, native smoke ok, drift-guard green. |