Compare commits

..

22 Commits

Author SHA1 Message Date
bigmerge 26af149aa1 lang: rebuild the bootstrap compiler against merged dev
El SDK CI - dev / build-and-test (pull_request) Failing after 4m46s
The binary was stamped before dev advanced (vindex publication landed in
el_runtime.c and engram_vindex.c). Rebuilt against the merged runtime so the
committed compiler matches the runtime it ships beside. Fixpoint re-verified
byte-identical; test_compiler 82/82; engram/src/server.el still compiles and
still emits its 18 config declarations.
2026-08-16 11:38:52 -05:00
bigmerge c18abf799c engram: declare configuration once instead of at every read site
Migrates engram to the `program` block. 18 configuration variables that each
carried their default inline at the point of use now declare it in one place,
and engram declares itself a singleton.

The read sites lose their defaults entirely: `let v = env("X")` followed by
`if str_eq(v,"") { "default" } else { v }` collapses to `config("X")`. The
guide_env_or(key, dflt) helper is deleted -- its whole job was supplying a
per-site default, which is the thing being removed.

Fixes ENGRAM_DATA_DIR, which was the clearest instance of the defect. It was
read at six sites. Five were dead: `let dir_raw = env("ENGRAM_DATA_DIR")`
immediately shadowed on the next line by `engram_resolve_data_dir()`. The sixth
was live and defaulted to /tmp/engram, contradicting the canonical resolver's
$HOME/.neuron/engram -- and its consumer is the pre-destructive reseed backup,
so with ENGRAM_DATA_DIR unset the safety copy was written to ephemeral storage
while the store it protected lived elsewhere. All six now go through
engram_resolve_data_dir().

ENGRAM_DATA_DIR is deliberately NOT declared in the program block, and the
source says why: engram_resolve_data_dir() already owns it, and a second
declaration would give it two owners that can disagree -- recreating the exact
defect being removed here. A variable belongs in the block when the block would
be its only owner. HOME stays a raw env() read; it is an environment fact, not
configuration.

singleton: "engram" matters more than it looks. Today a second engram whose
bind() fails merely returns from http_serve -- after it has already replayed
the WAL and written boot-time backup files -- and then exits 0, indistinguishable
from a clean run. That is how two instances came to share one data dir. Verified
that the second instance now refuses before any side effect: with instance 1
holding the lock (lsof pid, shell pid, and lock file contents all agreeing at
5946), the second start named that pid, exited 1, and left the data directory
untouched.

Verified by bijection on the generated C: 18 config() reads, 18 declarations,
no read without a declaration and no declaration without a read. Three bad Int
values are reported in a single run rather than costing one restart each.

ENGRAM_API_KEY keeps its permissive empty default, which disables auth -- that
is pre-existing behaviour and changing it is out of scope. The source marks
making it `required` as the obvious hardening follow-up.
2026-08-16 11:38:28 -05:00
bigmerge b305b49f40 lang: re-stamp the bootstrap compiler so the tree can compile its own source
server.el declares a `program` block, which the previously committed elc cannot
parse. Without this the tree is internally inconsistent: source in the repo that
the compiler in the repo rejects.

This is the documented re-stamp from BOOTSTRAP.md / AGENTS.md, and its
precondition is met -- the self-hosting fixpoint was verified byte-identical
(stage3 output == stage2 output) both before installing and again with the
installed binary. tests/native/test_compiler.el passes 82/82 against it.

Two pre-existing failures are unchanged and are NOT from this work, confirmed
by rebuilding them against the original runtime: test_env's
"state_keys returns JSON array" fails identically before and after, and
test_json/test_state fail to link on symbols (json_build_array, state_has) that
were never prototyped -- the same class of gap as config(), which this branch
fixed because it blocked the build.
2026-08-16 11:38:28 -05:00
bigmerge 8ae163e8e5 lang: give cross-cutting concerns an owner instead of a convention
El's units of encapsulation are the function and the module. Neither can hold
a concern that belongs to the process, so each one had been expressed the only
way it could be -- as a convention: call this at every site. Conventions of
that shape do not hold. Measured here: zero process-identity guards at any
layer, 20 environment variables each with its default written inline at the
read site, 62 persist call sites, 10 per-route auth checks. One absence, four
times.

Step 0 first, because the premise was wrong. El was believed to have no
middleware or effect mechanism. It has one, and it is already load-bearing:
codegen injects engram_boundary_beat at the entry of every @manager/@accessor
fn, decorators take arguments and stack, dharma_emit from a non-@manager fn is
a #error, and the cgi block injects el_cgi_init at the head of main(). So the
correct move was not to invent a mechanism but to generalize the seam that
already existed. The real gap is narrower and is now recorded: the seam is
prologue-only and its callee is a fixed builtin.

Adds a `program` block -- the third program-level declarative block. cgi and
service declare what a program may do; program declares what it is.

  program "engram" {
      singleton: "engram"
      env ENGRAM_BIND: String = ":8742"
      env GUIDE_PORT:  Int    = "8771"
  }

singleton takes an exclusive flock before any user statement runs and refuses a
second start, reporting the holder's pid. It is a lock rather than a pidfile so
the kernel releases it on death including SIGKILL -- no stale state, and so no
"delete the lock file to get unstuck" ritual, which would itself be a
convention. It reports the pid because "already running" is not actionable; a
pid is. That is the direct answer to a stale process surviving a pkill and
going on answering probes.

env entries resolve once at startup -- environment wins, declaration supplies
the fallback -- and validate as a whole, reporting every problem at once rather
than costing one restart per variable. config("X") for an undeclared X is
fatal, because an advisory schema is just another convention. Programs without
a program block are unaffected, so migration is per-program.

Only one keyword is added. `config` and `env` could not become keywords -- both
are real identifiers in the tree -- so the block's fields are read as
identifier token values by its own parse loop and stay usable everywhere else.

The init function is emitted at the block site and called from main() rather
than inlined into main(). The live backend is codegen_streaming, which emits in
source order and cannot hold the entry list alive until main(); this way only a
single bool has to survive.

Also fixes: config() was defined in el_runtime.c but never prototyped in
el_runtime.h, so any el program calling it failed to compile under C99.

Spec: section 18 documents what shipped. Section 9 is corrected -- it claimed
decorators had no structural meaning, which has not been true for some time.
Section 19 designs durability-as-an-epilogue-effect and route authorization
and states plainly why neither is implemented here: both land in files under
concurrent modification, and the prerequisite for both is lifting the seam
from prologue-only to prologue/epilogue.

Self-hosting fixpoint verified byte-identical.
2026-08-16 11:38:28 -05:00
will.anderson a6cef4b983 runtime: publish the vector index instead of guarding it (#143)
El SDK CI - dev / build-and-test (push) Failing after 10m29s
2026-08-16 16:33:28 +00:00
Neuron 8e9d88fc01 runtime: publish the vector index instead of guarding it
El SDK CI - dev / build-and-test (pull_request) Failing after 10m59s
The crash (SIGTRAP in engram_activate -> eg_vindex_sync -> vindex_insert ->
_realloc) had three read paths mutating five process-global statics.
engram_activate, eg_knn_for_node (whose own comment says "No writes.") and
engram_geo_reify_run_json all called eg_vindex_sync, which frees the index,
reallocs the seen-map and inserts — on a read.

Three moves, in decreasing order of how much they dissolve:

1. Misfiled scratch is not shared state. visited/visit_epoch/visited_cap
   were never owned by the index; they are one traversal's local, hoisted
   into struct VIndex as an allocation optimisation. They want neither a
   lock nor a capability nor a pool — just to go back in the call frame.
   Two concurrent READS stomped each other purely because of this.

2. const IS the capability. Once the scratch leaves the struct, search
   reads and nothing else, so vindex_search takes a const VIndex*. That is
   exactly what a capability-pointer ABI would have bought — a read path
   physically cannot call vindex_insert, enforced by the compiler on every
   future caller — for one qualifier instead of an ABI swept across
   hundreds of builtins.

3. What survives is publication, not ownership. HNSW insert is NOT an
   append: it rewires the neighbour links of already-existing elements and
   reallocs elems[], so the store's append-only property does not transfer
   to the index derived from it. eg_vindex_sync therefore splits into
   eg_vindex_maintain (exclusive, sole mutator) and eg_vindex_view (shared,
   returns const VIndex*). A read path may demand that a current snapshot
   exist — a request to the owner, not a mutation by the reader.

Write-side owner: eg_vindex_note_embedded hooks the embedding-ASSIGNMENT
sites rather than the append sites, because a node with no embedding cannot
be in a vector index — embedding assignment is the event that owns index
membership. One O(log n) insert, no O(node_count) presence scan. This also
retires the "STALENESS (honest tradeoff)" note where a lazily-embedded
older node stayed invisible to route_nearest/autoconnect until a full
rebuild (the embed-gap #20 shape).

Evidence. The existing harness conflated two hazards, which is why fixing
half of it read as failure. Split into four:

  single (3000 vec, ASan+UBSan)          clean  ->  clean
  readers (4 readers, no writer, TSan)   RACE   ->  clean
  unsynchronized (writer+reader, bare)   race   ->  race, expected forever
  published (owner + 4 readers)          n/a    ->  clean, 3000/3000 landed

RESULT: PASS. recall@10 = 0.9365 at ef_search=128 (gate >= 0.90);
determinism byte-identical across two independent builds.

The unsynchronized half is now permanently expected to race, deliberately:
it is the executable proof that the boundary must live above the data
structure, not inside it.

fb32d15's guard is KEPT, correcting this design's own section 5. Measured,
it guards TWO structures and only one was converted here: g->nodes/g->edges
are realloc'd in place (el_runtime.c:7618,7629) and engram_activate_inner's
embed-backfill writes n->emb through exactly such a borrowed pointer.
Deleting the guard reintroduces a measured 11171->9579 edge loss. Its
comment is narrowed to the RAM graph and the deletion precondition named.

That corrects the ordering claim too: the residual is not one ABI that
dissolves everything at once, it is a PROPERTY applied per structure.
Residues evaporate in the order the property is applied, and a residue
whose structure has not been converted must be left standing.
2026-08-16 11:29:17 -05:00
bigmerge e99a4640e2 test: regression harness for the vindex concurrency crash
Promotes the two throwaway sanitizer harnesses used to diagnose the
2026-08-16 soul crash into engram/test/ so the bug cannot silently regress.

The harness has two halves and the PAIR is the point — it is what localises
the defect to concurrency rather than to HNSW logic:

  single      3000 clustered vectors, one thread, ASan+UBSan. The CONTROL.
              Must always be clean. During diagnosis this cleared all 13,820
              real dim-768 vectors from the live store, which DISPROVED an
              inspection-derived hypothesis about an out-of-bounds
              reverse-link write at engram_vindex.c:340.

  concurrent  writer + reader on one shared index, TSan. Currently reports a
              race at engram_vindex.c:195 (visited_reset) reached from both
              vindex_search and vindex_insert, because VIndex still owns its
              visited[]/visit_epoch scratch — so even two concurrent READS
              corrupt each other's traversal.

Verified: half 1 passes, half 2 reproduces the race.

Gated on EXPECT_RACE, default 1, so the concurrent half documents the known
defect without failing the suite today. When the visited set moves to a
per-query checkout pool (hnswlib VisitedListPool style — NOT thread_local,
since http_worker is a thread per connection and a __thread buffer would leak
~55KB per connection), flip EXPECT_RACE=0 and it becomes a real gate.
2026-08-16 11:29:17 -05:00
bigmerge bdc1f99fb9 runtime: guard engram activation against the unsynchronized awareness thread
The soul daemon had two engram callers and only one of them locked.
soul.el:729 starts the HTTP server via http_serve_async (spawning
http_worker threads); soul.el:731 then runs awareness_run() on the MAIN
thread. awareness.el's perceive() -> engram_activate_json() ->
engram_activate() -> eg_vindex_sync() -> vindex_insert() mutates the same
g->nodes/g->edges and the process-global _eg_vindex HNSW index that the
workers touch. g_engram_req_lock existed to serialize exactly this, but it
was only ever taken inside http_worker: engram_req_lock/engram_req_unlock
appear in ZERO .el sources, so the awareness loop ran lock-free beside the
workers on every tick (SOUL_TICK_MS=1000).

Result was a crash-loop under launchd KeepAlive: five crashes in ~4 minutes
on 2026-08-16 with varying faulting frames -- search_layer<-vindex_insert
<-eg_vindex_sync, engram_activate, abort, and one inside xzm_realloc's own
freelist. Varying sites plus a fault in allocator metadata means heap
corruption. The SIGSEGV address 0x65646f4e6d617267 is little-endian ASCII
"gramNode": string bytes dereferenced as an Elem vector pointer.

Diagnosed by bisection rather than inspection:
  - Replaying all 13,820 real dim-768 vectors harvested from the live store
    through the index single-threaded under ASan is 100% clean, which rules
    out an HNSW logic/bounds bug.
  - Two threads on one index trip ThreadSanitizer immediately at
    engram_vindex.c:195 (visited_reset), reached from both vindex_search and
    vindex_insert. VIndex keeps a SHARED visited-epoch scratch buffer, so
    even two concurrent READS corrupt each other's traversal and walk bogus
    element indices.
So this is purely a concurrency defect, not an HNSW logic error. (An
inspection-derived hypothesis about an out-of-bounds reverse-link write at
engram_vindex.c:340 was disproved by the single-threaded run.)

Fix: a thread-local ownership depth (_eg_req_depth) lets engram entry points
self-guard. engram_activate() becomes a wrapper over engram_activate_inner()
that acquires g_engram_req_lock when called with depth 0 (the awareness
thread) and passes through when depth > 0 (nested inside an http_worker that
already holds it), so the non-recursive mutex cannot self-deadlock. The depth
is a plain counter, never a recursive-mutex count, preserving
engram_self_reify_beat_json's contract of genuinely releasing the lock
mid-beat.
2026-08-16 11:29:17 -05:00
will.anderson 44b621e551 runtime: anchor the think read, so Neuron can think at all (#142)
El SDK CI - dev / build-and-test (push) Failing after 13m0s
2026-08-16 16:26:00 +00:00
Neuron ded6ca546f runtime: anchor the think read, so Neuron can think at all
El SDK CI - dev / build-and-test (pull_request) Failing after 13m24s
engram_think_json passed NULL as the anchor. NULL is not "no opinion":
engram_think re-origins at `anchor ? anchor : region->centroid`, so NULL
means "read from the centroid" — and the centroid is the one point where
the gradient is zero by construction. r = x - centroid = 0, so every axis
projection is 0, grad is 0, and direction takes the "at rest" branch at
engram_cognition.c:137.

Measured consequence: EVERY faculty returned an identical null result,
differing only in its label —
  {"direction":[0,0,0,0,0,0,0,0],"spread":0,"magnitude":1,"confidence":0.5}
magnitude 1 is membership evaluated at the centroid, spread 0 is its
distance to itself, confidence 0.5 is the stance fallback. The geometry was
never at fault: /api/drift computes real values (centroid_sep 0.104,
core_disp 0.045) over the very same 87 members. Neuron could not think
because the read was always taken from the region's own centre.

The seeds choose WHICH region; they must also supply the VANTAGE. Anchor at
the first resolvable embedded seed — the same seed eg_geo_build_desc infers
dim from, so the two can never disagree. One seed still yields a real
gradient because the descriptor expands to that seed's neighbourhood, so
the seed's position is distinct from the neighbourhood centroid.

The vector is COPIED, never borrowed: g->nodes is realloc'd in place on
append, so a borrowed EngramNode* dangles across any concurrent write.

Verified against a clone of the production store (13,616 nodes / 37,865
edges):
  self anchor   n_support 87  magnitude 0.00282  spread 18.79
  values hub    n_support 28  magnitude 0.00318  spread 17.72
with distinct unit direction vectors. Previously both returned the zero
vector with magnitude 1 and spread 0.

STILL OPEN, now isolated by this fix: all five faculties return identical
numbers and confidence stays 0.5, because cog_stance_init is passed NULL
for the stance and the faculty enters the computation only through the
stance's axis_gain[] and bias_dir. The faculty label is inert until a
stance is loaded — which is what learn()'s correspondence-beat calibrates.
Same shape as this bug: a neutral parameter collapsing a capability to a
constant.
2026-08-16 11:25:24 -05:00
will.anderson b5b96c05ed runtime: let signal enter as geometry, not as prose about signal (#141)
El SDK CI - dev / build-and-test (push) Failing after 10m36s
2026-08-16 16:13:28 +00:00
Neuron c79033b749 runtime: let signal enter as geometry, not as prose about signal
El SDK CI - dev / build-and-test (pull_request) Failing after 10m55s
No ingest path could carry a vector. engram_node/_full/_layered take text
only, and a node acquired an embedding solely via engram_embed_backfill
DERIVING one from n->content. That made text the mandatory entry medium:
any non-text modality had to be described in prose first, so the geometry
we then reasoned over was the geometry OF THE DESCRIPTION, not of the
signal. Measured: POST /api/nodes accepted an "emb" field, returned 200
with a fresh id, and stored nothing — emb_dim=None, embedded=false.

engram_node_set_emb attaches a vector to an existing node. Off-dimension
vectors are stored but not indexed (the HNSW build loop already filters on
emb_dim), so modality geometry is durable and addressable without
perturbing the canonical index. Setting emb also makes the node ineligible
for embed_backfill, so a realizer's vector is never overwritten by a
text-derived one.

Two reporting fixes ride along, because both are how the drop stayed
invisible: the create response now reports emb_set instead of being
success-shaped regardless, and the node document now always emits emb_dim
and embedded — without which a genuine ingest drop and a mere reporting
gap are indistinguishable.

Verified live: voice node emb_dim=64 embedded=true; text control emb_dim=0
embedded=false; malformed hex, length mismatch and dim<=0 all reject.

KNOWN PLACEMENT DEFECT: this is at the consumer. Ingest is a language
concern, not an engram feature — every el program touching any modality
needs it. The vector also marshals as a hex STRING because el has no
first-class geometry value, which reintroduces text as the transport
medium one layer below the problem being fixed. The durable shape is
geometry as an el value plus declarable realizers, after which the engram
stops having an ingest concept at all. Landing this as the verified probe
that proves the path.
2026-08-16 11:12:48 -05:00
will.anderson 1119295238 Merge pull request 'runtime: state_get leaked its value on every call' (#140) from fix/state-get-leak into dev
El SDK CI - dev / build-and-test (push) Failing after 14m5s
2026-08-16 13:09:58 +00:00
bigmerge 9c07970943 runtime: state_get leaked its value on every call
El SDK CI - dev / build-and-test (pull_request) Failing after 14m26s
char* result = el_strdup_persist(e ? e->value : "");   // never freed
    pthread_mutex_unlock(&_state_mu);
    char* copy = el_strdup(result);                        // arena-tracked
    return el_wrap_str(copy);

Two copies were made. `result` existed only as the source for `copy` — never
returned, never freed — and el_strdup_persist bypasses the arena BY DESIGN
("state_set, engram internals"), so arena-pop could never reclaim it. Every
state_get leaked its full value string, permanently.

MEASURED: 200,000 state_get calls against a 64-byte value.
    before   15 MB peak RSS growth   (~75 bytes/call — the value plus overhead)
    after     0 MB

IMPACT. The soul's awareness loop has 68 state_get call sites and ticks every
200ms. Live measurement before the fix: RSS climbing 112 MB per 20s, about
19 GB/hour, in awareness_run -> one_cycle -> perceive, while node_count stayed
flat at ~13,479 — growth with no data behind it. It drove the host from 20 GB
free to 4.3 GB in roughly an hour.

WHY NOW, since the code is old: the soul used to restart constantly (no
write-through, divergent graph, 2.11 GB). Stabilising it (neuron #162) let it
stay up long enough to accumulate. The fix did not cause this leak; it removed
the crashes that were hiding it. Same pattern as the test framework surfacing
math_log — the defect was always there, something finally made it visible.

Found by Ishikawa rather than by reading the nearest code: method (arena
push/pop IS correctly paired per tick), material (node count flat, so not data
growth), environment (19 GB/hr / 18,000 ticks = ~1.1 MB per tick, so per-tick
not one-shot), machine (an allocator that bypasses the arena) — which is where
the evidence pointed.

el_strdup tracks into the thread-local arena, which touches no shared state, so
taking the single copy under _state_mu is safe and removes the temporary
entirely.

Verified: self-hosting fixpoint byte-identical; state round-trip correct for
hit, miss, and overwrite.
2026-08-16 08:09:32 -05:00
will.anderson 0832865952 Merge pull request 'test framework phase 3/4: black_box barrier + three-signal complexity gate, armed' (#139) from wt/soul-runtime-reconcile into dev
El SDK CI - dev / build-and-test (push) Failing after 11m32s
2026-08-16 03:02:18 +00:00
Neuron e0b2c0ea54 bench: arm the Phase 4 gate -- proven to pass clean AND fire on a quadratic
El SDK CI - dev / build-and-test (pull_request) Failing after 12m7s
Adds tests/native/test_lexer_scaling.el, the regression gate for el #132.

Both directions are proven on LIVE workloads, not synthetic series:
  healthy per-character scan  1821 3251 6007 10422 us -> O(n)   PASS
  rescan-from-zero (the #132 shape)  922 3667 13524 44792 -> O(n^2) FAIL

A gate only proven to pass is decoration. The quadratic specimen exists so
the gate is proven to FIRE.

Also fixes elb_spread_ok to judge the ASYMPTOTIC TAIL (last three ratios)
rather than the whole sweep. Measured on a genuinely linear scan the ratios
ran 3.37 2.92 1.76 1.65 -- the head looks quadratic because it is cold
cache, the tail is the truth. Whole-sweep spread rejected correct data. A
complexity bound is an asymptotic claim and must be judged asymptotically.

That fix came from the classifier refusing to rubber-stamp my own bad
measurement: it reported INDETERMINATE on an unwarmed sweep rather than
passing it. Warmup is now taken and discarded at every sweep point.

Reverts the == workarounds in test_elbench.el now that el #137 has landed;
the natural form generates no str_eq and all 13 fitter tests stay green.
The  workaround remains -- the Plus arm is still open.
2026-08-15 21:58:46 -05:00
bigmerge cf060adbfd Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile 2026-08-15 21:55:40 -05:00
will.anderson 63fe8a766d Merge pull request 'codegen: Bool is int-like, so Bool comparisons stop lowering to str_eq' (#138) from fix/bool-is-int-like into dev
El SDK CI - dev / build-and-test (push) Failing after 4m28s
2026-08-16 02:54:34 +00:00
Neuron 6a6b589ba0 bench: real black_box barrier + three-signal growth-curve gate
Adds el_black_box (inline asm, +r constraint, memory clobber) and
runtime/elbench.el: a growth-curve classifier that gates time AND
allocation-count AND allocation-bytes, failing if any exceeds its
declared curve.

Refusal is a first-class verdict. The classifier REFUSES rather than
classifying when the largest measurement is below the floor, or when a
series is hard-flat across an 8x input range -- the shape produced when
the optimiser deletes the work. Reporting O(1) there would be a
confident answer with nothing behind it. Disagreeing ratios report
INDETERMINATE rather than a guess.

Deviation from DESIGN.md 6.2, stated in the source: uses consecutive
ratios on a mandated geometric sweep rather than least-squares over
candidate curves. Ratios are directly interpretable on a doubling sweep
and need no floating point; the cost is weaker O(n) vs O(n log n)
separation, reported as an ambiguous band rather than guessed.

Documents the counter scope limit: engram_*.c and libcurl malloc are
NOT tracked, so a flat curve over engram/HTTP-dominated work is not
evidence of anything.

13 tests prove the classifier against real measured series from
fitprobe.el -- including that an accumulator's allocation COUNT is
linear while its bytes are quadratic, and that el #132's pure-CPU shape
reads FLAT on both allocation signals and is caught only by time.
2026-08-15 21:45:13 -05:00
bigmerge b5d1e53902 Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile 2026-08-15 21:38:11 -05:00
Neuron 6291a35bb9 design: gate on THREE signals -- the alloc gate would have missed el #132
el #132's quadratic (strlen per character in str_char_code/str_slice) is
pure CPU and allocates NOTHING. Measured on three controlled specimens:

  specimen  allocs        bytes         time
  linear    2.00 -> O(n)  2.16 -> O(n)  2.05 -> O(n)
  accum     2.00 -> O(n)  3.99 -> O(n2) noisy
  compute   FLAT          FLAT          3.96 -> O(n2)

'compute' is #132's shape. A gate fitting only allocation count and bytes
classifies it FLAT and passes -- it would not have caught the defect it
was created for. The gate now fits time AND count AND bytes, failing if
any exceeds its declared curve.

Also: black_box is mandatory and consuming the result is NOT sufficient.
The first 'compute' reported 0us at every n while returning a correct n2 --
clang closed the loop to a multiply. Only an opaque call restored the curve.

Adds lang/tests/bench/fitprobe.el as the fitter's known-good/known-bad set,
so the classifier is provable without depending on a real bug existing.
Marks DESIGN.md 1.3 stale: test_compiler 3.58s -> 0.03s (119x).
2026-08-15 21:34:39 -05:00
bigmerge 4c3414072b Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile 2026-08-15 21:30:52 -05:00
20 changed files with 2376 additions and 106 deletions
+49
View File
@@ -61,6 +61,13 @@ Per test file, current build model:
| `cc` test .c → .o | 0.02s |
| link | 0.02s |
> **STALE as of el #132 — re-measured 2026-08-16.** The `test_compiler` figure below was
> *entirely* the `strlen`-per-character quadratic, now fixed. Re-measured on the same host:
> **3.58s → 0.03s (119x)**, and the 422 KB compiler concatenation likewise compiles in 0.03s.
> The table is retained only as the historical record that motivated the gate. The remaining
> per-file cost is the redundant `el_runtime.c` rebuild, which §9's compile-once architecture
> addresses.
Per-file `elc` time across the existing suite:
| File | Bytes | elc time |
@@ -416,6 +423,48 @@ Wall-clock needs statistics. **Allocation counts do not.** They are perfectly de
> a level. That is why the gate fits a curve across a sweep instead of comparing one number to a
> threshold.
> **Second correction, same day — THE ALLOCATION GATE ALONE WOULD HAVE MISSED THE REAL BUG.**
>
> el #132 found the actual elc quadratic: `strlen()` called inside `str_char_code()` and
> `str_slice()`, so the lexer rescanned the remaining input on every character. Pure CPU.
> **Zero allocation.** `str_char_code` is a bounds check and an index — it allocates nothing.
>
> Measured on three controlled specimens (`lang/.work/fitprobe.el`), growth ratio per doubling of
> n across n = 200/400/800/1600:
>
> | specimen | allocs | bytes | time | what it proves |
> |---|---|---|---|---|
> | `linear` — one alloc per item | 2.00 2.00 2.00 → **O(n)** | 2.16 2.07 2.23 → **O(n)** | 0.83 2.00 2.05 → **O(n)** | clean baseline |
> | `accum` — rebuilds accumulator | 2.00 2.00 2.00 → **O(n)** | 3.97 3.99 3.99 → **O(n²)** | noisy | count misses, **bytes catches** |
> | `compute` — n scans over n chars | 0 → **FLAT** | 0 → **FLAT** | 3.93 4.01 3.96 → **O(n²)** | **both alloc signals blind; only time catches** |
>
> `compute` is el #132's shape exactly. A gate fitting only allocation count and bytes classifies
> it as FLAT and passes it. **The gate as originally specified would not have caught the defect it
> was created for.**
>
> Therefore the gate fits **THREE** signals and fails if ANY exceeds its declared curve:
>
> ```
> bench "elc_compile" over n in [...] expect time O(n) allocs O(n) bytes O(n) { ... }
> ```
>
> - **allocs (count)** — deterministic, zero-noise. Catches per-item allocation growth.
> - **allocs (bytes)** — deterministic, zero-noise. Catches accumulator-rebuild quadratics that
> count cannot see.
> - **time** — noisy, needs the sweep and statistics. The ONLY signal that sees pure-compute
> complexity regressions. Gate on the fitted *exponent*, never on absolute duration, so CI
> hardware variance scales the coefficient and leaves the classification intact.
>
> The deterministic signals remain preferable where they apply — they need no statistics and are
> correct on the first run. They are simply not sufficient.
>
> **`black_box` is mandatory, and consuming the result is NOT enough.** The first version of
> `compute` accumulated `total + 1` in a nested loop and reported **0 µs at every n** while
> returning a numerically correct n². Clang recognised the idiom and closed the loop to a
> multiply. Feeding the result into output did not prevent it. Only making the inner operation an
> opaque external call restored the real curve. A benchmark harness that trusts the user to defeat
> the optimiser will silently measure nothing — and report success while doing it.
Instrument the runtime with allocation counters and fit *those* against n instead of time:
```el
+109 -37
View File
@@ -10,10 +10,60 @@
// cc -std=c11 -O2 -lcurl -lpthread -o engram server.c el_runtime.c
// ./engram
//
// Configuration via environment:
// ENGRAM_BIND host:port (default :8742)
// ENGRAM_API_KEY bearer auth (optional)
// ENGRAM_DATA_DIR snapshot location (default ~/.neuron/engram)
// Configuration is DECLARED, not scattered. See the `program` block below:
// every knob's type and default lives there and nowhere else, is resolved from
// the environment (env wins, declaration is the fallback) and validated before
// any statement of this file runs. Read one with config("NAME") -> String.
//
// The one deliberate exception is ENGRAM_DATA_DIR see the note in the block.
// Program declaration (cross-cutting concerns)
//
// singleton: two engram processes against one data dir is data loss, not a
// warning. The runtime takes an exclusive flock at startup and a second start
// is refused loudly with the holder's pid.
//
// NOT declared here, on purpose: ENGRAM_DATA_DIR. Its resolution is owned by
// engram_resolve_data_dir() (el_runtime.c), which defaults to $HOME/.neuron/engram
// and fails LOUD rather than silently persisting to an ephemeral directory.
// Declaring a default for it here as well would put the data dir's fallback in
// two places which is precisely the defect this migration removes (until
// 2026-08-15 the reseed backup path carried its own "/tmp/engram" default that
// disagreed with the resolver, so the pre-destructive safety copy landed in /tmp).
// HOME is likewise not declared: it is a genuine environment read, not a knob.
program "engram" {
singleton: "engram"
// Core server
env ENGRAM_BIND: String = ":8742"
// Default "" leaves auth DISABLED (check_auth_ok short-circuits to true on an
// empty key). That is the pre-existing behaviour and is deliberately preserved
// here; making this `required` is the obvious hardening follow-up, but it is a
// behaviour change and out of scope for this migration.
env ENGRAM_API_KEY: String = ""
// Feature flags (bool-ish Strings; the predicate fns below own truthiness) ──
env ENGRAM_STORE: String = "off"
env ENGRAM_WAL: String = "off"
env ENGRAM_AUTOCONNECT: String = "off"
env ENGRAM_ISE_OFFGRAPH: String = "off"
// ISE telemetry
env ENGRAM_ISE_RETENTION_MS: Int = "172800000"
// Guide (local Qwen3 via llama-server)
env GUIDE_ENABLE: String = "off"
env GUIDE_TIER_FORCE: String = ""
env GUIDE_CACHE_DIR: String = ""
env GUIDE_RAM_GB_4B: Int = "16"
env GUIDE_RAM_GB_1P7B: Int = "8"
env GUIDE_BACKEND: String = "llama-server"
env GUIDE_HOST: String = "127.0.0.1"
env GUIDE_PORT: Int = "8771"
env GUIDE_LLAMA_SERVER_BIN: String = "llama-server"
env GUIDE_NGL: Int = "99"
env GUIDE_CTX: Int = "4096"
}
// Helpers
@@ -133,7 +183,7 @@ fn route_text_health(method: String, path: String, body: String) -> String {
// engram_store_enabled() in el_runtime.c EXACTLY (1 / on / true). Default off
// every persistence path below is byte-for-byte the historical snapshot behavior.
fn store_on() -> Bool {
let v: String = env("ENGRAM_STORE")
let v: String = config("ENGRAM_STORE")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -162,7 +212,6 @@ fn persist_canonical() -> Int {
if store_on() {
return engram_store_checkpoint()
}
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
// (2026-08-10 self-review) This returned a hardcoded 1, which made every
// caller's `let saved: Int = persist_canonical()` a dead variable six
@@ -176,7 +225,7 @@ fn persist_canonical() -> Int {
// per-write full-snapshot behavior. When ON, structural mutations append O(1)
// WAL records instead of rewriting the whole graph, with threshold compaction.
fn wal_on() -> Bool {
str_eq(env("ENGRAM_WAL"), "on")
str_eq(config("ENGRAM_WAL"), "on")
}
// autoconnect_on ENGRAM_AUTOCONNECT. Will's rule: "we shouldn't be inserting
@@ -184,7 +233,7 @@ fn wal_on() -> Bool {
// edge (kNN over embeddings) so no content node enters the graph edgeless.
// Default OFF -> byte-identical to prior behavior (node created, no auto edges).
fn autoconnect_on() -> Bool {
let v: String = env("ENGRAM_AUTOCONNECT")
let v: String = config("ENGRAM_AUTOCONNECT")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -197,7 +246,7 @@ fn autoconnect_on() -> Bool {
// separate state-event log tier instead of the node graph. Default OFF -> ISEs
// remain graph nodes exactly as before (with 48h prune).
fn ise_offgraph_on() -> Bool {
let v: String = env("ENGRAM_ISE_OFFGRAPH")
let v: String = config("ENGRAM_ISE_OFFGRAPH")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -288,6 +337,27 @@ fn route_create_node(method: String, path: String, body: String) -> String {
salience, importance, confidence,
tier, tags
)
// GEOMETRY INGEST (2026-08-16 self-review): this route accepted an "emb"
// field, returned 200 with a fresh id, and stored NOTHING engram_node_full
// has no vector parameter, so the caller's geometry was silently discarded
// and the node came back emb_dim=None / embedded:false. Measured live while
// trying to admit a voice signal. The consequence was structural, not
// cosmetic: text was the only entry medium, so any non-text modality had to
// be DESCRIBED in prose and what we then reasoned over was the geometry of
// the description, not of the signal.
//
// "emb" is little-endian float32 hex (dim*8 chars) the encoding the
// perception vessel's /voice/embed already emits, so a realizer's output
// moves in with no float-array round trip. "dim" defaults to the vector's
// implied width. Off-dimension vectors are stored but not inserted into the
// resident index (its build loop filters on emb_dim), so a modality vector
// is durable and addressable without perturbing the canonical index.
let emb_hex: String = json_get_string(body, "emb")
let emb_set: Int = if str_eq(emb_hex, "") { 0 } else {
let dim_raw: String = json_get_raw(body, "dim")
let dim: Int = if str_eq(dim_raw, "") { str_len(emb_hex) / 8 } else { json_get_int(body, "dim") }
engram_node_set_emb(id, emb_hex, dim)
}
let saved: Int = persist_node(id)
// ORPHAN PREVENTION (ENGRAM_AUTOCONNECT): connect the fresh node to its
// nearest embedded neighbors so it never enters the graph edgeless.
@@ -298,7 +368,11 @@ fn route_create_node(method: String, path: String, body: String) -> String {
if added > 0 { let sv2: Int = persist_edges_since(ec0) }
added
} else { 0 }
"{\"id\":\"" + id + "\",\"content\":\"" + content + "\",\"node_type\":\"" + node_type + "\",\"connected\":" + int_to_str(connected) + "}"
// Report whether the supplied geometry actually landed. The old response
// was success-shaped regardless 200 with an id while the vector was
// discarded which is how the drop went unnoticed. A caller can now
// assert on emb_set instead of trusting the status code.
"{\"id\":\"" + id + "\",\"content\":\"" + content + "\",\"node_type\":\"" + node_type + "\",\"connected\":" + int_to_str(connected) + ",\"emb_set\":" + int_to_str(emb_set) + "}"
}
fn route_get_node(method: String, path: String, body: String) -> String {
@@ -333,7 +407,6 @@ fn route_scan_nodes(method: String, path: String, body: String) -> String {
// process ever booted with a partial/empty store, the first read request
// clobbered the good snapshot. Read routes must never write the canonical path.)
fn route_scan_edges(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
let snap_path: String = dir + "/.scan-export.json"
engram_save(snap_path)
@@ -494,7 +567,6 @@ fn route_forget(method: String, path: String, body: String) -> String {
fn route_save(method: String, path: String, body: String) -> String {
let p_raw: String = json_get_string(body, "path")
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw }
// (2026-08-10 self-review) engram_save returns 0 on an empty path and the
@@ -578,7 +650,6 @@ fn route_drift(method: String, path: String, body: String) -> String {
fn route_load(method: String, path: String, body: String) -> String {
let p_raw: String = json_get_string(body, "path")
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw }
// (2026-08-10 self-review) This was a stub response over the single most
@@ -649,7 +720,6 @@ fn route_embed_backfill(method: String, path: String, body: String) -> String {
// (it skips nodes already present by ID). Auth-exempt: same-host internal call.
// (2026-06-27 self-review: added this route to fix silent 10-min sync failures)
fn route_sync(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
// 2026-07-21 self-review: export to a scratch path, never the canonical
// snapshot.json read routes must not be able to clobber the good snapshot.
@@ -725,8 +795,12 @@ fn route_reseed_nodes(method: String, path: String, body: String) -> String {
if str_eq(p, "") { return err_json("path is required") }
if str_eq(fs_read(p), "") { return err_json("file missing or empty") }
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw }
// (2026-08-15) This site carried its own "/tmp/engram" fallback, which
// DISAGREED with engram_resolve_data_dir() ($HOME/.neuron/engram, fail-loud).
// The consumer is the pre-destructive backup below, so with ENGRAM_DATA_DIR
// unset the safety copy taken before a reseed landed in an ephemeral /tmp
// while the store it was protecting lived elsewhere. One owner, one answer.
let dir: String = engram_resolve_data_dir()
let backup: String = dir + "/.reseed-backup.json"
let replace_raw: String = json_get_raw(body, "replace")
@@ -818,8 +892,7 @@ fn route_emit_ise(method: String, path: String, body: String) -> String {
sal, imp, conf,
"Episodic", "[\"internal-state\",\"InternalStateEvent\"]"
)
let ret_raw: String = env("ENGRAM_ISE_RETENTION_MS")
let ret_ms: Int = if str_eq(ret_raw, "") { 172800000 } else { str_to_int(ret_raw) }
let ret_ms: Int = str_to_int(config("ENGRAM_ISE_RETENTION_MS"))
let pruned: Int = engram_prune_telemetry(ret_ms)
"{\"ok\":true,\"id\":\"" + id + "\",\"pruned\":" + int_to_str(pruned) + "}"
}
@@ -1068,14 +1141,12 @@ fn route_correspondence_beat(method: String, path: String, body: String) -> Stri
// turns native thinking ON: the response carries reasoning_content (the thinking)
// alongside content (the answer).
fn guide_env_or(key: String, dflt: String) -> String {
let v: String = env(key)
if str_eq(v, "") { return dflt }
return v
}
// (2026-08-15) guide_env_or(key, dflt) lived here. Its whole job was supplying a
// per-call-site default, which is now the program block's job every GUIDE_* knob
// is declared once at the top of this file and read straight through config().
fn guide_enabled() -> Bool {
let v: String = env("GUIDE_ENABLE")
let v: String = config("GUIDE_ENABLE")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
@@ -1120,15 +1191,15 @@ fn guide_probe_metal() -> Bool {
// 2. Tier selection (config-driven thresholds, spec-autoselected)
fn guide_threshold_4b() -> Int {
return str_to_int(guide_env_or("GUIDE_RAM_GB_4B", "16"))
return str_to_int(config("GUIDE_RAM_GB_4B"))
}
fn guide_threshold_1p7b() -> Int {
return str_to_int(guide_env_or("GUIDE_RAM_GB_1P7B", "8"))
return str_to_int(config("GUIDE_RAM_GB_1P7B"))
}
// GUIDE_TIER_FORCE overrides the spec autoselect (used to prove cheaply on 0.6b).
fn guide_select_tier(ram_gb: Int) -> String {
let forced: String = env("GUIDE_TIER_FORCE")
let forced: String = config("GUIDE_TIER_FORCE")
if !str_eq(forced, "") { return forced }
if ram_gb >= guide_threshold_4b() { return "4b" }
if ram_gb >= guide_threshold_1p7b() { return "1.7b" }
@@ -1148,8 +1219,10 @@ fn guide_file(tier: String) -> String {
}
fn guide_cache_dir() -> String {
let c: String = env("GUIDE_CACHE_DIR")
let c: String = config("GUIDE_CACHE_DIR")
if !str_eq(c, "") { return c }
// HOME stays a raw env() read: it is the ambient environment, not a knob of
// this program, and it is deliberately absent from the program block.
let home: String = env("HOME")
if !str_eq(home, "") { return home + "/.neuron/guide/models" }
return engram_resolve_data_dir() + "/guide-models"
@@ -1190,9 +1263,9 @@ fn guide_fetch(tier: String) -> Bool {
}
// 4/5. Backend abstraction + BIND as an engageable interlocutor
fn guide_backend() -> String { return guide_env_or("GUIDE_BACKEND", "llama-server") }
fn guide_host() -> String { return guide_env_or("GUIDE_HOST", "127.0.0.1") }
fn guide_port() -> String { return guide_env_or("GUIDE_PORT", "8771") }
fn guide_backend() -> String { return config("GUIDE_BACKEND") }
fn guide_host() -> String { return config("GUIDE_HOST") }
fn guide_port() -> String { return config("GUIDE_PORT") }
fn guide_base_url() -> String { return "http://" + guide_host() + ":" + guide_port() }
// guide_healthy is the guide present and answering? llama-server's /health
@@ -1210,9 +1283,9 @@ fn guide_healthy() -> Bool {
fn guide_load(tier: String) -> Bool {
if guide_healthy() { return true }
let path: String = guide_model_path(tier)
let bin: String = guide_env_or("GUIDE_LLAMA_SERVER_BIN", "llama-server")
let ngl: String = guide_env_or("GUIDE_NGL", "99")
let ctx: String = guide_env_or("GUIDE_CTX", "4096")
let bin: String = config("GUIDE_LLAMA_SERVER_BIN")
let ngl: String = config("GUIDE_NGL")
let ctx: String = config("GUIDE_CTX")
let logf: String = guide_cache_dir() + "/llama-server." + guide_port() + ".log"
let cmd: String = bin + " -m '" + path + "' --host " + guide_host() + " --port " + guide_port() + " -c " + ctx + " -ngl " + ngl + " --jinja >> '" + logf + "' 2>&1"
let pid: String = exec_bg(cmd)
@@ -1608,7 +1681,7 @@ fn route_supersede(method: String, path: String, body: String) -> String {
// Auth
fn check_auth_ok(method: String, body: String) -> Bool {
let key: String = env("ENGRAM_API_KEY")
let key: String = config("ENGRAM_API_KEY")
if str_eq(key, "") { return true }
// Read-only methods don't require auth. Until http_serve surfaces
// request headers we can't accept a Bearer token cleanly; mutating
@@ -1871,8 +1944,7 @@ fn handle_request(method: String, path: String, body: String) -> String {
// Entry
let bind_raw: String = env("ENGRAM_BIND")
let bind_str: String = if str_eq(bind_raw, "") { ":8742" } else { bind_raw }
let bind_str: String = config("ENGRAM_BIND")
let port: Int = parse_port(bind_str)
// On startup, try to load any existing snapshot (best effort).
+107
View File
@@ -0,0 +1,107 @@
#!/usr/bin/env bash
# run_vindex_concurrency_tests.sh — regression harness for the 2026-08-16 soul crash.
#
# Four halves. The SET is the point: it separates two hazards the original two-half
# version conflated, and which have fixes in different files.
#
# 1. single ASan+UBSan, one thread. MUST be clean. Hard failure.
#
# 2. readers TSan, N readers, NO writer. Hazard (a): the visited set used
# to live on the index, so two pure READS stamped each other's
# epoch. Fixed in engram_vindex.c (frame-owned VVisit +
# `const VIndex*` search). MUST be clean. Hard failure.
#
# 3. unsynchronized TSan, writer + reader on a BARE index. Hazard (b): in-place
# HNSW insert rewires existing elements' neighbour lists and
# reallocs elems[]. EXPECTED TO RACE, PERMANENTLY. This is not
# a bug to fix inside engram_vindex.c — it is the executable
# proof that a publication boundary must exist above it.
# Not a failure. If it ever goes CLEAN, the test stopped
# interleaving and half 4 is no longer meaningful either.
#
# 4. published TSan, owner + N readers through a publication boundary
# (rwlock: readers shared, owner exclusive) mirroring
# eg_vindex_view / eg_vindex_maintain in lang/runtime/el_runtime.c.
# MUST be clean, and all inserts must land. Hard failure.
#
# See test_vindex_concurrency.c for the full story (SIGSEGV at ASCII address
# "gramNode", heap corruption in xzm_realloc, etc).
#
# usage: run_vindex_concurrency_tests.sh
set -uo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
RUNTIME="$(cd "$HERE/../../lang/runtime" && pwd)"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
SRC="$HERE/test_vindex_concurrency.c"
VINDEX="$RUNTIME/engram_vindex.c"
fail=0
echo "== [1/4] single-threaded control under AddressSanitizer =="
cc -std=c11 -g -O1 -fsanitize=address,undefined -fno-omit-frame-pointer \
-I"$RUNTIME" -o "$WORK/single" "$SRC" "$VINDEX" -lm || { echo "BUILD FAILED"; exit 2; }
if ASAN_OPTIONS=detect_leaks=0 "$WORK/single" single; then
echo " -> OK"
else
echo " -> FAIL: the single-threaded control must always be clean."
echo " If this fails the bug is NOT (only) concurrency — look for a real"
echo " out-of-bounds or lifetime error in engram_vindex.c."
fail=1
fi
cc -std=c11 -g -O1 -fsanitize=thread -fno-omit-frame-pointer \
-I"$RUNTIME" -o "$WORK/conc" "$SRC" "$VINDEX" -lm || { echo "BUILD FAILED"; exit 2; }
# run_tsan <mode> <logfile>; echoes nothing, sets $tsan_raced
run_tsan() {
TSAN_OPTIONS="halt_on_error=0" "$WORK/conc" "$1" >"$2" 2>&1
tsan_rc=$?
if grep -q "ThreadSanitizer: data race" "$2"; then tsan_raced=1; else tsan_raced=0; fi
}
echo
echo "== [2/4] concurrent READERS, no writer (visited-set gate) =="
run_tsan readers "$WORK/readers.log"
if [ "$tsan_raced" = "1" ]; then
echo " -> REGRESSION: two concurrent reads still race."
grep -m1 -A6 "ThreadSanitizer: data race" "$WORK/readers.log" | sed 's/^/ /'
echo " The visited set was supposed to be owned by the call frame."
fail=1
else
echo " -> clean (concurrent reads are safe)"
fi
echo
echo "== [3/4] writer+reader on a BARE index (expected-race probe) =="
run_tsan unsynchronized "$WORK/unsync.log"
if [ "$tsan_raced" = "1" ]; then
echo " -> RACE DETECTED, as expected:"
grep -m1 -A4 "ThreadSanitizer: data race" "$WORK/unsync.log" | sed 's/^/ /'
echo " In-place HNSW insert mutates existing elements. Not fixable inside"
echo " engram_vindex.c — this is why the publication boundary exists."
else
echo " -> NOTE: no race reported. The probe did not interleave; half 4's"
echo " clean result proves less than it should. Investigate."
fi
echo
echo "== [4/4] owner+readers through the publication boundary (boundary gate) =="
run_tsan published "$WORK/pub.log"
if [ "$tsan_raced" = "1" ]; then
echo " -> REGRESSION: the publication boundary did not serialize the owner."
grep -m1 -A6 "ThreadSanitizer: data race" "$WORK/pub.log" | sed 's/^/ /'
fail=1
elif [ "$tsan_rc" != "0" ]; then
echo " -> FAIL: boundary clean under TSan but the run failed:"
tail -3 "$WORK/pub.log" | sed 's/^/ /'
fail=1
else
echo " -> clean (readers project concurrently; the owner's inserts all landed)"
fi
echo
[ "$fail" -eq 0 ] && echo "RESULT: PASS" || echo "RESULT: FAIL"
exit "$fail"
+251
View File
@@ -0,0 +1,251 @@
/* test_vindex_concurrency.c — regression test for the 2026-08-16 soul crash.
*
* WHAT BROKE: the soul daemon crash-looped (5 crashes in ~100s) with SIGSEGV in
* search_layer <- vindex_insert <- eg_vindex_sync, a SIGABRT, and a fault inside
* xzm_realloc's own freelist i.e. heap corruption. The SIGSEGV address
* 0x65646f4e6d617267 is little-endian ASCII "gramNode": string bytes being
* dereferenced as an Elem vector pointer.
*
* ROOT CAUSE: VIndex owns its traversal scratch (visited[] + visit_epoch), and
* search_layer mutates it via visited_reset(). So the index is unsafe for ANY
* concurrent use including two concurrent READS. soul.el starts http_serve_async
* (a thread per connection) and then runs awareness_run() on the main thread, which
* reaches the same global index through engram_activate; nothing serialized them.
*
* Neither hnswlib nor FAISS puts the visited set on the index: hnswlib checks one
* out of a VisitedListPool per query, FAISS uses a thread_local VisitedTable.
*
* THE ORIGINAL `concurrent` HALF CONFLATED TWO DISTINCT HAZARDS (2026-08-16). It ran
* a writer against a reader on one bare index, so it could not tell apart:
*
* (a) READ/READ corruption two searches stamping each other's visited epoch.
* A defect INSIDE engram_vindex.c, fixable there, and now fixed: the visited
* set moved to the call frame and vindex_search takes a `const VIndex*`.
*
* (b) WRITE/READ corruption vindex_insert rewires the neighbour lists of
* EXISTING elements and reallocs elems[], so an insert is a mutation of the
* whole structure. This is NOT fixable inside engram_vindex.c at any price:
* it is inherent to in-place HNSW. It requires a publication boundary ABOVE
* the data structure (el_runtime.c: eg_vindex_view / eg_vindex_maintain).
*
* Conflating them made the suite unfailable-then-unpassable: fixing (a) left (b)
* still racing, which reads as "the fix did not work" when in fact a different,
* correctly-located fix is what (b) needs. So the halves are now separate:
*
* single N clustered vectors, ONE thread, ASan. The CONTROL. Must always
* be clean. When this passes and a concurrent half fails, the defect
* is concurrency, not an out-of-bounds/logic error in the graph code.
* (On 2026-08-16 this control cleared all 13,820 real dim-768 store
* vectors under ASan, which DISPROVED an inspection-derived hypothesis
* about an out-of-bounds reverse-link write at engram_vindex.c:340.)
*
* readers N reader threads, NO writer, one shared index, TSan. This is
* hazard (a) in isolation. It RACED before the visited set moved off
* the index struct and must be CLEAN now. Hard gate.
*
* unsynchronized writer + reader on a bare index, TSan. Hazard (b) in isolation.
* EXPECTED TO RACE, permanently it is the executable proof that
* the index cannot be made safe from the inside, and therefore that
* the publication boundary in el_runtime.c has to exist. If this
* ever goes clean, the test stopped interleaving; do not celebrate.
*
* published writer + readers through a publication boundary that mirrors
* eg_vindex_view / eg_vindex_maintain (rwlock: readers shared,
* the single owner exclusive), TSan. Must be CLEAN. Hard gate.
* This is what proves the shape of the runtime fix, in the same
* process, rather than asserting it.
*
* Absence of a crash does NOT mean absence of a race always read the sanitizer
* verdict, never just the exit code.
*
* Build/run: engram/test/run_vindex_concurrency_tests.sh
*/
#include "engram_vindex.h"
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#define DIM 128
#define NVEC 3000
#define SEED_N 50
static VIndex* g_ix;
static float* g_vecs;
/* Deterministic filler. Real embeddings are strongly correlated, not uniform noise;
* clustering keeps many candidates near-equidistant, which exercises the diversity
* heuristic and the visited set far harder than random vectors do. */
static void fill_vectors(void) {
g_vecs = (float*)malloc((size_t)NVEC * DIM * sizeof(float));
if (!g_vecs) { fprintf(stderr, "OOM\n"); exit(1); }
for (int i = 0; i < NVEC; i++) {
int cluster = i % 8;
for (int d = 0; d < DIM; d++)
g_vecs[(size_t)i * DIM + d] =
(float)(((d + cluster * 7) % 13) / 13.0) +
(float)(((i * 2654435761u + (unsigned)d) % 97) / 9700.0);
}
}
static void* writer_fn(void* arg) {
(void)arg;
for (int i = SEED_N; i < NVEC; i++)
(void)vindex_insert(g_ix, (uint64_t)i, g_vecs + (size_t)i * DIM);
return NULL;
}
static void* reader_fn(void* arg) {
(void)arg;
uint64_t ids[8]; float ds[8];
for (int i = 0; i < 20000; i++)
(void)vindex_search(g_ix, g_vecs + (size_t)(i % NVEC) * DIM, 8, 0, ids, ds);
return NULL;
}
static int run_single(void) {
printf("[single] inserting %d vectors on one thread (ASan control)\n", NVEC);
g_ix = vindex_create(DIM, 0, 0);
if (!g_ix) { fprintf(stderr, "[single] vindex_create failed\n"); return 1; }
for (int i = 0; i < NVEC; i++) {
if (vindex_insert(g_ix, (uint64_t)i, g_vecs + (size_t)i * DIM) != 0) {
fprintf(stderr, "[single] insert %d failed\n", i); return 1;
}
}
if (vindex_size(g_ix) != (size_t)NVEC) {
fprintf(stderr, "[single] size %zu != %d\n", vindex_size(g_ix), NVEC); return 1;
}
uint64_t ids[16]; float ds[16];
for (int q = 0; q < 200; q++) {
int k = vindex_search(g_ix, g_vecs + (size_t)((q * 7) % NVEC) * DIM, 16, 0, ids, ds);
if (k < 0) { fprintf(stderr, "[single] search failed at q=%d\n", q); return 1; }
}
vindex_free(g_ix); g_ix = NULL;
printf("[single] PASS — no memory error (this must ALWAYS pass)\n");
return 0;
}
/* Hazard (b) in isolation: writer + reader on a BARE index, no boundary. */
static int run_unsynchronized(void) {
printf("[unsynchronized] 1 writer + 1 reader on a BARE index (TSan probe)\n");
printf("[unsynchronized] a race here is EXPECTED and PERMANENT — in-place HNSW\n");
printf("[unsynchronized] insert rewires existing elements. This is the proof that\n");
printf("[unsynchronized] the publication boundary must live ABOVE engram_vindex.c.\n");
g_ix = vindex_create(DIM, 0, 0);
if (!g_ix) { fprintf(stderr, "[unsynchronized] vindex_create failed\n"); return 1; }
for (int i = 0; i < SEED_N; i++)
(void)vindex_insert(g_ix, (uint64_t)i, g_vecs + (size_t)i * DIM);
pthread_t w, r;
if (pthread_create(&w, NULL, writer_fn, NULL) ||
pthread_create(&r, NULL, reader_fn, NULL)) {
fprintf(stderr, "[unsynchronized] pthread_create failed\n"); return 1;
}
pthread_join(w, NULL);
pthread_join(r, NULL);
vindex_free(g_ix); g_ix = NULL;
printf("[unsynchronized] completed — CHECK THE SANITIZER VERDICT, not this line.\n");
return 0;
}
/* ── hazard (a) in isolation: concurrent READS only ───────────────────────────
* This is what the frame-owned visited set fixes. Before that change, two
* vindex_search calls on one index wrote each other's epoch stamp; TSan reported
* the race at visited_reset and the traversal then walked bogus element indices. */
#define NREADERS 4
static int run_readers(void) {
printf("[readers] %d concurrent readers, NO writer, one shared index (TSan)\n", NREADERS);
printf("[readers] this is the visited-set regression gate — must be CLEAN.\n");
g_ix = vindex_create(DIM, 0, 0);
if (!g_ix) { fprintf(stderr, "[readers] vindex_create failed\n"); return 1; }
for (int i = 0; i < NVEC; i++)
(void)vindex_insert(g_ix, (uint64_t)i, g_vecs + (size_t)i * DIM);
pthread_t t[NREADERS];
for (int i = 0; i < NREADERS; i++)
if (pthread_create(&t[i], NULL, reader_fn, NULL)) {
fprintf(stderr, "[readers] pthread_create failed\n"); return 1;
}
for (int i = 0; i < NREADERS; i++) pthread_join(t[i], NULL);
vindex_free(g_ix); g_ix = NULL;
printf("[readers] completed — CHECK THE SANITIZER VERDICT, not this line.\n");
return 0;
}
/* ── the publication boundary, mirroring el_runtime.c ─────────────────────────
* Readers take the boundary SHARED and hold it across the whole search; the one
* owner takes it EXCLUSIVE to extend. Same shape as eg_vindex_view /
* eg_vindex_maintain. Note the reader's index pointer is `const VIndex*` the
* compiler, not this comment, is what stops a reader inserting. */
static pthread_rwlock_t g_pub = PTHREAD_RWLOCK_INITIALIZER;
static void* pub_writer_fn(void* arg) {
(void)arg;
for (int i = SEED_N; i < NVEC; i++) {
pthread_rwlock_wrlock(&g_pub);
(void)vindex_insert(g_ix, (uint64_t)i, g_vecs + (size_t)i * DIM);
pthread_rwlock_unlock(&g_pub);
}
return NULL;
}
static void* pub_reader_fn(void* arg) {
(void)arg;
uint64_t ids[8]; float ds[8];
for (int i = 0; i < 5000; i++) {
pthread_rwlock_rdlock(&g_pub);
const VIndex* view = g_ix; /* immutable view */
(void)vindex_search(view, g_vecs + (size_t)(i % NVEC) * DIM, 8, 0, ids, ds);
pthread_rwlock_unlock(&g_pub);
}
return NULL;
}
static int run_published(void) {
printf("[published] 1 owner + %d readers through a publication boundary (TSan)\n", NREADERS);
printf("[published] this is the eg_vindex_view/eg_vindex_maintain gate — must be CLEAN.\n");
g_ix = vindex_create(DIM, 0, 0);
if (!g_ix) { fprintf(stderr, "[published] vindex_create failed\n"); return 1; }
for (int i = 0; i < SEED_N; i++)
(void)vindex_insert(g_ix, (uint64_t)i, g_vecs + (size_t)i * DIM);
pthread_t w, r[NREADERS];
if (pthread_create(&w, NULL, pub_writer_fn, NULL)) {
fprintf(stderr, "[published] pthread_create failed\n"); return 1;
}
for (int i = 0; i < NREADERS; i++)
if (pthread_create(&r[i], NULL, pub_reader_fn, NULL)) {
fprintf(stderr, "[published] pthread_create failed\n"); return 1;
}
pthread_join(w, NULL);
for (int i = 0; i < NREADERS; i++) pthread_join(r[i], NULL);
if (vindex_size(g_ix) != (size_t)NVEC) {
fprintf(stderr, "[published] size %zu != %d — the owner lost inserts\n",
vindex_size(g_ix), NVEC);
vindex_free(g_ix); g_ix = NULL; return 1;
}
vindex_free(g_ix); g_ix = NULL;
printf("[published] all %d inserts landed; CHECK THE SANITIZER VERDICT too.\n", NVEC);
return 0;
}
int main(int argc, char** argv) {
const char* mode = (argc > 1) ? argv[1] : "single";
fill_vectors();
int rc;
if (!strcmp(mode, "single")) rc = run_single();
else if (!strcmp(mode, "readers")) rc = run_readers();
else if (!strcmp(mode, "unsynchronized")) rc = run_unsynchronized();
else if (!strcmp(mode, "published")) rc = run_published();
/* back-compat: the pre-split name meant the bare writer+reader probe. */
else if (!strcmp(mode, "concurrent")) rc = run_unsynchronized();
else {
fprintf(stderr, "usage: %s [single|readers|unsynchronized|published]\n", argv[0]);
rc = 2;
}
free(g_vecs);
return rc;
}
BIN
View File
Binary file not shown.
+84
View File
@@ -2907,6 +2907,7 @@ fn builtin_arity(name: String) -> Int {
if str_eq(name, "el_alloc_count") { return 0 }
if str_eq(name, "el_alloc_bytes") { return 0 }
if str_eq(name, "el_peak_rss") { return 0 }
if str_eq(name, "el_black_box") { return 1 }
if str_eq(name, "engram_neighbors_json") { return 3 }
if str_eq(name, "engram_activate_json") { return 2 }
if str_eq(name, "engram_stats_json") { return 0 }
@@ -3264,6 +3265,7 @@ fn is_top_level_decl(stmt: Map<String, Any>) -> Bool {
if kind == "EnumDef" { return true }
if kind == "Import" { return true }
if kind == "CgiBlock" { return true }
if kind == "ProgramBlock" { return true }
if kind == "ExternFn" { return true }
false
}
@@ -3276,6 +3278,55 @@ fn cgi_arg(value: String, has_value: Bool) -> String {
return "EL_NULL"
}
// -- Program block: cross-cutting concerns injected at the process boundary ----
//
// emit_program_init emit the `static void __el_program_init(void)` that
// carries a program's declared cross-cutting concerns. Called from main()
// BEFORE any user statement runs, so the guarantees hold for the whole process
// rather than depending on each call site remembering to ask for them.
//
// This is emitted at the point the `program` block is encountered, not buffered
// until main(). The streaming backend emits in source order and cannot hold a
// declaration's entry list alive until main(); emitting a named function here
// and calling it from main() means only a single bool has to survive.
//
// Order matters and is deliberate:
// 1. singleton FIRST if another instance already holds the lock, refuse and
// exit before touching configuration, ports, or any data directory.
// 2. config declarations resolve env-or-default, one declaration per entry.
// 3. validate LAST report EVERY missing/ill-typed entry at once, then exit.
fn el_bool_arg(b: Bool) -> String {
if b { return "EL_INT(1)" }
return "EL_INT(0)"
}
fn emit_program_init(stmt: Map<String, Any>) -> Void {
let pname: String = stmt["name"]
emit_line("static void __el_program_init(void) {")
let has_singleton: Bool = stmt["has_singleton"]
if has_singleton {
let sid: String = stmt["singleton"]
emit_line(" el_singleton_acquire(EL_STR(" + c_str_lit(sid) + "));")
}
let entries = stmt["entries"]
let n: Int = native_list_len(entries)
let i = 0
while i < n {
let e = native_list_get(entries, i)
let ename: String = e["name"]
let etype: String = e["etype"]
let edefault: String = e["default"]
let has_default: Bool = e["has_default"]
let erequired: Bool = e["required"]
let arg_def: String = cgi_arg(edefault, has_default)
emit_line(" el_config_declare(EL_STR(" + c_str_lit(ename) + "), EL_STR(" + c_str_lit(etype) + "), " + arg_def + ", " + el_bool_arg(has_default) + ", " + el_bool_arg(erequired) + ");")
let i = i + 1
}
emit_line(" el_config_validate(EL_STR(" + c_str_lit(pname) + "));")
emit_line("}")
emit_blank()
}
// -- VBD role enforcement ------------------------------------------------------
//
// Scan a function body for direct calls to DHARMA-restricted builtins
@@ -3598,6 +3649,20 @@ fn codegen(stmts: [Map<String, Any>], source: String) -> String {
}
}
// Program block: emit the cross-cutting init function before the user's
// functions so main() can call it (see emit_program_init).
let prog_have: Bool = false
let i = 0
while i < n {
let stmt = native_list_get(stmts, i)
let sk4: String = stmt["stmt"]
if str_eq(sk4, "ProgramBlock") {
emit_program_init(stmt)
let prog_have = true
}
let i = i + 1
}
// Function definitions
let i = 0
while i < n {
@@ -3616,6 +3681,9 @@ fn codegen(stmts: [Map<String, Any>], source: String) -> String {
// with the C-side parameters when fn main()'s body is folded in below.
emit_line("int main(int _argc, char** _argv) {")
emit_line(" el_runtime_init_args(_argc, _argv);")
if prog_have {
emit_line(" __el_program_init();")
}
if cgi_count >= 1 {
let cname: String = cgi_block["name"]
let cdid: String = cgi_block["dharma_id"]
@@ -4209,6 +4277,7 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
// Fix: copy the values out BEFORE the release (strings, so no dangling reference)
// and emit from these. No search, so the failure mode is removed rather than moved.
let cgi_have: Bool = false
let prog_have: Bool = false
let cgi_name_v: String = ""
let cgi_did_v: String = ""
let cgi_prin_v: String = ""
@@ -4330,6 +4399,14 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
// These are no-ops in codegen (forward decls already emitted)
// except a CgiBlock, whose declared identity must survive
// this release to be emitted as a compiled constant.
// A ProgramBlock's cross-cutting declarations are
// emitted HERE, as a named init function, because the
// streaming backend cannot hold the entry list alive
// until main(). Only the bool survives.
if str_eq(sk, "ProgramBlock") {
emit_program_init(stmt)
let prog_have = true
}
if str_eq(sk, "CgiBlock") {
let cgi_have = true
let cgi_name_v = stmt["name"]
@@ -4476,6 +4553,13 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
let kind2: String = state_get("__program_kind")
emit_line("int main(int _argc, char** _argv) {")
emit_line(" el_runtime_init_args(_argc, _argv);")
// Cross-cutting concerns declared by a `program` block run BEFORE anything
// else a singleton violation must refuse the start before this process
// touches a port or a data directory, and configuration must be validated
// before the first read of it rather than at each read site.
if prog_have {
emit_line(" __el_program_init();")
}
// cgi init if needed
let ns2: Int = native_list_len(sigs)
+1
View File
@@ -184,6 +184,7 @@ fn keyword_kind(word: String) -> String {
if word == "false" { return "Bool" }
if word == "cgi" { return "Cgi" }
if word == "service" { return "Service" }
if word == "program" { return "Program" }
if word == "manager" { return "Manager" }
if word == "engine" { return "Engine" }
if word == "accessor" { return "Accessor" }
+124 -1
View File
@@ -1967,6 +1967,113 @@ fn parse_stmt(tokens: [Any], pos: Int) -> Map<String, Any> {
}, p)
}
// program block: program "name" { singleton: "id", env NAME: Type = "default", ... }
//
// The program block is El's declaration surface for CROSS-CUTTING CONCERNS
// properties of the whole process rather than of any one function, which
// otherwise degrade into "remember to call this at every site" conventions.
//
// singleton: "id" process identity. The runtime takes an exclusive
// lock at startup; a SECOND start is refused, loudly,
// instead of two processes sharing one data dir.
// env NAME: T = "d" one configuration entry. Its type and its default
// are declared ONCE, here, and resolved+validated
// before main() body runs.
// env NAME: T required
// no default; the program refuses to start unless the
// variable is set.
//
// Both compile into calls injected at the head of main() the same boundary
// seam `cgi` already uses (codegen.el emit_program_init). No call site in the
// program body has to remember anything, which is the whole point.
if k == "Program" {
let p = pos + 1
let name = tok_value(tokens, p)
let p = p + 1
let p = expect(tokens, p, "LBrace")
let singleton = ""
let has_singleton = false
let entries = native_list_empty()
// Entry-scratch declared at loop-body level (not inside the branch) so
// that inner `let` forms compile to assignment rather than a C-scoped
// redeclaration the same idiom the service block above relies on.
let ename = ""
let etype = ""
let edefault = ""
let has_default = false
let erequired = false
let fname = ""
let fval = ""
let running = true
while running {
let k2 = tok_kind(tokens, p)
if k2 == "RBrace" {
let running = false
} else {
if k2 == "Eof" {
let running = false
} else {
let fname = tok_value(tokens, p)
let p = p + 1
if str_eq(fname, "env") {
// env NAME: Type [= "default"] [required]
let ename = tok_value(tokens, p)
let p = p + 1
let p = expect(tokens, p, "Colon")
let etype = tok_value(tokens, p)
let p = p + 1
let edefault = ""
let has_default = false
let erequired = false
let k3 = tok_kind(tokens, p)
if str_eq(k3, "Eq") {
let p = p + 1
let edefault = tok_value(tokens, p)
let has_default = true
let p = p + 1
}
let k4 = tok_kind(tokens, p)
if str_eq(k4, "Ident") {
let w = tok_value(tokens, p)
if str_eq(w, "required") {
let erequired = true
let p = p + 1
}
}
let entries = native_list_append(entries, {
"name": ename,
"etype": etype,
"default": edefault,
"has_default": has_default,
"required": erequired
})
} else {
// scalar field: `name: "value"`
let p = expect(tokens, p, "Colon")
let fval = tok_value(tokens, p)
let p = p + 1
if str_eq(fname, "singleton") {
let singleton = fval
let has_singleton = true
}
}
let k5 = tok_kind(tokens, p)
if k5 == "Comma" {
let p = p + 1
}
}
}
}
let p = expect(tokens, p, "RBrace")
return make_result({
"stmt": "ProgramBlock",
"name": name,
"singleton": singleton,
"has_singleton": has_singleton,
"entries": entries
}, p)
}
// assert <cond_expr> [ , <msg_expr> ]
// The message is optional if the next token after the condition is not a
// Comma, emit an empty string placeholder so the test still works.
@@ -2419,6 +2526,7 @@ fn scan_params_c(tokens: [Any], pos: Int) -> Map<String, Any> {
// toplevel_let: { "kind": "toplevel_let", "name": String, "ltype": String }
// cgi_block: { "kind": "cgi_block", "name": String }
// service_block: { "kind": "service_block", "name": String }
// program_block: { "kind": "program_block", "name": String }
//
// Import/TypeDef/EnumDef nodes are skipped (codegen treats them as no-ops).
//
@@ -2546,13 +2654,28 @@ fn scan_fn_sigs(tokens: [Any]) -> [Map<String, Any>] {
"name": name
})
let pos = p
} else {
// --- program block ---
if str_eq(k, "Program") {
let p: Int = pos + 1
let name: String = tok_value(tokens, p)
let p = p + 1
let k2: String = tok_kind(tokens, p)
if str_eq(k2, "LBrace") {
let p = skip_to_rbrace(tokens, p)
}
let sigs = native_list_append(sigs, {
"kind": "program_block",
"name": name
})
let pos = p
} else {
// Import, Type, Enum, From, or any other token.
// Skip ahead to the next statement boundary.
let p: Int = pos + 1
let p = skip_expr_to_stmt_boundary(tokens, p)
let pos = p
}}}}}
}}}}}}
}
}
}
+563 -23
View File
@@ -43,6 +43,7 @@
#include <dlfcn.h> /* dlsym for http_set_handler fallback */
#include <unistd.h>
#include <fcntl.h>
#include <sys/file.h> /* flock — process-identity singleton (program block) */
#include <dirent.h>
#include <errno.h>
#include <pthread.h>
@@ -1600,8 +1601,64 @@ typedef struct {
* no longer blocks ingest/reads (measured: non-health latency during a beat
* 13.9s sub-second). */
static pthread_mutex_t g_engram_req_lock = PTHREAD_MUTEX_INITIALIZER;
void engram_req_unlock(void){ pthread_mutex_unlock(&g_engram_req_lock); }
void engram_req_lock(void){ pthread_mutex_lock(&g_engram_req_lock); }
/* ── AWARENESS-THREAD GUARD (2026-08-16 self-review) ─────────────────────────
* The request lock above serialized http_worker threads against EACH OTHER, but
* the soul daemon has a SECOND, unsynchronized engram caller: soul.el starts the
* HTTP server with http_serve_async (spawning worker threads) and then runs
* awareness_run() on the MAIN thread, whose perceive() -> engram_activate_json()
* -> engram_activate() -> eg_vindex_sync() path mutates the very same RAM graph
* and the process-global _eg_vindex HNSW index. Nothing in any .el source ever
* called engram_req_lock, so that whole loop ran lock-free beside the workers.
*
* Measured consequence (2026-08-16): five crashes in ~4 minutes, all one bug
* SIGSEGV in search_layer<-vindex_insert<-eg_vindex_sync at address
* 0x65646f4e6d617267 (little-endian ASCII "gramNode": a string being
* dereferenced as an Elem vector pointer), plus a SIGABRT and a fault inside
* xzm_realloc's freelist, i.e. corrupted allocator metadata. Confirmed by
* bisection: replaying ALL 13,820 real dim-768 store vectors through the index
* single-threaded under ASan is 100% clean, while two threads on one index trip
* ThreadSanitizer instantly at engram_vindex.c:195 (visited_reset) VIndex keeps
* a SHARED visited-epoch scratch buffer, so even two concurrent READS stomp each
* other's traversal state and walk bogus element indices. So this is purely a
* concurrency defect, not a logic error in the HNSW code.
*
* Fix: a thread-local ownership depth lets engram entry points self-guard. A call
* arriving on the awareness thread (depth 0) acquires the lock; one arriving from
* inside an http_worker that already holds it (depth > 0) is a no-op, so there is
* no self-deadlock on this NON-recursive mutex. Depth is a plain counter, never a
* recursive-mutex count, which preserves engram_self_reify_beat_json's contract of
* really releasing the lock mid-beat (see engram_req_unlock at the reify beat).
*
* SCOPE NARROWED (2026-08-16, vindex publication boundary): this guard originally
* covered TWO hazards the RAM graph AND the process-global _eg_vindex. The vindex
* half is retired: the index now has its own publication boundary (_eg_vindex_rw),
* search takes a `const VIndex*`, and no read path can mutate the index at all.
*
* What REMAINS load-bearing here is the RAM graph alone, and it is a genuine,
* measured hazard independent of the index: g->nodes / g->edges are realloc'd in
* place (el_runtime.c:7618, 7629), so an awareness-thread reader holding
* `EngramNode* n = &g->nodes[i]` across a concurrent append from an http_worker
* holds a dangling pointer and engram_activate_inner's embed-backfill WRITES
* n->emb through exactly such a pointer. That is a separate residue with its own
* fix (the resident graph wants the same publication treatment the index just got);
* until it lands, this guard stays. Do NOT delete it as "the fb32d15 vindex lock". */
static __thread int _eg_req_depth = 0;
void engram_req_unlock(void){ if(_eg_req_depth > 0) _eg_req_depth--; pthread_mutex_unlock(&g_engram_req_lock); }
void engram_req_lock(void){ pthread_mutex_lock(&g_engram_req_lock); _eg_req_depth++; }
/* Acquire only if this thread does not already hold the request lock.
* Returns 1 if this call took ownership (caller must release), 0 if nested. */
static int eg_guard_enter(void){
if (_eg_req_depth > 0) return 0;
pthread_mutex_lock(&g_engram_req_lock);
_eg_req_depth++;
return 1;
}
static void eg_guard_exit(int owned){
if (!owned) return;
if (_eg_req_depth > 0) _eg_req_depth--;
pthread_mutex_unlock(&g_engram_req_lock);
}
static void* http_worker(void* arg) {
HttpWorkerArg* a = (HttpWorkerArg*)arg;
@@ -1642,7 +1699,7 @@ static void* http_worker(void* arg) {
(plen == 1 && path[0] == '/'))
health_exempt = 1;
}
if (!health_exempt) pthread_mutex_lock(&g_engram_req_lock);
if (!health_exempt) engram_req_lock(); /* tracks _eg_req_depth for eg_guard_enter */
if (h) {
el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), EL_STR(body));
const char* rs = EL_CSTR(r);
@@ -1669,7 +1726,7 @@ static void* http_worker(void* arg) {
}
/* end of the engram critical section — the response is now a private malloc'd
* copy; arena teardown + socket write touch no shared engram state. */
if (!health_exempt) pthread_mutex_unlock(&g_engram_req_lock);
if (!health_exempt) engram_req_unlock();
el_request_end(); /* free all intermediate strings */
_tl_http_head_only = head_only;
http_send_response(fd, response);
@@ -5168,10 +5225,23 @@ el_val_t state_get(el_val_t key) {
if (!k) return el_wrap_str(el_strdup(""));
pthread_mutex_lock(&_state_mu);
StateEntry* e = state_find(k);
char* result = el_strdup_persist(e ? e->value : "");
/* ONE arena-tracked copy, taken under the lock.
*
* This used to make TWO copies: an el_strdup_persist temporary, then an
* arena-tracked copy of that temporary. The persistent one was never
* returned and never freed el_strdup_persist bypasses the arena by
* design ("state_set, engram internals"), so arena-pop could not reclaim
* it. Every state_get therefore leaked its full value string, permanently.
*
* The soul's awareness loop has 68 state_get call sites and ticks every
* 200ms; measured leak was ~1.1 MB per tick, about 19 GB/hour. It went
* unnoticed for as long as the soul restarted often enough to mask it.
*
* el_strdup tracks into the thread-local arena, which touches no shared
* state, so doing it under _state_mu is safe and removes the need for the
* temporary entirely. */
char* copy = el_strdup(e ? e->value : "");
pthread_mutex_unlock(&_state_mu);
/* wrap in arena-tracked copy for the caller's request lifetime */
char* copy = el_strdup(result);
return el_wrap_str(copy);
}
@@ -8494,6 +8564,80 @@ el_val_t engram_node_count(void) {
return (el_val_t)engram_get()->node_count;
}
/* engram_node_set_emb — attach GEOMETRY to an existing node.
*
* WHY THIS EXISTS (2026-08-16). Until now no ingest path could carry a
* vector. engram_node / engram_node_full / engram_node_layered take text
* only, and the sole way a node acquired an embedding was
* engram_embed_backfill DERIVING one from n->content. That made text the
* mandatory entry medium: any non-text modality (audio, image, sensor)
* had to be described in prose first, and the geometry we then reasoned
* over was the geometry OF THE DESCRIPTION, not of the signal. Measured
* consequence: POST /api/nodes accepted an "emb" field, returned 200 with
* a fresh id, and stored emb_dim=None / embedded:false the vector was
* silently discarded because no parameter existed to receive it.
*
* `hex` is little-endian float32, the encoding the perception vessel's
* /voice/embed already emits, so a realizer's output moves in without a
* JSON float-array round trip. Length must be exactly dim*8 hex chars.
*
* DIMENSION POLICY: dim need NOT equal the canonical text-embedding dim.
* A modality vector of a different width is stored and is simply not
* inserted into the resident HNSW index, whose build loop already filters
* on `n->emb_dim == dim`. So off-dimension geometry is durable and
* addressable without perturbing the canonical index.
*
* Setting emb also makes the node ineligible for embed_backfill (which
* only fills nodes with no emb), so a realizer's vector is never
* overwritten by a text-derived one.
*
* Returns 1 on success, 0 on unknown id / malformed hex / bad dim. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim) {
const char* sid = EL_CSTR(id);
const char* sh = EL_CSTR(hex);
int32_t d = (int32_t)(int64_t)dim;
/* Bound the allocation. No max-dim constant existed because no caller
* could supply a dim before this function; 8192 is generous for any
* realizer (canonical text embeddings are 768, MFCC voice stats 64)
* while keeping a malformed `dim` from requesting an unbounded malloc. */
if (!sid || !*sid || !sh || d <= 0 || d > 8192) return (el_val_t)0;
size_t need = (size_t)d * 8u; /* 4 bytes → 8 hex chars per float */
if (strlen(sh) != need) return (el_val_t)0;
EngramNode* n = engram_find_node(sid);
if (!n) return (el_val_t)0;
float* v = (float*)malloc(sizeof(float) * (size_t)d);
if (!v) return (el_val_t)0;
for (int32_t i = 0; i < d; i++) {
uint32_t w = 0;
for (int k = 0; k < 8; k++) {
char c = sh[(size_t)i * 8u + (size_t)k];
uint32_t nib;
if (c >= '0' && c <= '9') nib = (uint32_t)(c - '0');
else if (c >= 'a' && c <= 'f') nib = (uint32_t)(c - 'a' + 10);
else if (c >= 'A' && c <= 'F') nib = (uint32_t)(c - 'A' + 10);
else { free(v); return (el_val_t)0; }
w = (w << 4) | nib;
}
/* Hex is emitted little-endian byte order; rebuild the word. */
uint32_t le = ((w & 0x000000FFu) << 24) | ((w & 0x0000FF00u) << 8) |
((w & 0x00FF0000u) >> 8) | ((w & 0xFF000000u) >> 24);
float f;
memcpy(&f, &le, sizeof(f));
v[i] = f;
}
free(n->emb);
n->emb = v;
n->emb_dim = d;
n->updated_at = engram_now_ms();
if (engram_store_enabled()) eg_store_put_node(n);
return (el_val_t)1;
}
/* ── Telemetry retention ────────────────────────────────────────────────────
* (2026-07-16 self-review) InternalStateEvent nodes are append-only telemetry
* (heartbeat, curiosity_scan, engram_sync) written ~3/min by the awareness
@@ -9477,6 +9621,35 @@ static double engram_goal_bias(const EngramNode* n, const char* query) {
* the exact O(n) argmax scan tops up any seed slot the ANN leaves unfilled.
* Single-threaded, matching the adjacent query-embedding cache (no lock).
* Returns NULL when no index is available caller falls back to the O(n) scan. */
/* ── VINDEX PUBLICATION BOUNDARY (2026-08-16) ────────────────────────────────
* The index is DERIVED GEOMETRY: a projection of the store's embeddings. The
* store is append-only and superseding, so a reader must be able to project
* against geometry that does not move under it.
*
* The HNSW index is NOT itself append-only: vindex_insert rewires the neighbour
* lists of ALREADY-EXISTING elements and reallocs elems[]. So "extend" is a
* mutation of the whole structure, and a reader holding element pointers across
* one is unsafe no matter how pure search itself is (measured: TSan reports the
* elems[] race even after the visited set moved to the call frame).
*
* Hence a publication boundary rather than an ownership discipline:
*
* - eg_vindex_maintain() is the ONLY mutator of the five statics below. It
* takes _eg_vindex_rw EXCLUSIVELY, so it never runs beside a reader.
* - eg_vindex_view() hands back a `const VIndex*` with the boundary held for
* READ. N readers project concurrently; none can mutate, because search
* takes a const index and the compiler enforces it.
*
* A read path may DEMAND that a current snapshot exist that is a request to
* the owner, not a mutation by the reader. What it may not do is mutate the
* geometry it is projecting against. eg_vindex_view/eg_vindex_maintain is
* exactly that split.
*
* Lock ordering: request-outer -> vindex -> store-inner. The vindex boundary is
* never held across a call that can re-enter eg_vindex_view/maintain (verified:
* the four read regions each acquire, search, release without nesting). */
static pthread_rwlock_t _eg_vindex_rw = PTHREAD_RWLOCK_INITIALIZER;
static VIndex* _eg_vindex = NULL;
static int32_t _eg_vindex_dim = 0;
static int64_t _eg_vindex_built_nc = 0; /* g->node_count at last (re)build */
@@ -9496,8 +9669,10 @@ static int eg_vindex_seen_ensure(int64_t need) {
return 0;
}
static VIndex* eg_vindex_sync(EngramStore* g, int32_t dim) {
if (!g || dim <= 0) return _eg_vindex;
/* THE OWNER. The only function that mutates _eg_vindex* — must be called with
* _eg_vindex_rw held EXCLUSIVELY (see eg_vindex_maintain, the sole caller). */
static void eg_vindex_publish_locked(EngramStore* g, int32_t dim) {
if (!g || dim <= 0) return;
/* Drop a stale index: embedder dim changed, or the resident array shrank
* (indices may have been reused/reordered cached node_ids unsafe). */
if (_eg_vindex && (_eg_vindex_dim != dim || g->node_count < _eg_vindex_built_nc)) {
@@ -9507,8 +9682,8 @@ static VIndex* eg_vindex_sync(EngramStore* g, int32_t dim) {
}
if (!_eg_vindex) {
VIndex* idx = vindex_create((int)dim, 0, 0);
if (!idx) return NULL;
if (eg_vindex_seen_ensure(g->node_count)) { vindex_free(idx); return NULL; }
if (!idx) return;
if (eg_vindex_seen_ensure(g->node_count)) { vindex_free(idx); return; }
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
if (n->emb && n->emb_dim == dim && vindex_insert(idx, (uint64_t)i, n->emb) == 0)
@@ -9532,8 +9707,62 @@ static VIndex* eg_vindex_sync(EngramStore* g, int32_t dim) {
}
_eg_vindex_built_nc = g->node_count;
}
}
/* Owner-mediated publish. Takes the boundary EXCLUSIVELY, so it can never run
* beside a reader. Cheap no-op when the published snapshot is already current. */
static void eg_vindex_maintain(EngramStore* g, int32_t dim) {
if (!g || dim <= 0) return;
pthread_rwlock_wrlock(&_eg_vindex_rw);
eg_vindex_publish_locked(g, dim);
pthread_rwlock_unlock(&_eg_vindex_rw);
}
/* READ SIDE. Returns the published snapshot as an IMMUTABLE view, with the
* boundary held for READ the caller MUST pair every call with exactly one
* eg_vindex_view_release(), on every path including error returns.
*
* The returned pointer is `const`: a read path physically cannot call
* vindex_insert on it. That is the compile-time constraint, and it is why this
* replaces eg_vindex_sync rather than wrapping it. May return NULL (no index
* available -> caller falls back to the exact O(n) scan); the boundary is still
* held and still must be released. */
static const VIndex* eg_vindex_view(EngramStore* g, int32_t dim) {
if (g && dim > 0) {
/* Fast path: snapshot already current, take it read-only and go. */
pthread_rwlock_rdlock(&_eg_vindex_rw);
if (_eg_vindex && _eg_vindex_dim == dim && _eg_vindex_built_nc == g->node_count)
return _eg_vindex;
/* Stale or absent. Drop to no lock, ask the owner to publish, re-acquire.
* NEVER upgrade rdlock->wrlock in place: that self-deadlocks. */
pthread_rwlock_unlock(&_eg_vindex_rw);
eg_vindex_maintain(g, dim);
}
pthread_rwlock_rdlock(&_eg_vindex_rw);
return _eg_vindex;
}
static void eg_vindex_view_release(void) {
pthread_rwlock_unlock(&_eg_vindex_rw);
}
/* WRITE-SIDE MAINTENANCE HOOK. Call after an embedding becomes present on a
* resident ordinal. A node without an embedding cannot be in a vector index at
* all, so embedding-assignment not node append is the event that owns index
* membership. Cheap: one O(log n) HNSW insert, no O(node_count) presence scan.
* A no-op before the first publish (the cold build picks the node up) and on a
* dim mismatch. */
static void eg_vindex_note_embedded(EngramStore* g, int64_t ordinal) {
if (!g || ordinal < 0 || ordinal >= g->node_count) return;
EngramNode* n = &g->nodes[ordinal];
if (!n->emb || n->emb_dim <= 0) return;
pthread_rwlock_wrlock(&_eg_vindex_rw);
if (_eg_vindex && _eg_vindex_dim == n->emb_dim &&
eg_vindex_seen_ensure(g->node_count) == 0 && !_eg_vindex_seen[ordinal]) {
if (vindex_insert(_eg_vindex, (uint64_t)ordinal, n->emb) == 0)
_eg_vindex_seen[ordinal] = 1;
}
pthread_rwlock_unlock(&_eg_vindex_rw);
}
/* ── M9 GEOMETRY PRIMING (ENGRAM_GEOMETRY_PRIMING, default OFF) ──────────────
* Opt-in wiring of the centered relational-neighborhood geometry (engram_geometry.c)
@@ -9640,7 +9869,9 @@ static int64_t engram_activate_beam(void) {
v = d; return v;
}
el_val_t engram_activate(el_val_t query, el_val_t depth) {
/* Core activation. Callers must hold the engram request lock — reached only via
* the engram_activate() wrapper below, which self-guards (see eg_guard_enter). */
static el_val_t engram_activate_inner(el_val_t query, el_val_t depth) {
EngramStore* g = engram_get();
const char* q = EL_CSTR(query);
int64_t max_depth = (int64_t)depth; if (max_depth <= 0) max_depth = 2;
@@ -9681,6 +9912,12 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
float* v = eg_embed_fetch(n->content, &d);
if (!v) break; /* embedder down / breaker open — stop this call */
n->emb = v; n->emb_dim = d;
/* Write-side index maintenance: an embedding just became present on
* ordinal i, so the index's owner publishes it now. This is what
* retires the "STALENESS (honest tradeoff)" note above a lazily
* embedded OLDER node no longer waits for a full rebuild to become
* visible to route_nearest / autoconnect. */
eg_vindex_note_embedded(g, i);
backfilled++;
}
}
@@ -9879,7 +10116,9 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
* same budget as the exact scan's retry `guard` so dedup/threshold
* rejects still leave enough distinct seeds. */
{
VIndex* vx = eg_vindex_sync(g, q_dim);
/* Immutable view: the boundary is held for READ across the whole
* search + harvest, and released at the end of this block. */
const VIndex* vx = eg_vindex_view(g, q_dim);
if (vx && (int64_t)vindex_size(vx) >= ENGRAM_EMBED_SEED_K) {
const float* seed_qv = e_eff ? e_eff : q_emb;
int kreq = ENGRAM_EMBED_SEED_K * 8;
@@ -9923,6 +10162,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
}
free(aid); free(ad);
}
eg_vindex_view_release();
}
/* Exact O(n) argmax fallback / top-up (pre-M8 selection, verbatim).
@@ -10009,9 +10249,11 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
char** vids = malloc((size_t)g->node_count * sizeof(char*));
if (gmean && vids) {
for (int64_t i = 0; i < g->node_count; i++) vids[i] = g->nodes[i].id;
const VIndex* gvx = eg_vindex_view(g, q_dim);
geo = engram_geometry_descriptor(
g_engram_store, _eg_vindex, vids, (int)g->node_count,
g_engram_store, gvx, vids, (int)g->node_count,
seed_ids, (size_t)nsel, NULL, gmean);
eg_vindex_view_release();
}
free(vids);
if (geo && geo->n_members > 0) {
@@ -11289,6 +11531,15 @@ static void engram_emit_node_json(JsonBuf* b, const EngramNode* n, int include_e
snprintf(tmp, sizeof(tmp), ",\"wm_anchor\":%g", n->wm_anchor); jb_puts(b, tmp);
snprintf(tmp, sizeof(tmp), ",\"base_level\":%g",
engram_bll_base_level(n, engram_now_ms())); jb_puts(b, tmp);
/* GEOMETRY VISIBILITY (2026-08-16 self-review): the node document never
* said whether the node carried a vector, so a read-back could not tell
* "has geometry" from "text only". Not cosmetic it is exactly how a
* real ingest drop and a mere reporting gap became indistinguishable,
* and I misdiagnosed one as the other for an hour. Always emit the width
* and the boolean; the vector itself stays behind include_emb since it
* is large and most callers do not want it inline. */
snprintf(tmp, sizeof(tmp), ",\"emb_dim\":%d,\"embedded\":%s",
(int)n->emb_dim, (n->emb && n->emb_dim > 0) ? "true" : "false"); jb_puts(b, tmp);
/* Base-level access history: chronological (oldest→newest) compact
* string. Loaders replay it through engram_bll_record_access; absent
* field = empty ring (optimized-form fallback). (2026-07-22) */
@@ -13152,13 +13403,16 @@ static int eg_knn_for_node(EngramStore* g, int64_t self, int want, uint64_t* out
if(self < 0 || self >= g->node_count) return 0;
EngramNode* n = &g->nodes[self];
if(!n->emb || n->emb_dim <= 0) return 0;
VIndex* vx = eg_vindex_sync(g, n->emb_dim);
if(!vx) return 0;
/* Immutable view held for READ across the search; the harvest below reads
* only g->nodes, so the boundary is released as soon as the search returns. */
const VIndex* vx = eg_vindex_view(g, n->emb_dim);
if(!vx){ eg_vindex_view_release(); return 0; }
int K = want + 8;
uint64_t* ids = (uint64_t*)malloc(sizeof(uint64_t)*(size_t)K);
float* dist = (float*)malloc(sizeof(float)*(size_t)K);
if(!ids || !dist){ free(ids); free(dist); return 0; }
if(!ids || !dist){ eg_vindex_view_release(); free(ids); free(dist); return 0; }
int m = vindex_search(vx, n->emb, K, 0, ids, dist);
eg_vindex_view_release();
int c = 0;
for(int j=0; j<m && c<want; j++){
int64_t bi = (int64_t)ids[j];
@@ -13189,7 +13443,8 @@ el_val_t engram_autoconnect_node(el_val_t id_v, el_val_t k_v, el_val_t minsim_v)
EngramNode* n = &g->nodes[self];
if((!n->emb || n->emb_dim <= 0) && n->content && eg_embed_eligible(n)){
int32_t d = 0; float* v = eg_embed_fetch(n->content, &d);
if(v && d > 0){ n->emb = v; n->emb_dim = d; if(engram_store_enabled()) eg_store_put_node(n); }
if(v && d > 0){ n->emb = v; n->emb_dim = d; if(engram_store_enabled()) eg_store_put_node(n);
eg_vindex_note_embedded(g, self); }
else free(v);
}
if(!n->emb || n->emb_dim <= 0){ jb_puts(&b, "{\"connected\":0,\"reason\":\"unembedded\"}"); return el_wrap_str(b.buf); }
@@ -13324,8 +13579,10 @@ static GeoDescriptor* eg_geo_build_desc(const char* csv) {
char** vids = malloc((size_t)g->node_count * sizeof(char*));
if (gmean && vids) {
for (int64_t i = 0; i < g->node_count; i++) vids[i] = g->nodes[i].id;
geo = engram_geometry_descriptor(g_engram_store, _eg_vindex, vids, (int)g->node_count,
const VIndex* gvx = eg_vindex_view(g, dim);
geo = engram_geometry_descriptor(g_engram_store, gvx, vids, (int)g->node_count,
(const char* const*)ids, (size_t)ns, NULL, gmean);
eg_vindex_view_release();
}
free(vids);
for (int i = 0; i < ns; i++) free(ids[i]);
@@ -13362,12 +13619,16 @@ el_val_t engram_geo_reify_run_json(void){
int32_t dim = 0;
for(int64_t i = 0; i < g->node_count && dim == 0; i++)
if(g->nodes[i].emb && g->nodes[i].emb_dim > 0) dim = g->nodes[i].emb_dim;
VIndex* vx = (dim > 0) ? eg_vindex_sync(g, dim) : NULL;
char** vids = malloc((size_t)g->node_count * sizeof(char*));
if(!vids) return eg_geo_err("reify oom");
for(int64_t i = 0; i < g->node_count; i++) vids[i] = g->nodes[i].id;
/* Held for READ across the whole reify pass: it only searches the index.
* (The multi-second SELF-reify beat below builds a PRIVATE index instead and
* never touches this boundary at all.) */
const VIndex* vx = eg_vindex_view(g, dim);
int persisted = engram_geo_reify_store(g_engram_store, vx, vids,
(int)g->node_count, NULL);
eg_vindex_view_release();
free(vids);
int nested = 0;
if(persisted >= 0){
@@ -13679,7 +13940,65 @@ el_val_t engram_think_json(el_val_t seeds, el_val_t faculty) {
if (!g) return eg_geo_err("geometry unavailable");
CogStance st; cog_stance_init(&st, NULL, EL_CSTR(faculty), g->hub_id, NULL, g);
GeoGradient grad;
if (engram_think(g, NULL, &st, &grad) != 0) { cog_stance_free(&st); engram_geo_free(g); return eg_geo_err("think failed"); }
/* ANCHOR THE READ (2026-08-16 self-review). This passed NULL, and NULL is
* not "no opinion" engram_think re-origins at `anchor ? anchor :
* region->centroid`, so NULL means "read from the centroid", and the
* centroid is the ONE point where the gradient is zero by construction:
* r = x - centroid = 0, so every axis projection is 0, grad is 0, and
* direction takes the "at rest" branch. Measured consequence: EVERY
* faculty reason, abduce, induce, plan, analogize returned an
* identical null result, differing only in its label:
* {"direction":[0,0,...],"spread":0,"magnitude":1,"confidence":0.5}
* magnitude 1 is membership evaluated at the centroid, spread 0 is its
* distance to itself, and confidence 0.5 is the stance fallback. The
* geometry was never the problem /api/drift computes real values
* (centroid_sep 0.104, core_disp 0.045) over the very same 87 members.
* Neuron could not think because the read was always taken from the
* region's own centre.
*
* The seeds choose WHICH region; they must also supply the VANTAGE it is
* read from. Anchor at the first resolvable embedded seed the same seed
* eg_geo_build_desc infers `dim` from, so the two never disagree. A single
* seed still yields a real gradient because the descriptor expands to the
* seed's neighbourhood (87 members for the self anchor), so the seed's own
* position is distinct from the neighbourhood centroid.
*
* COPY the vector, never borrow it: g->nodes is realloc'd in place on
* append, so a borrowed EngramNode* is a dangling pointer across any
* concurrent write. 768 floats is 3 KB. */
float* anchor = NULL;
{
EngramStore* eg = engram_get();
const char* csv = EL_CSTR(seeds);
if (eg && csv) {
const char* p = csv;
while (*p && !anchor) {
while (*p == ' ' || *p == ',') p++;
const char* s = p;
while (*p && *p != ',') p++;
const char* e = p; while (e > s && e[-1] == ' ') e--;
if (e > s) {
char* id = strndup(s, (size_t)(e - s));
if (id) {
int64_t idx = engram_find_node_index(id);
if (idx >= 0 && idx < eg->node_count) {
EngramNode* n = &eg->nodes[idx];
if (n->emb && n->emb_dim == g->dim) {
anchor = malloc(sizeof(float) * (size_t)g->dim);
if (anchor) memcpy(anchor, n->emb,
sizeof(float) * (size_t)g->dim);
}
}
free(id);
}
}
}
}
}
if (engram_think(g, anchor, &st, &grad) != 0) { free(anchor); cog_stance_free(&st); engram_geo_free(g); return eg_geo_err("think failed"); }
free(anchor);
JsonBuf b; jb_init(&b); char t[256];
snprintf(t, sizeof t, "{\"faculty\":\"%s\",\"n_support\":%d,\"magnitude\":%.6g,\"spread\":%.6g,\"confidence\":%.6g,\"dim\":%d",
EL_CSTR(faculty), grad.n_support, grad.magnitude, grad.spread, grad.confidence, grad.dim);
@@ -14032,6 +14351,21 @@ el_val_t engram_neighbors_json(el_val_t node_id, el_val_t max_depth, el_val_t di
return el_wrap_str(b.buf);
}
/* Public activation entry point. Serializes against the http_worker threads that
* share g->nodes/g->edges this is the guard the awareness main thread
* (soul.el: awareness_run) was missing entirely. Nested calls from a worker that
* already holds the lock pass straight through.
*
* It no longer guards _eg_vindex: the index has its own publication boundary
* (eg_vindex_view / eg_vindex_maintain) and search cannot mutate it. This guard is
* now about the RAM graph's realloc-in-place ONLY. See the note at eg_guard_enter. */
el_val_t engram_activate(el_val_t query, el_val_t depth) {
int owned = eg_guard_enter();
el_val_t r = engram_activate_inner(query, depth);
eg_guard_exit(owned);
return r;
}
el_val_t engram_activate_json(el_val_t query, el_val_t depth) {
/* Run two-layer engram_activate and serialize the result list to JSON.
* Each entry includes both activation_strength (layer 1 background) and
@@ -14808,6 +15142,7 @@ el_val_t engram_embed_backfill(el_val_t count) {
float* v = eg_embed_fetch(n->content, &d);
if (!v) break; /* embedder down / breaker open — stop this call */
n->emb = v; n->emb_dim = d;
eg_vindex_note_embedded(g, i); /* write-side index maintenance */
done++;
}
int64_t total = 0;
@@ -18001,11 +18336,196 @@ void log_warn(el_val_t msg_v) {
fprintf(stderr, "[WARN] %s\n", msg ? msg : "");
}
/* config — read a configuration value from the environment.
* Returns "" if the variable is not set (same as __env_get). */
/* ── Cross-cutting concerns: process identity and configuration ──────────────
*
* These back the `program` block (see lang/spec/language.md §18). Both concerns
* were previously conventions "check nothing is already running first",
* "remember the right default at every read site" and conventions is exactly
* what they failed as. Here they are mechanisms, injected by the compiler at
* the process boundary, so no call site has to remember anything.
*/
/* -- Process identity ------------------------------------------------------- */
/* The lock fd is deliberately never closed. Holding it open for the process
* lifetime is what makes the guarantee work: the kernel drops an flock when the
* owning process dies, including on SIGKILL and on crash. That is why this is an
* flock and not a bare pidfile there is no stale-lock state to clean up, and
* therefore no "delete the pidfile to get unstuck" ritual that would itself
* become a convention. */
static int el_singleton_fd = -1;
static char el_singleton_path[1024];
static const char* el_singleton_dir(void) {
const char* d = getenv("EL_SINGLETON_DIR");
if (d && *d) return d;
d = getenv("TMPDIR");
if (d && *d) return d;
return "/tmp";
}
/* el_singleton_acquire — claim exclusive process identity, or refuse to start.
* Compiler-injected as the FIRST statement of main() for any program whose
* `program` block declares `singleton:`. */
el_val_t el_singleton_acquire(el_val_t id_v) {
const char* id = EL_CSTR(id_v);
if (!id || !*id) return EL_NULL;
/* Sanitise the id into a filename. */
char safe[256];
size_t si = 0;
for (const char* p = id; *p && si + 1 < sizeof(safe); p++) {
char c = *p;
int ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
|| (c >= '0' && c <= '9') || c == '-' || c == '_' || c == '.';
safe[si++] = (char)(ok ? c : '-');
}
safe[si] = '\0';
snprintf(el_singleton_path, sizeof(el_singleton_path),
"%s/el-singleton-%s.lock", el_singleton_dir(), safe);
int fd = open(el_singleton_path, O_RDWR | O_CREAT, 0644);
if (fd < 0) {
fprintf(stderr, "[el] FATAL: singleton '%s': cannot open lock file %s: %s\n",
id, el_singleton_path, strerror(errno));
exit(1);
}
if (flock(fd, LOCK_EX | LOCK_NB) != 0) {
/* Someone else holds it. Report WHO. A pid is actionable; "already
* running" is not — and the observed failure was precisely a stale
* process that `pkill -f` had silently failed to match, still answering
* probes while a fresh build was believed to be under test. */
char buf[64];
buf[0] = '\0';
ssize_t n = pread(fd, buf, sizeof(buf) - 1, 0);
if (n > 0) buf[n] = '\0';
long holder = strtol(buf, NULL, 10);
fprintf(stderr, "[el] FATAL: another instance of '%s' is already running", id);
if (holder > 0) fprintf(stderr, " (pid %ld)", holder);
fprintf(stderr, ".\n"
"[el] lock: %s\n"
"[el] Refusing to start a second instance against the same\n"
"[el] state. Stop the running one and VERIFY it is gone\n"
"[el] (ps -p <pid>) before retrying.\n",
el_singleton_path);
close(fd);
exit(1);
}
/* We own it. Record our pid so the next would-be starter can name us. */
if (ftruncate(fd, 0) != 0) { /* best effort — the lock is the guarantee */ }
char pidbuf[32];
int pn = snprintf(pidbuf, sizeof(pidbuf), "%ld\n", (long)getpid());
if (pn > 0) { ssize_t w = write(fd, pidbuf, (size_t)pn); (void)w; }
el_singleton_fd = fd; /* never closed, by design */
return EL_NULL;
}
/* -- Configuration ---------------------------------------------------------- */
#define EL_CONFIG_MAX 128
typedef struct {
char name[128];
char type[16];
char* value; /* resolved: env value, else default; NULL if unset */
int has_default;
int required;
} ElConfigEntry;
static ElConfigEntry el_config_tab[EL_CONFIG_MAX];
static int el_config_n = 0;
static int el_config_has_schema = 0; /* did this program declare one at all? */
static int el_config_is_int(const char* s) {
if (!s || !*s) return 0;
if (*s == '-' || *s == '+') s++;
if (!*s) return 0;
for (; *s; s++) if (*s < '0' || *s > '9') return 0;
return 1;
}
/* el_config_declare — record ONE configuration entry and resolve it now.
* The default lives here, in the declaration, and nowhere else. */
el_val_t el_config_declare(el_val_t name_v, el_val_t type_v, el_val_t def_v,
el_val_t has_default_v, el_val_t required_v) {
const char* name = EL_CSTR(name_v);
if (!name || !*name) return EL_NULL;
el_config_has_schema = 1;
if (el_config_n >= EL_CONFIG_MAX) {
fprintf(stderr, "[el] FATAL: more than %d config entries declared.\n", EL_CONFIG_MAX);
exit(1);
}
const char* type = EL_CSTR(type_v);
const char* def = (def_v == EL_NULL) ? NULL : EL_CSTR(def_v);
ElConfigEntry* e = &el_config_tab[el_config_n++];
snprintf(e->name, sizeof(e->name), "%s", name);
snprintf(e->type, sizeof(e->type), "%s", type ? type : "String");
e->has_default = (int)(long)has_default_v;
e->required = (int)(long)required_v;
/* Resolution order: environment wins, declaration supplies the fallback. */
const char* env = getenv(name);
if (env && *env) e->value = el_strdup_persist(env);
else if (e->has_default && def) e->value = el_strdup_persist(def);
else e->value = NULL;
return EL_NULL;
}
/* el_config_validate — check the whole schema at once, before main() runs.
* Reports EVERY problem, not just the first: a startup that fails one variable
* at a time costs one restart per variable. */
el_val_t el_config_validate(el_val_t program_v) {
const char* prog = EL_CSTR(program_v);
int bad = 0;
for (int i = 0; i < el_config_n; i++) {
ElConfigEntry* e = &el_config_tab[i];
if (!e->value) {
if (e->required) {
fprintf(stderr, "[el] config: %s is required but is not set "
"(no value in the environment, no default declared)\n", e->name);
bad++;
}
continue;
}
if (strcmp(e->type, "Int") == 0 && !el_config_is_int(e->value)) {
fprintf(stderr, "[el] config: %s is declared Int but its value is \"%s\"\n",
e->name, e->value);
bad++;
}
}
if (bad) {
fprintf(stderr, "[el] FATAL: program '%s' has %d invalid configuration "
"entr%s. Refusing to start.\n",
prog ? prog : "?", bad, bad == 1 ? "y" : "ies");
exit(1);
}
return EL_NULL;
}
/* config — read a configuration value.
*
* When the program declared a schema, that schema is authoritative: the value
* has already been resolved and validated at startup, so this is a lookup and
* NOT a place where a default gets decided. Reading a key that was never
* declared is a bug at the read site, and is reported as one that enforcement
* is what makes the declaration real rather than advisory.
*
* With no schema declared, behaviour is unchanged (plain getenv), so programs
* that have not migrated keep working. */
el_val_t config(el_val_t key_v) {
const char* key = EL_CSTR(key_v);
if (!key || !*key) return EL_STR("");
if (el_config_has_schema) {
for (int i = 0; i < el_config_n; i++) {
if (strcmp(el_config_tab[i].name, key) == 0) {
const char* v = el_config_tab[i].value;
return el_wrap_str(el_strdup(v ? v : ""));
}
}
fprintf(stderr, "[el] FATAL: config(\"%s\") is not declared in the "
"program block. Declare it there, with its default, or stop "
"reading it.\n", key);
exit(1);
}
const char* val = getenv(key);
if (!val) return EL_STR("");
return el_wrap_str(el_strdup(val));
@@ -18522,6 +19042,26 @@ el_val_t engram_pool_stats_json(void) {
el_val_t el_alloc_count(void) { return (el_val_t)(int64_t)_el_alloc_count; }
el_val_t el_alloc_bytes(void) { return (el_val_t)(int64_t)_el_alloc_bytes; }
/* el_black_box — optimisation barrier for benchmark bodies.
*
* WHY THIS IS NOT OPTIONAL. A benchmark whose result is unused is dead code,
* and CONSUMING THE RESULT IS NOT SUFFICIENT: clang recognises loop idioms and
* closes them to arithmetic. A nested `total = total + 1` loop measured at
* 0 microseconds for every n while returning a numerically correct n*n --
* the answer was right and the work never happened.
*
* That is the same failure shape as a test that never ran reporting pass. The
* harness must own the barrier rather than trusting the benchmark author to
* defeat the optimiser.
*
* The constraint "+r" forces the value through a register the compiler must
* treat as both read and written by opaque code; the "memory" clobber stops
* loads and stores being reordered across it or elided. Emits no instructions. */
el_val_t el_black_box(el_val_t v) {
__asm__ __volatile__("" : "+r"(v) : : "memory");
return v;
}
el_val_t el_peak_rss(void) {
struct rusage ru;
if (getrusage(RUSAGE_SELF, &ru) != 0) return (el_val_t)0;
+22
View File
@@ -613,6 +613,11 @@ void engram_strengthen(el_val_t node_id);
void engram_forget(el_val_t node_id);
el_val_t engram_prune_telemetry(el_val_t older_than_ms);
el_val_t engram_node_count(void);
/* Attach geometry to an existing node. `hex` is little-endian float32,
* exactly dim*8 hex chars the encoding realizers already emit. Lets a
* non-text modality enter as geometry instead of being described in prose
* and embedded as its description. Returns 1 on success, 0 otherwise. */
el_val_t engram_node_set_emb(el_val_t id, el_val_t hex, el_val_t dim);
el_val_t engram_search(el_val_t query, el_val_t limit);
el_val_t engram_scan_nodes(el_val_t limit, el_val_t offset);
void engram_connect(el_val_t from_id, el_val_t to_id, el_val_t weight, el_val_t relation);
@@ -952,6 +957,22 @@ el_val_t __url_decode(el_val_t s);
/* Environment */
el_val_t __env_get(el_val_t key);
/* Cross-cutting concerns declared by a `program` block (spec §18).
* All three are COMPILER-INJECTED at the head of main() they are not meant to
* be written by hand, which is the point: the guarantee cannot be forgotten at a
* call site because there is no call site. */
el_val_t el_singleton_acquire(el_val_t id); /* §18.1 process identity */
el_val_t el_config_declare(el_val_t name, el_val_t type,
el_val_t deflt, el_val_t has_default,
el_val_t required); /* §18.2 config schema */
el_val_t el_config_validate(el_val_t program_name); /* §18.2 startup validate */
/* config(key) — the READ side, and the only one programs write by hand. With a
* schema declared it is a validated lookup; without one it degrades to getenv.
* (Defined in el_runtime.c but previously never prototyped here, so any program
* calling it failed to compile under -Werror=implicit-function-declaration.) */
el_val_t config(el_val_t key);
/* Subprocess */
el_val_t __exec(el_val_t cmd);
el_val_t __exec_bg(el_val_t cmd);
@@ -1022,6 +1043,7 @@ el_val_t el_mem_check(void);
el_val_t el_alloc_count(void);
el_val_t el_alloc_bytes(void);
el_val_t el_peak_rss(void);
el_val_t el_black_box(el_val_t v);
/* Semantic retrieval surface. NOT interchangeable with engram_search_json,
* which is lexical by design see the note at the definition. */
+256
View File
@@ -0,0 +1,256 @@
// runtime/elbench.el growth-curve classifier and complexity gate.
//
// Given a geometric sweep of input sizes and the measurements taken at each,
// classify the growth curve and decide whether it violates a declared bound.
//
// Why this exists
//
// Constant-factor regressions are annoying. Complexity regressions are outages.
// An O(n) lookup inside an O(n) loop is invisible at n=100 in a unit test and
// catastrophic at n=100000 in production. el #132 was exactly that: a strlen()
// inside a per-character accessor, quadratic, shipped for months.
//
// THREE signals, not one
//
// The gate fits time AND allocation-count AND allocation-bytes, and fails if
// ANY of them exceeds its declared curve. This is not belt-and-braces; each
// signal is blind to a real defect class the others catch:
//
// * A copy-on-write accumulator rebuilding its buffer allocates ONCE per
// iteration count is exactly linear while bytes go quadratic.
// Count alone passes it.
// * el #132's strlen-per-character is pure CPU and allocates NOTHING.
// Both allocation signals read FLAT. Only time catches it.
//
// The deterministic signals (count, bytes) are preferable where they apply:
// no statistics, correct on the first run, machine-independent. They are
// simply not sufficient.
//
// SCOPE LIMIT read this before trusting a flat curve
//
// The allocation counters track EL-LEVEL allocation only: strings, ElList and
// ElMap bodies, their backing arrays, copy-on-write clones, and the realloc
// growth path. malloc inside engram_*.c and inside libcurl is NOT counted.
//
// A flat allocation curve over a workload dominated by engram or HTTP calls is
// therefore NOT evidence of anything. It means "no El-level allocation growth",
// not "no allocation growth". Gate El-level complexity with this; do not read
// third-party memory behaviour into it.
//
// Classification method
//
// Sizes must form a geometric sweep (each n double the last). On such a sweep
// the ratio between consecutive measurements IS the growth exponent, directly:
//
// O(1) -> 1.0 O(log n) -> ~1.1 O(n) -> 2.0
// O(n log n) -> ~2.2 O(n^2) -> 4.0 O(n^3) -> 8.0
//
// DEVIATION FROM DESIGN.md 6.2, stated plainly: that section specified Google
// Benchmark's one-parameter least-squares fit over candidate curves. This uses
// consecutive ratios instead. The sweep is mandated geometric either way, and
// on a geometric sweep ratios are directly interpretable and need no floating
// point. The cost is weaker separation between O(n) and O(n log n), which is
// reported honestly as an ambiguous band rather than guessed at. Least-squares
// remains the better answer if that band ever needs to be resolved.
//
// All arithmetic is fixed-point, scaled by 1000 ("milli-ratio"), so a ratio of
// 2.0 is 2000. El values are int64; this avoids float-in-list handling.
// Curve identifiers. Ordered by growth the ordering IS the comparison used
// by the gate, so an index comparison decides "worse than declared".
// 0 = O(1) 1 = O(log n) 2 = O(n) 3 = O(n log n) 4 = O(n^2) 5 = O(n^3)
fn elb_curve_name(c: Int) -> String {
if c == 0 { return "O(1)" }
if c == 1 { return "O(log n)" }
if c == 2 { return "O(n)" }
if c == 3 { return "O(n log n)" }
if c == 4 { return "O(n^2)" }
if c == 5 { return "O(n^3)" }
return "O(?)"
}
fn elb_curve_from_name(s: String) -> Int {
if str_eq(s, "O(1)") { return 0 }
if str_eq(s, "O(log n)") { return 1 }
if str_eq(s, "O(n)") { return 2 }
if str_eq(s, "O(n log n)") { return 3 }
if str_eq(s, "O(n^2)") { return 4 }
if str_eq(s, "O(n^3)") { return 5 }
return -1
}
// elb_classify_ratio map a milli-ratio-per-doubling onto a curve.
//
// Bands are deliberately wide at the top (a quadratic measured at 3.4x is
// still a quadratic) and deliberately overlap-averse at the bottom, where a
// misclassification between O(1) and O(log n) matters least.
fn elb_classify_ratio(milli: Int) -> Int {
if milli < 1300 { return 0 }
if milli < 1700 { return 1 }
if milli < 2400 { return 2 }
if milli < 3200 { return 3 }
if milli < 6000 { return 4 }
return 5
}
// elb_ratio milli-ratio between two consecutive measurements.
// Returns -1 when the earlier measurement is zero (ratio undefined).
fn elb_ratio(prev: Int, cur: Int) -> Int {
if prev <= 0 { return -1 }
return (cur * 1000) / prev
}
// The measurement floor
//
// A benchmark whose largest measurement is at or near zero has not been
// measured. Reporting it as O(1) would be a confident answer with nothing
// behind it the same failure as a test that never ran reporting pass, and
// exactly what happened when clang closed a nested loop to a multiply and the
// harness read 0 microseconds at every n.
//
// So: REFUSE. Never classify below the floor.
fn elb_below_floor(vals: [Int], floor: Int) -> Bool {
let n: Int = native_list_len(vals)
let i: Int = 0
let mx: Int = 0
while i < n {
let v: Int = native_list_get(vals, i)
if v > mx { let mx = v }
let i = i + 1
}
if mx < floor { return true }
return false
}
// elb_implausibly_flat a measurement that does not move across a sweep whose
// input grew by 8x or more is not a flat curve, it is a broken measurement.
// Genuine O(1) work still shows noise; a hard-flat series means the work was
// optimised away, the timer has insufficient resolution, or the benchmark body
// never executed.
fn elb_implausibly_flat(vals: [Int]) -> Bool {
let n: Int = native_list_len(vals)
if n < 3 { return false }
let first: Int = native_list_get(vals, 0)
let last: Int = native_list_get(vals, n - 1)
if first == 0 {
if last == 0 { return true }
return false
}
let r: Int = (last * 1000) / first
if r < 1100 { return true }
return false
}
// elb_spread_ok do the consecutive ratios agree with each other?
//
// This is the ratio-method analogue of a normalised-RMS threshold. If the
// doublings disagree wildly the data is noise, a cache cliff, or a phase
// change, and the honest report is INDETERMINATE rather than a classification.
// Applies to the ASYMPTOTIC TAIL only the last three ratios.
//
// The small-n end of any sweep is dominated by fixed overhead, cold caches and
// branch predictors that have not warmed. Measured on a genuinely linear
// character scan, the ratios ran 3.37, 2.92, 1.76, 1.65: the head looks
// quadratic, the tail is the truth. Checking spread across the whole sweep
// therefore rejects correct data. A complexity bound is an asymptotic claim, so
// it is judged on the asymptotic region the same reason a benchmark harness
// discards warmup rather than averaging it in.
fn elb_spread_ok(ratios: [Int]) -> Bool {
let total: Int = native_list_len(ratios)
if total < 2 { return true }
let start: Int = total - 3
if start < 0 { let start = 0 }
let n: Int = total
let lo: Int = 999999
let hi: Int = 0
let i: Int = start
while i < n {
let r: Int = native_list_get(ratios, i)
if r >= 0 {
if r < lo { let lo = r }
if r > hi { let hi = r }
}
let i = i + 1
}
if lo <= 0 { return false }
// Reject when the widest ratio is more than 2.2x the narrowest. That is
// enough slack for real timing noise and tight enough to separate a clean
// 2.0 series from a clean 4.0 series.
if (hi * 1000) / lo > 2200 { return false }
return true
}
// elb_ratios consecutive milli-ratios across the sweep.
fn elb_ratios(vals: [Int]) -> [Int] {
let out: [Int] = native_list_empty()
let n: Int = native_list_len(vals)
let i: Int = 1
while i < n {
let out = native_list_append(out,
elb_ratio(native_list_get(vals, i - 1), native_list_get(vals, i)))
let i = i + 1
}
return out
}
// elb_mean_tail_ratio mean of the LAST TWO ratios.
//
// The tail is used deliberately: asymptotic behaviour is what a complexity
// bound claims, and the small-n end of any sweep is dominated by fixed
// overhead. This is the same reason a benchmark harness discards warmup.
fn elb_mean_tail_ratio(ratios: [Int]) -> Int {
let n: Int = native_list_len(ratios)
if n == 0 { return -1 }
if n == 1 { return native_list_get(ratios, 0) }
let a: Int = native_list_get(ratios, n - 1)
let b: Int = native_list_get(ratios, n - 2)
if a < 0 { return b }
if b < 0 { return a }
return (a + b) / 2
}
// Verdicts
//
// 0 PASS measured curve is at or below the declared bound
// 1 FAIL measured curve is strictly worse than declared
// 2 INDETERMINATE ratios disagree; data is noise or a phase change
// 3 REFUSED below the measurement floor, or implausibly flat
// 4 BETTER measured strictly better than declared (warn, not fail)
fn elb_verdict_name(v: Int) -> String {
if v == 0 { return "PASS" }
if v == 1 { return "FAIL" }
if v == 2 { return "INDETERMINATE" }
if v == 3 { return "REFUSED" }
if v == 4 { return "BETTER" }
return "?"
}
// elb_gate classify one signal against its declared bound.
//
// vals measurements, one per sweep point, in sweep order
// expect declared curve index (see elb_curve_name)
// floor minimum largest-measurement below which we refuse to classify
fn elb_gate(vals: [Int], expect: Int, floor: Int) -> Int {
if elb_below_floor(vals, floor) { return 3 }
if elb_implausibly_flat(vals) { return 3 }
let ratios: [Int] = elb_ratios(vals)
if !elb_spread_ok(ratios) { return 2 }
let m: Int = elb_mean_tail_ratio(ratios)
if m < 0 { return 2 }
let got: Int = elb_classify_ratio(m)
if got > expect { return 1 }
if got < expect { return 4 }
return 0
}
// elb_measured_curve the classified curve for a signal, or -1 if unclassifiable.
fn elb_measured_curve(vals: [Int], floor: Int) -> Int {
if elb_below_floor(vals, floor) { return -1 }
if elb_implausibly_flat(vals) { return -1 }
let ratios: [Int] = elb_ratios(vals)
let m: Int = elb_mean_tail_ratio(ratios)
if m < 0 { return -1 }
return elb_classify_ratio(m)
}
+2 -2
View File
@@ -222,7 +222,7 @@ static double eff_w(double weight, double hebb){
}
GeoDescriptor* engram_geometry_descriptor(
EngramPagedStore* store, VIndex* vindex,
EngramPagedStore* store, const VIndex* vindex,
char** vids, int n_vids,
const char* const* seed_ids, size_t n_seeds,
const GeoParams* params,
@@ -1401,7 +1401,7 @@ static double geo_weighted_degree(EngramPagedStore* st, const char* id, double e
return deg;
}
int engram_geo_reify_store(EngramPagedStore* store, VIndex* vindex,
int engram_geo_reify_store(EngramPagedStore* store, const VIndex* vindex,
char** vids, int n_vids,
const GeoReifyParams* params){
if(!store) return -1;
+2 -2
View File
@@ -150,7 +150,7 @@ void engram_geo_mean_free(GeoMeanCache* c);
* Returns a malloc'd descriptor (free with engram_geo_free), or NULL on error
* (no seeds resolvable, OOM). */
GeoDescriptor* engram_geometry_descriptor(
EngramPagedStore* store, VIndex* vindex,
EngramPagedStore* store, const VIndex* vindex,
char** vids, int n_vids,
const char* const* seed_ids, size_t n_seeds,
const GeoParams* params,
@@ -375,7 +375,7 @@ void engram_geo_reify_default_params(GeoReifyParams* p);
* neighborhood (+ member edges), superseding any prior same-hub record with
* provenance. Read-then-write over `store`. Returns #neighborhoods persisted, or <0.
* Skips existing Neighborhood/GeoMeanFrame nodes when detecting (idempotent re-reify). */
int engram_geo_reify_store(EngramPagedStore* store, VIndex* vindex,
int engram_geo_reify_store(EngramPagedStore* store, const VIndex* vindex,
char** vids, int n_vids,
const GeoReifyParams* params);
+68 -34
View File
@@ -74,11 +74,6 @@ struct VIndex {
int entry; /* entry-point element index, -1 if empty */
int max_level; /* current top layer */
/* scratch: version-stamped visited set (O(1) reset). */
uint32_t* visited;
uint32_t visit_epoch;
size_t visited_cap;
};
/* ── small helpers ────────────────────────────────────────────────────────── */
@@ -166,37 +161,63 @@ static Pair heap_pop(Heap* h, int is_max){
return top;
}
/* ── visited set ──────────────────────────────────────────────────────────── */
static int visited_ensure(VIndex* ix){
if (ix->visited_cap >= ix->cap && ix->visited) return 0;
size_t nc = ix->cap ? ix->cap : 16;
uint32_t* nv = (uint32_t*)realloc(ix->visited, nc*sizeof(uint32_t));
if (!nv) return -1;
if (nc > ix->visited_cap) memset(nv + ix->visited_cap, 0, (nc-ix->visited_cap)*sizeof(uint32_t));
ix->visited = nv; ix->visited_cap = nc;
/* ── visited set — owned by the CALL FRAME, never by the index ──────────────
* This buffer is per-TRAVERSAL scratch. It used to live in struct VIndex as an
* allocation optimisation, which made every traversal a write to shared state:
* two concurrent vindex_search calls stamped each other's epoch and then walked
* each other's marks, so even two pure READS corrupted the traversal (measured
* 2026-08-16: TSan data race at visited_reset, reached from vindex_search on one
* thread and vindex_insert on another; downstream SIGSEGV dereferencing a bogus
* element index).
*
* It is not an ownership problem and it does not want a lock or a capability
* it was simply misfiled. A pure function's scratch belongs to the call. Moving
* it here is what lets vindex_search take a `const VIndex*`, which is in turn
* what makes "search does not mutate the index" a COMPILE-TIME property instead
* of a review comment.
*
* Cost: one calloc/free of cap*4 bytes per traversal (~55 KB at the live store's
* 13,820 elements), against thousands of dim-768 dot products in the same call.
* Deliberately NOT __thread: http_worker is a thread per connection, so a
* thread-local buffer would retain ~55 KB per connection for the process life. */
typedef struct {
uint32_t* mark; /* per-element epoch stamp */
uint32_t epoch; /* current traversal's stamp; 0 == "no traversal yet" */
size_t cap;
} VVisit;
/* calloc leaves every stamp 0 and epoch 0; the first visit_reset moves to
* epoch 1, so no element reads as visited before it is marked. */
static int visit_init(VVisit* v, size_t cap){
size_t nc = cap ? cap : 16;
v->mark = (uint32_t*)calloc(nc, sizeof(uint32_t));
if (!v->mark) return -1;
v->cap = nc; v->epoch = 0;
return 0;
}
static inline void visited_reset(VIndex* ix){
if (++ix->visit_epoch == 0){ /* wrapped: clear all */
memset(ix->visited, 0, ix->visited_cap*sizeof(uint32_t));
ix->visit_epoch = 1;
static void visit_dispose(VVisit* v){ free(v->mark); v->mark = NULL; v->cap = 0; }
static inline void visit_reset(VVisit* v){
if (++v->epoch == 0){ /* wrapped: clear all */
memset(v->mark, 0, v->cap*sizeof(uint32_t));
v->epoch = 1;
}
}
static inline int is_visited(VIndex* ix, int e){ return ix->visited[e]==ix->visit_epoch; }
static inline void mark_visited(VIndex* ix, int e){ ix->visited[e]=ix->visit_epoch; }
static inline int is_visited(const VVisit* v, int e){ return v->mark[e]==v->epoch; }
static inline void mark_visited(VVisit* v, int e){ v->mark[e]=v->epoch; }
/* ── search one layer (Algorithm 2): best-first, ef-bounded ───────────────── */
/* Returns results as an unsorted Heap (max-heap on distance, size<=ef). Caller
* owns res->a. `q` is a normalised query. */
static int search_layer(VIndex* ix, const float* q, const int* eps, int neps,
static int search_layer(const VIndex* ix, VVisit* vis, const float* q,
const int* eps, int neps,
int ef, int layer, Heap* res /*out, max-heap*/){
Heap cand = {0,0,0}; /* min-heap: nearest to expand */
res->a=NULL; res->n=0; res->cap=0;
visited_reset(ix);
visit_reset(vis);
for (int i=0;i<neps;i++){
int e = eps[i];
if (is_visited(ix,e)) continue;
mark_visited(ix,e);
if (is_visited(vis,e)) continue;
mark_visited(vis,e);
float d = vdist(ix, q, ix->elems[e].vec);
Pair p = { d, e };
if (heap_push(&cand,p,0) || heap_push(res,p,1)){ free(cand.a); return -1; }
@@ -212,8 +233,8 @@ static int search_layer(VIndex* ix, const float* q, const int* eps, int neps,
NeighList* nl = &ce->links[layer];
for (int i=0;i<nl->count;i++){
int e = nl->ids[i];
if (is_visited(ix,e)) continue;
mark_visited(ix,e);
if (is_visited(vis,e)) continue;
mark_visited(vis,e);
float d = vdist(ix, q, ix->elems[e].vec);
if (res->n < ef || d < res->a[0].d){
Pair p = { d, e };
@@ -232,7 +253,7 @@ static int search_layer(VIndex* ix, const float* q, const int* eps, int neps,
* Keep c only if it is nearer to q than to every already-chosen neighbour;
* backfill from the pruned set (nearest first) to reach M for connectivity.
* Writes chosen element indices into out[], returns the count. */
static int select_neighbors(VIndex* ix, const float* q, Pair* W, int nW, int M, int* out){
static int select_neighbors(const VIndex* ix, const float* q, Pair* W, int nW, int M, int* out){
(void)q; /* q's distances are precomputed in W[].d; kept for call-site clarity */
/* sort W ascending by (dist,elem) — deterministic. */
for (int i=1;i<nW;i++){ /* insertion sort (nW small) */
@@ -281,7 +302,7 @@ static int elems_reserve(VIndex* ix){
Elem* ne = (Elem*)realloc(ix->elems, nc*sizeof(Elem));
if (!ne) return -1;
ix->elems = ne; ix->cap = nc;
return visited_ensure(ix);
return 0;
}
int vindex_insert(VIndex* ix, uint64_t node_id, const float* vec){
@@ -307,13 +328,19 @@ int vindex_insert(VIndex* ix, uint64_t node_id, const float* vec){
return 0;
}
/* This call frame owns its traversal scratch for the whole insert. ix->cap
* already covers `cur` (elems_reserve ran above), so every reachable element
* index is in range. */
VVisit vis;
if (visit_init(&vis, ix->cap)) return -1;
int ep = ix->entry;
int L = ix->max_level;
/* greedy descent through layers above `level` to refine the entry point. */
for (int lc = L; lc > level; lc--){
Heap r = {0,0,0};
int eps1[1] = { ep };
if (search_layer(ix, el->vec, eps1, 1, 1, lc, &r)){ return -1; }
if (search_layer(ix, &vis, el->vec, eps1, 1, 1, lc, &r)){ visit_dispose(&vis); return -1; }
if (r.n){ ep = r.a[0].e; float bd=r.a[0].d;
for (int i=1;i<r.n;i++) if (r.a[i].d<bd){bd=r.a[i].d; ep=r.a[i].e;} }
free(r.a);
@@ -329,7 +356,7 @@ int vindex_insert(VIndex* ix, uint64_t node_id, const float* vec){
for (int lc = start; lc >= 0; lc--){
int Mmax = (lc==0) ? ix->M0 : ix->M;
Heap W = {0,0,0};
if (search_layer(ix, el->vec, eps, neps, ix->ef_construction, lc, &W)){ rc=-1; break; }
if (search_layer(ix, &vis, el->vec, eps, neps, ix->ef_construction, lc, &W)){ rc=-1; break; }
int* chosen = (int*)malloc((size_t)(W.n?W.n:1)*sizeof(int));
if (!chosen){ free(W.a); rc=-1; break; }
int nc = select_neighbors(ix, el->vec, W.a, W.n, Mmax, chosen);
@@ -357,13 +384,17 @@ int vindex_insert(VIndex* ix, uint64_t node_id, const float* vec){
}
done:
free(eps_owned);
visit_dispose(&vis);
if (rc) return -1;
if (level > ix->max_level){ ix->max_level = level; ix->entry = cur; }
return 0;
}
/* ── search ───────────────────────────────────────────────────────────────── */
int vindex_search(VIndex* ix, const float* query, int k, int ef_search,
/* `ix` is const: search is pure with respect to the index. That is enforced by
* the compiler, not by convention it is the whole point of moving the visited
* set into the frame below. */
int vindex_search(const VIndex* ix, const float* query, int k, int ef_search,
uint64_t* node_id_out, float* dist_out){
if (!ix || !query || k <= 0) return -1;
if (ix->entry < 0) return 0;
@@ -373,11 +404,15 @@ int vindex_search(VIndex* ix, const float* query, int k, int ef_search,
float* q = vec_normalise_copy(query, ix->dim);
if (!q) return -1;
/* This call frame owns its traversal scratch. */
VVisit vis;
if (visit_init(&vis, ix->cap)){ free(q); return -1; }
int ep = ix->entry;
for (int lc = ix->max_level; lc > 0; lc--){
Heap r = {0,0,0};
int eps[1] = { ep };
if (search_layer(ix, q, eps, 1, 1, lc, &r)){ free(q); return -1; }
if (search_layer(ix, &vis, q, eps, 1, 1, lc, &r)){ visit_dispose(&vis); free(q); return -1; }
if (r.n){ int b=r.a[0].e; float bd=r.a[0].d;
for (int i=1;i<r.n;i++) if (r.a[i].d<bd){bd=r.a[i].d; b=r.a[i].e;}
ep = b; }
@@ -385,7 +420,8 @@ int vindex_search(VIndex* ix, const float* query, int k, int ef_search,
}
Heap res = {0,0,0};
int eps[1] = { ep };
if (search_layer(ix, q, eps, 1, ef_search, 0, &res)){ free(res.a); free(q); return -1; }
if (search_layer(ix, &vis, q, eps, 1, ef_search, 0, &res)){ visit_dispose(&vis); free(res.a); free(q); return -1; }
visit_dispose(&vis);
free(q);
/* res is a max-heap of size<=ef; pop into ascending order, keep nearest k. */
@@ -419,7 +455,6 @@ VIndex* vindex_create(int dim, int M, int ef_construction){
ix->mL = 1.0 / log((double)M > 1.0 ? (double)M : 2.0);
ix->entry = -1;
ix->max_level = 0;
ix->visit_epoch = 0;
return ix;
}
@@ -432,7 +467,6 @@ void vindex_free(VIndex* ix){
free(e->vec);
}
free(ix->elems);
free(ix->visited);
free(ix);
}
+9 -2
View File
@@ -53,8 +53,15 @@ int vindex_insert(VIndex* idx, uint64_t node_id, const float* vec);
* first (ascending distance). Either out array may be NULL to skip it.
* ef_search search-time candidate width; larger == higher recall, slower.
* Pass <=0 for VINDEX_DEFAULT_EF_SEARCH. Internally clamped to >=k.
* Returns the number of results written, or <0 on error. */
int vindex_search(VIndex* idx, const float* query, int k, int ef_search,
* Returns the number of results written, or <0 on error.
*
* `idx` is const BY CONTRACT AND BY TYPE: search does not mutate the index. The
* traversal's visited set is owned by the call frame, so N threads may search one
* index concurrently. Concurrent search against a vindex_insert on the same index
* is still unsafe insert rewires existing elements' neighbour lists and reallocs
* elems[] so the index's owner must not extend a published index under a live
* reader. See eg_vindex_view / eg_vindex_maintain in el_runtime.c. */
int vindex_search(const VIndex* idx, const float* query, int k, int ef_search,
uint64_t* node_id_out, float* dist_out);
/* Number of vectors currently indexed. */
+169 -5
View File
@@ -29,6 +29,8 @@ This section is the **single source of truth** for what works and what is planne
- Lexer: keywords, identifiers, integer/float/string/bool literals, operators below.
- Parser: `let`, `return`, `fn`, `type`, `enum`, `import`, `from … import`, `while`, `for`, `if/else if/else`, `match`, `@decorator`, array/map literals, all listed operators, function calls, field access, index access, unary `!`/`-`, postfix `?`.
- Codegen: function definitions, top-level `main()`, all expression forms above, control flow, decorator-as-AST-attachment.
- Boundary seam: decorator arguments and stacking; VBD role enforcement via `#error`; `engram_boundary_beat` auto-emit at `@manager`/`@accessor` entry; `@route` dispatch tables (Section 9).
- Program-level declarative blocks: `cgi`, `service`, and `program` — the last carrying process identity and configuration (Section 18).
- C runtime: I/O, string operations, integer math, lists, maps, filesystem, command-line args, basic `json_get` substring lookup.
### Planned (in flight)
@@ -37,7 +39,7 @@ This section is the **single source of truth** for what works and what is planne
- **Match codegen.** Currently parsed; codegen does not emit. Adding `({ ... })` statement-expression emission.
- **`?` propagation.** Currently no-op. Adding nil-propagation semantics.
- **`cgi` block parsing.** Currently lexed (`cgi` is a keyword) but not parsed as a statement. Adding `parse_cgi_block` and codegen of `el_cgi_init` at the head of `main()`.
- **VBD role enforcement.** `@manager`/`@engine`/`@accessor` are accepted as decorators but not enforced. Adding compile-time check that `dharma_emit`/`dharma_field` only appear inside `@manager` functions.
- **Boundary epilogues.** The decorator seam injects a prologue only. Adding prologue/epilogue wrapping, the prerequisite for durability-as-an-effect (Section 19.1).
- **`vessel` keyword.** Replaces `package` in manifests. Adding to lexer.
- **Real `engram_*` runtime.** Currently stub. Adding in-process graph store with spreading activation, Hebbian strengthening, and disk persistence — see Section 16.4.
- **Real `dharma_*` runtime.** Currently stub. Adding network transport, channel registry, identity resolution.
@@ -96,8 +98,10 @@ The following words are reserved and cannot be used as identifiers. Each row not
| `while` | yes | Loop |
| `import` / `from` / `as` | yes | Module import |
| `true` / `false` | yes | Bool literals |
| `cgi` | planned | Top-level CGI declaration block |
| `manager` / `engine` / `accessor` | as decorators | VBD role marker on `fn` (enforcement planned) |
| `cgi` | yes | Top-level CGI declaration block |
| `service` | yes | Top-level capability-bounded declaration block |
| `program` | yes | Top-level cross-cutting declaration block (Section 18) |
| `manager` / `engine` / `accessor` | as decorators | VBD role marker on `fn`; enforcement and boundary auto-emit are live (Section 9) |
| `vessel` | planned | Manifest declaration (replaces `package`) |
| `activate` / `where` | planned | Spreading-activation construct |
| `sealed` | planned | Capability scope block |
@@ -446,9 +450,21 @@ Parsed. The module name is recorded; the brace-list is consumed. Both forms prod
fn handle(channel: String, msg: String) -> Void { … }
```
The `@` token followed by an identifier attaches a decorator name to the next `FnDef`. Decorators with structural meaning today: none. Planned enforcement (Section 16.2): VBD roles `@manager`, `@engine`, `@accessor`.
The `@` token followed by an identifier attaches a decorator to the next `FnDef`.
Non-VBD decorators are accepted and ignored.
**Decorators take arguments and they stack.** `@route("/p", "GET") @manager fn f()` attaches both to `f` as a `decorators` list of `{name, args}` records, topmost-first. Arguments are string literals only.
**Decorators have structural meaning today.** This is El's function-level boundary seam — the mechanism by which a cross-cutting concern is handled *at the boundary* rather than by a convention repeated at every call site:
| Decorator | Structural effect |
|---|---|
| `@manager` | Permits calls to `dharma_emit` / `dharma_field`. Calling either from a non-`@manager` fn emits a `#error` into the generated C — a compile-time failure, not a lint. |
| `@manager`, `@accessor` | Codegen injects one call to `engram_boundary_beat(<fn name>)` at function entry. The decorated op self-reports (chrono tick, afferent counter, self-activity strengthen, dharma bus event) with **zero** hand-written instrumentation in its body. |
| `@route(path, method, …)` | Records a route into a generated dispatch table. |
Decorators with no registered meaning are accepted and ignored.
**Limits of the seam, as it stands.** The injection is a *prologue only* — there is no epilogue, no wrapping of the call, and no way for a decorator to run code after the body returns. The injected callee is a fixed builtin chosen by the compiler, not derived from the decorator name or its arguments. Section 19 depends on lifting exactly these two limits.
---
@@ -1088,4 +1104,152 @@ The next minor version closes the implementation gaps named in this document. Tr
---
## 18. The Program Block — cross-cutting concerns [implemented]
### 18.0 Why this exists
A cross-cutting concern is one that belongs to the *process*, not to any function in it: only one of me may run; this is what my configuration is; every mutation must be durable; every request must be authorized.
El's units of encapsulation are the function and the module. Neither can hold a concern like that. So each one had been expressed the only way it could be — as a **convention**: *call this at every site.* Conventions of that shape do not hold. They are not enforced by anything, they are invisible in review, and they fail silently at the one site somebody forgot.
Measured in this codebase before this section existed:
| Concern | State | What the convention was |
|---|---|---|
| process identity | **zero** guards anywhere — no pidfile, no lock, no already-running check, at any layer | "check nothing is already running first" |
| configuration | **20** distinct environment variables in one program, each with its default written inline at the read site | "remember the right default here" |
| durability | **62** `persist_*` / `engram_save` / `wal_*` / `checkpoint` call sites | "after you mutate, remember to persist" |
| request auth | **10** per-route `_auth` checks | "check the token in this handler too" |
These are not four problems. They are one absence, four times.
That the convention form fails is observed, not predicted. Process identity failed three times in a single day: twice, two engram processes ran simultaneously against the same data directory; twice, a stale binary held a port and answered probes while a fresh build was believed to be under test, because `pkill -f` had silently failed to match its argv — which nearly produced a false "the fix does not work" conclusion. Configuration failed structurally: `ENGRAM_DATA_DIR` was read at six sites, five of them dead bindings, and the sixth defaulted to `/tmp/engram` — contradicting the canonical resolver's `$HOME/.neuron/engram` and landing a pre-destructive safety backup on ephemeral storage.
The `program` block is where a concern of this shape is declared once and enforced by the compiler at the process boundary.
### 18.1 Syntax
```
program "engram" {
singleton: "engram"
env ENGRAM_BIND: String = ":8742"
env GUIDE_PORT: Int = "8771"
env ENGRAM_API_KEY: String required
}
```
At most one `program` block per program. It composes with `cgi` and `service` — those declare what a program *may do*; `program` declares what a program *is*.
Grammar:
```ebnf
program_block = "program" string "{" { program_field } "}" ;
program_field = singleton_field | env_field ;
singleton_field = "singleton" ":" string [ "," ] ;
env_field = "env" ident ":" type
[ "=" string ] [ "required" ] [ "," ] ;
```
`singleton` and `env` are **not** reserved words. They are read as identifier token values by the block's own parse loop, so they remain usable as ordinary identifiers everywhere else. `program` is the only keyword this section adds.
### 18.2 Process identity — `singleton`
`singleton: "id"` compiles to an `el_singleton_acquire("id")` call injected as the **first statement of `main()`**, before any user statement runs.
The runtime takes an exclusive non-blocking `flock` on `<dir>/el-singleton-<id>.lock`, where `<dir>` is `$EL_SINGLETON_DIR`, else `$TMPDIR`, else `/tmp`. On success it writes its pid and holds the descriptor open for the life of the process. On contention it **refuses to start**: it reports the holder's pid, names the lock file, and exits 1.
Two properties are deliberate:
- **It is a lock, not a pidfile.** The kernel releases an `flock` when the owning process dies — including on `SIGKILL` and on crash. There is therefore no stale-lock state, and so no "delete the lock file to get unstuck" recovery ritual. Such a ritual would itself be a convention, which is the thing this section exists to remove.
- **It reports the holder's pid.** "Already running" is not actionable. A pid is. This is the direct answer to the observed failure where a stale process survived a `pkill` and went on answering probes.
Refusal is loud and total. It is not a warning, and the program does not continue degraded. This matters more than it looks: today a second engram whose `bind()` fails merely *returns* from `http_serve` — after it has already replayed the WAL and written boot-time backup files — and then exits **0**, indistinguishable from a clean run. `singleton` refuses before the first side effect.
### 18.3 Configuration — `env`
Each `env` entry declares one configuration variable: its name, its type (`Int` or `String`), and either a default or `required`.
Resolution happens once, at startup, in declaration order: **the environment wins; the declaration supplies the fallback.** Then `el_config_validate` checks the whole schema and reports *every* problem at once before exiting — a startup that fails one variable at a time costs one restart per variable.
Values are read with `config("NAME")`, which returns a `String`.
The enforcement that makes the declaration real: **once a program block exists, `config("X")` for an undeclared `X` is a fatal error.** Without that, the schema would be advisory, and an advisory schema is just another convention. Programs with no `program` block are unaffected — `config()` falls back to a plain environment read, so migration is incremental and per-program.
The point is not that configuration is now centralized. It is that **a default is no longer a decision made at a read site.** A read site cannot disagree with another read site about what a variable means, because a read site no longer says.
### 18.4 What is deliberately not declared here
Some values look like configuration and are not. `ENGRAM_DATA_DIR` already has a single owner — `engram_resolve_data_dir()`, which resolves it, creates the directory, and fails loud rather than silently persisting to an ephemeral path. Declaring it in the `program` block as well would give it two owners that can disagree, recreating the precise defect this section removes.
The rule: **a variable belongs in the program block when the block would be its only owner.** If a resolver already owns it, leave it there.
`HOME` is likewise not configuration. It is an environment fact, and stays a raw `env()` read.
---
## 19. Boundary Effects — durability and request authorization [design only, not implemented]
Sections 19.1 and 19.2 specify the two remaining concerns from the table in 18.0. Both are **designed and deliberately unimplemented.** The reason is stated in 19.3 and it is not difficulty.
### 19.1 Durability as an epilogue effect
**The defect.** 62 call sites carry the convention *"after you mutate, remember to persist."* This is structurally the same defect as the index bug being fixed elsewhere in this tree — *"after you append, remember to index"* — which failed at **9 of 9** sites. A convention that failed at 100% of its sites is the strongest available evidence about what this class of convention is worth.
**Why the existing seam cannot express it.** §9's injection is a prologue. Durability is inherently an *epilogue*: persist after the mutation succeeds, and not at all if it threw. The seam has no epilogue.
**Design.** Extend the decorator seam from prologue-only to prologue/epilogue, then declare durability as an effect on the mutating function:
```
@durable("engram")
fn engram_write_node(id: String, body: String) -> Bool { … }
```
Codegen wraps rather than prefixes:
```c
el_val_t engram_write_node(el_val_t id, el_val_t body) {
el_effect_enter(EL_STR("durable"), EL_STR("engram"));
el_val_t __r = /* original body */;
el_effect_exit(EL_STR("durable"), EL_STR("engram"), __r);
return __r;
}
```
`el_effect_exit` is where the persist happens, and it is the only place it happens. Two properties follow that the 62 hand-written sites cannot have:
- **Coalescing.** The epilogue is a single choke point, so N mutations inside one request can produce one fsync instead of N. The hand-written form cannot coalesce, because no site knows about the others.
- **Failure is not silent.** A persist that fails inside `el_effect_exit` can force the mutation's return value to failure. A forgotten `persist_*` call cannot fail — it simply does not happen, which is exactly why the defect is invisible.
**Enforcement, and this is the part that actually fixes it.** Mirroring §9's `#error` for `dharma_emit`: a function that calls a mutating primitive without carrying `@durable` is a **compile error**. Otherwise this is a 63rd thing to remember rather than a replacement for 62.
### 19.2 Request authorization as a route effect
**The defect.** 10 per-route `_auth` checks. The HTTP layer has no concept of authorization, so a new route is unauthenticated by default and silently so — the failure mode is a route that forgot, and nothing anywhere reports it.
**Design.** Authorization becomes an argument to the `@route` decorator, which already takes arguments and already builds a dispatch table:
```
@route("/api/write", "POST", auth: "required")
fn route_write(body: String) -> String { … }
```
The generated dispatcher performs the check **before** dispatch, so an unauthorized request never reaches the handler and the handler contains no auth code at all.
The default must be `required`. A route that says nothing gets authorization; opening one up takes an explicit `auth: "public"`. Defaulting to public preserves the current failure mode exactly — forgetting stays silent — and a default that preserves the defect is not a fix.
Route inventory falls out for free: the dispatch table already exists, so the compiler can emit the full route/auth matrix and make "which routes are public" a fact that is read rather than audited.
### 19.3 Why these are not implemented
Not difficulty — **collision**. Both land squarely in regions two other agents hold right now:
- **Durability** requires changing the mutation and persist paths in `lang/runtime/el_runtime.c` and `engram/src/server.el` — the same files and the same read/write paths being restructured by concurrent work on VIndex read-path mutation and memory ownership, and on geometry-as-an-el-value and `transduce`.
- **Request auth** requires changing route dispatch in `engram/src/server.el`, which the geometry/`transduce` work is actively reshaping.
Implementing either now would mean editing files under concurrent modification and resolving conflicts in exactly the paths whose correctness is currently under repair. The designs are recorded here so the work is not lost, and so that whoever lands them does so against a settled tree.
The prerequisite for 19.1 is the same in both cases: **lift the §9 seam from prologue-only to prologue/epilogue.** That change is independent of both collisions and can land first.
---
End of specification.
+180
View File
@@ -0,0 +1,180 @@
# El Runtime — Ownership and Capability ABI
**Status:** §0–§2 verified. §3 re-derived and **built** for the vector index (2026-08-16); not yet applied to the resident RAM graph.
**Date:** 2026-08-16
**Scope:** `lang/runtime/` — every El program (soul, engram, cgi-studio vessels) inherits this by rebuild. Nothing in this document is a change to any El *program*.
**Note on §1's line numbers:** they were read against a checkout that has since shifted by ~135 lines. Verified positions as of `a67452f` are in §2a.
---
## 0. The residual
> **Builtins own memory and reach process state directly.**
That is the residual — the generator. Everything below labelled a "residue" is a deposit left by it. The distinction matters because we have spent significant effort removing deposits, and deposits regenerate.
A residue is fixed. A residual is eliminated. Fixing residues while the residual stands produces exactly the pattern observed on 2026-08-15/16: a run of individually-correct patches, each verified, followed by a new defect of the same shape in a different file.
---
## 1. The residues, measured
Each of these is a distinct merged or proposed fix. Each addresses one deposit. None addresses the residual.
| residue | location | fix that was applied or proposed |
|---|---|---|
| `state_get` leaked its return value per call — 15 MB over 200k calls | builtin | el #140 (merged) |
| VIndex freed under a concurrent reader | `el_runtime.c:9424` | `fb32d15` guard (merged 08:46:43) |
| `_eg_vindex_seen` realloc'd on a read path | `el_runtime.c:9412` | same guard |
| `vindex_insert` on a read path | `el_runtime.c:9434`, `9450` | same guard |
| shared `visited` / epoch scratch stomped by concurrent searches | `engram_vindex.c:7981`, `169186`, `195` | proposed: move to per-search frame |
| nine append sites, none indexing → lazily-embedded nodes invisible | `el_runtime.c:7806, 7988, 8148, 8224, 11526, 11731, 12050, 15295, 15312` | "embed-gap #20", patched by making the *read* path catch up (`9439` comment) |
**Measured:** all file/line references above, read 2026-08-16. Crash frames `engram_activate → eg_vindex_sync → vindex_insert → _realloc → _xzm_xzone_malloc_freelist_outlined` are accounted for by rows 24.
**Inferred, not yet verified:** that the nine append sites do not share a single commit point. This needs one pass before Change C is sized.
---
## 2. Why these are one defect
`eg_vindex_sync` (`el_runtime.c:9419`) has exactly three callers, and **all three are reads**:
- `engram_activate``9802`
- `eg_knn_for_node``13075` (its own header comment states *"No writes."*)
- `engram_geo_reify_run_json``13285`
It mutates five process-global statics (`94009404`): `_eg_vindex`, `_eg_vindex_dim`, `_eg_vindex_built_nc`, `_eg_vindex_seen`, `_eg_vindex_seen_cap`.
Reads mutate because index maintenance was never given an owner on the write side. It got bolted onto reads, because a builtin *could* reach the globals — nothing prevented it. Likewise `state_get` leaked because a builtin *owned* the value it returned; nothing prevented that either.
The store is architecturally append-only and superseding. A read path that mutates contradicts that directly. The contradiction is expressible only because the ABI permits it.
---
## 2a. Verified positions and the fact §1 missed
Read directly at `a67452f`, 2026-08-16. §1's line numbers predate a ~135-line shift; these are current.
| thing | §1 said | actually |
|---|---|---|
| five process-global statics | 94009404 | **95359539** |
| `eg_vindex_seen_ensure` realloc | 9412 | **9547** |
| `eg_vindex_sync` | 9419 | **9554** |
| `vindex_free` on a read path | 9424 | **9559** |
| `vindex_insert` on a read path | 9434 / 9450 | **9569** (build) / **9585** (incremental) |
| caller: `engram_activate_inner` | 9802 | **9939** |
| caller: `eg_knn_for_node` | 13075 | **13212** |
| caller: `engram_geo_reify_run_json` | 13285 | **13422** |
| `fb32d15` guard | — | lock **1602**, depth **1631**, `eg_guard_enter` **1636**, `http_worker` acquire **1687**, `engram_activate` wrapper **14097** |
| VIndex scratch fields | 7981 | **7981** ✓ |
| `search_layer` race site | 195 | **195** ✓ |
**The structural fact §1 and §3 both missed:** *the index does not inherit the store's append-only property.* `vindex_insert` rewires the `NeighList` links of already-existing elements and reallocs `elems[]` — so extending the index mutates the whole structure, not just its tail. This is why "make reads pure" is necessary but **not sufficient**, and why §3 needed a publication boundary rather than only a capability split. It is reproduced as a standing test (`unsynchronized` half, §5).
---
## 3. The change
*(Re-derived 2026-08-16. The previous §3 — a runtime context struct carrying read/write **capability pointers** to every builtin — was written in mutable-store, C-ownership terms. It asked "who is permitted to mutate the shared thing?", which presupposes a shared mutable thing. The engram is immutable and recall is projection; what does not mutate needs no ownership discipline. So the question is not answered, it is dissolved. The implemented change is below.)*
### 3.1 Three moves, in decreasing order of how much they dissolve
**(1) Misfiled scratch is not shared state.** `visited` / `visit_epoch` were never conceptually owned by the index — they are one traversal's local, hoisted into `struct VIndex` as an allocation optimisation. Nothing about them is derived geometry. They want neither a lock nor a capability nor a checkout pool: a pure function's scratch belongs to its call frame, and the fix is to put it back there. This is not "the capability model applied by hand to one global"; it is the deletion of a false ownership claim.
**(2) `const` is the capability, and immutability hands it over for free.** Once the scratch leaves the struct, `search_layer` reads the index and nothing else — so `vindex_search` can take a `const VIndex*`. That is *precisely* the teeth old-§3 wanted from capability pointers: a read path physically cannot call `vindex_insert`, and it is a **compile error**, not a review comment. It costs one qualifier rather than a new ABI swept across hundreds of builtins. The compiler enforces it on every future caller for the same reason.
> The capability type was already in the language. It is spelled `const`.
**(3) What remains is a publication problem, not an ownership problem.** With scratch in the frame and reads const, one hazard survives, and it is real: **HNSW insert is not an append.** `vindex_insert` rewires the `NeighList` links of *already-existing* elements and reallocs `elems[]`. The store's append-only property does **not** transfer to the index derived from it. So a reader projecting against the index while its owner extends it is unsafe no matter how pure search is.
Immutability answers this too, and the answer is publication:
- **`eg_vindex_maintain`** — the sole mutator. Takes the boundary exclusively; never runs beside a reader.
- **`eg_vindex_view`** — returns a `const VIndex*` with the boundary held for read. N readers project concurrently; none can mutate.
A read path may **demand that a current snapshot exist** — that is a request to the owner, not a mutation by the reader. What it may not do is mutate the geometry it is projecting against. `view` / `maintain` is exactly that split, and it is why this replaces `eg_vindex_sync` rather than wrapping it.
**Write-side owner.** Index membership is owned by the event *"an embedding became present on this ordinal"* — not by node append, since a node without an embedding cannot be in a vector index at all. `eg_vindex_note_embedded` hooks the embedding-assignment sites: one O(log n) insert, no O(node_count) presence scan. This also retires the "STALENESS (honest tradeoff)" note in the old `eg_vindex_sync`, where a lazily-embedded *older* node stayed invisible to `route_nearest` / autoconnect until the next full rebuild.
### 3.2 What this does not claim
The **resident RAM graph** (`g->nodes` / `g->edges`) is a *separate* residue of the same residual and is untouched by this change. It is realloc'd in place (`el_runtime.c:7618`, `7629`), so an awareness-thread reader holding `EngramNode* n = &g->nodes[i]` across a concurrent append holds a dangling pointer — and `engram_activate_inner`'s embed-backfill writes `n->emb` through exactly such a pointer. It wants the same publication treatment the index just received. Until that lands, the `fb32d15` guard stays (see §5).
---
## 4. Why this is not a large change
The old §4 argued that El owning its compiler makes a capability-ABI sweep mechanical, since `elc` generates every builtin call site. That argument was load-bearing only for the ABI, and the ABI is gone.
The constraint now travels with the **type of the thing**, not the shape of every call site — so no sweep is needed at all. Measured extent of the implemented change: two qualifiers (`const VIndex*` on `vindex_search`, propagated to `engram_geometry_descriptor` and `engram_geo_reify_store`), one struct field group relocated to a call frame, one rwlock, and three read call sites converted from `eg_vindex_sync` to `view`/`release`.
The payoff of owning the language is unchanged and is now *cheaper*: introduced once, enforced by the compiler on every future builtin, cannot subsequently be forgotten. Contrast the current state, where the same discipline was maintained by hand across hundreds of builtins and demonstrably failed at least six times.
---
## 5. What this deletes
**Deleted (done, 2026-08-16):**
- `eg_vindex_sync` — the function itself. Not renamed: split into `eg_vindex_maintain` (mutating, exclusive, sole owner) and `eg_vindex_view` (const, shared). A name that meant "read paths repair the index" had to stop existing.
- `VIndex::visited` / `visit_epoch` / `visited_cap` — the struct fields, `visited_ensure`, its call from `elems_reserve`, `ix->visit_epoch = 0` in `vindex_create`, and `free(ix->visited)` in `vindex_free`.
- The **proposed** per-search scratch *struct on the index* (a checkout pool / `VisitedListPool`) — never built. The buffer is a plain frame local; a pool is machinery for an ownership question that no longer exists.
- The **proposed** reader-view / owner-handle split for VIndex specifically — superseded. `const` already is the reader view.
- `EXPECT_RACE` in `run_vindex_concurrency_tests.sh` — a knob that let a known defect ride as "expected". Replaced by four halves with real verdicts.
**NOT deleted — the design doc was wrong about this one:**
- `fb32d15` (`eg_guard_enter` / `engram_req_lock` / `_eg_req_depth`). §5 originally called for its removal as "a lock protecting a mutation that ceases to exist." **Measured, it guards two things, and only one of them ceases to exist.** Its own comment names both: the RAM graph *and* `_eg_vindex`. The vindex justification is retired; the RAM-graph justification is independently load-bearing (§3.2), and removing the guard reintroduces the measured 11171→9579 edge-loss defect from 2026-08-14. Its comment has been narrowed to state the RAM graph only. **Precondition for deleting it:** the resident graph gets the same publication boundary the index just got.
- el #140's hand-patch. Left in place — the leak stops being *expressible* only under the abandoned capability-ABI §3, which is not what was built.
**Ordering consequence (revised):** the original ordering claim — "the residual lands first, the residues evaporate rather than get fixed" — did not survive contact. The residual here is not a single ABI that dissolves everything at once; it is a *property* (derived state is published, never edited) applied per structure. The index now has it. The RAM graph does not yet. Residues evaporate **per structure, in the order the property is applied**, and a residue whose structure has not been converted must be left standing, not deleted on the strength of the plan.
---
## 6. Sequencing
1. **Read** how builtins are declared and dispatched, to confirm the call sites are compiler-generated in one place. *(This determines whether §4 holds. If dispatch is scattered, re-size before proceeding.)*
2. Introduce the context type and capability types.
3. Codegen emits the context at every builtin call site.
4. Mechanical sweep of builtin signatures.
5. Move index maintenance behind the write capability; the three read callers take the read capability.
6. Delete the residue-fixes listed in §5.
7. **One** build of soul from el dev — which resolves the `state_get` leak and the crash together, rather than deploying a leak fix that reintroduces the crash.
---
## 7. Open questions
**Answered 2026-08-16:**
- ~~Do the nine append sites share a commit point?~~ **Moot.** The question was mis-aimed: node append is not the event that owns index membership, because a node without an embedding cannot be in a vector index. The five *embedding-assignment* sites are the real owner points (`el_runtime.c:7091, 9839, 13362, 15002`, plus snapshot-restore at `7951`), and three of them carry the ordinal directly — which is all `eg_vindex_note_embedded` needs. The other two run before the node is resident, where the cold build picks it up.
- ~~Does anything outside `lang/runtime/` construct a second `VIndex`?~~ **No.** Swept: the only constructors outside the runtime are `engram/test/*` and `lang/runtime/vindex_bench.c`, all single-threaded and index-private. Inside the runtime, `engram_self_reify_beat_json` builds a **private** index deliberately and never touches the shared boundary — that was already correct and is unchanged.
- ~~Does the HTTP worker pool contend on the same globals?~~ **Yes, and it was never the whole story.** Workers serialize against each other on `engram_req_lock`, but the awareness main thread does not take it at all — that is the gap `fb32d15` closed. Now verified independent of that guard: the index boundary is its own rwlock, so worker/awareness contention on `_eg_vindex` is handled whether or not the request lock is held.
**Still open:**
- The resident RAM graph wants the same publication boundary (§3.2). Until it has one, `fb32d15` cannot be deleted.
- `eg_vindex_view` holds the boundary for read across `engram_geo_reify_store`, which is a long pass. Correct, but it stalls the owner for that duration. If reify latency becomes a problem the answer is a refcounted snapshot, not a shorter lock.
---
## 7a. Evidence (measured 2026-08-16, `engram/test/run_vindex_concurrency_tests.sh`)
| half | before | after |
|---|---|---|
| `single` — 3000 vectors, 1 thread, ASan+UBSan | clean | clean |
| `readers` — 4 readers, no writer, TSan | **race** at `engram_vindex.c:195` (`visited_reset``vindex_search`) | **clean** |
| `unsynchronized` — writer+reader, bare index, TSan | race | **race, expected and permanent** — now the proof the boundary must exist |
| `published` — owner + 4 readers through the boundary, TSan | *(did not exist)* | **clean**, all 3000 inserts landed |
No recall regression: `recall@10 = 0.9365` at `ef_search=128` (gate ≥ 0.90); the determinism test still yields byte-identical results across two independent builds.
Builds locally: all seven engram runtime translation units compile `-Wall -Wextra` clean, and the full engram binary links (`engram/dist/engram.c` + runtime, arm64). The one pre-existing `-Wcomment` warning in `el_runtime.c` is present at `a67452f` too.
---
## 8. What this document is not
It is not an argument for a memory model in general, a garbage collector, process isolation between soul and engram, or a client/server split of the store. Each of those was considered and each addresses mutation that this change removes. They are answers to a question that stops being asked.
+91
View File
@@ -0,0 +1,91 @@
// fitprobe.el controlled growth-curve specimens for validating the complexity fitter.
//
// Three deliberately-shaped workloads. None depends on a real defect existing,
// which is the point: the fitter must be provable against KNOWN curves.
//
// linear one allocation per item. count O(n), bytes O(n), time O(n)
// accum rebuilds its accumulator. count O(n), bytes O(n^2), time O(n^2)
// compute nested arithmetic, no alloc. count O(1), bytes O(1), time O(n^2)
//
// `compute` is the specimen that matters. It is the shape of el #132
// (strlen-per-character inside str_char_code): pure CPU, zero allocation.
// An allocation-only gate is structurally blind to it.
//
// No imports uses runtime builtins directly so nothing collides.
fn work_linear(n: Int) -> Int {
let parts: [String] = native_list_empty()
let i: Int = 0
while i < n {
let parts = native_list_append(parts, int_to_str(i))
let i = i + 1
}
return native_list_len(parts)
}
fn work_accum(n: Int) -> Int {
let acc: String = ""
let i: Int = 0
while i < n {
let acc = acc + "x"
let i = i + 1
}
return str_len(acc)
}
fn work_compute(n: Int) -> Int {
// str_char_code is an opaque external call, so the C optimiser cannot
// reduce this nest to a closed form the way it does with `total + 1`.
// This is the exact shape of el #132: n scans over n characters, pure
// CPU, ZERO allocation.
let s: String = "abcdefghij"
let total: Int = 0
let i: Int = 0
while i < n {
let j: Int = 0
while j < n {
let total = total + str_char_code(s, 0)
let j = j + 1
}
let i = i + 1
}
return total
}
fn run_one(mode: String, n: Int) {
let c0: Int = el_alloc_count()
let b0: Int = el_alloc_bytes()
let t0: Int = el_now_instant()
let r: Int = 0
if str_eq(mode, "linear") { let r = work_linear(n) }
if str_eq(mode, "accum") { let r = work_accum(n) }
if str_eq(mode, "compute") { let r = work_compute(n) }
let t1: Int = el_now_instant()
let c1: Int = el_alloc_count()
let b1: Int = el_alloc_bytes()
println(mode + "\t" + int_to_str(n)
+ "\t" + int_to_str(c1 - c0)
+ "\t" + int_to_str(b1 - b0)
+ "\t" + int_to_str((t1 - t0) / 1000)
+ "\t" + int_to_str(r))
return
}
fn sweep(mode: String) {
run_one(mode, 200)
run_one(mode, 400)
run_one(mode, 800)
run_one(mode, 1600)
return
}
fn main() -> Int {
println("mode\tn\tallocs\tbytes\tusec\tsink")
sweep("linear")
sweep("accum")
sweep("compute")
return 0
}
+111
View File
@@ -0,0 +1,111 @@
import "../../runtime/eltest.el"
import "../../runtime/elbench.el"
// test_elbench.el proves the growth-curve classifier against KNOWN curves.
//
// Every series below is real measured data from lang/tests/bench/fitprobe.el
// on a geometric sweep n = 200/400/800/1600. The classifier must be provable
// without depending on a live defect existing, which is the whole point of
// keeping controlled specimens.
fn _s4(a: Int, b: Int, c: Int, d: Int) -> [Int] {
let l: [Int] = native_list_empty()
let l = native_list_append(l, a)
let l = native_list_append(l, b)
let l = native_list_append(l, c)
let l = native_list_append(l, d)
return l
}
test "classifies a linear allocation series as O(n)" {
// fitprobe `linear`, allocation count
let v = _s4(208, 409, 810, 1611)
assert elb_measured_curve(v, 10) == 2, "linear allocs should classify O(n)"
}
test "classifies a linear byte series as O(n)" {
// fitprobe `linear`, allocation bytes
let v = _s4(4786, 9682, 19474, 39658)
assert elb_measured_curve(v, 10) == 2, "linear bytes should classify O(n)"
}
test "classifies a quadratic byte series as O(n^2)" {
// fitprobe `accum`, allocation bytes -- the accumulator-rebuild shape
let v = _s4(20300, 80600, 321200, 1282400)
assert elb_measured_curve(v, 10) == 4, "accum bytes should classify O(n^2)"
}
test "accumulator count is linear -- proves count alone misses it" {
// Same run as above. The COUNT is exactly linear while bytes are
// quadratic. A count-only gate passes this defect clean.
let v = _s4(200, 400, 800, 1600)
assert elb_measured_curve(v, 10) == 2, "accum count classifies O(n)"
assert elb_gate(v, 2, 10) == 0, "count-only gate PASSES the quadratic"
}
test "classifies a quadratic time series as O(n^2)" {
// fitprobe `compute` -- el #132's shape: n scans over n characters
let v = _s4(67, 205, 818, 3268)
assert elb_measured_curve(v, 10) == 4, "compute time should classify O(n^2)"
}
test "REFUSES an all-zero series instead of calling it O(1)" {
// fitprobe `compute` allocation count. Pure CPU, allocates nothing.
// Reporting O(1) here would be a confident answer with nothing behind it.
let v = _s4(0, 0, 0, 0)
assert elb_gate(v, 2, 10) == 3, "all-zero series must be REFUSED"
assert elb_measured_curve(v, 10) < 0, "unclassifiable returns -1"
}
test "REFUSES an implausibly flat series" {
// The shape produced when clang closes a loop to a multiply: a real
// answer, no work done, no movement across an 8x input range.
let v = _s4(1000, 1001, 1002, 1003)
assert elb_gate(v, 2, 10) == 3, "hard-flat series must be REFUSED"
}
test "gate FAILS a quadratic declared as linear" {
let v = _s4(20300, 80600, 321200, 1282400)
assert elb_gate(v, 2, 10) == 1, "O(n^2) measured vs O(n) declared must FAIL"
}
test "gate PASSES a linear series declared as linear" {
let v = _s4(208, 409, 810, 1611)
assert elb_gate(v, 2, 10) == 0, "O(n) measured vs O(n) declared must PASS"
}
test "gate reports BETTER when measured beats the declared bound" {
let v = _s4(208, 409, 810, 1611)
assert elb_gate(v, 4, 10) == 4, "O(n) measured vs O(n^2) declared is BETTER"
}
test "gate reports INDETERMINATE on disagreeing ratios" {
// fitprobe `linear` WALL TIME at these sizes: 26/19/43/78 microseconds.
// Ratios 0.73, 2.26, 1.81 disagree well past the noise threshold. The
// honest answer is "cannot tell", not a classification -- this is exactly
// why benchmarks need auto-scaled iteration counts rather than one shot.
let v = _s4(26, 19, 43, 78)
assert elb_gate(v, 2, 10) == 2, "disagreeing ratios must be INDETERMINATE"
}
test "black_box is a real barrier and returns its input" {
assert el_black_box(42) == 42, "black_box is value-preserving"
let s: Int = 0
let i: Int = 0
while i < 100 {
// Bind the call before using it in arithmetic: `x + call(...)`
// lowers to el_str_concat() on integers. Same inference defect
// as `call(...) == y` lowering to str_eq().
let bx: Int = el_black_box(1)
let s = s + bx
let i = i + 1
}
assert s == 100, "black_box does not disturb the computation"
}
test "curve names round-trip" {
assert elb_curve_from_name("O(n)") == 2, "O(n) parses"
assert elb_curve_from_name("O(n^2)") == 4, "O(n^2) parses"
assert str_eq(elb_curve_name(4), "O(n^2)"), "O(n^2) renders"
assert elb_curve_from_name("O(nonsense)") < 0, "unknown curve is -1"
}
+178
View File
@@ -0,0 +1,178 @@
import "../../runtime/eltest.el"
import "../../runtime/elbench.el"
// test_lexer_scaling.el THE ARMED GATE.
//
// This is the regression test that would have caught el #132.
//
// #132 was a strlen() inside str_char_code() and str_slice(). The lexer walks
// source one character at a time, so every character access rescanned the whole
// remaining input: O(n) per character over n characters = O(n^2). It shipped for
// months. It was found by a geometric sweep, not by reading code.
//
// So this test IS a geometric sweep. It scans a string of length n, character by
// character, at four doubling sizes, and asserts the cost is linear. If anyone
// reintroduces a per-character rescan in str_char_code, in str_slice, in any
// accessor the lexer leans on the measured curve becomes O(n^2) and this fails.
//
// The value is in it being ARMED, not in it currently failing. It passes today
// because #132 is fixed. That is the correct state for a regression gate.
//
// Note the deliberate `let c: Int = str_char_code(...)` binding in the scan loop.
// Inlining it as `total + str_char_code(s, i)` lowers to el_str_concat() on
// integers the Plus arm of the operator-typing family, still open at the time
// of writing. Binding first is the safe form.
// _mk_string build a string of length >= n by DOUBLING.
//
// Deliberately not `s = s + "x"` n times: that is itself quadratic in bytes and
// would contaminate the very measurement this test exists to take. Doubling
// allocates ~2n total.
fn _mk_string(n: Int) -> String {
let s: String = "abcdefgh"
while str_len(s) < n {
let s = s + s
}
return s
}
// _scan walk the string one character at a time, REPS times.
//
// This is the lexer's access pattern reduced to its essential shape. The
// repetitions lift the measurement clear of timer resolution; without them the
// smaller sizes land in noise and the classifier correctly reports
// INDETERMINATE rather than guessing.
fn _scan(s: String, n: Int, reps: Int) -> Int {
let total: Int = 0
let r: Int = 0
while r < reps {
let i: Int = 0
while i < n {
let c: Int = str_char_code(s, i)
let total = total + c
let i = i + 1
}
let r = r + 1
}
return total
}
// _measure_scan microseconds for a full scan sweep point.
fn _measure_scan(n: Int, reps: Int) -> Int {
let s: String = _mk_string(n)
// WARMUP, discarded. Without it the small-n end of the sweep is dominated
// by cold caches and reads as superlinear on genuinely linear work --
// measured ratios 3.37 2.92 1.76 1.65 on exactly this workload.
let w: Int = _scan(s, n, 2)
let wj: Int = el_black_box(w)
let t0: Int = el_now_instant()
let got: Int = _scan(s, n, reps)
let t1: Int = el_now_instant()
// Feed the result through the barrier so the scan cannot be elided.
let sink: Int = el_black_box(got)
if sink == 0 { println("") }
return (t1 - t0) / 1000
}
fn _series4(a: Int, b: Int, c: Int, d: Int) -> [Int] {
let l: [Int] = native_list_empty()
let l = native_list_append(l, a)
let l = native_list_append(l, b)
let l = native_list_append(l, c)
let l = native_list_append(l, d)
return l
}
test "character scan is LINEAR in time -- regression gate for el #132" {
let reps: Int = 40
let t1: Int = _measure_scan(16384, reps)
let t2: Int = _measure_scan(32768, reps)
let t3: Int = _measure_scan(65536, reps)
let t4: Int = _measure_scan(131072, reps)
let series: [Int] = _series4(t1, t2, t3, t4)
let verdict: Int = elb_gate(series, 2, 50)
let measured: Int = elb_measured_curve(series, 50)
// Report the actual numbers regardless of outcome. A gate that fires
// without showing its evidence is just an assertion.
println(" scan us: " + int_to_str(t1) + " " + int_to_str(t2) + " "
+ int_to_str(t3) + " " + int_to_str(t4)
+ " -> " + elb_curve_name(measured) + " [" + elb_verdict_name(verdict) + "]")
// PASS (0) or BETTER (4) are both acceptable. FAIL (1) means someone
// reintroduced superlinear per-character cost. REFUSED (3) or
// INDETERMINATE (2) mean the measurement is untrustworthy -- which is
// also a failure of this test, deliberately: a gate that cannot measure
// must not report success.
assert verdict == 0 || verdict == 4, "character scan must measure O(n) or better"
}
test "string building by doubling stays linear in allocated bytes" {
let b1: Int = el_alloc_bytes()
let s1: String = _mk_string(8192)
let b2: Int = el_alloc_bytes()
let s2: String = _mk_string(16384)
let b3: Int = el_alloc_bytes()
let s3: String = _mk_string(32768)
let b4: Int = el_alloc_bytes()
let s4: String = _mk_string(65536)
let b5: Int = el_alloc_bytes()
let series: [Int] = _series4(b2 - b1, b3 - b2, b4 - b3, b5 - b4)
let verdict: Int = elb_gate(series, 2, 1000)
let measured: Int = elb_measured_curve(series, 1000)
println(" bytes: " + int_to_str(b2 - b1) + " " + int_to_str(b3 - b2) + " "
+ int_to_str(b4 - b3) + " " + int_to_str(b5 - b4)
+ " -> " + elb_curve_name(measured) + " [" + elb_verdict_name(verdict) + "]")
assert verdict == 0 || verdict == 4, "doubling build must be O(n) in bytes"
assert str_len(s4) >= 65536, "final string reached the requested size"
}
// _scan_quadratic a DELIBERATELY quadratic scan: for each position, rescan
// from the start. This is precisely what el #132 did strlen() from offset 0
// on every character access reproduced here so the gate can be proven to
// FIRE, not merely to pass on healthy code. An unproven gate is decoration.
fn _scan_quadratic(s: String, n: Int) -> Int {
let total: Int = 0
let i: Int = 0
while i < n {
let j: Int = 0
while j < i {
let c: Int = str_char_code(s, j)
let total = total + c
let j = j + 1
}
let i = i + 1
}
return total
}
fn _measure_quadratic(n: Int) -> Int {
let s: String = _mk_string(n)
let w: Int = _scan_quadratic(s, 64)
let wj: Int = el_black_box(w)
let t0: Int = el_now_instant()
let got: Int = _scan_quadratic(s, n)
let t1: Int = el_now_instant()
let sink: Int = el_black_box(got)
return (t1 - t0) / 1000
}
test "the gate FIRES on a live quadratic scan -- proves it is armed" {
let q1: Int = _measure_quadratic(1024)
let q2: Int = _measure_quadratic(2048)
let q3: Int = _measure_quadratic(4096)
let q4: Int = _measure_quadratic(8192)
let series: [Int] = _series4(q1, q2, q3, q4)
let verdict: Int = elb_gate(series, 2, 50)
let measured: Int = elb_measured_curve(series, 50)
println(" quad us: " + int_to_str(q1) + " " + int_to_str(q2) + " "
+ int_to_str(q3) + " " + int_to_str(q4)
+ " -> " + elb_curve_name(measured) + " [" + elb_verdict_name(verdict) + "]")
assert measured == 4, "a rescan-from-zero workload must classify O(n^2)"
assert verdict == 1, "declared O(n) against measured O(n^2) must FAIL the gate"
}