59 Commits

Author	SHA1	Message	Date
Neuron	cace6a5ebf	runtime: a disconnecting client must not kill the server ... There was no SIGPIPE handling anywhere in this runtime: no signal disposition, no MSG_NOSIGNAL, no SO_NOSIGPIPE, and send() called with bare flags. The default disposition of SIGPIPE is to TERMINATE THE PROCESS, so any client that hangs up mid-response takes the whole engram with it. MEASURED, and it is not hypothetical. Production has restarted 254 times since 2026-08-13T19:37 at a flat ~10 minute cadence: 17:05:18 17:15:29 17:25:38 17:35:50 17:46:00 17:56:10 18:06:22 18:16:30 Intervals of 10m09s-10m12s, not 10m00s. That excess is the whole story: ai.neuron.engram-tick has StartInterval 600, and engram-tick.sh:13 calls curl -s -m10 -X POST .../api/tick The beat does not finish within 10s over 13,634 nodes, so curl waits its full timeout and closes. The engram then writes the tick response to a dead socket, takes SIGPIPE, and dies. launchd KeepAlive restarts it, so the failure presents as a mysterious restart rather than a crash — and ~/.neuron/logs/engram.log records nothing but "[http] listening on" 254 times, with no exit reason. launchctl list confirms the last exit as -13. Root cause is one level out: consolidation had no owner, so an external ticker was created to poke it, and the ticker is what kills it. The fix here does not address that; it makes the process survivable while it is addressed. Two layers, because neither alone is portable: - SO_NOSIGPIPE per accepted socket (Darwin/BSD) and MSG_NOSIGNAL per send (Linux), so the signal is never raised for socket writes at all. - A process-wide SIG_IGN backstop, installed once and idempotent, for platforms and paths with neither. With the signal ignored, send() returns -1/EPIPE and the existing error path closes the connection. Also retries send() on EINTR, which the previous loop treated as fatal. This is an exemption in the sense of lang/spec §8: the write never checked whether the peer was still there, and the consequence of not checking was fatal rather than merely wrong.	2026-08-16 13:25:14 -05:00
will.anderson	d41645388a	runtime: make valid UTF-8 the JSON emitter's contract (#148)	2026-08-16 17:03:44 +00:00
Neuron	8a307dfd42	runtime: make valid UTF-8 the JSON emitter's contract ... Three nodes in the live graph carry labels truncated to exactly 80 bytes ending in a lone 0xE2 — the first byte of an em-dash, cut mid-sequence. jb_emit_escaped copied every byte >= 0x20 through verbatim, so those three nodes made the ENTIRE /api/nodes/list response undecodable and no strict parser could read the graph at all. production binary 25,929,607 bytes INVALID at byte 89260 this build 26,338,389 bytes VALID, parses to 13,630 nodes The damage was NOT written by this runtime. No 80-byte truncation exists here (the only label truncation is engram_first_n_chars at 60), and the content of those nodes is 2572 and 2746 bytes. Some other producer wrote them. That is exactly why fixing a writer could not have fixed this: the store already holds the damage, and it accepts data from importers, other producers and older binaries. So the fix goes where the promise is made. A serializer that emits JSON owes valid UTF-8 whatever it is handed. jb_emit_escaped now validates each multi-byte sequence before emitting any of it and substitutes U+FFFD for a bad lead byte, a missing or malformed continuation, an overlong encoding, a UTF-16 surrogate, or a codepoint above U+10FFFF. Invalid bytes are REPLACED rather than dropped, so the damage stays visible in the output instead of being silently papered over. Well-formed input is byte-identical to before. Second, preventive and explicitly NOT the cause of the above: engram_first_n_chars truncated by BYTES despite its name, so content with a multi-byte character crossing byte 60 would produce a half codepoint in the label. It now uses el_utf8_safe_len, which returns the largest byte length <= max that does not split a codepoint. Bounded by bytes, not codepoints, so existing labels never grow — they only stop splitting. el_utf8_safe_len lives beside str_count_chars rather than in the engram because the rest of el's string layer is already codepoint-aware (str_count_chars counts codepoints, str_reverse walks codepoint lengths). Byte truncation was the outlier and the concern is a string concern. Note on the investigation: I first "fixed" the truncator and wrote a test that passed on the UNPATCHED build too, because route_create_node passes label = content when no label is supplied, so engram_first_n_chars is never reached over HTTP. The test proved nothing. The real cause was only found by decoding the actual failing bytes out of the live response.	2026-08-16 12:03:03 -05:00
will.anderson	616815b2ab	Give cross-cutting concerns an owner instead of a convention (#145)	2026-08-16 16:57:51 +00:00
will.anderson	1a8a966cb3	runtime: transduction is a language concern, so move it into the language (#144)	2026-08-16 16:57:35 +00:00
Neuron	317466e8f7	runtime: ground the node asked about, and refuse circular support ... engram_ground_json resolved each seed to a REGION, wrote the grounded-by edge between the two regions' HUBS, and then echoed those hubs back in the "claim"/"evidence" fields as if they were the caller's input: const char* cid = C->hub_id ? C->hub_id : EL_CSTR(claim); const char* eid = E->hub_id ? E->hub_id : EL_CSTR(evidence); cog_ground_edge(g_engram_store, cid, eid, grounding, fw); Three consequences, all measured against a clone of the live store: 1. The edge landed on a node the caller never named. Grounding 3b9ced5d against 6edf8c79 wrote an edge on the hubs of their regions instead. 2. When both seeds resolve into the same region the support is circular and scores near 1.0 for structural reasons, not evidential ones. Four probe nodes written together landed in one region, and every grounding among them returned 0.93-0.99 as if it were evidence. Two independent agents hit this and reported 0.885 / 0.909 self-groundings as confident. 3. The echo concealed both: the response was indistinguishable from a successful grounding of the ids that were passed in. The region is HOW a claim is evaluated; it is not WHAT the claim is about. So the edge now attaches to the requested ids, and the resolved hubs are reported separately as claim_region / evidence_region. Degeneracy is broader than hub == hub. Three circular shapes, all previously invisible: same-region both seeds resolve to one region claim-region-is-evidence the evidence IS the hub of the claim's own neighbourhood — measured at 0.98883 evidence-region-is-claim the mirror case Each sets grounding to 0 and writes no edge. Circular support is not support, and a grounding that is degenerate by construction must not enter the graph as though it were evidence. Verified: 6edf8c79 -> 6edf8c79 degenerate=same-region g=0 written=false 6edf8c79 -> d0406dfd degenerate=same-region g=0 written=false ebc1413e -> 64cc96ef degenerate=false g=0.774563 written=true 64cc96ef -> ebc1413e degenerate=false g=0.802896 written=true Legitimate grounding across distinct regions is unchanged and still writes; only circular support is refused. This is the same class as #142 and #146 — a value that looked like an answer with nothing behind it — except here it was also writing that non-answer into the canonical store.	2026-08-16 11:53:31 -05:00
Neuron	88e3008735	runtime: resume the learned stance in think, instead of discarding it ... engram_think_json built a NEUTRAL stance on every call — cog_stance_init with a NULL id, all axis_gain 1.0, bias_dir NULL, reliability 0.5 — and never loaded the stance the correspondence-beat had been persisting. That mattered because the faculty enters engram_think ONLY through the stance: axis_gain[k] warps the per-axis extents and bias_dir seeds the steering direction. cog_stance_init stores the faculty NAME and nothing reads it. So with a neutral stance, reason/abduce/induce/plan/analogize were byte-identical output under different labels, and confidence was pinned to 0.5 because GeoGradient.confidence IS stance->reliability. The machinery already existed and only this call site ignored it. engram_correspondence_beat_json resumes via cog_stance_from_node and persists via cog_stance_to_node under "stance-<faculty>-<hub>". Every beat's calibration was written and then thrown away on the next read. Same defect as the NULL anchor fixed in #142, one line below: a neutral argument collapsing a capability to a constant. Resume the same id the beat writes, so learning compounds across beats and cold boot. Fall back to neutral only when no stance exists — a genuine uninformed prior rather than a discarded informed one. Also emit stance_resumed, so confidence 0.5 from a learned-but-unreliable stance is distinguishable from confidence 0.5 from "no stance exists". That reporting gap is what let the neutral stance hide. Verified against a clone of the production store (13,627 nodes): before beat, no stance stance_resumed=false confidence=0.5 beat on a NON-keystone brier 0.00458568 -> 0.00329654 reduction 28.11%, n_trials 6000, reliability 0.930726, stance_written=true after beat stance_resumed=true confidence=0.930726 Confidence now equals the learned reliability instead of the uninformed prior. The keystone self-anchor correctly stays at 0.5 — calibration is deliberately refused on protected identity regions, and that refusal is now visible as resumed=true with confidence unchanged, rather than being indistinguishable from the bug. STILL OPEN: with no learned bias_dir the faculties remain identical in direction. What distinguishes abduce from induce geometrically is a design decision about how Neuron thinks, not a plumbing defect, and is deliberately left to Will.	2026-08-16 11:43:38 -05:00
bigmerge	8ae163e8e5	lang: give cross-cutting concerns an owner instead of a convention ... El's units of encapsulation are the function and the module. Neither can hold a concern that belongs to the process, so each one had been expressed the only way it could be -- as a convention: call this at every site. Conventions of that shape do not hold. Measured here: zero process-identity guards at any layer, 20 environment variables each with its default written inline at the read site, 62 persist call sites, 10 per-route auth checks. One absence, four times. Step 0 first, because the premise was wrong. El was believed to have no middleware or effect mechanism. It has one, and it is already load-bearing: codegen injects engram_boundary_beat at the entry of every @manager/@accessor fn, decorators take arguments and stack, dharma_emit from a non-@manager fn is a #error, and the cgi block injects el_cgi_init at the head of main(). So the correct move was not to invent a mechanism but to generalize the seam that already existed. The real gap is narrower and is now recorded: the seam is prologue-only and its callee is a fixed builtin. Adds a `program` block -- the third program-level declarative block. cgi and service declare what a program may do; program declares what it is. program "engram" { singleton: "engram" env ENGRAM_BIND: String = ":8742" env GUIDE_PORT: Int = "8771" } singleton takes an exclusive flock before any user statement runs and refuses a second start, reporting the holder's pid. It is a lock rather than a pidfile so the kernel releases it on death including SIGKILL -- no stale state, and so no "delete the lock file to get unstuck" ritual, which would itself be a convention. It reports the pid because "already running" is not actionable; a pid is. That is the direct answer to a stale process surviving a pkill and going on answering probes. env entries resolve once at startup -- environment wins, declaration supplies the fallback -- and validate as a whole, reporting every problem at once rather than costing one restart per variable. config("X") for an undeclared X is fatal, because an advisory schema is just another convention. Programs without a program block are unaffected, so migration is per-program. Only one keyword is added. `config` and `env` could not become keywords -- both are real identifiers in the tree -- so the block's fields are read as identifier token values by its own parse loop and stay usable everywhere else. The init function is emitted at the block site and called from main() rather than inlined into main(). The live backend is codegen_streaming, which emits in source order and cannot hold the entry list alive until main(); this way only a single bool has to survive. Also fixes: config() was defined in el_runtime.c but never prototyped in el_runtime.h, so any el program calling it failed to compile under C99. Spec: section 18 documents what shipped. Section 9 is corrected -- it claimed decorators had no structural meaning, which has not been true for some time. Section 19 designs durability-as-an-epilogue-effect and route authorization and states plainly why neither is implemented here: both land in files under concurrent modification, and the prerequisite for both is lifting the seam from prologue-only to prologue/epilogue. Self-hosting fixpoint verified byte-identical.	2026-08-16 11:38:28 -05:00
bigmerge	3fcc36c2f1	runtime: transduction is a language concern, so move it into the language ... #141 let signal enter as geometry and it worked, but it was placed at the CONSUMER and said so in its own commit message. This is the correction. Three defects, all of them placement: 1. It sat in the engram. Ingest is a LANGUAGE concern — every el program touching any modality needs it, and the engram is merely one el program that happens to hold a graph. The geometry surface is now defined in el_runtime.c immediately ABOVE the engram section and depends on nothing inside it. Delete the entire engram and geometry still enters el. 2. It marshalled the vector as a hex STRING, because el had no first-class geometry value — which reintroduced text as the TRANSPORT medium one layer below the problem being fixed. Geometry is now an el value: a magic-tagged heap object carried in el_val_t, same discipline as List/Map. Hex survives only as an adapter at the edge, which is all an encoding should ever be. 3. It needed an arbitrary `dim <= 8192` bound purely to size an allocation from a caller's CLAIM about a string's length. A value carries its own width, so the width is derived and never asserted. The bound is gone, not raised — there is nothing left to validate. Language surface, none of it engram-prefixed: geometry_new / _dim / _is / _get / _set / _norm / _free, geometry_from_f32le_hex + geometry_to_f32le_hex as the wire adapters, realizer_register(modality, fn_name), realizer_has, and transduce(signal, modality) -> Geometry. REALIZERS ARE DECLARABLE IN EL. This is the part that makes the move real rather than nominal: registration resolves a name with dlsym against the running binary, the identical mechanism http_set_handler already relies on, because every el `fn name(...)` compiles to a global C symbol with that exact name. So an ordinary el function IS a realizer and a new modality needs no runtime patch. Verified end to end in lang/examples/transduce.el: an el-defined tone_realizer is registered by name, transduce dispatches to it, and the signal demonstrably reaches it (distinct signals produce distinct geometry). A modality with no realizer transduces to NOTHING. There is deliberately no built-in realizer, not even for text — silently embedding a description of a signal and calling that perception is the exact defect this ends. engram/src/server.el is migrated: POST /api/nodes decodes "emb" hex exactly once, at the edge, into a Geometry, and everything below that line moves geometry. The wire is unchanged because production clients speak it. "dim" is now an ASSERTION about the vector, not the source of its width; disagreement is a rejected ingest, not a silent reinterpretation. #141's engram_node_set_emb becomes a DEPRECATED WRAPPER over geometry_from_f32le_hex + node_attach_geometry — kept only because the runtime ships as an SDK asset and a downstream binary may link the symbol. Its exact contract, negative cases included, is preserved and re-verified. ingest.el's `fn transduce` is renamed transduce_manifold. Mechanically it had to yield the name (duplicate C symbol, a hard compile error, measured). But it was never signal->geometry: it chunks already-extracted content into a node+edge manifold, one layer up, and had taken the name belonging to the primitive underneath it. Behaviour unchanged. PROPERTIES FROM #141 PRESERVED, each re-measured on a scratch engram (:8971, never prod :8742): - off-dimension vectors stored but NOT indexed — the HNSW build loop still filters on n->emb_dim == dim at four sites, so a 64-dim voice vector is durable and addressable without perturbing the 768-dim canonical index - geometry makes a node ineligible for embed_backfill: after backfill the 64-dim voice node was still 64-dim while the text control acquired 768 - the create response reports whether geometry landed, and the node document always emits emb_dim and embedded Read-back with control and negatives, all verified against a PID-confirmed fresh binary: geometry node emb_dim=64 embedded=true / emb_set=1; text-only control emb_dim=0 embedded=false / emb_set=0; malformed hex, ragged length, and dim-disagreement each emb_set=0. Two compiler landmines found by reading the generated C rather than trusting a successful build, both documented at their sites: elc lowers `a == b` to str_eq unless both operand NAMES are in the per-function int-name set (which does NOT propagate into nested if-expression blocks — the first cut would have strcmp'd two integers as pointers on the first geometry-bearing request), and `+` lowers to string concat when either operand is a user-defined call.	2026-08-16 11:37:27 -05:00
Neuron	8e9d88fc01	runtime: publish the vector index instead of guarding it ... The crash (SIGTRAP in engram_activate -> eg_vindex_sync -> vindex_insert -> _realloc) had three read paths mutating five process-global statics. engram_activate, eg_knn_for_node (whose own comment says "No writes.") and engram_geo_reify_run_json all called eg_vindex_sync, which frees the index, reallocs the seen-map and inserts — on a read. Three moves, in decreasing order of how much they dissolve: 1. Misfiled scratch is not shared state. visited/visit_epoch/visited_cap were never owned by the index; they are one traversal's local, hoisted into struct VIndex as an allocation optimisation. They want neither a lock nor a capability nor a pool — just to go back in the call frame. Two concurrent READS stomped each other purely because of this. 2. const IS the capability. Once the scratch leaves the struct, search reads and nothing else, so vindex_search takes a const VIndex*. That is exactly what a capability-pointer ABI would have bought — a read path physically cannot call vindex_insert, enforced by the compiler on every future caller — for one qualifier instead of an ABI swept across hundreds of builtins. 3. What survives is publication, not ownership. HNSW insert is NOT an append: it rewires the neighbour links of already-existing elements and reallocs elems[], so the store's append-only property does not transfer to the index derived from it. eg_vindex_sync therefore splits into eg_vindex_maintain (exclusive, sole mutator) and eg_vindex_view (shared, returns const VIndex*). A read path may demand that a current snapshot exist — a request to the owner, not a mutation by the reader. Write-side owner: eg_vindex_note_embedded hooks the embedding-ASSIGNMENT sites rather than the append sites, because a node with no embedding cannot be in a vector index — embedding assignment is the event that owns index membership. One O(log n) insert, no O(node_count) presence scan. This also retires the "STALENESS (honest tradeoff)" note where a lazily-embedded older node stayed invisible to route_nearest/autoconnect until a full rebuild (the embed-gap #20 shape). Evidence. The existing harness conflated two hazards, which is why fixing half of it read as failure. Split into four: single (3000 vec, ASan+UBSan) clean -> clean readers (4 readers, no writer, TSan) RACE -> clean unsynchronized (writer+reader, bare) race -> race, expected forever published (owner + 4 readers) n/a -> clean, 3000/3000 landed RESULT: PASS. recall@10 = 0.9365 at ef_search=128 (gate >= 0.90); determinism byte-identical across two independent builds. The unsynchronized half is now permanently expected to race, deliberately: it is the executable proof that the boundary must live above the data structure, not inside it. fb32d15's guard is KEPT, correcting this design's own section 5. Measured, it guards TWO structures and only one was converted here: g->nodes/g->edges are realloc'd in place (el_runtime.c:7618,7629) and engram_activate_inner's embed-backfill writes n->emb through exactly such a borrowed pointer. Deleting the guard reintroduces a measured 11171->9579 edge loss. Its comment is narrowed to the RAM graph and the deletion precondition named. That corrects the ordering claim too: the residual is not one ABI that dissolves everything at once, it is a PROPERTY applied per structure. Residues evaporate in the order the property is applied, and a residue whose structure has not been converted must be left standing.	2026-08-16 11:29:17 -05:00
bigmerge	bdc1f99fb9	runtime: guard engram activation against the unsynchronized awareness thread ... The soul daemon had two engram callers and only one of them locked. soul.el:729 starts the HTTP server via http_serve_async (spawning http_worker threads); soul.el:731 then runs awareness_run() on the MAIN thread. awareness.el's perceive() -> engram_activate_json() -> engram_activate() -> eg_vindex_sync() -> vindex_insert() mutates the same g->nodes/g->edges and the process-global _eg_vindex HNSW index that the workers touch. g_engram_req_lock existed to serialize exactly this, but it was only ever taken inside http_worker: engram_req_lock/engram_req_unlock appear in ZERO .el sources, so the awareness loop ran lock-free beside the workers on every tick (SOUL_TICK_MS=1000). Result was a crash-loop under launchd KeepAlive: five crashes in ~4 minutes on 2026-08-16 with varying faulting frames -- search_layer<-vindex_insert <-eg_vindex_sync, engram_activate, abort, and one inside xzm_realloc's own freelist. Varying sites plus a fault in allocator metadata means heap corruption. The SIGSEGV address 0x65646f4e6d617267 is little-endian ASCII "gramNode": string bytes dereferenced as an Elem vector pointer. Diagnosed by bisection rather than inspection: - Replaying all 13,820 real dim-768 vectors harvested from the live store through the index single-threaded under ASan is 100% clean, which rules out an HNSW logic/bounds bug. - Two threads on one index trip ThreadSanitizer immediately at engram_vindex.c:195 (visited_reset), reached from both vindex_search and vindex_insert. VIndex keeps a SHARED visited-epoch scratch buffer, so even two concurrent READS corrupt each other's traversal and walk bogus element indices. So this is purely a concurrency defect, not an HNSW logic error. (An inspection-derived hypothesis about an out-of-bounds reverse-link write at engram_vindex.c:340 was disproved by the single-threaded run.) Fix: a thread-local ownership depth (_eg_req_depth) lets engram entry points self-guard. engram_activate() becomes a wrapper over engram_activate_inner() that acquires g_engram_req_lock when called with depth 0 (the awareness thread) and passes through when depth > 0 (nested inside an http_worker that already holds it), so the non-recursive mutex cannot self-deadlock. The depth is a plain counter, never a recursive-mutex count, preserving engram_self_reify_beat_json's contract of genuinely releasing the lock mid-beat.	2026-08-16 11:29:17 -05:00
Neuron	ded6ca546f	runtime: anchor the think read, so Neuron can think at all ... engram_think_json passed NULL as the anchor. NULL is not "no opinion": engram_think re-origins at `anchor ? anchor : region->centroid`, so NULL means "read from the centroid" — and the centroid is the one point where the gradient is zero by construction. r = x - centroid = 0, so every axis projection is 0, grad is 0, and direction takes the "at rest" branch at engram_cognition.c:137. Measured consequence: EVERY faculty returned an identical null result, differing only in its label — {"direction":[0,0,0,0,0,0,0,0],"spread":0,"magnitude":1,"confidence":0.5} magnitude 1 is membership evaluated at the centroid, spread 0 is its distance to itself, confidence 0.5 is the stance fallback. The geometry was never at fault: /api/drift computes real values (centroid_sep 0.104, core_disp 0.045) over the very same 87 members. Neuron could not think because the read was always taken from the region's own centre. The seeds choose WHICH region; they must also supply the VANTAGE. Anchor at the first resolvable embedded seed — the same seed eg_geo_build_desc infers dim from, so the two can never disagree. One seed still yields a real gradient because the descriptor expands to that seed's neighbourhood, so the seed's position is distinct from the neighbourhood centroid. The vector is COPIED, never borrowed: g->nodes is realloc'd in place on append, so a borrowed EngramNode* dangles across any concurrent write. Verified against a clone of the production store (13,616 nodes / 37,865 edges): self anchor n_support 87 magnitude 0.00282 spread 18.79 values hub n_support 28 magnitude 0.00318 spread 17.72 with distinct unit direction vectors. Previously both returned the zero vector with magnitude 1 and spread 0. STILL OPEN, now isolated by this fix: all five faculties return identical numbers and confidence stays 0.5, because cog_stance_init is passed NULL for the stance and the faculty enters the computation only through the stance's axis_gain[] and bias_dir. The faculty label is inert until a stance is loaded — which is what learn()'s correspondence-beat calibrates. Same shape as this bug: a neutral parameter collapsing a capability to a constant.	2026-08-16 11:25:24 -05:00
Neuron	c79033b749	runtime: let signal enter as geometry, not as prose about signal ... No ingest path could carry a vector. engram_node/_full/_layered take text only, and a node acquired an embedding solely via engram_embed_backfill DERIVING one from n->content. That made text the mandatory entry medium: any non-text modality had to be described in prose first, so the geometry we then reasoned over was the geometry OF THE DESCRIPTION, not of the signal. Measured: POST /api/nodes accepted an "emb" field, returned 200 with a fresh id, and stored nothing — emb_dim=None, embedded=false. engram_node_set_emb attaches a vector to an existing node. Off-dimension vectors are stored but not indexed (the HNSW build loop already filters on emb_dim), so modality geometry is durable and addressable without perturbing the canonical index. Setting emb also makes the node ineligible for embed_backfill, so a realizer's vector is never overwritten by a text-derived one. Two reporting fixes ride along, because both are how the drop stayed invisible: the create response now reports emb_set instead of being success-shaped regardless, and the node document now always emits emb_dim and embedded — without which a genuine ingest drop and a mere reporting gap are indistinguishable. Verified live: voice node emb_dim=64 embedded=true; text control emb_dim=0 embedded=false; malformed hex, length mismatch and dim<=0 all reject. KNOWN PLACEMENT DEFECT: this is at the consumer. Ingest is a language concern, not an engram feature — every el program touching any modality needs it. The vector also marshals as a hex STRING because el has no first-class geometry value, which reintroduces text as the transport medium one layer below the problem being fixed. The durable shape is geometry as an el value plus declarable realizers, after which the engram stops having an ingest concept at all. Landing this as the verified probe that proves the path.	2026-08-16 11:12:48 -05:00
bigmerge	9c07970943	runtime: state_get leaked its value on every call ... char* result = el_strdup_persist(e ? e->value : ""); // never freed pthread_mutex_unlock(&_state_mu); char* copy = el_strdup(result); // arena-tracked return el_wrap_str(copy); Two copies were made. `result` existed only as the source for `copy` — never returned, never freed — and el_strdup_persist bypasses the arena BY DESIGN ("state_set, engram internals"), so arena-pop could never reclaim it. Every state_get leaked its full value string, permanently. MEASURED: 200,000 state_get calls against a 64-byte value. before 15 MB peak RSS growth (~75 bytes/call — the value plus overhead) after 0 MB IMPACT. The soul's awareness loop has 68 state_get call sites and ticks every 200ms. Live measurement before the fix: RSS climbing 112 MB per 20s, about 19 GB/hour, in awareness_run -> one_cycle -> perceive, while node_count stayed flat at ~13,479 — growth with no data behind it. It drove the host from 20 GB free to 4.3 GB in roughly an hour. WHY NOW, since the code is old: the soul used to restart constantly (no write-through, divergent graph, 2.11 GB). Stabilising it (neuron #162) let it stay up long enough to accumulate. The fix did not cause this leak; it removed the crashes that were hiding it. Same pattern as the test framework surfacing math_log — the defect was always there, something finally made it visible. Found by Ishikawa rather than by reading the nearest code: method (arena push/pop IS correctly paired per tick), material (node count flat, so not data growth), environment (19 GB/hr / 18,000 ticks = ~1.1 MB per tick, so per-tick not one-shot), machine (an allocator that bypasses the arena) — which is where the evidence pointed. el_strdup tracks into the thread-local arena, which touches no shared state, so taking the single copy under _state_mu is safe and removes the temporary entirely. Verified: self-hosting fixpoint byte-identical; state round-trip correct for hit, miss, and overwrite.	2026-08-16 08:09:32 -05:00
Neuron	e0b2c0ea54	bench: arm the Phase 4 gate -- proven to pass clean AND fire on a quadratic ... Adds tests/native/test_lexer_scaling.el, the regression gate for el #132. Both directions are proven on LIVE workloads, not synthetic series: healthy per-character scan 1821 3251 6007 10422 us -> O(n) PASS rescan-from-zero (the #132 shape) 922 3667 13524 44792 -> O(n^2) FAIL A gate only proven to pass is decoration. The quadratic specimen exists so the gate is proven to FIRE. Also fixes elb_spread_ok to judge the ASYMPTOTIC TAIL (last three ratios) rather than the whole sweep. Measured on a genuinely linear scan the ratios ran 3.37 2.92 1.76 1.65 -- the head looks quadratic because it is cold cache, the tail is the truth. Whole-sweep spread rejected correct data. A complexity bound is an asymptotic claim and must be judged asymptotically. That fix came from the classifier refusing to rubber-stamp my own bad measurement: it reported INDETERMINATE on an unwarmed sweep rather than passing it. Warmup is now taken and discarded at every sweep point. Reverts the == workarounds in test_elbench.el now that el #137 has landed; the natural form generates no str_eq and all 13 fitter tests stay green. The workaround remains -- the Plus arm is still open.	2026-08-15 21:58:46 -05:00
Neuron	6a6b589ba0	bench: real black_box barrier + three-signal growth-curve gate ... Adds el_black_box (inline asm, +r constraint, memory clobber) and runtime/elbench.el: a growth-curve classifier that gates time AND allocation-count AND allocation-bytes, failing if any exceeds its declared curve. Refusal is a first-class verdict. The classifier REFUSES rather than classifying when the largest measurement is below the floor, or when a series is hard-flat across an 8x input range -- the shape produced when the optimiser deletes the work. Reporting O(1) there would be a confident answer with nothing behind it. Disagreeing ratios report INDETERMINATE rather than a guess. Deviation from DESIGN.md 6.2, stated in the source: uses consecutive ratios on a mandated geometric sweep rather than least-squares over candidate curves. Ratios are directly interpretable on a doubling sweep and need no floating point; the cost is weaker O(n) vs O(n log n) separation, reported as an ambiguous band rather than guessed. Documents the counter scope limit: engram_*.c and libcurl malloc are NOT tracked, so a flat curve over engram/HTTP-dominated work is not evidence of anything. 13 tests prove the classifier against real measured series from fitprobe.el -- including that an accumulator's allocation COUNT is linear while its bytes are quadratic, and that el #132's pure-CPU shape reads FLAT on both allocation signals and is caught only by time.	2026-08-15 21:45:13 -05:00
bigmerge	a8908908df	runtime: count container allocations too, not just strings ... el #131 instrumented the four string allocators, which meant list- and map-heavy code reported ZERO allocations — a benchmark over lists would have been fitted against a flat line and passed anything. Caught during framework work: a "linear" specimen read 0 allocs until it was rewritten to allocate strings. A gate is only as good as its blind spots are small, and a signal that silently reads zero is worse than no signal: it produces a confident pass. Now counted at every container allocation — ElList and ElMap bodies, their backing arrays, the copy-on-write clones, and the realloc growth path. Verified on an append loop (n = 100..800): allocs 7, 8, 9, 10 +1 per doubling = O(log n) reallocations bytes 2048, 4096, 8192, 16384 exactly 2x per doubling = O(n) Both curves are what correct amortized growth should look like, and both read zero before this change. Known remaining scope, stated rather than left implicit: these counters cover the runtime's own allocations. They do not see malloc inside engram_*.c or libcurl, which is correct — the gate is for El-level complexity, not for third-party memory behaviour.	2026-08-15 21:36:52 -05:00
bigmerge	edafd8cce8	runtime: math_log is base-10, not natural log ... el_val_t math_log(el_val_t f) { return el_from_float(log(el_to_float(f))); } el_val_t math_ln(el_val_t f) { return el_from_float(log(el_to_float(f))); } Both were natural log, so math_log and math_ln were the same function. log10(100) returned 4.605 instead of 2. Three sources already agreed it should be base-10 and were being contradicted by this one line: - runtime/math.el:55 "// math_log — base-10 logarithm." - el_seed.c:1278 __log_f -> log10() (the path math.el actually calls) - tests/native/test_math.el:133 asserts log10(100) == 2 FOUND BY THE NEW TEST FRAMEWORK ON ITS FIRST RUN (el #133). The assertion had been sitting in the suite the whole time; nothing could report it. The old harness printed "N passed, M failed" with no per-test detail, and half the suites were not compiling at all — so a failing assertion in a suite nobody could run was indistinguishable from no failure. That is the entire argument for the framework, demonstrated on day one: this is not a bug the framework introduced, it is a bug the framework made VISIBLE. Verified: tests/native/test_math.el goes 12/13 -> 13/13, math-log passing.	2026-08-15 21:33:20 -05:00
will.anderson	5e3e69d326	Merge pull request 'test framework phase 1: compile-time registry + El-side runner with per-test timing' (#133) from wt/soul-runtime-reconcile into dev	2026-08-16 02:31:08 +00:00
Neuron	3e7ab07e82	test framework phase 1: forward decls, void-return fix, suite migration ... Completes the Phase 1 runner and migrates the 11 test files onto it. - forward-declare the registry accessors in the test preamble; they are defined at the end of the unit but the El runner is compiled in between - eltest.el: explicit trailing return in the void emit_* helpers, which otherwise lower to 'return println(...)' and fail to compile - test files import runtime/eltest.el explicitly, using the language's own textual import mechanism rather than compiler-side auto-injection - DESIGN.md 6.5: gate on allocation COUNT AND BYTES, not count alone Verified: self-hosting fixpoint byte-identical (gen2 == gen3). 6 of 11 suites run and report per-test timing. The other 5 fail to COMPILE, and fail identically under the committed compiler -- pre-existing breakage this framework makes visible for the first time.	2026-08-15 21:28:30 -05:00
bigmerge	d231b7e5e7	compiler: fix the quadratic — strlen() on every character access ... THE BUG. str_char_code() and str_slice() each called strlen() on every invocation. The lexer walks source one character at a time, so every character access rescanned the whole remaining input: O(n) per character over n characters = O(n^2). el_val_t str_char_code(el_val_t s, el_val_t i) { ... int64_t n = (int64_t)strlen(str); // <- O(n), every call if (idx < 0 || idx >= n) return 0; return str[idx]; } HOW IT WAS FOUND. Not by reading code — by sampling the running process, which is the same method that resolved tonight's engram outage after four wrong theories. A geometric sweep of synthetic sources showed wall-clock rising 3.0x, 3.0x, 4.0x, 4.14x per doubling (converging on 4x = quadratic), and a stack sample put 779 of 779 samples inside lex(), every one bottoming out in _platform_strlen via str_char_code and str_slice. THE FIX. Remember the length instead of recomputing it. The subtlety is INVALIDATION: El strings are arena-allocated, so a freed pointer can be reused for a different string at the same address, and a naive pointer-keyed cache would hand back a stale length and read past the end of the new string — trading a performance bug for a memory-safety one. So entries carry a generation, a hit requires pointer AND generation to match, and every path that frees or mutates a runtime string bumps the generation: el_arena_pop, seed_request_end, __str_set_char. Stale entries cannot be believed; they miss and recompute. MEASURED, same host, same inputs: n(fns) before after 512 0.10s 0.01s 1024 0.37s 0.02s 2048 1.51s 0.03s 50x the compiler's own 422 KB source concatenated (DESIGN.md's 3.58s case): 3.55s -> 0.03s 118x The speedup GROWS with input size, which is the signature of removing a complexity class rather than a constant factor. After the fix each doubling adds ~0.01s: linear. CORRECTNESS, verified rather than assumed: - byte-identical output on every sweep input (n = 128..2048) - byte-identical output on the 422 KB compiler concatenation - byte-identical output on tests/runtime/string_test.el - self-hosting fixpoint byte-identical - new tests/runtime/str_cache_test.el: 17 assertions covering bounds, empty strings, negative indices, slice clamping, distinct strings not sharing a cached length, 1000 interleaved strings forcing cache-slot collisions, and a grown string not reporting its old length. All pass. This is the defect that made dist/soul.c a committed artifact: elc could not run in CI because it needed 24 GB+ and minutes. It needs neither now.	2026-08-15 21:28:23 -05:00
bigmerge	a668062e38	Merge remote-tracking branch 'origin/dev' into wt/soul-runtime-reconcile	2026-08-15 21:24:03 -05:00
Neuron	24fac765a6	test framework phase 1: compile-time registry + El-side runner ... Replace the hardcoded test harness main() with a generated static registry and index-based accessors, and move all reporting into runtime/eltest.el. The old harness inlined direct calls into main() and counted assertions in two globals. That shape cannot report which test failed, how long any test took, or whether a test ran at all -- a misspelled registration reported success for a test that never executed. - assertions record into per-test state instead of global counters - registry table emitted at compile time; discovery strictly precedes execution, which is what later enables --list, filtering and sharding - per-test wall timing on CLOCK_MONOTONIC, taken in C around the call - runner in El: structured NDJSON events as source of truth, human output rendered from the same fields	2026-08-15 21:24:03 -05:00
bigmerge	37bcf7eb74	runtime: allocation accounting — the deterministic signal for complexity gating ... Implements the three primitives the test-framework design (DESIGN.md §6.5) requires for gating on growth curves: el_alloc_count, el_alloc_bytes, el_peak_rss. Registered in codegen's builtin_arity and wrapped in el_seed.c per the project's C-builtin recipe. WHY COUNTS AND NOT WALL-CLOCK: a growth-curve gate has to be a hard build failure, which means the signal cannot flake. Wall-clock needs warmup, statistics, and a quiet machine; on shared CI it is unusable as a gate. Allocation counts are perfectly deterministic — same input, same number, every machine, every run. Fit them against n and a complexity regression becomes a build failure with zero noise. All four runtime string allocators (el_strdup, el_strbuf, and their _persist variants) funnel every allocation the language performs, so instrumenting there counts everything. WHY BYTES AS WELL AS COUNT — this is not redundancy, it is the whole gate. Measured with two El programs, one allocating once per item, one rebuilding its accumulator each iteration: n linear allocs / bytes quadratic allocs / bytes 100 100 / 290 100 / 5,150 200 200 / 690 200 / 20,300 400 400 / 1,490 400 / 80,600 800 800 / 3,090 800 / 321,200 The quadratic program's allocation COUNT is exactly linear — identical to the healthy one. Counting allocations alone would have missed it completely. Bytes catch it: each doubling of n quadruples bytes (ratios 3.94, 3.97, 3.99 -> converging on 4.0, i.e. O(n^2)), while the linear case converges on 2.0. That shape — count linear, per-allocation size growing — is the classic accidental quadratic, and it is exactly elc's defect: quadratic allocation VOLUME, which the old shipped compiler paid in RSS (27 GB, OOM) and the rebuilt one pays in malloc/free churn (42s on 1.4 MB). Volume was the invariant across both; RSS and wall-clock were just the two ways it surfaced. el_peak_rss is exported for context and is explicitly NOT a gating signal — it is perturbed by allocator internals, the page cache, and the OS. Gate on the deterministic numbers; report the physical one. Counters are unsynchronised by design: this is measurement, and a lock would change the thing being measured. Exact on the single-threaded compile path, approximate under threads.	2026-08-15 21:21:42 -05:00
bigmerge	19cc99e57d	store: judge memory pressure by swap RATE, not swap level ... The guard I added minutes ago checked swap availability as a level (avail < total/8 -> report zero available). That is the wrong signal, and the same host proved it twice within minutes: 47.65 / 48.00 GiB swap used, 2047 swapouts/s -> genuinely thrashing 26.67 / 28.00 GiB swap used, 0 swapouts/s -> healthy, 15.6 GiB free Both are ~97% "used". macOS grows swap files on demand and trims them lazily, so the level says almost nothing about now — it is a high-water mark. The level check calls the second state an emergency and starves the pool for no reason, which is its own failure mode: a guard that fires on healthy machines gets disabled, and then guards nothing. What separates the two is whether pages are moving. So sample the swapout counter across calls and judge the delta: - > 200 pages/s (~3 MiB/s) sustained outward paging => report zero available; callers refuse to grow and pc_relieve_pressure hands frames back. - The first call primes the baseline and reports no pressure. One sample cannot have a rate, and inferring one from a single reading is exactly the mistake this commit removes. Measured thresholds, not guessed: idle sat at 0/s, recovery burst hit 24,845/s while the compressor drained (transient, correctly not a growth decision since growth is only evaluated on eviction passes), and real thrash held ~2000/s. 200/s sits clearly above noise and far below either. The compressor-footprint subtraction stays: that RAM is genuinely spoken for regardless of paging rate.	2026-08-15 21:12:00 -05:00
bigmerge	e52415f0e0	store: bound the pool by AVAILABLE memory and let it shrink ... The adaptive budget I added an hour ago could only grow, and grew toward a share of TOTAL ram (80%, ~38 GiB on a 48 GB host). That is a memory leak with extra steps: total never shrinks when other processes need memory, so the pool had no way to notice it was starving the machine it runs on. Deployed briefly; caught as memory pressure on the host. A control loop with only one direction is not a control loop. - pc_available_ram(): free + inactive + purgeable via host_statistics64 on Darwin, MemAvailable on Linux. Availability is the quantity that moves when the machine is under pressure; total is not. Returns 0 when it cannot be read, and callers then refuse to grow — a cache is never worth swapping the host, so unknown means no. - Growth is bounded by availability minus a free-memory floor (2 GiB default, ENGRAM_POOL_FREE_FLOOR_MB), not by total. The share-of-total ceiling stays as a second bound and drops 80% -> 50%. - pc_relieve_pressure(): the missing direction. On every eviction pass, if available memory is under the floor, hand back ~25% of held frames; the resident set follows on the next pass so the memory is actually returned rather than merely re-labelled. Counted as adapt_shrinks alongside adapt_grows so both directions are visible in the same report. - pc_default_cap() also clamps the STARTING budget to what is spare right now, so a cold boot on a loaded machine does not open at a size the host cannot afford. Verified on a 48 GB host: engram boots in ~30s, RSS settles at 2.22 GiB (the store's actual size, resident, not creeping), 0.0% CPU, 13,439 nodes / 37,670 edges, embeddings complete. Guard reports 9.71 GiB available against a 2.00 GiB floor — 7.71 GiB of headroom it is permitted to use and no more.	2026-08-15 21:02:05 -05:00
bigmerge	e917b3d439	store: make the buffer pool sense its own state and correct from it ... Follow-on to the edge write barrier. That fix removed the full-store walk; this one makes the pool able to notice if anything like it happens again. WHAT WENT WRONG, precisely: the pool thrashed the live engram to a standstill twice on 2026-08-15 and said nothing. From outside it was indistinguishable from "busy loading" — 100% CPU, flat RSS, no output — so four wrong theories got tried (bad binary, corrupt snapshot, WAL replay, feature flags), each costing a deploy or a rollback. The whole time, hits/misses/evictions were already being counted in PgCache, and the struct comment read: /* stats (introspection only — never affect semantics) */ That comment was the bug. Self-measurement treated as decoration is why the pool could not correct itself and why no one outside could see what it was doing. A system that cannot read its own state cannot correct, and neither can anyone watching it. - pc_adapt_budget(): the loop, closed. Over a sliding window, evictions running at a large fraction of accesses WHILE reuse is real means the working set exceeds the budget — so grow it, geometrically, bounded by a LIVE re-read of physical memory. Evictions alone are not pressure (a scan evicts and never returns); evictions with reuse are. An explicit ENGRAM_POOL_FRAMES still wins — an operator override must not be silently overruled. - Budget derived, not declared. A constant cannot be right: 16 GiB of frames is arbitrary on a 48 GB host and suicidal on a 16 GB one. Even "60% of RAM at startup" is a guess about the future — it cannot know the store grew or the machine changed. Hence the live re-read. - pc_report(): ONE structured emission carrying the entire sensed state, through emit_log — El's existing telemetry, already exporting to OTLP. Deliberately not a function per stat, and deliberately not a bespoke /api/pool endpoint: both make observability something hand-written per noun instead of the uniform mechanism every component already has. - engram_pool_stats_json(): the same state readable live, wired through the normal builtin path (codegen arity + el_seed wrapper), so the pool can be observed in real time rather than reconstructed afterward from a stack sample. Verified: with the exact configuration that took production down (ENGRAM_POOL_FRAMES=65536 → 1 GiB cache against a 2 GiB store) the engram boots clean and serves — 0.0% CPU, 13,436 nodes / 37,663 edges, embeddings complete — and NO pressure event fires, because the barrier removed the walk that caused it. The controller is defense in depth; the barrier is the fix.	2026-08-15 20:44:23 -05:00
bigmerge	777ccc02f0	store: extend the durable-hash write barrier to edges (kills the full-store walk) ... Checkpointing pushes the ENTIRE resident graph through store_put_node and store_put_edge (see engram_store_checkpoint). Nodes were cheap: a durable-hash compare skipped unchanged records with zero page I/O. Edges had no barrier at all — struct comment at PgCache.barrier_on even says "node durable-hash barrier" — so every edge was rewritten on every checkpoint, and each rewrite runs the idempotency probe max_page_lsn_for_id -> btree lookup -> page_read. Edges outnumber nodes ~3:1 here (37,663 vs 13,436), so routine checkpointing degenerated into a FULL-STORE WALK in id order: random page access across the whole 2 GiB store, repeated, overwhelmingly to rediscover nothing had changed. LRU is worst-case under exactly that pattern — it evicts the page it is about to want — so once the page cache was smaller than the store, the walk collapsed into thrashing: 100% CPU, flat RSS, no forward progress, port never bound. That took the live engram down twice on 2026-08-15. The walk is the defect. Sizing the cache to survive it treats the symptom. Changes: - dh_edge_hash(): edge counterpart of dh_node_hash, with a kind discriminator byte so an edge can never collide with a node of the same id in the shared map. created_at/updated_at/last_fired are excluded deliberately: last_fired is touched by activation without changing what the edge IS, and folding it in would defeat the barrier on precisely the hot edges that most need it. - store_put_edge(): barrier check + dh_set on success, mirroring store_put_node exactly. - store_scan_edges(): seed the barrier map from on-disk truth at load, so the FIRST post-boot checkpoint already skips unchanged edges. store_scan_nodes already did this and its comment says why; edges were simply never done. Verified: with the exact configuration that killed production (ENGRAM_POOL_FRAMES=65536 -> 1 GiB cache against a 2 GiB store), the engram now boots clean and serves — LISTENING, 13,436 nodes / 37,663 edges, embeddings complete, 0.0% CPU, RSS 1.14 GiB (cache resting at its budget rather than thrashing against it). Same small cache, same store, no walk.	2026-08-15 20:38:11 -05:00
bigmerge	4e24d7d3f1	runtime: engram_edges_json — read edges without a whole-graph file round trip ... /api/graph/edges answered a read query by calling engram_save() to serialize the ENTIRE graph to disk (128 MB) and then fs_read-ing it back. Two defects in one line, and both bit production on 2026-08-15: 1. The path it wrote was ~/.neuron/engram/snapshot.json — the engram server's CANONICAL store. A READ route overwriting the persistence owner's canonical file. This defect had been fixed once (export moved to a scratch path); it came back when the hand-written dispatch block was replaced by @route dispatch and the unfixed copy is the one that survived the merge. 2. Cost: a full snapshot write, a 128 MB read, and a parse of the whole graph, per request, to return a bounded slice. Calling it tonight overwrote the canonical snapshot and immediately preceded an engram crash loop. engram_edges_json(limit, offset) is the builtin that route's own TODO asked for ("Future: add an engram_edges_json() builtin and drop the file round trip entirely"). It walks g->edges directly and emits every persisted field. limit <= 0 defaults to 1000, not unbounded: this is the endpoint that fell over, and an unbounded default would preserve the failure mode under a new name. Callers page explicitly. Registered in codegen.el's builtin_arity (both plain and __ spellings) and wrapped in el_seed.c per the project's C-builtin recipe.	2026-08-15 20:10:48 -05:00
bigmerge	7351fb0a8d	runtime: restore engram_recall_json + cgi_* accessors ... neuron's soul calls engram_recall_json (neuron-api.el:618, memory.el:80) and cgi_principal (studio.el:72). Both existed in the runtime neuron vendored (v1.0.0-20260501) and were absent here, so the soul could not link against current el at all. The dangerous part is what the obvious "fix" would have done. These look like redundant wrappers over one impl: engram_search_json(q, limit) -> eg_search_json_impl(q, limit, 0) LEXICAL engram_recall_json(q, limit) -> eg_search_json_impl(q, limit, 1) SEMANTIC They are not interchangeable, and the split is documented at neuron-api.el:613: search stays LEXICAL because ~40 internal call sites pass a KEY and seven of them DELETE every record returned. Point those at a semantic matcher and they delete fuzzy matches. Conversely, pointing recall at search silently downgrades the mind's entire retrieval surface from semantic to lexical — no error, just permanently worse recall. Implemented over engram_activate(), which in this runtime already IS the semantic path the old with_legs=1 branch built by hand (embeds the query via eg_embed_fetch, scores by cosine, then spreads activation one hop). Output shape matches engram_search_json — a flat array via engram_emit_node_json — because callers parse search's shape, not activate's envelope. Verified: neuron's soul now compiles and links against current el, boots, and serves /health with layers initialized. NOTE for follow-up: current el also ships engram_retrieve_geometric_json, a structure-first retrieval that appears to be the intended successor to recall. Repointing the two recall call sites at it may well be the right end state and would remove the two-wrapper shape entirely — but that is a behavioral change that must be measured against neuron/tools/retrieval-eval/'s gold set, not assumed. This commit preserves existing behavior exactly; it does not decide that question.	2026-08-15 19:56:43 -05:00
bigmerge	598915cc61	runtime: restore the three builtins that made elc unrebuildable ... The committed elc binary could not be refreshed from its own source. Rebuilding failed with three implicit-declaration errors: el_mem_check, stdout_to_file, stdout_restore. The compiler's own source calls all three (compiler.el:472,479,574 and codegen.el:4248) and two are registered in codegen.el's builtin_arity table — but none were defined in this runtime. They were found intact in ui/examples/native-hello-ios/NativeHello/el_runtime.c, a divergent private copy of this runtime that still carried them. Ported verbatim. Consequence of them being missing: the canonical elc binary was frozen. Source gained @route dispatch codegen (emit_route_dispatch, codegen.el:3948) and the @manager boundary-beat seam, but no rebuilt binary could carry them, so neuron's soul — whose routes.el now calls the compiler-synthesized el_route_dispatch — could not be built at all. Verified after the fix: - elc rebuilds from current source, clean. - Self-hosting fixpoint byte-identical (stage3 == stage2). - The rebuilt elc emits el_route_dispatch (2 occurrences in the soul amalgam, previously 0) and injects engram_boundary_beat at @manager boundaries, i.e. the decorator seam is live rather than inert. el_mem_check is itself the compiler's memory guard (ELC_MAX_MEM_MB, default 512MB, self-terminates before the OS OOM-killer fires) — so the runtime was missing the very guard that would have surfaced the compiler's memory blowup as a clean error instead of a 27GB host-killer.	2026-08-15 19:50:17 -05:00
bigmerge	40eb48e92f	engram: fix silently-wrong query params, and make el_seed.o + el_runtime.o link ... Three real bugs, all found by actually running the thing rather than reading it. 1. query_param never URL-decoded. A GET of /api/search?q=neural%20network searched for the literal string "neural%20network" and returned []. Every multi-word search against the live engram has been silently returning empty results — not an error, an empty result, which is why it went unnoticed. Affects every GET route that reads query params, not just search. 2. query_param matched key names unanchored. str_index_of(qs, "q=") matches inside "faq=", so "?faq=X&q=Y" returned X for key "q". Verified live before the fix. Now searches for "&key=" against "&"+querystring so a match can only land on a real parameter boundary. 3. el_request_start/el_request_end were defined in BOTH el_seed.c and el_runtime.c, so linking the two objects together — which is exactly what the product build does — failed with duplicate symbols. el_seed.c's own comment already says these moved there ("formerly defined in el_runtime.c. Now self-contained in el_seed.c"); the el_runtime.c copies were left behind during that move. Removed them, kept declarations since http_worker calls them. Also added the three missing prototypes (engram_op_assert_json, engram_node_full_in, engram_connect_in) that el_seed.c wraps but never declared, which made it fail to compile standalone under C99+. Verified: engram builds and links clean from canonical source; before/after comparison on a copy of the real store shows "neural network" returning a real match where the live build returns [], and "?faq=WRONG&q=MetaColloc" now resolving to MetaColloc. Live engram on :8742 was never touched.	2026-08-15 19:34:04 -05:00
bigmerge	09dade0613	Merge remote-tracking branch 'origin/pr/103' into HEAD ... # Conflicts: # lang/AGENTS.md # lang/runtime/el_runtime.h	2026-08-15 18:26:57 -05:00
will.anderson	38a8e32d6c	Merge pull request 'engram: batch-cosine Adapter/Strategy/Factory over ggml (supersedes #114)' (#116) from feat/engram-ggml-cosine-batch into dev	2026-08-15 23:22:56 +00:00
bigmerge	9883aa7564	Merge remote-tracking branch 'origin/wt/swarm-ccr' into merge-swarm-ccr-v2 ... # Conflicts: # lang/runtime/el_runtime.c # lang/runtime/el_runtime.h	2026-08-15 18:17:33 -05:00
bigmerge	c008b7228a	engram: make the ggml batch-cosine strategy actually compute in fp32 ... #116 shipped the ggml strategy at 0.9933 id-recall against the CPU oracle while the hand-rolled Metal kernel it replaced scored 0.9997 — a ~150x worse error margin. That was not an inherent property of ggml. It was a usage bug in this file, and this commit fixes it. ggml-metal has two F32xF32 matmul kernels and picks between them purely on ne11, the number of B rows, which for us is the query-batch size: ne11 <= 8 -> kernel_mul_mv_ext_f32_f32_* / kernel_mul_mv_f32_f32_*, templated <float, float> — genuine F32. ne11 > 8 -> kernel_mul_mm_f32_f32, templated <half, half4x4, simdgroup_half8x8, half, half2x4, ...> — BOTH operands narrowed to F16, despite F32 tensors on both sides. The old code issued one ggml_mul_mat with ne11 = nq (300 in the benchmark), landing squarely on the F16 path. The file's own header comment asserted the opposite ("computes in F32 on the Metal backend"); that claim was wrong and is replaced with the measurement. Fix: emit ceil(nq/8) mul_mats over ne11<=8 ggml_view_2d slices of one query tensor, all expanded into ONE graph and one ggml_backend_graph_compute, so the node matrix is still uploaded and shared exactly once. EL_GGML_MULMAT_CHUNK overrides the 8; setting it >= nq reproduces the old behaviour exactly, which is also how the before/after below was measured in a single binary. Measured, real store snapshot, 13415 live embedded nodes, dim=768, 300 real queries, vs the CPU double-accumulated oracle (vindex_bench, offline copy of the store — no live service touched): id-recall same-rank |Δdist| max mean old (ne11=300) 0.9933 6.80e-05 1.43e-05 new (ne11<=8) 0.9987 4.77e-07 9.30e-08 hand-rolled 0.9997 3.58e-07 7.55e-08 ~145x better max error, ~154x better mean — now the same order of magnitude as the hand-rolled kernel rather than 150x off it. The cost is real and is documented rather than buried. Median of 15 reps of the whole batch_multi() call, three runs: 13.2-14.4ms unchunked, 19.9-20.2ms chunked, 17.7-18.0ms hand-rolled. Correctness costs ~+6.7ms per 300-query batch and leaves ggml ~12% behind the hand-rolled kernel instead of ~35% ahead. It cannot be recovered inside ggml: an fp32 matmul on Metal must re-stream the node matrix once per <=8 queries, and ggml's Metal backend ships no fp32 TILED matmul, so "fast" and "fp32" are genuinely exclusive there. Two things that did NOT work, recorded so nobody retries them: - ggml_mul_mat_set_prec(t, GGML_PREC_F32) does nothing here. Error was bit-identical with and without it (1.038e-05 either way) — ggml-metal has no F32-accumulating mul_mm kernel to switch to. ne11 is the only lever. - The ACCEL/BLAS device looked excellent in an isolated compute-only probe (3.4-4.0ms, mean |Δdot| 1.5e-08) but is dominated on BOTH axes end-to-end (0.191 ms/query at 0.9973 recall vs 0.125-0.142 at 0.9987), because the probe was not competing for the same CPU cores the real call path is. It stays reachable via EL_GGML_DEVICE as a no-Metal fallback, labelled as measured-and-rejected, not as a recommendation. Also corrected: the ~7.8s "cold start" blamed on this file is not this file re-initialising per call — init was already cached. It is Apple's shader cache missing on ggml's embedded metallib (~650 kernels), keyed on the library and shared across processes: the first load on a machine reports "loaded in 7.670 sec", the next run of a *different* binary reports 0.009 sec. Once per machine per ggml version, not once per process, and not ours to fix. Warm ggml init is 44-53ms vs 36-117ms for the hand-rolled strategy. Loading only libggml-metal.so instead of every plugin in the directory is kept for tidiness, and explicitly documented as NOT a speedup: 44.7-52.4ms against 46.9-58.9ms, the same number inside noise. The -2.0 sentinel contract is unchanged and re-verified at batch sizes that straddle the chunk boundary (1,7,8,9,16,17,33), plus NULL rows, dim mismatches, zero-norm rows, and an all-invalid population. Notably the old ne11=300 path fails that same check at a 2e-6 cosine tolerance with 2299 mismatches, which is an independent confirmation of the defect.	2026-08-15 17:57:09 -05:00
bigmerge	3718bf0380	runtime: port missing __channel_* primitives into el_seed.c ... runtime/channel.el has always called __channel_new/__channel_send/ __channel_recv/__channel_try_recv/__channel_close, but these were only ever implemented in the pre-restructure lang/el-compiler/runtime/el_runtime.c. When the canonical runtime was consolidated onto the release copy (lang/runtime/el_runtime.c) and el_seed.c became the sole C dependency, the channel implementation was never carried forward — __mutex_new made the move, __channel_* did not. Any El program using Go-style channels currently fails to link on dev. Ported the working buffered-MPMC-channel implementation (mutex+condvar+ circular buffer, bounded and unbounded modes) from the old el_runtime.c verbatim, adapted only to el_seed.c's arena API (seed_arena_track in place of el_arena_track). Declared in el_seed.h alongside the existing mutex primitives.	2026-08-15 17:50:04 -05:00
bigmerge	9d40f87926	ingest: unify transduce_prose/transduce_structured into one transduce() ... transduce() is now THE single mechanism: one function, no content-type branch inside it. It never asks whether `source` is prose, JSON, or raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally: split on "\n\n" as a universal boundary-marker check, and if that finds no boundary, fall back to fixed 4096-char windows. Same node/edge wiring (root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets a heading/section_of link) regardless of what's inside a chunk. Dedup is the existing find_existing_by_content path via merge_manifold, applied uniformly. The old transduce_structured JSON dataset/records/feature-node interpretation is deleted outright, not just unused — a JSON file now gets chunked and deduped like anything else, with no pre-computed structure. All five ingest_* entry points still exist unchanged in name and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one transduce() (ingest_stream builds its own turn-nodes directly and never called either old function, so it's untouched). This unlocks raw/opaque content (audio, or anything else with no natural text/JSON shape) without any DSP, LLM call, or external API: transduce() chunks it exactly like it chunks anything else. There is zero semantic understanding of audio (or any payload) claimed or built here — any meaning is expected to emerge later from Neuron's own existing mechanisms (embedding, spreading activation, dedup) acting on this real geometry over time. Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk) because El strings are NUL-unsafe under strlen-based ops and fs_read()'s result silently truncates at the first embedded NUL, which is routine in real binary/audio bytes. ingest_file compares fs_read()'s string length against a real fs_size() stat() count; on mismatch it rebuilds the payload as base64-encoded fixed 3072-byte windows read directly off disk (binary-safe in C, verbatim, no invention), joined with the same "\n\n" marker transduce()'s boundary scan already looks for. This is a mechanical fidelity fix, not interpretation of content — transduce() never learns a fallback happened. Registered both builtins' arity in codegen.el; did not rebuild the elc compiler binary itself (unrelated, pre-existing gap: self-hosting elc via el_seed.c fails on this worktree independent of this change, reproduced with codegen.el reverted) — the existing elc binary compiles calls to unregistered builtins via its already-existing arity=-1 passthrough, confirmed by an actual clean `elc ingest.el` + `cc` build against the modified el_runtime.c. INGEST_KIND keeps existing only as an acquisition-mechanism selector (dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a content-type flag; the redundant "structured" value (an alias for "file" that hinted the now-deleted JSON branch) is removed. ingest_dir drops its file-extension filter for the same reason: transduce() takes anything now. Verification: local manifold construction confirmed correct against a real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte real prefix slice) — exact expected node/edge counts both times (101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64 content confirmed decoding back to the actual WAV header bytes. Compiles clean via the real elc + the modified el_runtime.c/engram_*.c (built and booted an actual sandbox engram off this exact source with `nsbx create --branch`). NOT verified this session, disclosed rather than papered over: end-to-end server-confirmed persistence (a real before/after /api/stats delta, and a fetched node by id) for the audio, prose, and JSON-fixture cases. Every local nsbx sandbox engram tried tonight (two stock pre-#109 binaries hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW binary built from current dev) took minutes-to indefinitely long on the final /api/load-merge write's embedding step and hit the client's 60s HTTP timeout before responding, even for a 5-node write. This is confirmed as real (if slow) forward progress, not a hang: the sandbox's WAL file was observed growing steadily across every attempt. The code's own pre-existing HONESTY GATE correctly refused to report success in every case, returning "load-merge failed: ..." with a "nothing below this manifold was confirmed persisted by the server" note instead — exactly as designed. This is an environment/infrastructure limitation, not a defect introduced by this change: the engram server binary itself is untouched by this commit.	2026-08-15 17:50:04 -05:00
bigmerge	90d3f0bc76	engram: port PR #105's 3 genuine wins onto dev's existing cosq/e_eff semantic layer ... Reconciles PR #105 ("fix: engram search latency — pin embed model, cache query embeddings, bound activate BFS") with dev's ACTUAL current engram_activate, rather than the ancient pre-restructure snapshot #105 was built against. WHY THIS NEEDED RECONCILIATION, NOT A DIRECT PORT: #105's single commit (1dc49b1) modifies `lang/el-compiler/runtime/el_runtime.c` — a path that does not exist on dev (dev has `lang/runtime/el_runtime.c`; the restructure that renamed it happened after #105's branch point, which traces to a July 22 merge-base, weeks before the M8/M8.1/qgate/fan-effect/adjacency-index work this file has grown since). #105's own engram_activate is consequently the PRE-restructure version: no adjacency index (O(E) full edge scan per hop), no query-aware qgate, no ACT-R fan effect, no eg_edge_eff_weight, and no awareness of dev's cosq/e_eff embedding-blend semantic layer — it built a parallel `g_qcache`/`engram_embed_raw` mechanism from scratch against code that no longer exists at that path. A raw merge/cherry-pick was not possible and would have been wrong even if it were: taking #105's tree wholesale would have thrown away everything dev grew in the meantime (qgate, fan effect, adjacency index, and this session's own M8 HNSW vindex integration). RECONCILIATION: kept dev's cosq/e_eff mechanism as the semantic layer entirely intact (unchanged by this commit) and ported #105's three genuinely additive wins on TOP of it, at their equivalent sites in the CURRENT eg_embed_fetch/engram_activate: 1. keep_alive:-1 on the Ollama embed request body (eg_embed_fetch) — pins the embed model resident so a larger generation model loading under unified-memory pressure can't evict it and force a cold reload on the next search (#105 measured ~2.2s cold vs ~0.02-0.05s warm). 2. Query-embedding cache upgraded from dev's single-slot (`_eg_qcache_text`, only ever remembered the LAST query) to a direct-mapped, FNV-1a-keyed, 1024-slot cache (reusing the existing engram_id_hash) — so the curiosity loop's rotating phrases actually hit the cache instead of evicting each other every call. Same "pointer owned by the cache, not freed by caller" contract as before, just per-slot instead of global. 3. Beam cap on the layer-1 spreading-activation BFS (new engram_activate_beam(), tunable via ENGRAM_ACTIVATE_BEAM, default 128). The FIFO frontier is processed in hop-level batches (entries sharing .hops are provably contiguous — see the code comment); when a level exceeds the beam width, only the top-`beam` by activation actually EXPAND. Every node in an oversized level still gets reached[]/best_bg[] recorded (that happens at enqueue time, one level up) and appears in the reported/promoted set — the cap bounds associative SPREAD width only, never recall of what was already found. Kept as a genuine additional bound even though the adjacency index + qgate + fan effect already mitigate #105's original "hub-node explosion" failure mode for a different reason: those prune WHICH targets matter; this bounds worst-case width regardless. Everything else in dev's engram_activate — cosq/e_eff, the qgate rescale, the fan effect, eg_edge_eff_weight, the M8 HNSW vindex seed discovery from the #109 reconciliation earlier this session — is untouched. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): cc -std=c11 -O2 clean build; booted in an isolated sandbox against a real cloned production snapshot (13,424 nodes / 37,656 edges); ran 5 activate() calls across rotating queries at depth 3, including the same query issued twice non-consecutively (2nd hit landed at 476ms vs the 1st at 483ms — consistent with a cache hit once Ollama's own warm-model latency is accounted for; no crash, correct varied result counts (367-2610 nodes) each call; act-stats JSON read correctly throughout. Built on top of the M8/#109 reconciliation (bacaf3d, merged to dev as #109) — dev's current HEAD at the time of this commit.	2026-08-15 17:50:04 -05:00
bigmerge	2d0aef4ef8	lang: declare the el_runtime.c symbols el_seed.c's wrappers call ... runtime/el_seed.c does not compile standalone via the exact command tools/install.sh uses (`cc -std=c11 -O2 -I runtime -c runtime/el_seed.c`): 51 of its __-prefixed wrapper functions (http serving, JSON access, key-val state, URL/HTML escaping, and the whole engram_* node/edge/layer/search surface) call unprefixed counterparts that are implemented in el_runtime.c, not in el_seed.c itself, and el_seed.c never declared them -- a toolchain that treats an implicit function declaration as a hard error under C11 fails the compile outright. install.sh already compiles el_seed.c and el_runtime.c as separate objects and archives both into libel.a, so the symbols are always present at link time; el_seed.c alone was just missing the prototypes. A plain `#include "el_runtime.h"` was tried first and rejected: it redefines el_to_float/el_from_float, which el_seed.h already provides -- a real compile error, not a style preference. Added narrow prototypes instead, copied verbatim from el_runtime.h, for exactly the 51 symbols el_seed.c's wrappers reference and nothing else. Verified clean: - `cc -std=c11 -O2 -I runtime -c runtime/el_seed.c` (install.sh's exact per-file compile) -- 0 errors, 0 warnings, even with -ferror-limit=0. - full `tools/install.sh` run -- compiles both objects and archives them into libel.a successfully. Separately (not fixed here, out of scope): AGENTS.md's documented compiler self-rebuild command links elc-new.c against el_seed.c, but elc-new.c's own generated `#include "el_runtime.h"` line and 3 undeclared symbols (el_mem_check, stdout_to_file, stdout_restore -- present in neither el_runtime.c nor el_seed.c) mean that command fails regardless of which runtime file it's linked against; and install.sh's libel.a only archives el_seed.o + el_runtime.o, so any program that calls into the engram_* surface fails to link against it (el_runtime.c's engram_* wrappers need engram_store.c/engram_geometry.c/engram_reason.c/engram_cognition.c/ engram_vindex.c, none of which install.sh compiles in). Both are real, pre-existing, and independent of this fix -- worth their own look.	2026-08-15 17:32:58 -05:00
bigmerge	b3f410fc91	engram: batch-cosine Adapter/Strategy/Factory over ggml, supersedes hand-rolled PR #114 ... Stop hand-rolling GPU kernels for batch cosine similarity — use ggml (the MIT-licensed compute library underneath llama.cpp, installed standalone via Homebrew) as the preferred backend, without ripping out PR #114's carefully-verified hand-rolled Metal shader. Structure: one stable public adapter (eg_cosine_batch.h, zero #ifdef at call sites) backed by three selectable concrete Strategies behind an internal vtable (eg_cosine_batch_strategy.h) chosen by a Factory (eg_cosine_batch.c): - eg_cosine_batch_strategy_ggml.c — NEW. ggml + dynamically-loaded Metal backend plugin (ggml_backend_load_all_from_path + ggml_mul_mat for the batched dot product), gather/scatter around the -2.0 sentinel contract. - eg_cosine_batch_strategy_metal_hand.m — PR #114's original hand-rolled Metal shader bridge, preserved almost verbatim, now one strategy among several rather than the only option. eg_cosine_batch.metal kept byte-identical to the original. - eg_cosine_batch_strategy_cpu.c — universal always-false fallback (direct descendant of PR #114's eg_metal_cosine_stub.c). Selection: EL_COSINE_BATCH_STRATEGY=ggml|metal|cpu|auto (default: ggml first, then hand-rolled Metal, then CPU — first available wins), plus back-compat EL_METAL_COSINE=0 to disable every GPU-backed strategy. build_vindex_bench.sh compiles all three strategies on Darwin, CPU-fallback-only elsewhere. vindex_bench.c now reports BRUTE-GGML and BRUTE-METAL side by side against the same CPU oracle, on the same dataset, in one run (real numbers vs. real store snapshot in the PR body).	2026-08-15 17:16:41 -05:00
bigmerge	bacaf3d39c	engram: reconcile M8 HNSW vindex (#109) onto current dev, restore 3 fixes the branch predated ... Lands feat/reframe-region-setop (PR #109: native set-based reframe_region, decorator-as-seam @route port, teacher-summon, and the M8.1 activate-latency work — lazy-memoized cosq via eg_cosq_at + engram_vindex HNSW-accelerated seed discovery + vindex_harvest_from_store/vindex_bench oracle) onto dev's actual current HEAD, plus engram-tiered-storage's still-unique test suite. RECONCILING #109 WITH engram-tiered-storage (M4-M10 HNSW/geometry/reason/ verify work): not a two-way merge. engram_vindex.c's HNSW core (search_layer/ select_neighbors/prune_links/insert) is BYTE-IDENTICAL between the two branches; #109's copy is a strict superset (adds vindex_harvest_from_store, used by vindex_bench.c's brute-force-vs-HNSW oracle). engram_reason.c and engram_verify.c are also byte-identical. #109's own branch point already carried engram-tiered-storage's M4-M10 lineage forward, so there was nothing left to merge into #109 for those files. The one thing engram-tiered-storage had that #109's tree dropped: its full test suite (test_vindex.c, test_geometry.c, test_reason.c, test_verify.c, test_m7_traversal.c, the interoception P0-P5 tests, bufpool/compaction tests, and their run_*.sh harnesses) — ported over here unchanged. WHY THIS NEEDED HAND RECONCILIATION, NOT A MECHANICAL MERGE: #109's branch forked from dev on 2026-08-14 15:40 (before restructure-adjacent history diverged the file's merge-base for `git merge` — it presented as an add/add conflict). A straight two-dot diff (dev tip -> PR tip) applied cleanly, but it silently reverted THREE dev fixes landed on 2026-08-14/15, after the branch point, that the PR's diff had no way to know about: 1. qgate rescale (2026-08-14 self-review): PR's lazy eg_cosq_at rewrite of the query-aware propagation gate dropped the shift-and-floor rescale about ENGRAM_EMBED_S0 (measured: unrelated-pair median 0.562->raw gate 0.67, i.e. "a small tax, not a gate"). Restored the rescale, wrapped around the lazy accessor -- the PR's actual improvement (WHEN cosq[oi] is computed) is orthogonal to WHAT it gates on and both are kept. 2. Eviction cause decomposition (2026-08-14 self-review): dev decomposes wm_evicted into evict_floor/evict_cap/evict_bll so WM churn is diagnosable (identity: evicted == floor+cap+bll+dup_wm+dup_wm_global). PR's tree predates this and dropped all three counters + their JSON stats fields. Restored declarations, all 4 direct increment sites, the eg_wm_carry_over bll increment, and the act-stats JSON fields -- alongside (not instead of) the PR's own P4 afferent / API-reshape counters already in that same struct/JSON. 3. Hebbian link-formation selection (2026-08-15 self-review, TODAY): dev selects the STRONGEST qualifying candidate for consolidation each call; PR's tree predates this and reverted to hash-slot order (arbitrary wrt association strength) for edge formation -- the one path that writes PERMANENT structure. Restored the strongest-candidate while-loop, keeping the PR's own genuine improvement at that site (engram_adj_on_edge_added incremental-index append instead of a bare adj_dirty=1 full-rebuild flag). engram/src/server.el's 3-way conflicts (autoconnect_on/ise_offgraph_on env flags, /api/nodes connected-count in responses) were pure additive: dev's side was empty, PR's side added the feature. Took PR's side whole. VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): - cc -std=c11 -O2, clean link against the real engram/src/server.el via elc, zero errors. - vindex_bench (built standalone, read-only harvest) against the real production store clone (13,671 embedded nodes, 768-dim nomic-embed-text): recall@10 = 1.0000 at ef 64/128/200; HNSW search 0.28-0.79ms/query vs 2.03ms/query brute-force oracle (2.6x-7.2x). HNSW build itself: 46.5s for the full 13,671-node set -- see the flagged risk below. - Booted the reconciled binary in an isolated nsbx sandbox (:8905, cloned snapshot of the live store, 13,424 nodes / 37,656 edges) and called /api/activate for real: first call after boot 41.5s (pays the one-time HNSW build inline -- matches the standalone bench), second/third calls 356ms/605ms, no crash, correct results, act-stats JSON (including the restored evict_floor/cap/bll fields) reads correctly. KNOWN RISK TO FLAG BEFORE ANY LIVE CUTOVER (not fixed here; out of scope for this dev-only land per instructions not to touch :8742/:7770): eg_vindex_sync builds the HNSW index synchronously, inline, on the first engram_activate() call after every process start (or index invalidation). On the real node count that is a ~46s blocking stall on a single-threaded server -- the first request after every restart (or its concurrent siblings) waits the full build. Recommend a background/incremental build (or a bounded per-call build budget) before this ever reaches the live daemon. See PR description / final report for the fuller writeup.	2026-08-15 16:46:44 -05:00
will.anderson	e34ebd4b3d	Merge pull request 'nsbx + cognitive architecture design + engram self-review series' (#113) from feat/neuron-sandbox into dev	2026-08-15 19:59:07 +00:00
will.anderson	d4a04bb944	Merge pull request 'swarm: native interruptibility for dispatched agent workers' (#107) from worktree-agent-a6177cda24c71d1df into dev	2026-08-15 19:57:10 +00:00
will.anderson	09ae14a970	Merge pull request 'fix: float arithmetic codegen (segfault/garbage) and math_log aliasing' (#104) from worktree-agent-a456e0cf8cd2ee361 into dev	2026-08-15 19:54:46 +00:00
will.anderson	fdf0d6cb64	Merge pull request 'nsbx: one-command dev onboarding (branch + worktree + isolated engram)' (#99) from feat/nsbx-dev-env into dev	2026-08-15 19:52:35 +00:00
bigmerge	08cbcef5d9	engram: fix lazy-embed index gap (#20) and make activate's cosine scan lazy; extract vindex harvest primitive with a bench/oracle harness ... Adds an O(1) "seen" bitmap so lazily-embedded older nodes get picked up incrementally instead of only on a full rebuild (embed-gap #20). Replaces engram_activate's O(N*D) cosine prescan with a lazy-memoized cosine cache (eg_cosq_at), proven bit-identical to the old path. Extracts a clean vindex_harvest_from_store primitive (read-only vector harvest, careful malloc/ownership/error-path handling) reused by both index-build and the new vindex_bench.c — a read-only proof harness comparing brute-force vs HNSW recall/latency on both the real store and synthetic data. .nsbx-env intentionally excluded — local sandbox config (ports, paths, dev-only placeholder key), not checked in.	2026-08-15 14:26:16 -05:00
bigmerge	708722b7ff	Add native interruptibility for dispatched agent workers ... Cancellation-token control channel checked at every step boundary lets a coordinator PAUSE/RESUME/REDIRECT/KILL a running worker mid-task instead of waiting for the whole (possibly wrong) plan to finish. Bounded purviews mean no half-committed state to unwind on interrupt. Includes a proof harness (proof.el, run.sh) comparing a broken non-interruptible worker against the new one under identical kill/redirect/pause timing. Distinct from the already-preserved swarm-ccr orchestrator (fan-out/ converge dispatch): this is single-worker interruptibility, a complementary mechanism, not a duplicate.	2026-08-15 14:25:55 -05:00
bigmerge	01826421c4	seam: implement decorated-fn boundary auto-emit; prove on clone ... Will waived diff review -> build it for real. Add engram_boundary_beat() to the runtime (afferent counter++ + engram_chrono_tick + engram_strengthen(self-anchor) + dharma_emit) and two act-stats counters (aff_boundary_ops, dharma_emits). codegen cg_fn injects ONE engram_boundary_beat(op) at the entry of every @manager/@accessor fn (fn_has_decorator, so it fires under @route @manager too) — a decorated op self-reports with ZERO hand-written instrumentation. Rebuilt elc self-host + the cognition engram in the worktree; ran it as the clone daemon on :8900. Proof (/api/boundary-proof, @manager, empty body, 5x): aff_boundary_ops 0->5, dharma_emits 0->5, self activation_count 1510->1513, chrono stamp advanced. Brought in feat/cognitive-architecture engram runtime+server for the build. strengthen = activation bump (not content/edge write) -> identity protection intact. Live :8742 untouched; no push, no cutover.	2026-08-14 21:20:18 -05:00
will.anderson	7aa847e32a	Merge origin/dev into engram-tiered-storage ... Resolve 3 conflicts: - lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork; single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh). - lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated artifact folder, not a source path; a release is a git tag, not a folder). - lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL. Nothing in dev's build consumes the deprecated fork or releases/ folder.	2026-08-12 15:23:02 -05:00