nsbx: fail loud on daemon-not-ready + el_seed.c standalone compile #118

2026-08-15T22:33:23Z

will.anderson commented

2026-08-15 22:33:23 +00:00

nsbx tooling hardening

Three confirmed-live bugs from tonight's session, each verified with real before/after command output (not just code review):

False "ready" banner. nsbx up on an existing-but-dead sandbox printed the readiness-failure warning immediately followed by a green "your sandbox is ready" success banner and exited 0. Root cause: daemon_alive "$name" || start_daemon "$name" in cmd_up never checked start_daemon's return code. Same unguarded pattern existed in cmd_build, cmd_run, cmd_validate. All four now || die with a pointer to daemon.log.
Stale liveness reporting. nsbx status/nsbx list reported bare state: running for a process that's alive (kill -0 passes) but not actually answering /api/stats -- pegged, hung, or mid-boot. Added daemon_health(), which does the real stats fetch (short 2s timeout) and distinguishes stopped/running/unresponsive. Reproduced live against another agent's actively-running (CPU-pinned, 100% CPU, LISTENing but not responding) sandbox tonight, and again via a deliberate SIGSTOP on a throwaway sandbox of my own.
No visibility into stale binaries. status/list now show the binary's sha + real build timestamp (mtime survives cp -p, so it's the original build time even for stock-prod clones), plus a best-effort staleness note: for stock-prod clones, compare against the currently-configured live binary; for source/branch builds, compare the recorded source commit against local origin/dev via merge-base --is-ancestor.

Also found and fixed while verifying the above (live, not manufactured):

A cold boot under concurrent sandbox/CPU load can legitimately take past the old hardcoded 15s readiness window -- observed directly tonight with 2-3 sandboxes competing for CPU. Made it configurable (NSBX_READY_TIMEOUT_SECS) instead of blindly widening the default.
cmd_create's post-boot baseline capture could silently record sbx_baseline as 0/0 when the stats fetch came back empty right after the auto-remerge step -- which would make every future nsbx validate zero-loss/reboot-prove check trivially PASS regardless of real data loss. Added a bounded retry + loud warning.
Sharpened a handful of "no such sandbox" / missing-binary errors to name the next command.

lang: el_seed.c standalone compile

runtime/el_seed.c didn't compile via the exact command tools/install.sh uses. 51 of its __-prefixed wrappers (http serving, JSON, key-val state, URL/HTML escaping, the whole engram_* surface) call unprefixed counterparts implemented in el_runtime.c but never declared in el_seed.c. install.sh already compiles both files separately and archives them into libel.a, so the symbols are always present at link time -- el_seed.c was just missing the prototypes. A plain #include "el_runtime.h" was tried and rejected (redefines el_to_float/el_from_float, already provided by el_seed.h); added narrow prototypes for exactly the 51 symbols instead.

Verified: cc -std=c11 -O2 -I runtime -c runtime/el_seed.c (0 errors/warnings, -ferror-limit=0) and a full tools/install.sh run (libel.a built successfully).

Confirmed before editing: the AGENTS.md warning about a protected el-compiler/runtime/el_seed.c path describes a layout that doesn't exist in this worktree (lang/el-compiler/runtime/ is absent) -- the real, current file at lang/runtime/el_seed.c isn't what that warning protects.

Noted but explicitly NOT fixed here (separate, pre-existing, unrelated to this change): AGENTS.md's documented compiler self-rebuild command fails regardless of which runtime file it links against (elc-new.c references 3 symbols that exist in neither el_runtime.c nor el_seed.c), and install.sh's libel.a doesn't archive the engram_*.c engine files, so any program that calls into the engram_* surface fails to link against it. Both worth their own look.

Sandboxes touched

All verification used my own throwaway sandboxes (created and destroyed within this session). dev-api-reshape (stopped) and transduce-verify (running, another agent's active work) were left untouched throughout -- transduce-verify was torn down by its own owning session partway through mine, not by me.

## nsbx tooling hardening Three confirmed-live bugs from tonight's session, each verified with real before/after command output (not just code review): 1. **False "ready" banner.** `nsbx up` on an existing-but-dead sandbox printed the readiness-failure warning immediately followed by a green "your sandbox is ready" success banner and exited 0. Root cause: `daemon_alive "$name" || start_daemon "$name"` in `cmd_up` never checked `start_daemon`'s return code. Same unguarded pattern existed in `cmd_build`, `cmd_run`, `cmd_validate`. All four now `|| die` with a pointer to `daemon.log`. 2. **Stale liveness reporting.** `nsbx status`/`nsbx list` reported bare `state: running` for a process that's alive (`kill -0` passes) but not actually answering `/api/stats` -- pegged, hung, or mid-boot. Added `daemon_health()`, which does the real stats fetch (short 2s timeout) and distinguishes stopped/running/unresponsive. Reproduced live against another agent's actively-running (CPU-pinned, 100% CPU, LISTENing but not responding) sandbox tonight, and again via a deliberate `SIGSTOP` on a throwaway sandbox of my own. 3. **No visibility into stale binaries.** `status`/`list` now show the binary's sha + real build timestamp (mtime survives `cp -p`, so it's the original build time even for stock-prod clones), plus a best-effort staleness note: for stock-prod clones, compare against the currently-configured live binary; for source/branch builds, compare the recorded source commit against local `origin/dev` via `merge-base --is-ancestor`. Also found and fixed while verifying the above (live, not manufactured): - A cold boot under concurrent sandbox/CPU load can legitimately take past the old hardcoded 15s readiness window -- observed directly tonight with 2-3 sandboxes competing for CPU. Made it configurable (`NSBX_READY_TIMEOUT_SECS`) instead of blindly widening the default. - `cmd_create`'s post-boot baseline capture could silently record `sbx_baseline` as `0/0` when the stats fetch came back empty right after the auto-remerge step -- which would make every future `nsbx validate` zero-loss/reboot-prove check trivially PASS regardless of real data loss. Added a bounded retry + loud warning. - Sharpened a handful of "no such sandbox" / missing-binary errors to name the next command. ## lang: el_seed.c standalone compile `runtime/el_seed.c` didn't compile via the exact command `tools/install.sh` uses. 51 of its `__`-prefixed wrappers (http serving, JSON, key-val state, URL/HTML escaping, the whole `engram_*` surface) call unprefixed counterparts implemented in `el_runtime.c` but never declared in `el_seed.c`. `install.sh` already compiles both files separately and archives them into `libel.a`, so the symbols are always present at link time -- `el_seed.c` was just missing the prototypes. A plain `#include "el_runtime.h"` was tried and rejected (redefines `el_to_float`/`el_from_float`, already provided by `el_seed.h`); added narrow prototypes for exactly the 51 symbols instead. Verified: `cc -std=c11 -O2 -I runtime -c runtime/el_seed.c` (0 errors/warnings, `-ferror-limit=0`) and a full `tools/install.sh` run (`libel.a` built successfully). Confirmed before editing: the `AGENTS.md` warning about a protected `el-compiler/runtime/el_seed.c` path describes a layout that doesn't exist in this worktree (`lang/el-compiler/runtime/` is absent) -- the real, current file at `lang/runtime/el_seed.c` isn't what that warning protects. Noted but explicitly NOT fixed here (separate, pre-existing, unrelated to this change): `AGENTS.md`'s documented compiler self-rebuild command fails regardless of which runtime file it links against (elc-new.c references 3 symbols that exist in neither `el_runtime.c` nor `el_seed.c`), and `install.sh`'s `libel.a` doesn't archive the `engram_*.c` engine files, so any program that calls into the `engram_*` surface fails to link against it. Both worth their own look. ## Sandboxes touched All verification used my own throwaway sandboxes (created and destroyed within this session). `dev-api-reshape` (stopped) and `transduce-verify` (running, another agent's active work) were left untouched throughout -- `transduce-verify` was torn down by its own owning session partway through mine, not by me.

will.anderson added 2 commits 2026-08-15 22:33:24 +00:00

nsbx: fail loud on daemon-not-ready instead of printing a false success banner 979e820f68

Three confirmed-live bugs tonight:

- `nsbx up` printed "daemon did not become ready" immediately followed by a
  green "your sandbox is ready" banner and exited 0, because the existing-
  sandbox restart path (`daemon_alive || start_daemon`) never checked
  start_daemon's return code. `cmd_build` had the identical unguarded
  pattern, plus `cmd_run`/`cmd_validate`'s own start-if-dead calls. All four
  now `|| die` with a message pointing at daemon.log.

- `nsbx status`/`nsbx list` reported bare "state: running" for a process
  that's alive (passes kill -0) but not actually answering /api/stats --
  pegged, hung, or mid-boot. Added daemon_health(), which does the real
  stats fetch and distinguishes stopped/running/unresponsive; both commands
  now say "running but NOT RESPONDING" with a next-step hint instead of
  silently going quiet on the stats field. Reproduced live against another
  agent's actively-running (CPU-pinned, non-responsive) sandbox tonight, and
  again via a deliberate SIGSTOP on a throwaway sandbox.

- Sandboxes carried no visible signal that their binary predated a relevant
  fix. `status`/`list` now show the binary's sha + real build timestamp
  (mtime survives `cp -p`), plus a best-effort staleness note: for
  stock-prod clones, compare against the currently-configured live binary;
  for source/branch builds, compare the recorded source commit against
  local origin/dev via merge-base --is-ancestor.

Also, found live while verifying the above:

- A cold boot under concurrent sandbox/CPU load can legitimately take past
  the old hardcoded 15s readiness window. Made it configurable
  (NSBX_READY_TIMEOUT_SECS) rather than just widening the default blindly.

- cmd_create's post-boot baseline capture could silently record sbx_baseline
  as 0/0 when the stats fetch came back empty right after the auto-remerge
  step -- which would make every future `nsbx validate` zero-loss/reboot-
  prove check trivially PASS regardless of real data loss. Added a bounded
  retry and a loud warning if it still comes back empty.

- Sharpened a handful of "no such sandbox" / missing-binary errors to name
  the next command instead of just stating the failure.

lang: declare the el_runtime.c symbols el_seed.c's wrappers call

El SDK CI - dev / build-and-test (pull_request) Failing after 3m55s

Details

2d0aef4ef8

runtime/el_seed.c does not compile standalone via the exact command
tools/install.sh uses (`cc -std=c11 -O2 -I runtime -c runtime/el_seed.c`):
51 of its __-prefixed wrapper functions (http serving, JSON access, key-val
state, URL/HTML escaping, and the whole engram_* node/edge/layer/search
surface) call unprefixed counterparts that are implemented in el_runtime.c,
not in el_seed.c itself, and el_seed.c never declared them -- a toolchain
that treats an implicit function declaration as a hard error under C11
fails the compile outright.

install.sh already compiles el_seed.c and el_runtime.c as separate objects
and archives both into libel.a, so the symbols are always present at link
time; el_seed.c alone was just missing the prototypes.

A plain `#include "el_runtime.h"` was tried first and rejected: it redefines
el_to_float/el_from_float, which el_seed.h already provides -- a real
compile error, not a style preference. Added narrow prototypes instead,
copied verbatim from el_runtime.h, for exactly the 51 symbols el_seed.c's
wrappers reference and nothing else.

Verified clean:
  - `cc -std=c11 -O2 -I runtime -c runtime/el_seed.c` (install.sh's exact
    per-file compile) -- 0 errors, 0 warnings, even with -ferror-limit=0.
  - full `tools/install.sh` run -- compiles both objects and archives them
    into libel.a successfully.

Separately (not fixed here, out of scope): AGENTS.md's documented compiler
self-rebuild command links elc-new.c against el_seed.c, but elc-new.c's own
generated `#include "el_runtime.h"` line and 3 undeclared symbols
(el_mem_check, stdout_to_file, stdout_restore -- present in neither
el_runtime.c nor el_seed.c) mean that command fails regardless of which
runtime file it's linked against; and install.sh's libel.a only archives
el_seed.o + el_runtime.o, so any program that calls into the engram_*
surface fails to link against it (el_runtime.c's engram_* wrappers need
engram_store.c/engram_geometry.c/engram_reason.c/engram_cognition.c/
engram_vindex.c, none of which install.sh compiles in). Both are real,
pre-existing, and independent of this fix -- worth their own look.

will.anderson added 3 commits 2026-08-15 22:51:45 +00:00

engram: port PR #105 's 3 genuine wins onto dev's existing cosq/e_eff semantic layer 90d3f0bc76

Reconciles PR #105 ("fix: engram search latency — pin embed model, cache
query embeddings, bound activate BFS") with dev's ACTUAL current
engram_activate, rather than the ancient pre-restructure snapshot #105 was
built against.

WHY THIS NEEDED RECONCILIATION, NOT A DIRECT PORT: #105's single commit
(1dc49b1) modifies `lang/el-compiler/runtime/el_runtime.c` — a path that does
not exist on dev (dev has `lang/runtime/el_runtime.c`; the restructure that
renamed it happened after #105's branch point, which traces to a July 22
merge-base, weeks before the M8/M8.1/qgate/fan-effect/adjacency-index work
this file has grown since). #105's own engram_activate is consequently the
PRE-restructure version: no adjacency index (O(E) full edge scan per hop),
no query-aware qgate, no ACT-R fan effect, no eg_edge_eff_weight, and no
awareness of dev's cosq/e_eff embedding-blend semantic layer — it built a
parallel `g_qcache`/`engram_embed_raw` mechanism from scratch against code
that no longer exists at that path. A raw merge/cherry-pick was not possible
and would have been wrong even if it were: taking #105's tree wholesale would
have thrown away everything dev grew in the meantime (qgate, fan effect,
adjacency index, and this session's own M8 HNSW vindex integration).

RECONCILIATION: kept dev's cosq/e_eff mechanism as the semantic layer
entirely intact (unchanged by this commit) and ported #105's three genuinely
additive wins on TOP of it, at their equivalent sites in the CURRENT
eg_embed_fetch/engram_activate:

  1. keep_alive:-1 on the Ollama embed request body (eg_embed_fetch) — pins
     the embed model resident so a larger generation model loading under
     unified-memory pressure can't evict it and force a cold reload on the
     next search (#105 measured ~2.2s cold vs ~0.02-0.05s warm).
  2. Query-embedding cache upgraded from dev's single-slot (`_eg_qcache_text`,
     only ever remembered the LAST query) to a direct-mapped, FNV-1a-keyed,
     1024-slot cache (reusing the existing engram_id_hash) — so the
     curiosity loop's rotating phrases actually hit the cache instead of
     evicting each other every call. Same "pointer owned by the cache, not
     freed by caller" contract as before, just per-slot instead of global.
  3. Beam cap on the layer-1 spreading-activation BFS (new
     engram_activate_beam(), tunable via ENGRAM_ACTIVATE_BEAM, default 128).
     The FIFO frontier is processed in hop-level batches (entries sharing
     .hops are provably contiguous — see the code comment); when a level
     exceeds the beam width, only the top-`beam` by activation actually
     EXPAND. Every node in an oversized level still gets reached[]/best_bg[]
     recorded (that happens at enqueue time, one level up) and appears in
     the reported/promoted set — the cap bounds associative SPREAD width
     only, never recall of what was already found. Kept as a genuine
     additional bound even though the adjacency index + qgate + fan effect
     already mitigate #105's original "hub-node explosion" failure mode for
     a different reason: those prune WHICH targets matter; this bounds
     worst-case width regardless.

Everything else in dev's engram_activate — cosq/e_eff, the qgate rescale,
the fan effect, eg_edge_eff_weight, the M8 HNSW vindex seed discovery from
the #109 reconciliation earlier this session — is untouched.

VERIFIED (nsbx sandbox only, live :8742/:7770 never touched): cc -std=c11
-O2 clean build; booted in an isolated sandbox against a real cloned
production snapshot (13,424 nodes / 37,656 edges); ran 5 activate() calls
across rotating queries at depth 3, including the same query issued twice
non-consecutively (2nd hit landed at 476ms vs the 1st at 483ms — consistent
with a cache hit once Ollama's own warm-model latency is accounted for; no
crash, correct varied result counts (367-2610 nodes) each call; act-stats
JSON read correctly throughout.

Built on top of the M8/#109 reconciliation (bacaf3d, merged to dev as
#109) — dev's current HEAD at the time of this commit.

ingest: unify transduce_prose/transduce_structured into one transduce() 9d40f87926

transduce() is now THE single mechanism: one function, no content-type
branch inside it. It never asks whether `source` is prose, JSON, or
raw/opaque bytes (audio, etc.) — it runs one algorithm unconditionally:
split on "\n\n" as a universal boundary-marker check, and if that finds
no boundary, fall back to fixed 4096-char windows. Same node/edge wiring
(root -contains-> chunk, chunk -precedes-> next, "#"-prefixed chunk gets
a heading/section_of link) regardless of what's inside a chunk. Dedup is
the existing find_existing_by_content path via merge_manifold, applied
uniformly. The old transduce_structured JSON dataset/records/feature-node
interpretation is deleted outright, not just unused — a JSON file now
gets chunked and deduped like anything else, with no pre-computed
structure. All five ingest_* entry points still exist unchanged in name
and role; ingest_file/ingest_dir/ingest_url/ingest_llm now call the one
transduce() (ingest_stream builds its own turn-nodes directly and never
called either old function, so it's untouched).

This unlocks raw/opaque content (audio, or anything else with no natural
text/JSON shape) without any DSP, LLM call, or external API: transduce()
chunks it exactly like it chunks anything else. There is zero semantic
understanding of audio (or any payload) claimed or built here — any
meaning is expected to emerge later from Neuron's own existing mechanisms
(embedding, spreading activation, dedup) acting on this real geometry
over time.

Two small C builtins added to el_runtime.c/h (fs_size, fs_read_b64_chunk)
because El strings are NUL-unsafe under strlen-based ops and fs_read()'s
result silently truncates at the first embedded NUL, which is routine in
real binary/audio bytes. ingest_file compares fs_read()'s string length
against a real fs_size() stat() count; on mismatch it rebuilds the
payload as base64-encoded fixed 3072-byte windows read directly off disk
(binary-safe in C, verbatim, no invention), joined with the same "\n\n"
marker transduce()'s boundary scan already looks for. This is a
mechanical fidelity fix, not interpretation of content — transduce()
never learns a fallback happened. Registered both builtins' arity in
codegen.el; did not rebuild the elc compiler binary itself (unrelated,
pre-existing gap: self-hosting elc via el_seed.c fails on this worktree
independent of this change, reproduced with codegen.el reverted) — the
existing elc binary compiles calls to unregistered builtins via its
already-existing arity=-1 passthrough, confirmed by an actual clean
`elc ingest.el` + `cc` build against the modified el_runtime.c.

INGEST_KIND keeps existing only as an acquisition-mechanism selector
(dir/file/url/llm/stream — which RPC to use to fetch bytes), not as a
content-type flag; the redundant "structured" value (an alias for "file"
that hinted the now-deleted JSON branch) is removed. ingest_dir drops its
file-extension filter for the same reason: transduce() takes anything now.

Verification: local manifold construction confirmed correct against a
real captured audio file (will_clean.wav, 304288 bytes, and a 12288-byte
real prefix slice) — exact expected node/edge counts both times
(101 nodes/199 edges full file; 5 nodes/7 edges for the slice, matching
ceil(bytes/3072)+1 nodes and 2n-1 edges), with real, verbatim base64
content confirmed decoding back to the actual WAV header bytes. Compiles
clean via the real elc + the modified el_runtime.c/engram_*.c (built and
booted an actual sandbox engram off this exact source with `nsbx create
--branch`).

NOT verified this session, disclosed rather than papered over: end-to-end
server-confirmed persistence (a real before/after /api/stats delta, and a
fetched node by id) for the audio, prose, and JSON-fixture cases. Every
local nsbx sandbox engram tried tonight (two stock pre-#109 binaries
hitting the known O(N*D) brute-force scan bug, then a fresh #109/HNSW
binary built from current dev) took minutes-to indefinitely long on the
final /api/load-merge write's embedding step and hit the client's 60s
HTTP timeout before responding, even for a 5-node write. This is
confirmed as real (if slow) forward progress, not a hang: the sandbox's
WAL file was observed growing steadily across every attempt. The code's
own pre-existing HONESTY GATE correctly refused to report success in
every case, returning "load-merge failed: ..." with a
"nothing below this manifold was confirmed persisted by the server" note
instead — exactly as designed. This is an environment/infrastructure
limitation, not a defect introduced by this change: the engram server
binary itself is untouched by this commit.

runtime: port missing __channel_* primitives into el_seed.c

El SDK CI - dev / build-and-test (pull_request) Failing after 3m44s

Details

3718bf0380

runtime/channel.el has always called __channel_new/__channel_send/
__channel_recv/__channel_try_recv/__channel_close, but these were only ever
implemented in the pre-restructure lang/el-compiler/runtime/el_runtime.c.
When the canonical runtime was consolidated onto the release copy
(lang/runtime/el_runtime.c) and el_seed.c became the sole C dependency,
the channel implementation was never carried forward — __mutex_new made the
move, __channel_* did not. Any El program using Go-style channels currently
fails to link on dev.

Ported the working buffered-MPMC-channel implementation (mutex+condvar+
circular buffer, bounded and unbounded modes) from the old el_runtime.c
verbatim, adapted only to el_seed.c's arena API (seed_arena_track in place
of el_arena_track). Declared in el_seed.h alongside the existing mutex
primitives.

will.anderson merged commit d45a0882f3 into dev

2026-08-15 23:06:02 +00:00

will.anderson referenced this pull request

2026-08-15 23:17:50 +00:00

swarm: land wt/swarm-ccr onto dev (clean re-merge) #122

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#118