Three confirmed-live bugs tonight:
- `nsbx up` printed "daemon did not become ready" immediately followed by a
green "your sandbox is ready" banner and exited 0, because the existing-
sandbox restart path (`daemon_alive || start_daemon`) never checked
start_daemon's return code. `cmd_build` had the identical unguarded
pattern, plus `cmd_run`/`cmd_validate`'s own start-if-dead calls. All four
now `|| die` with a message pointing at daemon.log.
- `nsbx status`/`nsbx list` reported bare "state: running" for a process
that's alive (passes kill -0) but not actually answering /api/stats --
pegged, hung, or mid-boot. Added daemon_health(), which does the real
stats fetch and distinguishes stopped/running/unresponsive; both commands
now say "running but NOT RESPONDING" with a next-step hint instead of
silently going quiet on the stats field. Reproduced live against another
agent's actively-running (CPU-pinned, non-responsive) sandbox tonight, and
again via a deliberate SIGSTOP on a throwaway sandbox.
- Sandboxes carried no visible signal that their binary predated a relevant
fix. `status`/`list` now show the binary's sha + real build timestamp
(mtime survives `cp -p`), plus a best-effort staleness note: for
stock-prod clones, compare against the currently-configured live binary;
for source/branch builds, compare the recorded source commit against
local origin/dev via merge-base --is-ancestor.
Also, found live while verifying the above:
- A cold boot under concurrent sandbox/CPU load can legitimately take past
the old hardcoded 15s readiness window. Made it configurable
(NSBX_READY_TIMEOUT_SECS) rather than just widening the default blindly.
- cmd_create's post-boot baseline capture could silently record sbx_baseline
as 0/0 when the stats fetch came back empty right after the auto-remerge
step -- which would make every future `nsbx validate` zero-loss/reboot-
prove check trivially PASS regardless of real data loss. Added a bounded
retry and a loud warning if it still comes back empty.
- Sharpened a handful of "no such sandbox" / missing-binary errors to name
the next command instead of just stating the failure.
Assembles every constituent repo of a stack into one combined worktree
workspace, laid out at natural relpaths so cross-repo ../foundation/el
imports resolve to the sandbox copy. Sibling of nsbx; pure bash + git
worktree; never touches live :8742/:7770; isolated engram delegated to nsbx.
Add 'nsbx dev <name>' / 'nsbx dev-down <name>' plus a Makefile so a newcomer
goes from clone to coding on an isolated cloned engram in one command. The
worktree is created on a real named branch at a persistent path (never /tmp,
guarded), and the whole worktree is pinned to the clone via an emitted .nsbx-env
so live :8742 / ~/.neuron/engram is unreachable by accident. Optimizes the El
edit->build->run loop so provisional work is built in El against a throwaway
clone instead of prototyped in Python and re-ported. Additive over the proven
primitives; no live cutover.
Generalise the ad-hoc cog-arch (worktree+build+store-clone+C-tests) and
store-fix (secondary soul + launchctl rails cutover) proto-sandboxes into one
reproducible primitive: run experiments and code changes against the REAL
engram runtime on an isolated snapshot of the live mind, with a gated
promote-to-prod path.
Dev environment as a primitive — any team member gets a private, isolated copy
of the mind (separate port/store/process); prod on :8742/:7770 is untouchable
from a sandbox. Wraps the real binary; never reimplements engram logic.
Lifecycle: create/up (consistent store+WAL+config snapshot; place OR build the
runtime from --source/--branch/--binary; boot on an isolated port) · build ·
run · validate (rails as checks: zero-loss under load+reboot, reboot-prove, RSS
bound, retrieval parity, keystone integrity) · promote (gated rails cutover:
snapshot-first, additive binary swap, bootout→settle-poll→bootstrap, verify,
auto-rollback; never pkill/kickstart -k; dry-run unless approved) · destroy.
Dogfooded: reproduced retrieval-parity 25/25 vs baseline and the cog-arch
correspondence-loop known result (Brier 0.028648->0.000586, reboot-proven) and
real-store reboot-prove at 10994-node scale, all inside a sandbox; prod
untouched.