bigmerge
979e820f68
Three confirmed-live bugs tonight: - `nsbx up` printed "daemon did not become ready" immediately followed by a green "your sandbox is ready" banner and exited 0, because the existing- sandbox restart path (`daemon_alive || start_daemon`) never checked start_daemon's return code. `cmd_build` had the identical unguarded pattern, plus `cmd_run`/`cmd_validate`'s own start-if-dead calls. All four now `|| die` with a message pointing at daemon.log. - `nsbx status`/`nsbx list` reported bare "state: running" for a process that's alive (passes kill -0) but not actually answering /api/stats -- pegged, hung, or mid-boot. Added daemon_health(), which does the real stats fetch and distinguishes stopped/running/unresponsive; both commands now say "running but NOT RESPONDING" with a next-step hint instead of silently going quiet on the stats field. Reproduced live against another agent's actively-running (CPU-pinned, non-responsive) sandbox tonight, and again via a deliberate SIGSTOP on a throwaway sandbox. - Sandboxes carried no visible signal that their binary predated a relevant fix. `status`/`list` now show the binary's sha + real build timestamp (mtime survives `cp -p`), plus a best-effort staleness note: for stock-prod clones, compare against the currently-configured live binary; for source/branch builds, compare the recorded source commit against local origin/dev via merge-base --is-ancestor. Also, found live while verifying the above: - A cold boot under concurrent sandbox/CPU load can legitimately take past the old hardcoded 15s readiness window. Made it configurable (NSBX_READY_TIMEOUT_SECS) rather than just widening the default blindly. - cmd_create's post-boot baseline capture could silently record sbx_baseline as 0/0 when the stats fetch came back empty right after the auto-remerge step -- which would make every future `nsbx validate` zero-loss/reboot- prove check trivially PASS regardless of real data loss. Added a bounded retry and a loud warning if it still comes back empty. - Sharpened a handful of "no such sandbox" / missing-binary errors to name the next command instead of just stating the failure.
180 lines
10 KiB
Markdown
180 lines
10 KiB
Markdown
# nsbx — the Neuron Sandbox
|
|
|
|
**Dev environment as a primitive.** A reproducible way to run experiments *and code
|
|
changes* against the **real** engram runtime on an isolated snapshot of the live
|
|
mind — with a gated promote-to-prod path built on the proven rails.
|
|
|
|
Everyone (Tim, any team member, any agent) gets their own private, safe copy of the
|
|
mind to build against. **Prod — the live Neuron on `:8742` (engram) / `:7770`
|
|
(soul) — is untouchable from a sandbox.** A sandbox runs a *separate* engram
|
|
process, on a *separate* port, against a *separate* clone of the store. The only op
|
|
that can ever reach prod is `promote`, which is explicit, gated, and per-use
|
|
approved.
|
|
|
|
It **wraps the real engram binary** — it never reimplements any engram logic. It
|
|
generalises two proven proto-sandboxes into one primitive:
|
|
|
|
- the **cog-arch** build — isolated git worktree + build + clone of the live `.egm` + real C tests
|
|
- the **store-fix** cutover — secondary soul + launchctl `bootout → settle → bootstrap` rails
|
|
|
|
## Quickstart
|
|
|
|
```bash
|
|
export PATH="$PWD:$PATH" # or symlink nsbx onto your PATH
|
|
|
|
nsbx up # your private copy of the mind (auto-named <user>-dev)
|
|
nsbx run <name> api /api/stats # poke it
|
|
nsbx validate <name> # prove it: zero-loss, reboot, RSS, retrieval, keystones
|
|
nsbx destroy <name> # cheap teardown; live untouched
|
|
```
|
|
|
|
That is the whole loop. Sane defaults: stock prod binary, auto-allocated port
|
|
(`8900+`, never `8742`/`7770`), snapshot of the live store.
|
|
|
|
## One-command dev onboarding — `nsbx dev` (start here)
|
|
|
|
Going from a clone to *coding on the mind* is a single command. It creates a git
|
|
**branch**, a persistent git **worktree**, and an **isolated engram** (a clone of the
|
|
live store on a non-default port) — and wires the whole worktree to that clone so you
|
|
**cannot hit live `:8742` by accident**.
|
|
|
|
```bash
|
|
make dev NAME=tim # branch wt/tim + worktree + isolated engram, in one shot
|
|
cd ~/Development/neuron-technologies/el-worktrees/tim
|
|
source .nsbx-env # every ENGRAM_* var now points at YOUR clone
|
|
|
|
# edit El in the worktree, then the fast loop:
|
|
make build NAME=tim # compile your El change into the isolated engram
|
|
make run NAME=tim # poke it (API=/api/stats by default)
|
|
make test NAME=tim # run the safety rails as checks
|
|
make destroy NAME=tim # tear it all down (branch kept; live untouched)
|
|
```
|
|
|
|
**Why this exists:** so provisional/experimental work is built **directly in El against a
|
|
throwaway cloned engram** — not prototyped in Python and re-ported later. The El
|
|
edit → `make build` → `make run` loop is the path of least resistance; that double-work is
|
|
what stranded the translation faculty for weeks.
|
|
|
|
### What `nsbx dev <name>` does, in order
|
|
|
|
1. **branch** — `git worktree add -b <prefix><name>` (default prefix `dev/`; a *real named
|
|
branch*, never detached HEAD).
|
|
2. **worktree** — at a **persistent** path (default `…/el-worktrees/<name>`, override
|
|
`NSBX_DEV_WT_ROOT`). It **refuses** `/tmp` — temp dirs are ablated on compaction, which
|
|
is the exact "worktree in /tmp + no branch = lost work" failure this designs out.
|
|
3. **isolated engram** — `nsbx create` under the hood: clone of the live store + WAL +
|
|
config, booted on an auto-allocated port (`8900+`, never `:8742`/`:7770`). Stock prod
|
|
binary by default (instant); `--build` compiles the worktree's own runtime instead.
|
|
4. **env pin** — writes `.nsbx-env` (+ `.envrc` for direnv) into the worktree exporting
|
|
`ENGRAM_URL / ENGRAM_PORT / ENGRAM_DATA_DIR / ENGRAM_API_KEY / NEURON_ENGRAM_URL …` — all
|
|
pointing at the clone. Nothing references live.
|
|
|
|
```
|
|
nsbx dev <name> [--base REF] [--worktree DIR] [--port N] [--prefix P] [--build] [--no-engram] [--repo R]
|
|
nsbx dev-down <name> [--delete-branch] [--repo R] # destroy engram + remove worktree
|
|
```
|
|
|
|
### Makefile targets
|
|
|
|
| target | does |
|
|
|--------|------|
|
|
| `make dev NAME=x` | branch + worktree + isolated engram (one shot) |
|
|
| `make bt NAME=x` | fast El loop: `build` then `run` |
|
|
| `make build NAME=x` | recompile the engram from the worktree's El source |
|
|
| `make run NAME=x` | poke the isolated engram (`API=/api/stats`) |
|
|
| `make test NAME=x` | rails as checks (`nsbx validate`) |
|
|
| `make status [NAME=x]` | inspect one, or `list` all |
|
|
| `make destroy NAME=x` | tear down (add `ARGS=--delete-branch` to drop the branch) |
|
|
|
|
> Note: if a bare `dev` branch already exists in the repo, git can't create `dev/*` names —
|
|
> pass `--prefix wt/` (or delete the stray `dev` branch). The tool surfaces git's exact error.
|
|
|
|
## The code-change dev loop (first-class)
|
|
|
|
Run *your changed runtime*, not just the stock binary, against a snapshot:
|
|
|
|
```bash
|
|
# build a runtime from a working tree, a git branch, or a prebuilt binary:
|
|
nsbx create feat --source /path/to/worktree # elc + cc build from source
|
|
nsbx create feat --branch feat/my-change --repo <r> # worktree the branch, then build
|
|
nsbx create feat --binary /path/to/engram # use a prebuilt binary
|
|
|
|
nsbx build feat --source /path/to/worktree # rebuild + hot-restart in place
|
|
nsbx validate feat # prove the change is safe
|
|
nsbx promote feat --i-approve-prod-cutover # gated rails cutover (see below)
|
|
```
|
|
|
|
The build replicates the engram release recipe exactly:
|
|
`elc engram/src/server.el > engram.c` then
|
|
`cc -std=c11 -O2 -I lang/runtime engram.c el_runtime.c engram_*.c -lcurl -lpthread`.
|
|
|
|
## Lifecycle
|
|
|
|
| op | what it does |
|
|
|----|--------------|
|
|
| `create <name> [--port N] [--source\|--branch\|--binary]` | consistent snapshot of the live store+WAL+config into an isolated dir; place or **build** the runtime; boot the real engram daemon on an isolated port. Named, versioned (binary sha + egm sha in `manifest.json`), reproducible. |
|
|
| `up [name]` | one command: create-if-missing then start; prints the URL. |
|
|
| `build <name> --source\|--branch` | rebuild the runtime from a code change and hot-restart on the same clone+port. |
|
|
| `run <name> <cmd…>` / `run <name> api <path> [json]` | run an experiment against the real runtime; capture output + before/after stats + wall time. Env: `$SBX_URL $SBX_PORT $SBX_KEY $SBX_DATA $SBX_BIN`. |
|
|
| `validate <name>` | the rails as first-class checks (below). |
|
|
| `promote <name> [--data] [--i-approve-prod-cutover]` | **the only prod-touching op.** Gated rails cutover. DRY-RUN plan unless approved. |
|
|
| `destroy <name>` | stop the isolated daemon, free the port, remove the clone. Live untouched. |
|
|
| `list` / `status <name>` | inspect. |
|
|
|
|
## `validate` — the rails as checks
|
|
|
|
- **zero-loss-under-load** — node/edge counts hold at/above baseline through ~15s of sustained tick+read load
|
|
- **reboot-prove** — counts survive a real stop→start of the daemon
|
|
- **rss-bound** — daemon RSS under `NSBX_RSS_BOUND_MB` (default 550 MB, from the store-fix reboot-proof)
|
|
- **retrieval-parity** — top-k node ids for a fixed probe set match the create-time baseline
|
|
- **keystone-integrity** — `kn-efeb4a5b…` and `kn-5b606390…` present and intact
|
|
|
|
A PASS writes `validate.json` stamped with the binary sha; `promote` refuses unless
|
|
the current binary has a fresh PASS on record.
|
|
|
|
## `promote` — gated cutover (rails only)
|
|
|
|
Default is a **dry-run plan**. With `--i-approve-prod-cutover` it, in order:
|
|
|
|
1. **snapshot-first** — back up live `egm`+`wal`+`plist` to `~/.neuron/backups/promote-<name>-<ts>/` with a `rollback.txt`
|
|
2. **additive** binary install — copy the validated binary to a *new* file, update the plist `ENGRAM_REAL_BIN` (old binary retained — additive/supersede, never destructive)
|
|
3. **rails cutover** — `launchctl bootout` → **settle-poll** (prints until the job is gone) → `launchctl bootstrap`. Never `pkill`, never `kickstart -k`.
|
|
4. **verify** — `/api/stats` returns, edges ≥ baseline, keystones intact
|
|
5. **auto-rollback armed** — any verify failure restores the plist (and data, if `--data`) and boots the prior binary back via the same rails
|
|
|
|
## Isolation guarantees
|
|
|
|
- separate **port** (`8900+`; refuses `8742`/`7770`), separate **store clone**, separate **process**
|
|
- a hard guard refuses to boot a sandbox daemon whose data dir resolves to the live store
|
|
- sandboxes are plain supervised background processes (not launchd), so teardown is a signal + settle-poll — it can never touch the prod launchd job
|
|
- prod is read exactly twice: once for the snapshot, and (only if you approve) during `promote`
|
|
|
|
## Layout
|
|
|
|
- tool: `tools/neuron-sandbox/nsbx` (this repo, branch `feat/neuron-sandbox`)
|
|
- runtime state: `~/.neuron/sandboxes/<name>/` — `data/` (clone), `bin/engram`, `build/`, `logs/`, `manifest.json`, `validate.json`, `baseline/`
|
|
|
|
## Validated (dogfood)
|
|
|
|
Standing up a sandbox from a live-store clone and reproducing a **known** result:
|
|
|
|
- **retrieval-parity 25/25** top-k id overlap vs baseline; sandbox boot-stats exactly matched the live baseline captured at snapshot time (10 672 nodes / 32 439 edges) — the wrapped real binary faithfully reloads the live mind
|
|
- reboot-prove + zero-loss PASS; RSS 379 MB < 550 MB; keystones intact
|
|
- the **cog-arch correspondence-loop** re-run *inside* the sandbox reproduced the known calibration numbers exactly: held-Brier **0.028648 → 0.000586** (98.0% reduction), monotone, **reboot bit-identical**, metastability holds; and the real-store Stance persistence reboot-proved at **10 994-node** scale (`think()` on real 768-dim embeddings) against a scratch copy of the sandbox's own clone — never live
|
|
- `promote` dry-run refused to touch prod; teardown freed the port; live `:8742`/`:7770` never perturbed (soul uptime unbroken)
|
|
|
|
## Migrating existing experiments
|
|
|
|
Each ad-hoc harness becomes `nsbx run <name> …` (or `--source` build) against a sandbox:
|
|
|
|
- **cog-arch** — `nsbx create x --source <worktree>` then `nsbx run x -- bash cogarch_dogfood.sh` (compiles + runs the real C cognition tests against `$SBX_DATA`)
|
|
- **codec / ingest / faculty** — `nsbx run x api /api/<endpoint> '<json>'` against the isolated daemon, or a script using `$SBX_URL`/`$SBX_KEY`; measure with the built-in before/after stats
|
|
|
|
## Env knobs
|
|
|
|
`NSBX_ROOT`, `NSBX_PORT_BASE`, `NSBX_RSS_BOUND_MB`, `NSBX_REMERGE_THRESHOLD`,
|
|
`NSBX_READY_TIMEOUT_SECS` (default 15 — how long `up`/`create`/`build` wait for a
|
|
daemon to answer `/api/stats` before reporting failure; raise it if a boot is
|
|
legitimately slow under concurrent sandbox/CPU load rather than actually broken),
|
|
`EL_REPO` (for `elc` + runtime sources), `ENGRAM_LIVE_DATA_DIR`, `ENGRAM_LIVE_PLIST`.
|