el/tools/neuron-sandbox/README.md

# nsbx — the Neuron Sandbox

**Dev environment as a primitive.** A reproducible way to run experiments *and code
changes* against the **real** engram runtime on an isolated snapshot of the live
mind — with a gated promote-to-prod path built on the proven rails.

Everyone (Tim, any team member, any agent) gets their own private, safe copy of the
mind to build against. **Prod — the live Neuron on `:8742` (engram) / `:7770`
(soul) — is untouchable from a sandbox.** A sandbox runs a *separate* engram
process, on a *separate* port, against a *separate* clone of the store. The only op
that can ever reach prod is `promote`, which is explicit, gated, and per-use
approved.

It **wraps the real engram binary** — it never reimplements any engram logic. It
generalises two proven proto-sandboxes into one primitive:

- the **cog-arch** build — isolated git worktree + build + clone of the live `.egm` + real C tests
- the **store-fix** cutover — secondary soul + launchctl `bootout → settle → bootstrap` rails

## Quickstart

```bash
export PATH="$PWD:$PATH"          # or symlink nsbx onto your PATH

nsbx up                          # your private copy of the mind (auto-named <user>-dev)
nsbx run <name> api /api/stats   # poke it
nsbx validate <name>             # prove it: zero-loss, reboot, RSS, retrieval, keystones
nsbx destroy <name>              # cheap teardown; live untouched
```

That is the whole loop. Sane defaults: stock prod binary, auto-allocated port
(`8900+`, never `8742`/`7770`), snapshot of the live store.

## One-command dev onboarding — `nsbx dev` (start here)

Going from a clone to *coding on the mind* is a single command. It creates a git
**branch**, a persistent git **worktree**, and an **isolated engram** (a clone of the
live store on a non-default port) — and wires the whole worktree to that clone so you
**cannot hit live `:8742` by accident**.

```bash
make dev NAME=tim        # branch wt/tim + worktree + isolated engram, in one shot
cd ~/Development/neuron-technologies/el-worktrees/tim
source .nsbx-env         # every ENGRAM_* var now points at YOUR clone

# edit El in the worktree, then the fast loop:
make build NAME=tim      # compile your El change into the isolated engram
make run   NAME=tim      # poke it (API=/api/stats by default)
make test  NAME=tim      # run the safety rails as checks
make destroy NAME=tim    # tear it all down (branch kept; live untouched)
```

**Why this exists:** so provisional/experimental work is built **directly in El against a
throwaway cloned engram** — not prototyped in Python and re-ported later. The El
edit → `make build` → `make run` loop is the path of least resistance; that double-work is
what stranded the translation faculty for weeks.

### What `nsbx dev <name>` does, in order

1. **branch** — `git worktree add -b <prefix><name>` (default prefix `dev/`; a *real named
   branch*, never detached HEAD).
2. **worktree** — at a **persistent** path (default `…/el-worktrees/<name>`, override
   `NSBX_DEV_WT_ROOT`). It **refuses** `/tmp` — temp dirs are ablated on compaction, which
   is the exact "worktree in /tmp + no branch = lost work" failure this designs out.
3. **isolated engram** — `nsbx create` under the hood: clone of the live store + WAL +
   config, booted on an auto-allocated port (`8900+`, never `:8742`/`:7770`). Stock prod
   binary by default (instant); `--build` compiles the worktree's own runtime instead.
4. **env pin** — writes `.nsbx-env` (+ `.envrc` for direnv) into the worktree exporting
   `ENGRAM_URL / ENGRAM_PORT / ENGRAM_DATA_DIR / ENGRAM_API_KEY / NEURON_ENGRAM_URL …` — all
   pointing at the clone. Nothing references live.

```
nsbx dev <name> [--base REF] [--worktree DIR] [--port N] [--prefix P] [--build] [--no-engram] [--repo R]
nsbx dev-down <name> [--delete-branch] [--repo R]      # destroy engram + remove worktree
```

### Makefile targets

| target | does |
|--------|------|
| `make dev NAME=x` | branch + worktree + isolated engram (one shot) |
| `make bt NAME=x` | fast El loop: `build` then `run` |
| `make build NAME=x` | recompile the engram from the worktree's El source |
| `make run NAME=x` | poke the isolated engram (`API=/api/stats`) |
| `make test NAME=x` | rails as checks (`nsbx validate`) |
| `make status [NAME=x]` | inspect one, or `list` all |
| `make destroy NAME=x` | tear down (add `ARGS=--delete-branch` to drop the branch) |

> Note: if a bare `dev` branch already exists in the repo, git can't create `dev/*` names —
> pass `--prefix wt/` (or delete the stray `dev` branch). The tool surfaces git's exact error.

## The code-change dev loop (first-class)

Run *your changed runtime*, not just the stock binary, against a snapshot:

```bash
# build a runtime from a working tree, a git branch, or a prebuilt binary:
nsbx create feat --source /path/to/worktree           # elc + cc build from source
nsbx create feat --branch feat/my-change --repo <r>   # worktree the branch, then build
nsbx create feat --binary /path/to/engram             # use a prebuilt binary

nsbx build feat --source /path/to/worktree            # rebuild + hot-restart in place
nsbx validate feat                                    # prove the change is safe
nsbx promote feat --i-approve-prod-cutover            # gated rails cutover (see below)
```

The build replicates the engram release recipe exactly:
`elc engram/src/server.el > engram.c` then
`cc -std=c11 -O2 -I lang/runtime engram.c el_runtime.c engram_*.c -lcurl -lpthread`.

## Lifecycle

| op | what it does |
|----|--------------|
| `create <name> [--port N] [--source\|--branch\|--binary]` | consistent snapshot of the live store+WAL+config into an isolated dir; place or **build** the runtime; boot the real engram daemon on an isolated port. Named, versioned (binary sha + egm sha in `manifest.json`), reproducible. |
| `up [name]` | one command: create-if-missing then start; prints the URL. |
| `build <name> --source\|--branch` | rebuild the runtime from a code change and hot-restart on the same clone+port. |
| `run <name> <cmd…>` / `run <name> api <path> [json]` | run an experiment against the real runtime; capture output + before/after stats + wall time. Env: `$SBX_URL $SBX_PORT $SBX_KEY $SBX_DATA $SBX_BIN`. |
| `validate <name>` | the rails as first-class checks (below). |
| `promote <name> [--data] [--i-approve-prod-cutover]` | **the only prod-touching op.** Gated rails cutover. DRY-RUN plan unless approved. |
| `destroy <name>` | stop the isolated daemon, free the port, remove the clone. Live untouched. |
| `list` / `status <name>` | inspect. |

## `validate` — the rails as checks

- **zero-loss-under-load** — node/edge counts hold at/above baseline through ~15s of sustained tick+read load
- **reboot-prove** — counts survive a real stop→start of the daemon
- **rss-bound** — daemon RSS under `NSBX_RSS_BOUND_MB` (default 550 MB, from the store-fix reboot-proof)
- **retrieval-parity** — top-k node ids for a fixed probe set match the create-time baseline
- **keystone-integrity** — `kn-efeb4a5b…` and `kn-5b606390…` present and intact

A PASS writes `validate.json` stamped with the binary sha; `promote` refuses unless
the current binary has a fresh PASS on record.

## `promote` — gated cutover (rails only)

Default is a **dry-run plan**. With `--i-approve-prod-cutover` it, in order:

1. **snapshot-first** — back up live `egm`+`wal`+`plist` to `~/.neuron/backups/promote-<name>-<ts>/` with a `rollback.txt`
2. **additive** binary install — copy the validated binary to a *new* file, update the plist `ENGRAM_REAL_BIN` (old binary retained — additive/supersede, never destructive)
3. **rails cutover** — `launchctl bootout` → **settle-poll** (prints until the job is gone) → `launchctl bootstrap`. Never `pkill`, never `kickstart -k`.
4. **verify** — `/api/stats` returns, edges ≥ baseline, keystones intact
5. **auto-rollback armed** — any verify failure restores the plist (and data, if `--data`) and boots the prior binary back via the same rails

## Isolation guarantees

- separate **port** (`8900+`; refuses `8742`/`7770`), separate **store clone**, separate **process**
- a hard guard refuses to boot a sandbox daemon whose data dir resolves to the live store
- sandboxes are plain supervised background processes (not launchd), so teardown is a signal + settle-poll — it can never touch the prod launchd job
- prod is read exactly twice: once for the snapshot, and (only if you approve) during `promote`

## Layout

- tool: `tools/neuron-sandbox/nsbx` (this repo, branch `feat/neuron-sandbox`)
- runtime state: `~/.neuron/sandboxes/<name>/` — `data/` (clone), `bin/engram`, `build/`, `logs/`, `manifest.json`, `validate.json`, `baseline/`

## Validated (dogfood)

Standing up a sandbox from a live-store clone and reproducing a **known** result:

- **retrieval-parity 25/25** top-k id overlap vs baseline; sandbox boot-stats exactly matched the live baseline captured at snapshot time (10 672 nodes / 32 439 edges) — the wrapped real binary faithfully reloads the live mind
- reboot-prove + zero-loss PASS; RSS 379 MB < 550 MB; keystones intact
- the **cog-arch correspondence-loop** re-run *inside* the sandbox reproduced the known calibration numbers exactly: held-Brier **0.028648 → 0.000586** (98.0% reduction), monotone, **reboot bit-identical**, metastability holds; and the real-store Stance persistence reboot-proved at **10 994-node** scale (`think()` on real 768-dim embeddings) against a scratch copy of the sandbox's own clone — never live
- `promote` dry-run refused to touch prod; teardown freed the port; live `:8742`/`:7770` never perturbed (soul uptime unbroken)

## Migrating existing experiments

Each ad-hoc harness becomes `nsbx run <name> …` (or `--source` build) against a sandbox:

- **cog-arch** — `nsbx create x --source <worktree>` then `nsbx run x -- bash cogarch_dogfood.sh` (compiles + runs the real C cognition tests against `$SBX_DATA`)
- **codec / ingest / faculty** — `nsbx run x api /api/<endpoint> '<json>'` against the isolated daemon, or a script using `$SBX_URL`/`$SBX_KEY`; measure with the built-in before/after stats

## Env knobs

`NSBX_ROOT`, `NSBX_PORT_BASE`, `NSBX_RSS_BOUND_MB`, `NSBX_REMERGE_THRESHOLD`,
`EL_REPO` (for `elc` + runtime sources), `ENGRAM_LIVE_DATA_DIR`, `ENGRAM_LIVE_PLIST`.