fix(engine): the daemon survives a client leaving, and says it is working while it works #127

2026-08-07T01:47:06Z

tim.lingo commented

2026-08-07 01:47:06 +00:00

Two small engine changes behind one measured incident: on 2026-08-06 the desktop app cancelled a request 25 s into a healthy 116.9-second call; the brain finished its work, wrote the reply to the departed client — and died.

exited due to SIGPIPE | sent by neuron[3953], ran for 361177ms. launchd respawned it 4 ms later; the user was never told. Every other in-flight session on that daemon died with it.

D1 — the daemon must survive a client leaving

Root cause is in the vendored C runtime, not .el. The whole HTTP server lives in el_runtime.c; .el has no socket or signal primitive, so chat.el/routes.el could not have fixed this. http_send_all() called send() and nothing anywhere in the runtime ever set a SIGPIPE disposition, so the default — terminate the process — applied. One reply is written in four sequential send() calls: after a departing client's FIN the first succeeds, the peer answers RST, the second raises SIGPIPE and kills the daemon.

Fix: SIG_IGN at el_runtime_init_args and each http_serve* entry; per-connection SO_NOSIGPIPE/MSG_NOSIGNAL; http_send_all retries EINTR and preserves errno; http_send_response classifies once — a departure logs client left before the reply was written … reply discarded, and any other errno still logs a real send failed: <strerror>, so the routine path cannot mask a genuine fault. (Routine branch is E2E-exercised; the genuine-fault branch is code-inspected only — stated rather than implied.)

Discrimination, which is the part worth reading:

brain	result
round-9 shipped `4402179554…`	DIED, exit 141 (128+13), first disconnect
round-9 rebuilt from source under my exact recipe `e6130002…`	DIED, exit 141, first disconnect
this build `fdd0c74a…`	SURVIVED 3/3, `/health` 200 after, still serving, three honest "client left" lines naming Broken pipe / Connection reset by peer

The middle row is the point: it rules out the build recipe as the variable.

D2 — emit the round-start marker the app already listens for

The app handles a run_progress entry with tool __working__ as a life signal, and a spec document claims the engine emits it. It does not — 0 occurrences across the engine's .el sources and 0 in the shipped binary. That client branch has been dead code, which is why a long first leg looked identical to a dead brain.

This appends {"i":N,"t":"","tool":"__working__"} to the existing run_progress_<session_id> ledger before each round's outbound call. No new key, no new route, no run registry — a strict subset of item 3 in docs/specs/WS3-daemon-owned-runs-for-will.md. WS3 itself is untouched and remains yours.

Measured: round-9 baseline ledger empty for a whole 59.7 s leg; this build shows the marker at 18.6 s of a 70.0 s leg. Both returned correct ~4.9 KB answers.

Residual named honestly: ~17 s of agentic pre-flight (engram/session preload, safety screen) runs before round 0's marker, so that window is still unlit. Not fixed here.

Regression + gates

Prompt-matrix gate 32/32 on this build and 32/32 on the rebuilt round-9 baseline under the identical recipe. verify-soul-contract.sh: PASS, 27/27 routes + immutability (needs /opt/homebrew/bin/bash — the script uses local -n, which macOS bash 3.2 rejects with a false red). neuron#111 miscompile guard: 0 sites in the generated amalgam, while the stale committed dist/soul.c still shows its 4 — proving the grep discriminates.

tests/run-el-test.sh (from neuron#122) is not in this base; borrowed and run against scratch copies of both trees, it fails identically in both (LINK FAILED: _generate, _generate_lang), so it discriminates nothing about this change and I claim no green from it.

Two things for you specifically

dist/soul.c is deliberately NOT committed. CI compiles that file, so production stays exposed until a regen — same standing ask as neuron#111 / ui#209. Landing it is your call, every time.
We can now regenerate the amalgam locally, and the reason it kept failing is worth knowing: elc trusts a module's .elh header when present and inlines only when absent (proven both directions — hiding chat.elh alone took output from 42 KB to 210 KB), and bare elb in a worktree clobbers dist/soul.c with the small non-inlined unit. Recipe compiled to gen-soul-amalgam.sh; output 1,163,886 B / 1225 inlined bodies. This supersedes our standing belief that only your toolchain could produce it — authority is still yours, capability is no longer the blocker.

Base is round 9's engine tip. Deliberately branched as a sibling of fix/resume-server-tool-replay so it does not move the base under neuron#122; land order is #115 → #122 → this, or rebase as you prefer.

Also relevant to the OpenAI lane: D1 protects it identically — a client disconnect there would have killed the daemon the same way.

🤖 Generated with Claude Code

**Two small engine changes behind one measured incident: on 2026-08-06 the desktop app cancelled a request 25 s into a healthy 116.9-second call; the brain finished its work, wrote the reply to the departed client — and died.** `exited due to SIGPIPE | sent by neuron[3953], ran for 361177ms`. launchd respawned it 4 ms later; the user was never told. Every other in-flight session on that daemon died with it. ## D1 — the daemon must survive a client leaving **Root cause is in the vendored C runtime, not `.el`.** The whole HTTP server lives in `el_runtime.c`; `.el` has no socket or signal primitive, so `chat.el`/`routes.el` could not have fixed this. `http_send_all()` called `send()` and **nothing anywhere in the runtime ever set a SIGPIPE disposition**, so the default — terminate the process — applied. One reply is written in four sequential `send()` calls: after a departing client's FIN the first succeeds, the peer answers RST, the second raises SIGPIPE and kills the daemon. Fix: `SIG_IGN` at `el_runtime_init_args` and each `http_serve*` entry; per-connection `SO_NOSIGPIPE`/`MSG_NOSIGNAL`; `http_send_all` retries `EINTR` and preserves errno; `http_send_response` classifies once — a departure logs `client left before the reply was written … reply discarded`, and **any other errno still logs a real `send failed: <strerror>`**, so the routine path cannot mask a genuine fault. (Routine branch is E2E-exercised; the genuine-fault branch is code-inspected only — stated rather than implied.) **Discrimination, which is the part worth reading:** | brain | result | |---|---| | round-9 shipped `4402179554…` | **DIED**, exit 141 (128+13), first disconnect | | round-9 **rebuilt from source under my exact recipe** `e6130002…` | **DIED**, exit 141, first disconnect | | this build `fdd0c74a…` | **SURVIVED 3/3**, `/health` 200 after, still serving, three honest "client left" lines naming Broken pipe / Connection reset by peer | The middle row is the point: it rules out the build recipe as the variable. ## D2 — emit the round-start marker the app already listens for The app handles a `run_progress` entry with tool `__working__` as a life signal, and a spec document claims the engine emits it. It does not — 0 occurrences across the engine's `.el` sources **and** 0 in the shipped binary. That client branch has been dead code, which is why a long first leg looked identical to a dead brain. This appends `{"i":N,"t":"","tool":"__working__"}` to the existing `run_progress_<session_id>` ledger **before** each round's outbound call. No new key, no new route, no run registry — a strict subset of item 3 in `docs/specs/WS3-daemon-owned-runs-for-will.md`. WS3 itself is untouched and remains yours. Measured: round-9 baseline ledger **empty for a whole 59.7 s leg**; this build shows the marker at **18.6 s of a 70.0 s leg**. Both returned correct ~4.9 KB answers. **Residual named honestly:** ~17 s of agentic pre-flight (engram/session preload, safety screen) runs *before* round 0's marker, so that window is still unlit. Not fixed here. ## Regression + gates Prompt-matrix gate **32/32** on this build and **32/32** on the rebuilt round-9 baseline under the identical recipe. `verify-soul-contract.sh`: **PASS, 27/27 routes + immutability** (needs `/opt/homebrew/bin/bash` — the script uses `local -n`, which macOS bash 3.2 rejects with a false red). neuron#111 miscompile guard: **0 sites** in the generated amalgam, while the stale committed `dist/soul.c` still shows its 4 — proving the grep discriminates. `tests/run-el-test.sh` (from neuron#122) is not in this base; borrowed and run against scratch copies of both trees, it fails identically in both (`LINK FAILED: _generate, _generate_lang`), so it discriminates nothing about this change and **I claim no green from it**. ## Two things for you specifically 1. **`dist/soul.c` is deliberately NOT committed.** CI compiles that file, so production stays exposed until a regen — same standing ask as neuron#111 / ui#209. Landing it is your call, every time. 2. **We can now regenerate the amalgam locally, and the reason it kept failing is worth knowing:** `elc` trusts a module's `.elh` header when present and inlines only when absent (proven both directions — hiding `chat.elh` alone took output from 42 KB to 210 KB), and **bare `elb` in a worktree clobbers `dist/soul.c` with the small non-inlined unit**. Recipe compiled to `gen-soul-amalgam.sh`; output 1,163,886 B / 1225 inlined bodies. This supersedes our standing belief that only your toolchain could produce it — authority is still yours, capability is no longer the blocker. Base is round 9's engine tip. Deliberately branched as a **sibling** of `fix/resume-server-tool-replay` so it does not move the base under neuron#122; land order is #115 → #122 → this, or rebase as you prefer. Also relevant to the OpenAI lane: D1 protects it identically — a client disconnect there would have killed the daemon the same way. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tim.lingo changed target branch from fix/resume-server-tool-replay to main

2026-08-07 15:54:05 +00:00

tim.lingo added 1 commit 2026-08-07 15:54:05 +00:00

fix(engine): a client that leaves must not kill the daemon, and a long round must say it started 98ccbd4704

Round 9.1, spec §3 D + ADR 0006 items 2 and 4. Two small changes, both proven
by measurement, both E2E-verified locally against a rebuilt brain.

D1 — SIGPIPE/EPIPE survival (vendor/el-runtime el_runtime.c).
Root cause, at the layer that owns it: the whole HTTP server lives in the C
runtime; .el has no socket primitive. http_send_all() called send() with flags
0 and nothing anywhere in the runtime set a SIGPIPE disposition, so the default
disposition — terminate the process — applied. When a handler finished after
its client had gone (Tim's VM: reply at 116.9 s, client cancelled at 25.0 s),
the second of the four sends that write one reply raised SIGPIPE and the daemon
died: `exited due to SIGPIPE ... ran for 361177ms`, launchd respawn 4 ms later,
every other in-flight session's work lost, user never told.

Fix: SIGPIPE -> SIG_IGN at runtime init and at each http_serve* entry, plus
per-connection SO_NOSIGPIPE / MSG_NOSIGNAL so the guard survives an embedder
resetting dispositions. http_send_all now retries EINTR and preserves errno;
http_send_response classifies it once — a departure is logged as routine
("client left before the reply was written ... reply discarded") and ANY other
errno is logged as a real "send failed: <strerror>". Spec §5.3: the routine
case must not mask a genuine write fault, and it does not.

Proof (scratch HOME + free port, 3 disconnects mid-reply):
  round-9 shipped brain 4402179554… — DIED, exit 141 (128+13 = SIGPIPE), round 1
  round-9 sources rebuilt with this exact recipe — DIED, exit 141, round 1
  this build — SURVIVED 3/3, /health 200 after, still serving the full graph,
  three honest "client left" lines in the log naming Broken pipe / Connection
  reset by peer.

D2 — the round-start marker (chat.el, agentic_loop).
The ledger only ever appended AFTER a round returned, so a healthy first leg
produced zero progress by construction; since server-side web_search moved
inside the outbound call that leg is 60-120 s of silence, which is how a 25 s
client watchdog came to kill a healthy mission. One entry,
{"i":N,"t":"","tool":"__working__"}, written to the existing
run_progress_<session_id> ledger BEFORE each round's outbound call — the wire
shape ChatView.kt:1148 has handled as a life signal since 2026-07-13 and never
received. No new key, no new route, no new lifecycle: a strict subset of WS3
item 3. WS3's run registry is untouched and stays Will's.

Proof (live Anthropic key, real research mission, scratch HOME + free port):
  round-9 baseline — ledger EMPTY for the whole 59.7 s leg
  this build       — {"i":0,"t":"","tool":"__working__"} visible at 18.6 s of a
                     70.0 s leg; both builds returned correct ~4.9 KB answers

Regression: prompt-matrix gate 32/32 on this build (round-9 baseline also 32/32
under the same recipe, so the score is not a build artifact). Soul contract
gate PASS — 27/27 routes, immutability clean. neuron#111 miscompile guard: 0
sites in the generated amalgam this binary was compiled from.

NOT included, deliberately: the regenerated dist/soul.c. CI compiles that file,
so production stays exposed until it is regenerated — the same open ask as
neuron#111 / ui#209. The regen recipe is now known and recorded; landing it is
Will's call, per BUILD-HYGIENE.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

tim.lingo merged commit 4936099c39 into main

2026-08-07 15:54:20 +00:00

tim.lingo referenced this issue from a commit

2026-08-07 16:16:50 +00:00

chore(engine): regenerate dist/soul.c from merged main — the build input was 4 days behind the source

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/neuron#127