runtime: a client hanging up must not kill the server #153

2026-08-16T18:38:17Z

will.anderson commented

2026-08-16 18:38:17 +00:00

Prod has been crash-looping every ~10 minutes for three days. This is the cause, and it is one line.

Root cause

lang/runtime/el_runtime.c:1341

ssize_t w = send(fd, p, left, 0);

Flags 0, and zero SIGPIPE suppression anywhere in the runtime — grep -rn "SIGPIPE\|MSG_NOSIGNAL\|SO_NOSIGPIPE\|sigaction" lang/runtime/ engram/src/ returns 0 hits. send() to a socket whose peer has closed raises SIGPIPE; default disposition is terminate. Any abandoned request kills the server.

The chain, each link measured

#	link	evidence
1	`ai.neuron.engram-tick.plist` has `StartInterval 600`	matches the restart cadence exactly
2	it runs `curl -s -m10 -X POST /api/tick`	hard 10s client timeout
3	`/api/tick` regularly exceeds 10s, so curl hangs up	tick log shows an empty response in 279 of 448 ticks
4	engram later writes into the closed socket	`http_send_all` → `send(fd, …, 0)`
5	SIGPIPE, unhandled → terminate	`launchctl list ai.neuron.engram` → `LastExitStatus = 13` (raw wait status 13 = signal 13)
6	`KeepAlive: true` restarts it	254 restarts, 2026-08-13T19:37Z → 2026-08-16T18:16Z

Why nothing was ever logged: a signal-killed process never reaches a line where it could log. ~/.neuron/logs/engram.log is 2.7 MB of nothing but repeated [http] listening on [::]:8742. That is this bug's signature, not a logging gap.

The detail worth pausing on

http_send_all's dead-peer branch was already correct:

if (w <= 0) return -1;

It had never once executed. The signal killed the process before send() could return -1/EPIPE. The error handling was written and unreachable.

The fix

Per-socket / per-call suppression — SO_NOSIGPIPE (macOS/BSD), MSG_NOSIGNAL (Linux) — not a global signal(SIGPIPE, SIG_IGN). Both stop the crash; only this one has zero blast radius. A global handler would also change the disposition of writes to ordinary pipes in the exec/fs paths, which this fix has no business touching. Every HTTP response funnels through http_send_all, so one function covers all three accept loops.

Negative control (invariant §8.6)

Same data, same endpoint, same probe, two builds:

unpatched  ->  SERVER DIED 43.3s after the hang-up          (exit 1)
patched    ->  survived 2 rounds, still listening, serving   (exit 0)

The obvious version of this probe is useless, and I shipped it first. Closing with SO_LINGER=0 emits RST; the server's first write then returns ECONNRESET rather than raising SIGPIPE, http_send_all stops cleanly, and the test passes on the unpatched build. It has to be a normal close (FIN): first write succeeds, peer answers RST, second write raises the signal. And liveness must be checked ~45s later — the kill lands when the server reaches its write, not when the client leaves. My first probe got both wrong and reported a false pass on the broken binary. Both traps are documented in the test.

engram/test/test_http_sigpipe.c + run_http_sigpipe_test.sh. C and shell, no Python.

Not fixed here

/api/activate takes ~16s and /api/tick exceeds 10s at all; the heartbeat's -m10 is shorter than the work it asks for. Fixing SIGPIPE stops the crash but not the empty responses — that is a separate and still-open problem.

Also worth someone deciding: prod runs ~/.neuron/bin/engram.memguard-20260815-210116, a hand-labelled binary with no commit behind it. "memguard" was a guess that this was a memory leak. It was not — RSS 2.34 GB is just the resident 2.2 GB .egm, stable across restarts. The wrong hypothesis has been carrying prod for a day because a filename got the same durability as a fact.

Nothing was deployed. All work ran against APFS clones on ports 18752/18753.

Prod has been crash-looping every ~10 minutes for three days. This is the cause, and it is one line. ## Root cause `lang/runtime/el_runtime.c:1341` ```c ssize_t w = send(fd, p, left, 0); ``` Flags `0`, and **zero** SIGPIPE suppression anywhere in the runtime — `grep -rn "SIGPIPE\|MSG_NOSIGNAL\|SO_NOSIGPIPE\|sigaction" lang/runtime/ engram/src/` returns 0 hits. `send()` to a socket whose peer has closed raises SIGPIPE; default disposition is *terminate*. Any abandoned request kills the server. ## The chain, each link measured | # | link | evidence | |---|---|---| | 1 | `ai.neuron.engram-tick.plist` has `StartInterval 600` | matches the restart cadence exactly | | 2 | it runs `curl -s **-m10** -X POST /api/tick` | hard 10s client timeout | | 3 | `/api/tick` regularly exceeds 10s, so curl hangs up | tick log shows an **empty** response in **279 of 448** ticks | | 4 | engram later writes into the closed socket | `http_send_all` → `send(fd, …, 0)` | | 5 | SIGPIPE, unhandled → terminate | `launchctl list ai.neuron.engram` → **`LastExitStatus = 13`** (raw wait status 13 = signal 13) | | 6 | `KeepAlive: true` restarts it | **254 restarts**, 2026-08-13T19:37Z → 2026-08-16T18:16Z | **Why nothing was ever logged:** a signal-killed process never reaches a line where it could log. `~/.neuron/logs/engram.log` is 2.7 MB of nothing but repeated `[http] listening on [::]:8742`. That is this bug's signature, not a logging gap. ## The detail worth pausing on `http_send_all`'s dead-peer branch was already correct: ```c if (w <= 0) return -1; ``` It had **never once executed**. The signal killed the process before `send()` could return `-1/EPIPE`. The error handling was written and unreachable. ## The fix Per-socket / per-call suppression — `SO_NOSIGPIPE` (macOS/BSD), `MSG_NOSIGNAL` (Linux) — not a global `signal(SIGPIPE, SIG_IGN)`. Both stop the crash; only this one has zero blast radius. A global handler would also change the disposition of writes to ordinary pipes in the exec/fs paths, which this fix has no business touching. Every HTTP response funnels through `http_send_all`, so one function covers all three accept loops. ## Negative control (invariant §8.6) Same data, same endpoint, same probe, two builds: ``` unpatched -> SERVER DIED 43.3s after the hang-up (exit 1) patched -> survived 2 rounds, still listening, serving (exit 0) ``` **The obvious version of this probe is useless, and I shipped it first.** Closing with `SO_LINGER=0` emits RST; the server's *first* write then returns `ECONNRESET` rather than raising SIGPIPE, `http_send_all` stops cleanly, and the test **passes on the unpatched build**. It has to be a normal close (FIN): first write succeeds, peer answers RST, second write raises the signal. And liveness must be checked ~45s later — the kill lands when the server reaches its write, not when the client leaves. My first probe got both wrong and reported a false pass on the broken binary. Both traps are documented in the test. `engram/test/test_http_sigpipe.c` + `run_http_sigpipe_test.sh`. C and shell, no Python. ## Not fixed here `/api/activate` takes ~16s and `/api/tick` exceeds 10s at all; the heartbeat's `-m10` is shorter than the work it asks for. Fixing SIGPIPE stops the crash but not the empty responses — that is a separate and still-open problem. Also worth someone deciding: prod runs `~/.neuron/bin/engram.memguard-20260815-210116`, a hand-labelled binary with no commit behind it. "memguard" was a guess that this was a memory leak. It was not — RSS 2.34 GB is just the resident 2.2 GB `.egm`, stable across restarts. The wrong hypothesis has been carrying prod for a day because a filename got the same durability as a fact. Nothing was deployed. All work ran against APFS clones on ports 18752/18753.

will.anderson added 1 commit 2026-08-16 18:38:19 +00:00

runtime: a client hanging up must not kill the server

El SDK CI - dev / build-and-test (pull_request) Failing after 3m47s

Details

dab081c8f8

send() to a socket whose peer has closed raises SIGPIPE, whose default
disposition is terminate, and nothing in this runtime suppressed it. grep for
SIGPIPE|MSG_NOSIGNAL|SO_NOSIGPIPE|sigaction across lang/runtime/ and engram/src/
returned zero hits. So any abandoned request could kill the process, and one
did, every ten minutes, for three days.

Measured in production, not inferred:
  - 254 restarts between 2026-08-13T19:37Z and 2026-08-16T18:16Z
  - `launchctl list ai.neuron.engram` -> LastExitStatus = 13
    (raw wait status 13 = killed by signal 13 = SIGPIPE)
  - trigger: ai.neuron.engram-tick.plist, StartInterval 600 — matching the
    restart cadence exactly — running `curl -s -m10 -X POST /api/tick`. When
    /api/tick exceeded curl's 10s timeout the client hung up, and the eventual
    response write killed the server.
  - the tick log recorded an EMPTY response in 279 of 448 ticks.
  - launchd KeepAlive:true restarted it each time, so the loop was invisible
    except as a PID that kept changing.

Nothing was ever logged about it and nothing could have been: a signal-killed
process never reaches a line where it could write one. engram.log is 2.7 MB of
nothing but repeated "[http] listening on [::]:8742". That is the signature of
this bug, not a gap in logging.

Note what was already correct: http_send_all's `w <= 0` branch handles a dead
peer properly. It had never once executed, because the signal killed the process
before send() could return -1/EPIPE. The error handling was written and
unreachable.

The suppression is per-socket/per-call (SO_NOSIGPIPE on macOS/BSD, MSG_NOSIGNAL
on Linux) rather than a global signal(SIGPIPE, SIG_IGN). Both fix the crash;
only this one has zero blast radius. A global handler would also change the
disposition of writes to ordinary pipes in the exec/fs paths, which this fix has
no business touching. Every HTTP response funnels through http_send_all, so one
function covers all three accept loops.

Negative control (invariant 8.6). Same data, same endpoint, same probe:
  unpatched -> SERVER DIED 43.3s after the hang-up   (exit 1)
  patched   -> survived 2 rounds, still listening, still serving (exit 0)

The probe is C, not Python, and it is worth saying why the obvious version of it
is useless: closing with SO_LINGER=0 emits RST, the server's FIRST write returns
ECONNRESET rather than raising SIGPIPE, http_send_all stops cleanly, and the test
PASSES ON THE UNPATCHED BUILD. It has to be a normal close (FIN) — first write
succeeds, peer answers RST, second write raises the signal. And liveness must be
checked ~45s later, not immediately: the kill lands when the server reaches its
write, not when the client leaves. My first probe got both wrong and reported a
false pass.

Not fixed here, and worth separate attention: /api/activate takes ~16s and
/api/tick exceeds 10s at all, and the heartbeat's -m10 is shorter than the work
it asks for.

will.anderson closed this pull request

2026-08-16 18:41:57 +00:00

will.anderson commented

2026-08-16 18:42:00 +00:00

Closing as a duplicate. #151 landed the same fix at 13:25:14 while I was building this, from another session under the same identity — same diagnosis, same LastExitStatus 13, same StartInterval 600 trigger. Theirs is a superset of mine: per-socket SO_NOSIGPIPE + MSG_NOSIGNAL, plus a process-wide SIG_IGN backstop I had deliberately left out for blast radius. Their portability argument is better than my purity argument. Nothing here needs to land.

Two things survive and I am handling them separately:

The fix shipped with no test. git show cace6a5 --name-only is one file, el_runtime.c, and engram/test/ has nothing matching sigpipe or http. That is invariant §8.6 unmet on a fix for a bug that killed production 254 times. I have a probe that fails on the pre-fix build (SERVER DIED 43.3s after the hang-up) and passes on merged dev (survived 2 rounds, still serving) — so #151 is now independently verified, which it was not before. I am holding it until the testing-framework question below is settled, because EL now has native testing and a C probe may be the wrong artifact.

Two sessions did this same work in parallel, neither aware of the other. That is the same failure the spec names in §7 — 'dreaming has seven implementations' — arriving in the workflow rather than the code. Worth someone's attention independently of this PR.

Closing as a duplicate. #151 landed the same fix at 13:25:14 while I was building this, from another session under the same identity — same diagnosis, same LastExitStatus 13, same StartInterval 600 trigger. Theirs is a superset of mine: per-socket SO_NOSIGPIPE + MSG_NOSIGNAL, plus a process-wide SIG_IGN backstop I had deliberately left out for blast radius. Their portability argument is better than my purity argument. Nothing here needs to land. Two things survive and I am handling them separately: **The fix shipped with no test.** `git show cace6a5 --name-only` is one file, `el_runtime.c`, and `engram/test/` has nothing matching sigpipe or http. That is invariant §8.6 unmet on a fix for a bug that killed production 254 times. I have a probe that fails on the pre-fix build (SERVER DIED 43.3s after the hang-up) and passes on merged dev (survived 2 rounds, still serving) — so #151 is now independently verified, which it was not before. I am holding it until the testing-framework question below is settled, because EL now has native testing and a C probe may be the wrong artifact. **Two sessions did this same work in parallel, neither aware of the other.** That is the same failure the spec names in §7 — 'dreaming has seven implementations' — arriving in the workflow rather than the code. Worth someone's attention independently of this PR.

This repo is archived. You cannot comment on pull requests.

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el-retired#153