runtime: a client hanging up must not kill the server #153

2026-08-16T18:38:17Z

will.anderson commented

2026-08-16 18:38:17 +00:00

Prod has been crash-looping every ~10 minutes for three days. This is the cause, and it is one line.

Root cause

lang/runtime/el_runtime.c:1341

ssize_t w = send(fd, p, left, 0);

Flags 0, and zero SIGPIPE suppression anywhere in the runtime — grep -rn "SIGPIPE\|MSG_NOSIGNAL\|SO_NOSIGPIPE\|sigaction" lang/runtime/ engram/src/ returns 0 hits. send() to a socket whose peer has closed raises SIGPIPE; default disposition is terminate. Any abandoned request kills the server.

The chain, each link measured

#	link	evidence
1	`ai.neuron.engram-tick.plist` has `StartInterval 600`	matches the restart cadence exactly
2	it runs `curl -s -m10 -X POST /api/tick`	hard 10s client timeout
3	`/api/tick` regularly exceeds 10s, so curl hangs up	tick log shows an empty response in 279 of 448 ticks
4	engram later writes into the closed socket	`http_send_all` → `send(fd, …, 0)`
5	SIGPIPE, unhandled → terminate	`launchctl list ai.neuron.engram` → `LastExitStatus = 13` (raw wait status 13 = signal 13)
6	`KeepAlive: true` restarts it	254 restarts, 2026-08-13T19:37Z → 2026-08-16T18:16Z

Why nothing was ever logged: a signal-killed process never reaches a line where it could log. ~/.neuron/logs/engram.log is 2.7 MB of nothing but repeated [http] listening on [::]:8742. That is this bug's signature, not a logging gap.

The detail worth pausing on

http_send_all's dead-peer branch was already correct:

if (w <= 0) return -1;

It had never once executed. The signal killed the process before send() could return -1/EPIPE. The error handling was written and unreachable.

The fix

Per-socket / per-call suppression — SO_NOSIGPIPE (macOS/BSD), MSG_NOSIGNAL (Linux) — not a global signal(SIGPIPE, SIG_IGN). Both stop the crash; only this one has zero blast radius. A global handler would also change the disposition of writes to ordinary pipes in the exec/fs paths, which this fix has no business touching. Every HTTP response funnels through http_send_all, so one function covers all three accept loops.

Negative control (invariant §8.6)

Same data, same endpoint, same probe, two builds:

unpatched  ->  SERVER DIED 43.3s after the hang-up          (exit 1)
patched    ->  survived 2 rounds, still listening, serving   (exit 0)

The obvious version of this probe is useless, and I shipped it first. Closing with SO_LINGER=0 emits RST; the server's first write then returns ECONNRESET rather than raising SIGPIPE, http_send_all stops cleanly, and the test passes on the unpatched build. It has to be a normal close (FIN): first write succeeds, peer answers RST, second write raises the signal. And liveness must be checked ~45s later — the kill lands when the server reaches its write, not when the client leaves. My first probe got both wrong and reported a false pass on the broken binary. Both traps are documented in the test.

engram/test/test_http_sigpipe.c + run_http_sigpipe_test.sh. C and shell, no Python.

Not fixed here

/api/activate takes ~16s and /api/tick exceeds 10s at all; the heartbeat's -m10 is shorter than the work it asks for. Fixing SIGPIPE stops the crash but not the empty responses — that is a separate and still-open problem.

Also worth someone deciding: prod runs ~/.neuron/bin/engram.memguard-20260815-210116, a hand-labelled binary with no commit behind it. "memguard" was a guess that this was a memory leak. It was not — RSS 2.34 GB is just the resident 2.2 GB .egm, stable across restarts. The wrong hypothesis has been carrying prod for a day because a filename got the same durability as a fact.

Nothing was deployed. All work ran against APFS clones on ports 18752/18753.

Prod has been crash-looping every ~10 minutes for three days. This is the cause, and it is one line. ## Root cause `lang/runtime/el_runtime.c:1341` ```c ssize_t w = send(fd, p, left, 0); ``` Flags `0`, and **zero** SIGPIPE suppression anywhere in the runtime — `grep -rn "SIGPIPE\|MSG_NOSIGNAL\|SO_NOSIGPIPE\|sigaction" lang/runtime/ engram/src/` returns 0 hits. `send()` to a socket whose peer has closed raises SIGPIPE; default disposition is *terminate*. Any abandoned request kills the server. ## The chain, each link measured | # | link | evidence | |---|---|---| | 1 | `ai.neuron.engram-tick.plist` has `StartInterval 600` | matches the restart cadence exactly | | 2 | it runs `curl -s **-m10** -X POST /api/tick` | hard 10s client timeout | | 3 | `/api/tick` regularly exceeds 10s, so curl hangs up | tick log shows an **empty** response in **279 of 448** ticks | | 4 | engram later writes into the closed socket | `http_send_all` → `send(fd, …, 0)` | | 5 | SIGPIPE, unhandled → terminate | `launchctl list ai.neuron.engram` → **`LastExitStatus = 13`** (raw wait status 13 = signal 13) | | 6 | `KeepAlive: true` restarts it | **254 restarts**, 2026-08-13T19:37Z → 2026-08-16T18:16Z | **Why nothing was ever logged:** a signal-killed process never reaches a line where it could log. `~/.neuron/logs/engram.log` is 2.7 MB of nothing but repeated `[http] listening on [::]:8742`. That is this bug's signature, not a logging gap. ## The detail worth pausing on `http_send_all`'s dead-peer branch was already correct: ```c if (w <= 0) return -1; ``` It had **never once executed**. The signal killed the process before `send()` could return `-1/EPIPE`. The error handling was written and unreachable. ## The fix Per-socket / per-call suppression — `SO_NOSIGPIPE` (macOS/BSD), `MSG_NOSIGNAL` (Linux) — not a global `signal(SIGPIPE, SIG_IGN)`. Both stop the crash; only this one has zero blast radius. A global handler would also change the disposition of writes to ordinary pipes in the exec/fs paths, which this fix has no business touching. Every HTTP response funnels through `http_send_all`, so one function covers all three accept loops. ## Negative control (invariant §8.6) Same data, same endpoint, same probe, two builds: ``` unpatched -> SERVER DIED 43.3s after the hang-up (exit 1) patched -> survived 2 rounds, still listening, serving (exit 0) ``` **The obvious version of this probe is useless, and I shipped it first.** Closing with `SO_LINGER=0` emits RST; the server's *first* write then returns `ECONNRESET` rather than raising SIGPIPE, `http_send_all` stops cleanly, and the test **passes on the unpatched build**. It has to be a normal close (FIN): first write succeeds, peer answers RST, second write raises the signal. And liveness must be checked ~45s later — the kill lands when the server reaches its write, not when the client leaves. My first probe got both wrong and reported a false pass on the broken binary. Both traps are documented in the test. `engram/test/test_http_sigpipe.c` + `run_http_sigpipe_test.sh`. C and shell, no Python. ## Not fixed here `/api/activate` takes ~16s and `/api/tick` exceeds 10s at all; the heartbeat's `-m10` is shorter than the work it asks for. Fixing SIGPIPE stops the crash but not the empty responses — that is a separate and still-open problem. Also worth someone deciding: prod runs `~/.neuron/bin/engram.memguard-20260815-210116`, a hand-labelled binary with no commit behind it. "memguard" was a guess that this was a memory leak. It was not — RSS 2.34 GB is just the resident 2.2 GB `.egm`, stable across restarts. The wrong hypothesis has been carrying prod for a day because a filename got the same durability as a fact. Nothing was deployed. All work ran against APFS clones on ports 18752/18753.

will.anderson added 1 commit 2026-08-16 18:38:19 +00:00

runtime: a client hanging up must not kill the server

El SDK CI - dev / build-and-test (pull_request) Failing after 3m47s

Details

dab081c8f8

send() to a socket whose peer has closed raises SIGPIPE, whose default
disposition is terminate, and nothing in this runtime suppressed it. grep for
SIGPIPE|MSG_NOSIGNAL|SO_NOSIGPIPE|sigaction across lang/runtime/ and engram/src/
returned zero hits. So any abandoned request could kill the process, and one
did, every ten minutes, for three days.

Measured in production, not inferred:
  - 254 restarts between 2026-08-13T19:37Z and 2026-08-16T18:16Z
  - `launchctl list ai.neuron.engram` -> LastExitStatus = 13
    (raw wait status 13 = killed by signal 13 = SIGPIPE)
  - trigger: ai.neuron.engram-tick.plist, StartInterval 600 — matching the
    restart cadence exactly — running `curl -s -m10 -X POST /api/tick`. When
    /api/tick exceeded curl's 10s timeout the client hung up, and the eventual
    response write killed the server.
  - the tick log recorded an EMPTY response in 279 of 448 ticks.
  - launchd KeepAlive:true restarted it each time, so the loop was invisible
    except as a PID that kept changing.

Nothing was ever logged about it and nothing could have been: a signal-killed
process never reaches a line where it could write one. engram.log is 2.7 MB of
nothing but repeated "[http] listening on [::]:8742". That is the signature of
this bug, not a gap in logging.

Note what was already correct: http_send_all's `w <= 0` branch handles a dead
peer properly. It had never once executed, because the signal killed the process
before send() could return -1/EPIPE. The error handling was written and
unreachable.

The suppression is per-socket/per-call (SO_NOSIGPIPE on macOS/BSD, MSG_NOSIGNAL
on Linux) rather than a global signal(SIGPIPE, SIG_IGN). Both fix the crash;
only this one has zero blast radius. A global handler would also change the
disposition of writes to ordinary pipes in the exec/fs paths, which this fix has
no business touching. Every HTTP response funnels through http_send_all, so one
function covers all three accept loops.

Negative control (invariant 8.6). Same data, same endpoint, same probe:
  unpatched -> SERVER DIED 43.3s after the hang-up   (exit 1)
  patched   -> survived 2 rounds, still listening, still serving (exit 0)

The probe is C, not Python, and it is worth saying why the obvious version of it
is useless: closing with SO_LINGER=0 emits RST, the server's FIRST write returns
ECONNRESET rather than raising SIGPIPE, http_send_all stops cleanly, and the test
PASSES ON THE UNPATCHED BUILD. It has to be a normal close (FIN) — first write
succeeds, peer answers RST, second write raises the signal. And liveness must be
checked ~45s later, not immediately: the kill lands when the server reaches its
write, not when the client leaves. My first probe got both wrong and reported a
false pass.

Not fixed here, and worth separate attention: /api/activate takes ~16s and
/api/tick exceeds 10s at all, and the heartbeat's -m10 is shorter than the work
it asks for.

will.anderson closed this pull request

2026-08-16 18:41:57 +00:00

will.anderson commented

2026-08-16 18:42:00 +00:00

Closing as a duplicate. #151 landed the same fix at 13:25:14 while I was building this, from another session under the same identity — same diagnosis, same LastExitStatus 13, same StartInterval 600 trigger. Theirs is a superset of mine: per-socket SO_NOSIGPIPE + MSG_NOSIGNAL, plus a process-wide SIG_IGN backstop I had deliberately left out for blast radius. Their portability argument is better than my purity argument. Nothing here needs to land.

Two things survive and I am handling them separately:

The fix shipped with no test. git show cace6a5 --name-only is one file, el_runtime.c, and engram/test/ has nothing matching sigpipe or http. That is invariant §8.6 unmet on a fix for a bug that killed production 254 times. I have a probe that fails on the pre-fix build (SERVER DIED 43.3s after the hang-up) and passes on merged dev (survived 2 rounds, still serving) — so #151 is now independently verified, which it was not before. I am holding it until the testing-framework question below is settled, because EL now has native testing and a C probe may be the wrong artifact.

Two sessions did this same work in parallel, neither aware of the other. That is the same failure the spec names in §7 — 'dreaming has seven implementations' — arriving in the workflow rather than the code. Worth someone's attention independently of this PR.

Closing as a duplicate. #151 landed the same fix at 13:25:14 while I was building this, from another session under the same identity — same diagnosis, same LastExitStatus 13, same StartInterval 600 trigger. Theirs is a superset of mine: per-socket SO_NOSIGPIPE + MSG_NOSIGNAL, plus a process-wide SIG_IGN backstop I had deliberately left out for blast radius. Their portability argument is better than my purity argument. Nothing here needs to land. Two things survive and I am handling them separately: **The fix shipped with no test.** `git show cace6a5 --name-only` is one file, `el_runtime.c`, and `engram/test/` has nothing matching sigpipe or http. That is invariant §8.6 unmet on a fix for a bug that killed production 254 times. I have a probe that fails on the pre-fix build (SERVER DIED 43.3s after the hang-up) and passes on merged dev (survived 2 rounds, still serving) — so #151 is now independently verified, which it was not before. I am holding it until the testing-framework question below is settled, because EL now has native testing and a C probe may be the wrong artifact. **Two sessions did this same work in parallel, neither aware of the other.** That is the same failure the spec names in §7 — 'dreaming has seven implementations' — arriving in the workflow rather than the code. Worth someone's attention independently of this PR.

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#153