runtime: a client hanging up must not kill the server
El SDK CI - dev / build-and-test (pull_request) Failing after 3m47s

send() to a socket whose peer has closed raises SIGPIPE, whose default
disposition is terminate, and nothing in this runtime suppressed it. grep for
SIGPIPE|MSG_NOSIGNAL|SO_NOSIGPIPE|sigaction across lang/runtime/ and engram/src/
returned zero hits. So any abandoned request could kill the process, and one
did, every ten minutes, for three days.

Measured in production, not inferred:
  - 254 restarts between 2026-08-13T19:37Z and 2026-08-16T18:16Z
  - `launchctl list ai.neuron.engram` -> LastExitStatus = 13
    (raw wait status 13 = killed by signal 13 = SIGPIPE)
  - trigger: ai.neuron.engram-tick.plist, StartInterval 600 — matching the
    restart cadence exactly — running `curl -s -m10 -X POST /api/tick`. When
    /api/tick exceeded curl's 10s timeout the client hung up, and the eventual
    response write killed the server.
  - the tick log recorded an EMPTY response in 279 of 448 ticks.
  - launchd KeepAlive:true restarted it each time, so the loop was invisible
    except as a PID that kept changing.

Nothing was ever logged about it and nothing could have been: a signal-killed
process never reaches a line where it could write one. engram.log is 2.7 MB of
nothing but repeated "[http] listening on [::]:8742". That is the signature of
this bug, not a gap in logging.

Note what was already correct: http_send_all's `w <= 0` branch handles a dead
peer properly. It had never once executed, because the signal killed the process
before send() could return -1/EPIPE. The error handling was written and
unreachable.

The suppression is per-socket/per-call (SO_NOSIGPIPE on macOS/BSD, MSG_NOSIGNAL
on Linux) rather than a global signal(SIGPIPE, SIG_IGN). Both fix the crash;
only this one has zero blast radius. A global handler would also change the
disposition of writes to ordinary pipes in the exec/fs paths, which this fix has
no business touching. Every HTTP response funnels through http_send_all, so one
function covers all three accept loops.

Negative control (invariant 8.6). Same data, same endpoint, same probe:
  unpatched -> SERVER DIED 43.3s after the hang-up   (exit 1)
  patched   -> survived 2 rounds, still listening, still serving (exit 0)

The probe is C, not Python, and it is worth saying why the obvious version of it
is useless: closing with SO_LINGER=0 emits RST, the server's FIRST write returns
ECONNRESET rather than raising SIGPIPE, http_send_all stops cleanly, and the test
PASSES ON THE UNPATCHED BUILD. It has to be a normal close (FIN) — first write
succeeds, peer answers RST, second write raises the signal. And liveness must be
checked ~45s later, not immediately: the kill lands when the server reaches its
write, not when the client leaves. My first probe got both wrong and reported a
false pass.

Not fixed here, and worth separate attention: /api/activate takes ~16s and
/api/tick exceeds 10s at all, and the heartbeat's -m10 is shorter than the work
it asks for.
This commit is contained in:
Neuron
2026-08-16 13:37:49 -05:00
parent d41645388a
commit dab081c8f8
3 changed files with 207 additions and 2 deletions
+56 -2
View File
@@ -1335,10 +1335,64 @@ static const char* http_reason_phrase(int status) {
}
}
/* Best-effort send with retry on partial writes. */
/* ── SIGPIPE MUST NEVER REACH THE PROCESS (2026-08-16) ───────────────────────
*
* A client that hangs up before we finish writing its response was KILLING THE
* SERVER. `send()` to a socket whose peer has closed raises SIGPIPE, whose
* default disposition is terminate, and nothing in this runtime suppressed it:
* `grep -rn "SIGPIPE\|MSG_NOSIGNAL\|SO_NOSIGPIPE\|sigaction" lang/runtime/`
* returned zero hits.
*
* MEASURED IN PRODUCTION. The engram restarted 254 times between 2026-08-13T19:37
* and 2026-08-16T18:16 on a ~10 minute cadence, with `launchctl list
* ai.neuron.engram` reporting `LastExitStatus = 13` raw wait status 13, i.e.
* killed by signal 13, SIGPIPE. The trigger was ai.neuron.engram-tick.plist
* (`StartInterval 600`, matching the cadence exactly) running
* `curl -s -m10 -X POST /api/tick`: when /api/tick exceeded curl's 10 second
* timeout the client hung up, and the response write then killed the process.
* The tick log recorded an EMPTY response curl having returned nothing in
* 279 of 448 ticks. launchd `KeepAlive: true` restarted it each time, so the
* loop was invisible except as a PID that kept changing.
*
* Nothing was ever logged about it, and could not have been: a signal-killed
* process never reaches a line where it could write one. ~/.neuron/logs/engram.log
* is 2.7 MB of nothing but repeated "[http] listening on [::]:8742" that is the
* signature of this bug, not a logging gap.
*
* Note what was already correct below: the `w <= 0` branch handles a dead peer
* properly. It had simply never once executed, because the signal killed the
* process before send() could return -1/EPIPE. The error handling was written
* and unreachable.
*
* The suppression is deliberately PER-SOCKET / PER-CALL rather than a global
* signal(SIGPIPE, SIG_IGN). Both forms fix the crash; only this one has zero
* blast radius. A global handler would also change the disposition of writes to
* ordinary pipes in the exec/fs paths, which is a semantic change to code this
* fix has no business touching. SO_NOSIGPIPE (macOS/BSD) and MSG_NOSIGNAL
* (Linux) each affect only the socket and call sites named here, and each is a
* no-op on the platform that lacks it. Every HTTP response in the runtime funnels
* through this one function, so covering it covers all three accept loops. */
#ifdef MSG_NOSIGNAL
# define EL_SEND_FLAGS MSG_NOSIGNAL /* Linux: suppress per send() call */
#else
# define EL_SEND_FLAGS 0
#endif
static void el_sock_nosigpipe(int fd) {
#ifdef SO_NOSIGPIPE /* macOS/BSD: suppress per socket */
int on = 1;
setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &on, sizeof(on));
#else
(void)fd;
#endif
}
/* Best-effort send with retry on partial writes. A peer that has gone away now
* surfaces as a -1 return (EPIPE) and drops the connection, instead of killing
* the server. */
static int http_send_all(int fd, const char* p, size_t left) {
el_sock_nosigpipe(fd);
while (left > 0) {
ssize_t w = send(fd, p, left, 0);
ssize_t w = send(fd, p, left, EL_SEND_FLAGS);
if (w <= 0) return -1;
p += w; left -= (size_t)w;
}