runtime: a disconnecting client must not kill the server #151
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: neuron-technologies/el#151
Reference in New Issue
Block a user
Delete Branch "fix/sigpipe"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
There was no SIGPIPE handling anywhere in this runtime — no signal disposition, no
MSG_NOSIGNAL, noSO_NOSIGPIPE, andsend()called with bare flags. The default disposition of SIGPIPE is to terminate the process, so any client that hangs up mid-response takes the whole engram with it.This is not hypothetical
Production has restarted 254 times since 2026-08-13T19:37, at a flat ~10 minute cadence:
Intervals of 10m09s–10m12s, not 10m00s. That excess is the whole story:
ai.neuron.engram-tickhasStartInterval600engram-tick.sh:13—curl -s -m10 -X POST .../api/tickThe beat does not finish within 10s over 13,634 nodes. Curl waits its full timeout and closes. The engram writes the tick response to a dead socket, takes SIGPIPE, and dies.
launchdKeepAlive restarts it — so the failure presents as a mysterious restart rather than a crash, andengram.logrecords nothing but[http] listening on254 times with no exit reason.launchctl listconfirms the last exit as -13.The ticker has been killing the engram every ten minutes for three days.
Root cause is one level out
Consolidation had no owner, so an external ticker was created to poke it — and the ticker is what kills it. This PR does not fix that; it makes the process survivable while it is fixed. See
lang/spec/correspondence-and-censorship.md§5, §7.The fix
Two layers, because neither alone is portable:
SO_NOSIGPIPEper accepted socket (Darwin/BSD) andMSG_NOSIGNALper send (Linux) — the signal is never raised for socket writes at all.SIG_IGNbackstop, installed once and idempotent, for platforms and paths with neither. With the signal ignored,send()returns-1/EPIPEand the existing error path closes the connection.Also retries
send()onEINTR, which the previous loop treated as fatal.This is an exemption in the §8 sense: the write never checked whether the peer was still there, and the consequence of not checking was fatal rather than merely wrong.
There was no SIGPIPE handling anywhere in this runtime: no signal disposition, no MSG_NOSIGNAL, no SO_NOSIGPIPE, and send() called with bare flags. The default disposition of SIGPIPE is to TERMINATE THE PROCESS, so any client that hangs up mid-response takes the whole engram with it. MEASURED, and it is not hypothetical. Production has restarted 254 times since 2026-08-13T19:37 at a flat ~10 minute cadence: 17:05:18 17:15:29 17:25:38 17:35:50 17:46:00 17:56:10 18:06:22 18:16:30 Intervals of 10m09s-10m12s, not 10m00s. That excess is the whole story: ai.neuron.engram-tick has StartInterval 600, and engram-tick.sh:13 calls curl -s -m10 -X POST .../api/tick The beat does not finish within 10s over 13,634 nodes, so curl waits its full timeout and closes. The engram then writes the tick response to a dead socket, takes SIGPIPE, and dies. launchd KeepAlive restarts it, so the failure presents as a mysterious restart rather than a crash — and ~/.neuron/logs/engram.log records nothing but "[http] listening on" 254 times, with no exit reason. launchctl list confirms the last exit as -13. Root cause is one level out: consolidation had no owner, so an external ticker was created to poke it, and the ticker is what kills it. The fix here does not address that; it makes the process survivable while it is addressed. Two layers, because neither alone is portable: - SO_NOSIGPIPE per accepted socket (Darwin/BSD) and MSG_NOSIGNAL per send (Linux), so the signal is never raised for socket writes at all. - A process-wide SIG_IGN backstop, installed once and idempotent, for platforms and paths with neither. With the signal ignored, send() returns -1/EPIPE and the existing error path closes the connection. Also retries send() on EINTR, which the previous loop treated as fatal. This is an exemption in the sense of lang/spec §8: the write never checked whether the peer was still there, and the consequence of not checking was fatal rather than merely wrong.Negative control fires.
Same data (separate APFS clones), same induction, PID verified against the bound port before each probe.
A first attempt did NOT reproduce and is worth recording, because it nearly caused a merge on no evidence:
curl -m1against a 26 MB GET left both builds alive. A client timing out politely often loses the race — the response reaches the socket buffer before the FIN lands, which is exactly why 279 timed-out ticks produced only 254 restarts.The control that works uses
SO_LINGER 0, soclose()sends an RST and the peer is provably gone before the server writes: 12 aborted requests per build, across both the large-GET path and the real production trigger (POST /api/tick).