Neuron
cace6a5ebf
El SDK CI - dev / build-and-test (pull_request) Failing after 13m45s
There was no SIGPIPE handling anywhere in this runtime: no signal
disposition, no MSG_NOSIGNAL, no SO_NOSIGPIPE, and send() called with bare
flags. The default disposition of SIGPIPE is to TERMINATE THE PROCESS, so
any client that hangs up mid-response takes the whole engram with it.
MEASURED, and it is not hypothetical. Production has restarted 254 times
since 2026-08-13T19:37 at a flat ~10 minute cadence:
17:05:18 17:15:29 17:25:38 17:35:50 17:46:00 17:56:10 18:06:22 18:16:30
Intervals of 10m09s-10m12s, not 10m00s. That excess is the whole story:
ai.neuron.engram-tick has StartInterval 600, and engram-tick.sh:13 calls
curl -s -m10 -X POST .../api/tick
The beat does not finish within 10s over 13,634 nodes, so curl waits its
full timeout and closes. The engram then writes the tick response to a dead
socket, takes SIGPIPE, and dies. launchd KeepAlive restarts it, so the
failure presents as a mysterious restart rather than a crash — and
~/.neuron/logs/engram.log records nothing but "[http] listening on" 254
times, with no exit reason. launchctl list confirms the last exit as -13.
Root cause is one level out: consolidation had no owner, so an external
ticker was created to poke it, and the ticker is what kills it. The fix
here does not address that; it makes the process survivable while it is
addressed.
Two layers, because neither alone is portable:
- SO_NOSIGPIPE per accepted socket (Darwin/BSD) and MSG_NOSIGNAL per send
(Linux), so the signal is never raised for socket writes at all.
- A process-wide SIG_IGN backstop, installed once and idempotent, for
platforms and paths with neither. With the signal ignored, send()
returns -1/EPIPE and the existing error path closes the connection.
Also retries send() on EINTR, which the previous loop treated as fatal.
This is an exemption in the sense of lang/spec §8: the write never checked
whether the peer was still there, and the consequence of not checking was
fatal rather than merely wrong.