Compare commits

...

3 Commits

Author SHA1 Message Date
Neuron fe820928b0 docs: the builtin recipe never required a test
El SDK CI - dev / build-and-test (pull_request) Failing after 10m55s
lang/AGENTS.md:71-77 gives four steps for adding a C builtin and ends at
'confirm the self-host fixpoint is byte-identical'. No step asks for a test.
The only 'verify' in the file is that fixpoint, which proves the COMPILER
REPRODUCES ITSELF and says nothing about whether the builtin works — so the
recipe reads as complete while having checked nothing about the thing just
added.

Measured on 2026-08-16: engram_node_set_emb, engram_curiosity_json and
dream_set_handler were all added in a single session with zero tests, by an
agent following this recipe. Separately a UTF-8 fix was written and tested
and THE TEST PASSED ON THE UNPATCHED BUILD — the real defect was elsewhere,
and only building the pre-fix binary exposed it. Without a negative control
that fix would have merged as verified.

Adds step 5 with the two failure shapes actually encountered: a test that
never exercises the change (a route default bypassed the code under test),
and an induction that loses a race (curl --max-time left BOTH builds alive;
only SO_LINGER 0, a real RST, reproduced it). Plus the port-binding check,
because a stale instance answering has silently produced false results here
more than once and pkill -f does not reliably match argv './engram'.

Documentation only. Does not touch the (a) split-the-C / (b) close-the-
compiler-gap question, which is a separate decision.
2026-08-16 13:53:08 -05:00
will.anderson 385c18442d runtime: a disconnecting client must not kill the server (#151)
El SDK CI - dev / build-and-test (push) Failing after 3m53s
2026-08-16 18:33:50 +00:00
Neuron cace6a5ebf runtime: a disconnecting client must not kill the server
El SDK CI - dev / build-and-test (pull_request) Failing after 13m45s
There was no SIGPIPE handling anywhere in this runtime: no signal
disposition, no MSG_NOSIGNAL, no SO_NOSIGPIPE, and send() called with bare
flags. The default disposition of SIGPIPE is to TERMINATE THE PROCESS, so
any client that hangs up mid-response takes the whole engram with it.

MEASURED, and it is not hypothetical. Production has restarted 254 times
since 2026-08-13T19:37 at a flat ~10 minute cadence:

  17:05:18  17:15:29  17:25:38  17:35:50  17:46:00  17:56:10  18:06:22  18:16:30

Intervals of 10m09s-10m12s, not 10m00s. That excess is the whole story:
ai.neuron.engram-tick has StartInterval 600, and engram-tick.sh:13 calls

  curl -s -m10 -X POST .../api/tick

The beat does not finish within 10s over 13,634 nodes, so curl waits its
full timeout and closes. The engram then writes the tick response to a dead
socket, takes SIGPIPE, and dies. launchd KeepAlive restarts it, so the
failure presents as a mysterious restart rather than a crash — and
~/.neuron/logs/engram.log records nothing but "[http] listening on" 254
times, with no exit reason. launchctl list confirms the last exit as -13.

Root cause is one level out: consolidation had no owner, so an external
ticker was created to poke it, and the ticker is what kills it. The fix
here does not address that; it makes the process survivable while it is
addressed.

Two layers, because neither alone is portable:
  - SO_NOSIGPIPE per accepted socket (Darwin/BSD) and MSG_NOSIGNAL per send
    (Linux), so the signal is never raised for socket writes at all.
  - A process-wide SIG_IGN backstop, installed once and idempotent, for
    platforms and paths with neither. With the signal ignored, send()
    returns -1/EPIPE and the existing error path closes the connection.

Also retries send() on EINTR, which the previous loop treated as fatal.

This is an exemption in the sense of lang/spec §8: the write never checked
whether the peer was still there, and the consequence of not checking was
fatal rather than merely wrong.
2026-08-16 13:25:14 -05:00
2 changed files with 70 additions and 2 deletions
+11
View File
@@ -73,6 +73,17 @@ When you add a C builtin (verbatim-emit recipe — the El name is emitted as the
2. Add a `__`-prefixed thin wrapper in `el_seed.c` and declare it in `el_seed.h`. 2. Add a `__`-prefixed thin wrapper in `el_seed.c` and declare it in `el_seed.h`.
3. Add the name to `builtin_arity` in `el-compiler/src/codegen.el` — add **both** the plain and `__`-prefixed spellings. 3. Add the name to `builtin_arity` in `el-compiler/src/codegen.el` — add **both** the plain and `__`-prefixed spellings.
4. Rebuild the elc binary (see below) and confirm the self-host fixpoint is byte-identical. 4. Rebuild the elc binary (see below) and confirm the self-host fixpoint is byte-identical.
5. **Prove it with a NEGATIVE CONTROL.** Show the test FAILING on a build without your change, then passing with it. A test that has never been seen to fail has proven nothing.
> **Step 5 is not optional, and step 4 does not cover it.** The fixpoint proves the *compiler reproduces itself*. It says nothing whatsoever about whether your builtin works. A recipe ending at "byte-identical" reads as complete while having verified nothing about the thing just added — which is why this file, until 2026-08-16, produced builtins with no tests at all.
>
> Measured cost of the omission (2026-08-16): `engram_node_set_emb`, `engram_curiosity_json` and `dream_set_handler` were all added in one session with zero tests. Separately, a UTF-8 fix was written, tested, and **the test passed on the unpatched build too** — the defect was elsewhere entirely, and only building the pre-fix binary exposed it. Without a negative control that fix would have merged as verified.
>
> Two shapes that pass while proving nothing, both hit the same day:
> - A test that never exercises your change (the route supplied a default that bypassed the code under test).
> - An induction that loses a race. `curl --max-time` on a large response left *both* builds alive; only `SO_LINGER 0` — a genuine RST, so the peer is provably gone — reproduced the failure. Six of ten attempts is not a control.
>
> Before every probe, confirm **your** process bound the port (`lsof -nP -iTCP:<port>`, match the PID). A stale instance answering on the port has silently produced false results here more than once, and `pkill -f` does not reliably match an argv like `./engram`.
Worked example: the `engram_assert_json` (op_assert seam) and `engram_node_full_in`/`engram_connect_in` (purview write-side) primitives added 2026-08-15 follow exactly this recipe. Worked example: the `engram_assert_json` (op_assert seam) and `engram_node_full_in`/`engram_connect_in` (purview write-side) primitives added 2026-08-15 follow exactly this recipe.
+59 -2
View File
@@ -40,6 +40,7 @@
#include <sys/stat.h> #include <sys/stat.h>
#include <netinet/in.h> #include <netinet/in.h>
#include <arpa/inet.h> #include <arpa/inet.h>
#include <signal.h> /* SIGPIPE disposition: a hung-up client must not kill us */
#include <dlfcn.h> /* dlsym for http_set_handler fallback */ #include <dlfcn.h> /* dlsym for http_set_handler fallback */
#include <unistd.h> #include <unistd.h>
#include <fcntl.h> #include <fcntl.h>
@@ -1335,10 +1336,63 @@ static const char* http_reason_phrase(int status) {
} }
} }
/* Best-effort send with retry on partial writes. */ /* A DISCONNECTING CLIENT MUST NOT KILL THE SERVER (2026-08-16).
*
* There was no SIGPIPE handling anywhere in this runtime: no signal disposition,
* no MSG_NOSIGNAL, no SO_NOSIGPIPE, and send() called with bare flags. The
* default disposition of SIGPIPE is to TERMINATE THE PROCESS, so any client that
* hung up mid-response a curl that hit its timeout, a browser tab closed
* during a large read, a proxy giving up took the whole engram down with it.
*
* Measured on the live instance: 18 boots in the log, and `launchctl list`
* reporting the previous exit for ai.neuron.engram as -13, i.e. killed by
* signal 13 = SIGPIPE. Reproduced by the cause: pulling /api/nodes/list (26 MB)
* with a client-side timeout. launchd's KeepAlive then restarts it, so the
* failure looks like a mysterious restart rather than a crash, and the graph
* silently reloads under whatever was mid-flight.
*
* This is an exemption in the §8 sense: the write never checked whether the
* peer was still there, and the consequence of not checking was fatal rather
* than merely wrong.
*
* Two layers, because neither alone is portable:
* - SO_NOSIGPIPE per socket (Darwin/BSD) and MSG_NOSIGNAL per send (Linux),
* so the signal is never raised for socket writes in the first place.
* - A process-wide SIG_IGN as the backstop for platforms/paths with neither,
* installed once and idempotent. With the signal ignored, send() returns
* -1/EPIPE and the existing error path closes the connection. */
#ifndef MSG_NOSIGNAL
#define MSG_NOSIGNAL 0
#endif
static void el_ignore_sigpipe_once(void) {
static int done = 0;
if (done) return;
done = 1;
#ifndef _WIN32
signal(SIGPIPE, SIG_IGN);
#endif
}
/* Per-socket suppression where the platform offers it. Best-effort: a failure
* here is not fatal because el_ignore_sigpipe_once() already covers the case. */
static void el_sock_nosigpipe(int fd) {
#if defined(SO_NOSIGPIPE)
int on = 1;
setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &on, sizeof(on));
#else
(void)fd;
#endif
}
/* Best-effort send with retry on partial writes. EPIPE/ECONNRESET are a client
* that left, not a server fault: return -1 so the caller closes the connection,
* and never let it reach the process as a signal. */
static int http_send_all(int fd, const char* p, size_t left) { static int http_send_all(int fd, const char* p, size_t left) {
el_ignore_sigpipe_once();
while (left > 0) { while (left > 0) {
ssize_t w = send(fd, p, left, 0); ssize_t w = send(fd, p, left, MSG_NOSIGNAL);
if (w < 0 && errno == EINTR) continue;
if (w <= 0) return -1; if (w <= 0) return -1;
p += w; left -= (size_t)w; p += w; left -= (size_t)w;
} }
@@ -1788,6 +1842,7 @@ void http_serve(el_val_t port, el_val_t handler) {
pthread_mutex_unlock(&_http_conn_mu); pthread_mutex_unlock(&_http_conn_mu);
HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg)); HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg));
if (!arg) { el_closesocket(cfd); continue; } if (!arg) { el_closesocket(cfd); continue; }
el_sock_nosigpipe(cfd);
arg->fd = cfd; arg->fd = cfd;
pthread_t tid; pthread_t tid;
if (pthread_create(&tid, NULL, http_worker, arg) != 0) { if (pthread_create(&tid, NULL, http_worker, arg) != 0) {
@@ -1834,6 +1889,7 @@ static void* _http_serve_async_loop(void* raw) {
pthread_mutex_unlock(&_http_conn_mu); pthread_mutex_unlock(&_http_conn_mu);
HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg)); HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg));
if (!arg) { close(cfd); continue; } if (!arg) { close(cfd); continue; }
el_sock_nosigpipe(cfd);
arg->fd = cfd; arg->fd = cfd;
pthread_t tid; pthread_t tid;
if (pthread_create(&tid, NULL, http_worker, arg) != 0) { if (pthread_create(&tid, NULL, http_worker, arg) != 0) {
@@ -2134,6 +2190,7 @@ void http_serve_v2(el_val_t port, el_val_t handler) {
pthread_mutex_unlock(&_http_conn_mu); pthread_mutex_unlock(&_http_conn_mu);
HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg)); HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg));
if (!arg) { el_closesocket(cfd); continue; } if (!arg) { el_closesocket(cfd); continue; }
el_sock_nosigpipe(cfd);
arg->fd = cfd; arg->fd = cfd;
pthread_t tid; pthread_t tid;
if (pthread_create(&tid, NULL, http_worker_v2, arg) != 0) { if (pthread_create(&tid, NULL, http_worker_v2, arg) != 0) {