Compare commits

...

4 Commits

Author SHA1 Message Date
Neuron 8a307dfd42 runtime: make valid UTF-8 the JSON emitter's contract
El SDK CI - dev / build-and-test (pull_request) Failing after 10m36s
Three nodes in the live graph carry labels truncated to exactly 80 bytes
ending in a lone 0xE2 — the first byte of an em-dash, cut mid-sequence.
jb_emit_escaped copied every byte >= 0x20 through verbatim, so those three
nodes made the ENTIRE /api/nodes/list response undecodable and no strict
parser could read the graph at all.

  production binary   25,929,607 bytes   INVALID at byte 89260
  this build          26,338,389 bytes   VALID, parses to 13,630 nodes

The damage was NOT written by this runtime. No 80-byte truncation exists
here (the only label truncation is engram_first_n_chars at 60), and the
content of those nodes is 2572 and 2746 bytes. Some other producer wrote
them. That is exactly why fixing a writer could not have fixed this: the
store already holds the damage, and it accepts data from importers, other
producers and older binaries.

So the fix goes where the promise is made. A serializer that emits JSON
owes valid UTF-8 whatever it is handed. jb_emit_escaped now validates each
multi-byte sequence before emitting any of it and substitutes U+FFFD for a
bad lead byte, a missing or malformed continuation, an overlong encoding, a
UTF-16 surrogate, or a codepoint above U+10FFFF. Invalid bytes are REPLACED
rather than dropped, so the damage stays visible in the output instead of
being silently papered over. Well-formed input is byte-identical to before.

Second, preventive and explicitly NOT the cause of the above:
engram_first_n_chars truncated by BYTES despite its name, so content with a
multi-byte character crossing byte 60 would produce a half codepoint in the
label. It now uses el_utf8_safe_len, which returns the largest byte length
<= max that does not split a codepoint. Bounded by bytes, not codepoints,
so existing labels never grow — they only stop splitting.

el_utf8_safe_len lives beside str_count_chars rather than in the engram
because the rest of el's string layer is already codepoint-aware
(str_count_chars counts codepoints, str_reverse walks codepoint lengths).
Byte truncation was the outlier and the concern is a string concern.

Note on the investigation: I first "fixed" the truncator and wrote a test
that passed on the UNPATCHED build too, because route_create_node passes
label = content when no label is supplied, so engram_first_n_chars is never
reached over HTTP. The test proved nothing. The real cause was only found
by decoding the actual failing bytes out of the live response.
2026-08-16 12:03:03 -05:00
will.anderson 1f70b9fa18 runtime: ground the node asked about, and refuse circular support (#147)
El SDK CI - dev / build-and-test (push) Failing after 14m46s
2026-08-16 16:54:17 +00:00
Neuron 317466e8f7 runtime: ground the node asked about, and refuse circular support
El SDK CI - dev / build-and-test (pull_request) Failing after 15m5s
engram_ground_json resolved each seed to a REGION, wrote the grounded-by
edge between the two regions' HUBS, and then echoed those hubs back in the
"claim"/"evidence" fields as if they were the caller's input:

    const char* cid = C->hub_id ? C->hub_id : EL_CSTR(claim);
    const char* eid = E->hub_id ? E->hub_id : EL_CSTR(evidence);
    cog_ground_edge(g_engram_store, cid, eid, grounding, fw);

Three consequences, all measured against a clone of the live store:

1. The edge landed on a node the caller never named. Grounding 3b9ced5d
   against 6edf8c79 wrote an edge on the hubs of their regions instead.
2. When both seeds resolve into the same region the support is circular
   and scores near 1.0 for structural reasons, not evidential ones. Four
   probe nodes written together landed in one region, and every grounding
   among them returned 0.93-0.99 as if it were evidence. Two independent
   agents hit this and reported 0.885 / 0.909 self-groundings as confident.
3. The echo concealed both: the response was indistinguishable from a
   successful grounding of the ids that were passed in.

The region is HOW a claim is evaluated; it is not WHAT the claim is about.
So the edge now attaches to the requested ids, and the resolved hubs are
reported separately as claim_region / evidence_region.

Degeneracy is broader than hub == hub. Three circular shapes, all
previously invisible:
    same-region                both seeds resolve to one region
    claim-region-is-evidence   the evidence IS the hub of the claim's own
                               neighbourhood — measured at 0.98883
    evidence-region-is-claim   the mirror case
Each sets grounding to 0 and writes no edge. Circular support is not
support, and a grounding that is degenerate by construction must not
enter the graph as though it were evidence.

Verified:
  6edf8c79 -> 6edf8c79   degenerate=same-region   g=0        written=false
  6edf8c79 -> d0406dfd   degenerate=same-region   g=0        written=false
  ebc1413e -> 64cc96ef   degenerate=false         g=0.774563 written=true
  64cc96ef -> ebc1413e   degenerate=false         g=0.802896 written=true
Legitimate grounding across distinct regions is unchanged and still
writes; only circular support is refused.

This is the same class as #142 and #146 — a value that looked like an
answer with nothing behind it — except here it was also writing that
non-answer into the canonical store.
2026-08-16 11:53:31 -05:00
will.anderson eb3e6d7c1f runtime: resume the learned stance in think (#146)
El SDK CI - dev / build-and-test (push) Failing after 3m54s
2026-08-16 16:44:15 +00:00
2 changed files with 164 additions and 26 deletions
+160 -26
View File
@@ -3478,28 +3478,74 @@ static void jb_puts(JsonBuf* b, const char* s) {
b->buf[b->len] = '\0';
}
/* UTF-8 VALIDITY IS THE EMITTER'S CONTRACT (2026-08-16 self-review).
*
* This copied every byte >= 0x20 through verbatim, so a malformed sequence
* anywhere in the store became malformed output. Measured against the live
* graph: three nodes carry labels truncated to exactly 80 bytes ending in a
* lone 0xE2 the first byte of an em-dash, cut mid-sequence by some producer
* that is NOT this runtime (no 80-byte truncation exists here; the content
* itself is 2572 and 2746 bytes). Those three nodes made the ENTIRE 26 MB
* /api/nodes/list response undecodable, so a strict parser could not read the
* graph at all.
*
* Fixing only the writer would not have helped: the store already contains the
* damage, and it accepts data from importers, other producers and older
* binaries. A serializer that promises JSON owes valid UTF-8 regardless of what
* it is handed so validate here, at the boundary that makes the promise.
* Invalid bytes become U+FFFD rather than being dropped, so damage stays
* visible in the output instead of being silently papered over.
*
* Well-formed input is byte-identical to before: valid sequences are copied
* verbatim, and only structurally invalid ones (bad lead byte, missing or bad
* continuation, overlong encoding, UTF-16 surrogate, or > U+10FFFF) are
* replaced. */
static void jb_emit_escaped(JsonBuf* b, const char* s) {
jb_putc(b, '"');
for (; *s; s++) {
unsigned char c = (unsigned char)*s;
const unsigned char* p = (const unsigned char*)s;
while (*p) {
unsigned char c = *p;
switch (c) {
case '"': jb_puts(b, "\\\""); break;
case '\\': jb_puts(b, "\\\\"); break;
case '\b': jb_puts(b, "\\b"); break;
case '\f': jb_puts(b, "\\f"); break;
case '\n': jb_puts(b, "\\n"); break;
case '\r': jb_puts(b, "\\r"); break;
case '\t': jb_puts(b, "\\t"); break;
default:
if (c < 0x20) {
char tmp[8];
snprintf(tmp, sizeof(tmp), "\\u%04x", c);
jb_puts(b, tmp);
} else {
jb_putc(b, (char)c);
}
break;
case '"': jb_puts(b, "\\\""); p++; continue;
case '\\': jb_puts(b, "\\\\"); p++; continue;
case '\b': jb_puts(b, "\\b"); p++; continue;
case '\f': jb_puts(b, "\\f"); p++; continue;
case '\n': jb_puts(b, "\\n"); p++; continue;
case '\r': jb_puts(b, "\\r"); p++; continue;
case '\t': jb_puts(b, "\\t"); p++; continue;
default: break;
}
if (c < 0x20) {
char tmp[8];
snprintf(tmp, sizeof(tmp), "\\u%04x", c);
jb_puts(b, tmp);
p++;
continue;
}
if (c < 0x80) { jb_putc(b, (char)c); p++; continue; }
/* Multi-byte: validate the whole sequence before emitting any of it. */
int len; unsigned int cp;
if ((c & 0xE0) == 0xC0) { len = 2; cp = c & 0x1Fu; }
else if ((c & 0xF0) == 0xE0) { len = 3; cp = c & 0x0Fu; }
else if ((c & 0xF8) == 0xF0) { len = 4; cp = c & 0x07u; }
else { jb_puts(b, "\\ufffd"); p++; continue; }
int ok = 1;
for (int i = 1; i < len; i++) {
if ((p[i] & 0xC0) != 0x80) { ok = 0; break; } /* also catches NUL */
cp = (cp << 6) | (unsigned int)(p[i] & 0x3F);
}
if (ok) {
if (len == 2 && cp < 0x80) ok = 0; /* overlong */
else if (len == 3 && cp < 0x800) ok = 0; /* overlong */
else if (len == 4 && cp < 0x10000) ok = 0; /* overlong */
else if (cp >= 0xD800 && cp <= 0xDFFF) ok = 0; /* UTF-16 surrogate */
else if (cp > 0x10FFFF) ok = 0; /* out of range */
}
if (!ok) { jb_puts(b, "\\ufffd"); p++; continue; }
for (int i = 0; i < len; i++) jb_putc(b, (char)p[i]);
p += len;
}
jb_putc(b, '"');
}
@@ -5516,6 +5562,45 @@ el_val_t str_count(el_val_t sv, el_val_t subv) {
return (el_val_t)count;
}
/* el_utf8_safe_len — the largest byte length <= max_bytes that does NOT split a
* UTF-8 codepoint.
*
* WHY (2026-08-16 self-review): engram_first_n_chars truncated with a plain
* `if (l > n) l = n; memcpy(...)`, i.e. by BYTES despite its name. Any content
* carrying a multi-byte character across the 60-byte boundary produced a label
* ending in a half codepoint. That label is copied verbatim into every JSON
* document containing the node, so a single such node makes the WHOLE response
* invalid UTF-8 /api/nodes/list failed to decode at byte 89261 against the
* live store, which breaks any strict parser reading the graph.
*
* This lives beside str_count_chars rather than in the engram because the rest
* of el's string layer is already codepoint-aware (str_count_chars counts
* codepoints, str_reverse walks codepoint lengths). Byte-truncation was the
* outlier, and the concern is a string concern. Bounded by BYTES, not
* codepoints, so existing labels never grow only stop splitting.
*
* A lead byte with no room for its full sequence is dropped entirely; a stray
* continuation byte (already-invalid input) is passed through unchanged rather
* than silently repaired, so this never manufactures data. */
size_t el_utf8_safe_len(const char* s, size_t max_bytes) {
if (!s) return 0;
size_t len = strlen(s);
if (len <= max_bytes) return len;
size_t i = 0;
while (i < max_bytes) {
unsigned char c = (unsigned char)s[i];
size_t cp_len;
if ((c & 0x80) == 0x00) cp_len = 1;
else if ((c & 0xE0) == 0xC0) cp_len = 2;
else if ((c & 0xF0) == 0xE0) cp_len = 3;
else if ((c & 0xF8) == 0xF0) cp_len = 4;
else cp_len = 1; /* stray continuation: passthrough */
if (i + cp_len > max_bytes) break; /* would split — stop before it */
i += cp_len;
}
return i;
}
/* Codepoint count: walk bytes, count those NOT matching 10xxxxxx. */
el_val_t str_count_chars(el_val_t sv) {
const char* s = EL_CSTR(sv);
@@ -7714,10 +7799,14 @@ static double engram_decode_score(el_val_t v) {
return (double)n;
}
/* Truncate to at most n BYTES without splitting a UTF-8 codepoint. The old
* implementation was `if (l > n) l = n;` a byte cut that could land inside a
* multi-byte character and emit a half codepoint into the node's label, which
* then propagated into every JSON document containing that node. See
* el_utf8_safe_len for the measurement. */
static char* engram_first_n_chars(const char* s, size_t n) {
if (!s) return el_strdup("");
size_t l = strlen(s);
if (l > n) l = n;
size_t l = el_utf8_safe_len(s, n);
char* out = el_strbuf(l);
memcpy(out, s, l);
out[l] = '\0';
@@ -14062,12 +14151,57 @@ el_val_t engram_ground_json(el_val_t claim, el_val_t evidence, el_val_t for_whom
double grounding = (rc == 0) ? gr.grounding : 0.0;
if (rc == 0) engram_verify_grounding_free(&gr);
const char* fw = EL_CSTR(for_whom); if (fw && !*fw) fw = NULL;
const char* cid = C->hub_id ? C->hub_id : EL_CSTR(claim);
const char* eid = E->hub_id ? E->hub_id : EL_CSTR(evidence);
int wr = cog_ground_edge(g_engram_store, cid, eid, grounding, fw);
JsonBuf b; jb_init(&b); char t[256];
snprintf(t, sizeof t, "{\"relation\":\"grounded-by\",\"claim\":\"%s\",\"evidence\":\"%s\",\"for_whom\":\"%s\",\"grounding\":%.6g,\"written\":%s}",
cid, eid, fw ? fw : "-", grounding, wr == 0 ? "true" : "false");
/* GROUND THE NODE ASKED ABOUT, AND SAY WHAT WAS RESOLVED (2026-08-16
* self-review). This wrote the grounded-by edge between the two REGION
* HUBS and then echoed those hubs back in the "claim"/"evidence" fields
* as though they were the caller's input. Three consequences, all measured
* against the live store:
*
* 1. The edge landed on a node the caller never named. Asking to ground
* 3b9ced5d against 6edf8c79 wrote an edge on 6edf8c79 -> d0406dfd,
* because those were the hubs of the two regions.
* 2. When both seeds resolve into the same region, the hubs coincide and
* the call grounds a node against ITSELF, returning grounding = 1
* a perfect score with no evidence behind it. Two independent agents
* hit this and reported 0.885 / 0.909 self-groundings as confident.
* 3. The echo concealed both, because the response looked exactly like a
* successful grounding of the ids that were passed in.
*
* The region is HOW a claim is evaluated; it is not WHAT the claim is
* about. So the edge attaches to the requested ids, and the resolved hubs
* are reported separately under claim_region / evidence_region. When the
* two regions coincide, the grounding is degenerate by construction and is
* reported as such rather than as a confident 1.0. */
const char* cid = EL_CSTR(claim);
const char* eid = EL_CSTR(evidence);
const char* chub = C->hub_id ? C->hub_id : cid;
const char* ehub = E->hub_id ? E->hub_id : eid;
/* Degeneracy is broader than chub == ehub. Three circular shapes, each of
* which yields a high score for structural reasons rather than evidential
* ones, and all three were previously invisible:
* same-region both seeds resolve to one region grounding a thing
* against itself.
* claim-in-ev the claim's region hub IS the evidence node: the evidence
* sits at the centre of the claim's own neighbourhood.
* ev-in-claim the mirror case.
* Measured: grounding 3b9ced5d against 6edf8c79 scored 0.98883 purely
* because 6edf8c79 is the hub of 3b9ced5d's region. */
const char* degenerate = NULL;
if (chub && ehub && strcmp(chub, ehub) == 0) degenerate = "same-region";
else if (chub && eid && strcmp(chub, eid) == 0) degenerate = "claim-region-is-evidence";
else if (ehub && cid && strcmp(ehub, cid) == 0) degenerate = "evidence-region-is-claim";
if (degenerate) grounding = 0.0; /* circular support is not support */
/* Do not write an edge for a grounding that is degenerate by construction. */
int wr = degenerate ? -1 : cog_ground_edge(g_engram_store, cid, eid, grounding, fw);
JsonBuf b; jb_init(&b); char t[512];
snprintf(t, sizeof t, "{\"relation\":\"grounded-by\",\"claim\":\"%s\",\"evidence\":\"%s\","
"\"claim_region\":\"%s\",\"evidence_region\":\"%s\",\"degenerate\":%s%s%s,"
"\"for_whom\":\"%s\",\"grounding\":%.6g,\"written\":%s}",
cid ? cid : "", eid ? eid : "", chub ? chub : "", ehub ? ehub : "",
degenerate ? "\"" : "false", degenerate ? degenerate : "", degenerate ? "\"" : "",
fw ? fw : "-", grounding, wr == 0 ? "true" : "false");
jb_puts(&b, t);
engram_geo_free(C); engram_geo_free(E);
return el_wrap_str(b.buf);
+4
View File
@@ -612,6 +612,10 @@ el_val_t engram_get_node(el_val_t id);
void engram_strengthen(el_val_t node_id);
void engram_forget(el_val_t node_id);
el_val_t engram_prune_telemetry(el_val_t older_than_ms);
/* Largest byte length <= max_bytes that does not split a UTF-8 codepoint.
* Bounded by bytes, not codepoints, so truncated strings never grow. */
size_t el_utf8_safe_len(const char* s, size_t max_bytes);
el_val_t engram_node_count(void);
/* Attach geometry to an existing node. `hex` is little-endian float32,
* exactly dim*8 hex chars — the encoding realizers already emit. Lets a