P0: recalled-memory truncation splits UTF-8 characters, poisoning the request body (both wires, ships today) #123

New Issue

2026-08-06T21:49:09Z

tim.lingo commented

2026-08-06 21:49:09 +00:00

Affects every shipped build today, including Tim's own running Neuron. Fix is written and under review in PR #122 but reaches no one until that merges and a brain is rebuilt.

Symptom

Chat fails with {"error":"llm unavailable"} and no explanation. Intermittent-looking; actually deterministic per conversation.

Root cause

The session preload cuts recalled memory content at fixed byte lengths — the continuity snippet (str_slice(acc, 0, 350)) and session_preload_bullets' per-bullet cut. str_slice/str_len count bytes, so a cut landing inside a multi-byte UTF-8 character leaves a dangling lead byte in the system prompt, making the entire request body invalid UTF-8. Providers reject it outright.

Evidence (captured body, not inferred)

18,710 bytes; decode fails at offset 18,248 on 0xe2:

bytes at fault: 94 80 e2 5c 6e 5c 6e
context after : '<U+FFFD>\n\n\n\n[RECEIPTS - permanent]...'

A run of box-drawing rules (U+2500 = E2 94 80) sliced mid-character, sourced from a stored memory containing table borders.

Why it matters more than it looks

Ordinary content triggers it: em dash, curly quote, accented name, emoji, table border.
It gets MORE likely as a user's memory grows — the opposite of what a memory product can afford.
Shared code: hits the Anthropic wire too, not just the OpenAI lane.
It was invisible because nothing decoded the outgoing body strictly; found only when a new gate did.

Fix (in PR #122)

utf8_safe_slice(s, n) — walks back over UTF-8 continuation bytes to the lead byte and keeps the character only if all its bytes survived — applied at both cut sites. Covered by tests/test_utf8_slice.el (18/18: 2-, 3- and 4-byte characters, boundary-exact keeps, the real box-rule shape), which also pins the runtime semantics the fix depends on (str_len counts bytes; str_char_code returns the byte value) so a future runtime change fails loudly instead of silently rotting.

🤖 Generated with Claude Code

**Affects every shipped build today, including Tim's own running Neuron.** Fix is written and under review in PR #122 but reaches no one until that merges and a brain is rebuilt. ## Symptom Chat fails with `{"error":"llm unavailable"}` and no explanation. Intermittent-looking; actually deterministic per conversation. ## Root cause The session preload cuts recalled memory content at fixed **byte** lengths — the continuity snippet (`str_slice(acc, 0, 350)`) and `session_preload_bullets`' per-bullet cut. `str_slice`/`str_len` count bytes, so a cut landing inside a multi-byte UTF-8 character leaves a dangling lead byte **in the system prompt**, making the entire request body invalid UTF-8. Providers reject it outright. ## Evidence (captured body, not inferred) 18,710 bytes; decode fails at offset 18,248 on `0xe2`: ``` bytes at fault: 94 80 e2 5c 6e 5c 6e context after : '<U+FFFD>\n\n\n\n[RECEIPTS - permanent]...' ``` A run of box-drawing rules (U+2500 = `E2 94 80`) sliced mid-character, sourced from a stored memory containing table borders. ## Why it matters more than it looks - **Ordinary content triggers it**: em dash, curly quote, accented name, emoji, table border. - **It gets MORE likely as a user's memory grows** — the opposite of what a memory product can afford. - **Shared code**: hits the Anthropic wire too, not just the OpenAI lane. - It was invisible because nothing decoded the outgoing body strictly; found only when a new gate did. ## Fix (in PR #122) `utf8_safe_slice(s, n)` — walks back over UTF-8 continuation bytes to the lead byte and keeps the character only if all its bytes survived — applied at **both** cut sites. Covered by `tests/test_utf8_slice.el` (18/18: 2-, 3- and 4-byte characters, boundary-exact keeps, the real box-rule shape), which also pins the runtime semantics the fix depends on (`str_len` counts bytes; `str_char_code` returns the byte value) so a future runtime change fails loudly instead of silently rotting. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/neuron#123