neuron-technologies/neuron
3
0
Fork 0
Code Issues 44 Pull Requests Actions Packages Projects Releases Wiki Activity

soul: /api/neuron/memory mangles non-ASCII on store (em dash → literal 'u2014', backslash swallowed) — data corruption on the primary save route #87

New Issue
Open
opened 2026-07-18 19:54:57 +00:00 by tim.lingo · 2 comments
tim.lingo commented 2026-07-18 19:54:57 +00:00
Member
Copy Link
Copy Source

Found 2026-07-18 during the split-brain repair, verified on the installed app-soul (:7770, local mode).

Repro: POST /api/neuron/memory with content containing an em dash (—). Stored node content comes back with the character replaced by the literal text 'u2014' (the escape's backslash is swallowed somewhere between json_escape and store). Diff of a 1,027-char save vs its source shows first divergence exactly at the em dash; all non-ASCII appears affected.

Impact: silent content corruption on the designed save path (MCP remember → /memory). Also breaks exact-content dedupe for sync tooling (we switched to src-tag dedupe as a workaround). Likely same family as the recall-output id/label mojibake in #72.

Machine: Tim's Mac, /Applications/Neuron.app soul, engram ~/.neuron/engram/snapshot.json.

Found 2026-07-18 during the split-brain repair, verified on the installed app-soul (:7770, local mode). Repro: POST /api/neuron/memory with content containing an em dash (—). Stored node content comes back with the character replaced by the literal text 'u2014' (the escape's backslash is swallowed somewhere between json_escape and store). Diff of a 1,027-char save vs its source shows first divergence exactly at the em dash; all non-ASCII appears affected. Impact: silent content corruption on the designed save path (MCP remember → /memory). Also breaks exact-content dedupe for sync tooling (we switched to src-tag dedupe as a workaround). Likely same family as the recall-output id/label mojibake in #72. Machine: Tim's Mac, /Applications/Neuron.app soul, engram ~/.neuron/engram/snapshot.json.
will.anderson commented 2026-08-03 19:08:02 +00:00
Owner
Copy Link
Copy Source

GROOMING — 2026-08-03
Status: Open — verified silent data corruption on the primary save route: /api/neuron/memory mangles non-ASCII on store (em dash stored as literal 'u2014', backslash swallowed); likely the same family as the #72 recall mojibake.
Next action: Will to fix the json_escape-to-store path so non-ASCII content round-trips byte-for-byte.
Owner: Will
Priority: BETA-CRITICAL

**GROOMING — 2026-08-03** **Status:** Open — verified silent data corruption on the primary save route: /api/neuron/memory mangles non-ASCII on store (em dash stored as literal 'u2014', backslash swallowed); likely the same family as the #72 recall mojibake. **Next action:** Will to fix the json_escape-to-store path so non-ASCII content round-trips byte-for-byte. **Owner:** Will **Priority:** BETA-CRITICAL
will.anderson added the BETA-CRITICAL label 2026-08-03 19:08:04 +00:00
tim.lingo commented 2026-08-07 04:32:55 +00:00
Author
Member
Copy Link
Copy Source

New direct evidence for this bug, found 2026-08-06 while building a UTF-8 test fixture — and it is worse than the title suggests.

The engine's JSON reader does not decode \uXXXX escapes at all: it drops the backslash and stores the literal text u2500. That means any client encoding JSON with the default ensure_ascii=True (Python's json.dumps, and the default in most languages) silently corrupts every non-ASCII character it saves — not just em dashes, and not just via /api/neuron/memory: /api/neuron/node/create takes the same path.

Why it is worth re-reading now: it nearly cost us a real bug. Our first fixture run came back green for the wrong reason — the non-ASCII test corpus we were saving had been silently ASCII-flattened on store, so the test was measuring nothing. A test that can only ever pass is as worthless as one that can never pass; this bug manufactures exactly that condition for anything downstream that reads memory back.

Repertoire confirmed affected: em dash (U+2014), curly quotes, accented Latin, emoji (4-byte), CJK, and box-drawing characters (U+2500 — the same character class implicated in neuron#122's byte-slice bug).

Related but distinct: neuron#122 fixes truncation cutting inside a multi-byte character on the outbound side. This issue is the inbound store side. Both must be closed for non-ASCII memory to survive a round trip; fixing either alone leaves the other's corruption in place.

Filed from the round-9.1 terminal; evidence and the reproducing corpus generator are in our fixture set (memory-fixtures.py --selftest).

**New direct evidence for this bug, found 2026-08-06 while building a UTF-8 test fixture — and it is worse than the title suggests.** The engine's JSON reader does not decode `\uXXXX` escapes at all: it drops the backslash and stores the literal text `u2500`. That means **any** client encoding JSON with the default `ensure_ascii=True` (Python's `json.dumps`, and the default in most languages) silently corrupts every non-ASCII character it saves — not just em dashes, and not just via `/api/neuron/memory`: `/api/neuron/node/create` takes the same path. Why it is worth re-reading now: it nearly cost us a real bug. Our first fixture run came back **green for the wrong reason** — the non-ASCII test corpus we were saving had been silently ASCII-flattened on store, so the test was measuring nothing. A test that can only ever pass is as worthless as one that can never pass; this bug manufactures exactly that condition for anything downstream that reads memory back. Repertoire confirmed affected: em dash (U+2014), curly quotes, accented Latin, emoji (4-byte), CJK, and box-drawing characters (U+2500 — the same character class implicated in neuron#122's byte-slice bug). Related but distinct: **neuron#122** fixes truncation cutting *inside* a multi-byte character on the outbound side. This issue is the *inbound* store side. Both must be closed for non-ASCII memory to survive a round trip; fixing either alone leaves the other's corruption in place. Filed from the round-9.1 terminal; evidence and the reproducing corpus generator are in our fixture set (`memory-fixtures.py --selftest`).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BETA-CRITICAL
On the beta critical path — must resolve before ship
 blocks-public-beta
Must be fixed before any public beta
 HELD
Blocked pending an external decision (counsel / dual sign-off)
 ORTHOGONAL
Parallel workstream (testing/hardening) — safe to assign off critical path
 P0
Drop-everything severity
 POST-BETA
Important but not blocking the beta ship
 security
Security vulnerability
No labels BETA-CRITICAL
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
andre.almeida (Andre Almeida) tim.lingo (Tim Lingo) will.anderson (Will Anderson)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/neuron#87
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?