self-review 2026-08-08: stop the JSON parser destroying every non-ASCII character
jp_parse_string_raw handled \uXXXX by skipping the four hex digits and emitting a literal '?'. JSON writers escape non-ASCII by default (Python's json.dumps ships ensure_ascii=True; MCP clients do the same), so every em dash, curly quote, accented letter and emoji arriving over MCP or HTTP was silently replaced by one question mark on the way in. Measured on the live store: 3,119 of 4,081 non-telemetry nodes carried the damage, including the self traversal root and all 13 values nodes. Contents split cleanly into fully-clean or fully-mangled with zero overlap, which is the tell that it was one write path rather than gradual rot. No snapshot on disk predates it, and 3 bytes collapsing to 1 is not invertible, so the existing damage is permanent; only the forward path could be fixed. Decode properly instead: 4 hex digits, surrogate-pair reassembly for astral codepoints, U+FFFD for lone surrogates, UTF-8 encode. Malformed escapes keep the old '?' so a truncated body still parses. The deeper failure was that nothing measured this for two months. Every gauge in the system reports whether the machinery is running; none reported whether the text it carries is intact. Adds both halves: engram_text_health_json() / GET /api/text-health for the daily census, and a txt_damaged counter on the heartbeat for live regression. Verified in both directions - clean UTF-8 does not trip it, a deliberately damaged node does.
This commit is contained in:
@@ -90,6 +90,24 @@ fn route_act_stats(method: String, path: String, body: String) -> String {
|
||||
engram_act_stats_json()
|
||||
}
|
||||
|
||||
// route_text_health — GET /api/text-health
|
||||
// (2026-08-08 self-review) The daily census half of the text-integrity gauge.
|
||||
// Today's review found that the JSON parser had been replacing every \uXXXX
|
||||
// escape with a literal '?' for at least two months: 3,119 of 4,081
|
||||
// non-telemetry nodes (76%) were damaged, including the self traversal root
|
||||
// and every values node, and NOTHING detected it — because every gauge in the
|
||||
// system measured whether the machinery was running, and none measured whether
|
||||
// the text it carried was intact. No snapshot on disk predates the damage, so
|
||||
// it cannot be undone; it can only be made impossible to repeat quietly.
|
||||
//
|
||||
// The parser is fixed. This route is the standing check: `damaged` should now
|
||||
// hold flat at its historical floor and never climb. `write_damaged` (also on
|
||||
// the heartbeat as txt_damaged) is the live regression signal — non-zero means
|
||||
// a write path is mangling text right now.
|
||||
fn route_text_health(method: String, path: String, body: String) -> String {
|
||||
engram_text_health_json()
|
||||
}
|
||||
|
||||
// (2026-07-18 self-review) Scoping sweep: `let` inside an if-block creates an
|
||||
// inner scope only — it does NOT mutate the outer binding (documented with
|
||||
// evidence in awareness.el, 2026-05-25). Every default/reassignment below used
|
||||
@@ -573,6 +591,9 @@ fn handle_request(method: String, path: String, body: String) -> String {
|
||||
if str_eq(method, "GET") && (str_eq(clean, "/api/act-stats") || str_eq(clean, "/act-stats")) {
|
||||
return route_act_stats(method, path, body)
|
||||
}
|
||||
if str_eq(method, "GET") && (str_eq(clean, "/api/text-health") || str_eq(clean, "/text-health")) {
|
||||
return route_text_health(method, path, body)
|
||||
}
|
||||
|
||||
// Nodes
|
||||
if str_eq(method, "POST") && (str_eq(clean, "/api/nodes") || str_eq(clean, "/nodes")) {
|
||||
|
||||
Reference in New Issue
Block a user