fix(engine): approving a researched mission completes — the resume replay read a tool id out of the conversation (BUG-42, both faces) #115
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: neuron-technologies/neuron#115
Reference in New Issue
Block a user
Delete Branch "fix/resume-server-tool-replay"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
BUG-42 — the beta's core use case ("research, then produce a file") was 100% fatal on approve→resume. One mechanism, two faces, fixed at the root.
Root cause
bridge_saveserialized the raw replayedmessagesarray before thetool_use_idscalar, andjson_getis a first-substring-match scanner (strstr("\"key\":")). So on resume,json_get(blob, "tool_use_id")returned the firsttool_use_idoccurrence inside the conversation, not the field:web_search, that first hit is aweb_search_tool_result'ssrvtoolu_…id. Anthropic rejects the replay:messages.2.content.0: unexpected tool_use_id found in tool_result blocks: srvtoolu_…, whichchat.el:2888maps to{"error":"llm unavailable"}— surfaced in the app as a false "check your key".tool_resultid, killing 10+-round missions after ~2 files. Found by the new prompt-matrix gate, not by a human.Both resume-guard branches reduced to
saved_use_id(eff_use_id = if use_id == saved { use_id } else { saved }≡saved), so the client's correctcall_idcould never win.The
server_tool_use↔web_search_tool_resultpairs replay intact — this was a cross-field blob misread, not a pairing defect. Confirming the mechanism: the single-cycle-no-search shape passes on round 8 precisely because its messages contain notool_use_idsubstring (tool_useblocks carryid).This is the third instance of the first-match-scanner class (cf. BUG-6, and round 8's citation-block fix). Postmortem + failure-pattern entry owed on our side; if the class deserves a scanner-level fix in the runtime, that's your call.
The change (
chat.el, +33/−8)bridge_saveblob order made load-bearing and documented — everyjson_safe'd scalar before both raw fields, andtools_raw(fixed schema) beforemessages_raw(arbitrary conversation).agentic_resumehonors the client's echoedcall_idwhen present (clean provenance, never blob-round-tripped), saved id as fallback — binds each approve cycle to its own round's id, so it also repairs pre-fix blobs.Pattern sweep: the legacy synthetic blob in
sessions.elis all-escaped (no hazard); no other first-match read of a raw-embedding container exists.How to test
web_searches → bridges onwrite_file→ approve exactly as the app does → real completion (2,967 / 3,269 / 2,608 chars) with the file on disk (12–13 KB). Zerollm errorlines.llm unavailable,unexpected tool_use_id … srvtoolu_…), and a same-toolchain unpatched baseline build fails identically. The one-commit diff is the only variable.scripts/gate9/prompt-matrix-gate.shin neuron-ui, 8 behavior classes × 4 phrasings against the packaged brain via a deterministic stub-LLM, no key): round-8 brain 24/32 → this build 32/32.Built with the documented local recipe (
elb+cc -std=c11 -O1 -DHAVE_CURL … -include dist/elp-c-decls.h); BUG-PLAINCHAT-1 miscompile guard verified 0 sites in the generated C (neuron#111). As-built44021795…, as-bundled15cf7d1b…— the latter ships inNeuron-round9-27b4ff80.dmg.Base is
8f3a478(fix/soul-history-provenance-20260805), the true round-8-built engine state.Will: this is soul-core, so it needs your review and, to reach production, a
dist/soul.cregen (neuron#209). Related open items this build touches or corroborates: neuron#110 (BUG-41 — the shipped brain still binds all interfaces with no auth; independently re-confirmed in the guest log this morning), neuron#111 (guard verified clean here), neuron#112 (non-Anthropic providers remain toolless — the app now tells the user the truth about that rather than pretending).🤖 Generated with Claude Code
The agentic loop keeps only the FIRST tool_use block per round (chat.el:2281, "Capture first tool_use block only"). Anthropic lets a model emit several tool_use blocks in one message and requires a tool_result for every one, so a parallel-tool turn is answered once, the rest are dropped, and the next request dies with: tool_use ids were found without tool_result blocks immediately after (neuron#78 quotes this as "tool_use ids found without tool_result"; the above is the API's actual wording - recorded so the next person's grep matches.) This constrains the wire to match what the loop can assemble: "tool_choice":{"type":"auto","disable_parallel_tool_use":true} STOPGAP - AND THE DURABLE FIX ALREADY EXISTS. A correct multi-tool loop is already in Will's EL runtime, in C, and the soul does not call it. Verified on el:origin/main lang/el-compiler/runtime/el_runtime.c: llm_register_tool:9616, llm_build_tool_results:9743 - which walks EVERY content block, emits one tool_result per tool_use, and sets is_error for an unregistered tool - llm_call_agentic:9817 calling it at :9918, iteration cap 10 at :9847. Will's commit 12d5e77 (2026-04-30). grep for llm_call_agentic/llm_register_tool across every neuron/*.el returns nothing; dist/soul.c has zero references. chat.el hand-rolls its own single-tool loop instead, and that is the one that breaks. The durable fix is therefore to register the soul's tools via llm_register_tool and call llm_call_agentic - deleting a loop, not writing one. See ADR 0005. Our own approved spec called this seven weeks ago: docs/research/agentic-tool-approval-design.md (2026-06-12, "Approved for build"), line 20 on the defect, line 30 on the goal ("Execute all tool_use blocks in a turn (one result per block)"). Two edits, because dist/soul.c cannot be regenerated here (Will's gated elc/elb toolchain is not on this machine): (a) chat.el:2255 - source of truth, so a later regen carries the fix. One edit covers all three routes: agentic_loop is called from chat.el:2152 (/api/chat agentic), :2676 (dharma room) and :2496 (agentic_resume). (b) dist/soul.c:28173 - generated form, hand-spliced. Line 27624 is the non-agentic/OpenAI-compat req_body (no tools) and was left untouched. Prior art reused rather than reinvented: soul-narrated-runs-20260713.patch (27,824 bytes) line 78 spliced this same string into the same concat chain on 2026-07-13. Deliberately NOT ported from that patch, having read it: max_tokens is not changed by it (16384 sits on both sides of the hunk; our main's 4096 is a separate output-truncation concern), and its pause_turn pairing fix - same defect class - is unreachable today because no server-side web_search is wired (agentic_tools_with_web at chat.el:1418 is never called), so it is untestable and logged instead. Proven E2E on a scratch profile and port 7791, never the live chain. A/B against a pristine origin/main control built from the same vendored runtime: fixed completed the mission (tools_used read_file x3, 4 iterations, correct answer); control failed 3/3. Direct API probe confirmed the mechanism - without the field the model emits 3 parallel tool_use blocks and replaying the unfixed loop's next turn returns HTTP 400; with it, exactly 1 block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>DESIGN FIT: three of round 7's five defects share ONE root — the conversation-history layer persists only {role, content}, discarding tool provenance, session scoping, and the distinction between a real user turn and an internal utility call. Fixes A and B RESTORE Will's design rather than extend it: his agentic path already scopes history per session, the plain path never got it, and his own source carries the TODO admitting the resulting race (chat.el, handle_chat: "process-global key; concurrent /api/chat requests without session_id race on this read-append-write"). Fix C repairs one join Will wrote that was correct for a year and one we added last week. E1/E2 are ours. FIX A — tool provenance in history (kills the FALSE CONFESSION) Root cause, EXECUTED-verified: handle_chat_agentic recorded turns via hist_append, which emits {"role","content"} only. server_tool_use blocks, web_search_tool_result blocks and every citation were discarded, then replayed as text. On the next turn the model saw a data-rich answer with zero evidence a search had happened, and its own permanent rule ("never describe a search you did not perform") left one conclusion available: that it had fabricated the data. It apologised for a search it HAD run — four independent lines of evidence confirm the search was real. The defect is not the model's honesty. It is that we deleted the evidence and then asked it to account for itself. Change: agentic_loop accumulates the source URLs it already walks past (citations and web_search_tool_result content) and returns them as "sources"; handle_chat_agentic folds tools_used + sources into a receipt line stored WITH the assistant turn. Receipts are unconditional — a negative receipt ("no tools ran") is the other half of the guarantee, because "no evidence of a tool" and "evidence of no tool" were previously identical in the transcript. conv_history_block splits the receipt off before snipping so a long answer cannot truncate away the evidence. The user never sees it: it is appended to the history copy, not the reply. FIX B — one history key for both paths (kills the BLANK STARE) Root cause, EXECUTED-verified: the agentic path keyed history on session_hist_<id>; the plain path was hard-wired to the process-global conv_history and never read session_id. One conversation, two buckets. Proven in the guest engram: the scoped node held exactly two turns starting at "Try again" while the earlier exchanges sat unscoped. Change: conv_hist_key/conv_hist_label are now the single definition, used by BOTH paths; session_id is threaded route -> layered_cycle -> layered_generate / conv_history_record. The 2-line fallback (plain path reads the agentic key) was REJECTED: it keeps the process-global bucket as a live write target, which is the bleed the TODO describes. Also found and closed while threading: layered_cycle read session_id from the state key "current_session_id", which is read here and WRITTEN NOWHERE in the entire source. It was unconditionally "", so TODO(reliability #4) — per-session steward continuity — was dead code that could never fire. It fires now. LAZY SESSION, decided explicitly: we create the session EAGERLY at the door (app half, ui#223) rather than migrating orphaned turns. Migration would copy the CONTENTS of a process-global bucket, possibly another conversation's, into a named session — the bleed, performed deliberately. Eager creation makes the situation impossible instead. Migration is deliberately not implemented and must not be added without solving provenance first. FIX C — the two text-join seams ("to.Good", byte-verified 0x77 0x2e 0x47) Two bare `+` joins, written a year apart, had drifted into two answers to one question: within-response block joins (Will's, 2026-05-03, latent until server-side web_search began interleaving non-text blocks) and across-round joins (ours,62af564). Change: one named rule, text_join_sep, at both sites. NOT a blanket separator — a cited answer splits MID-SENTENCE ("The current temperature is " + "86°F" + ", with "), so a blanket separator shatters every sourced sentence. The rule takes the one bit that distinguishes the cases: whether a NON-TEXT block intervened. Hoisting it also makes the fix verifiable in the shipped binary, which an inline `+` is not. FIX E1 — utility generations stay out of the transcript Title generation ("Write a 3-6 word title...") and insight passes ran down the same plain door as a real message and were recorded as if the user had typed them; the same calls are the "model":"unknown" rows in usage.jsonl. is_utility_request reads an explicit utility flag from the app, with the __title__/__insight__ id prefixes as a fallback for older clients. Answered normally, never recorded. FIX E2 — OPERATOR IDENTITY is scoped to tool-capable turns The block (env USER/HOME, closing "This is a hard rule") was prepended to EVERY system prompt including chat mode. On a Tools:Off turn there is no filesystem in reach, so it governed nothing and merely supplied the loudest fact in the prompt — which is why the model opened a fresh conversation with "You're test, on your machine at /Users/test". Hoisted to operator_identity_block() and gated on !chat_mode. Unchanged wherever a file or command tool can actually be reached. ALSO: agentic_loop's per-session history persist had a second hand-rolled copy of conv_history_persist with a different label expression, different salience scores and different tags for the same node. Since both now derive the label from conv_hist_label and engram_node_full upserts by label, two score policies were writing one node. Collapsed to one writer. BUILD NOTE: dist/elp-c-decls.h is force-included by the documented link recipe and carried the OLD C arities, so it is updated here. This is the build-support header, NOT the stale generated dist/soul.c — no dist/*.c was read or edited; all engine changes are .el source. chat.elh/soul.elh are committed because a first-pass build against the old signatures FAILS (measured); the other regenerated headers are reverted as unrelated churn. BUILT: 887,000 bytes, sha256 d632b061ad75269d6adeb52578d030eaf49e895d91289d7f946b19c08450d728 Zero el_str_concat(<int>, str_len(...)) sites (the BUG-PLAINCHAT-1 miscompile guard). web_search_20250305 and disable_parallel_tool_use both still present — PR #108's web search and the ADR-0005 stopgap are intact. Refs neuron#109 (builds on it), neuron#78 (Receipt Contract — the real fix A is a stopgap for) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>FOUND BY E2E, NOT BY REASONING. The previous commit's design note asserted the provenance receipt "never reaches the user: it is appended to the history copy, not the reply." That was FALSE, and only running the thing showed it. On the first live run against the built DMG brain, two agentic turns out of two came back with "Your favourite colour is chartreuse and your project is called Perihelion. [[RECEIPT - recorded by the soul, not written by the model: no tools ran on this turn.]]" — the receipt in the user-visible reply. MECHANISM: the receipt is stored inside the assistant turn, and the agentic path replays history VERBATIM as Anthropic message objects. So the model sees its own previous answers ending in [[RECEIPT ...]] and does the obvious thing — it imitates the format and signs the next answer the same way. The plain path did NOT leak, which is the tell: there, history is rendered into the SYSTEM prompt as labelled lines rather than replayed as assistant turns, and a model imitates its own turns far more readily than a transcript. FIX, two layers, because one of them is not a guarantee: - receipt_rule() names the marker in both system prompts (plain and agentic): these lines are written by the system, read them as evidence, never write one. Reduces occurrence. - receipt_strip() truncates any [[RECEIPT ...]] out of model output before it becomes the reply — plain path in layered_generate, agentic path on final_text in agentic_loop. Deterministic. A guard that depends on the model choosing to obey is exactly the class of thing round 8 exists to stop shipping, so the instruction is the optimisation and the strip is the guarantee. Placed ABOVE agentic_loop's empty-check on purpose: a turn whose entire output was an imitated receipt has produced no answer, and must be reported as no answer. The receipt stays in HISTORY, which is the whole point and is proven to work: asked "What source did you use for that?" one turn after a live web_search, this brain answered "I used Weather Underground (https://www.wunderground.com/weather/is/reykjav%C3%ADk) for the current temperature in Reykjavik" — a real source, no apology. That is the false confession dead, and it is dead BECAUSE the model can read the receipt. BUILT: 887,112 bytes, sha256 54a2eff84d4fa44f8d2db6781dcf225df4b5075a8ec5f1058018bd40cc1af10b BUG-PLAINCHAT-1 miscompile guard: zero sites. Refs neuron#109 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>CAUGHT BY A/B, AND ONLY BY A/B. The previous commit's receipt_strip assumed the receipt is always TERMINAL and cut everything from the marker onward. It is not always terminal: once receipt_rule told the model what [[RECEIPT ...]] means, the model sometimes LED with one and wrote the answer underneath. Cutting at the marker then deleted the entire answer and the turn returned {"error":"no response"}. MEASURED, same prompt (two web searches, cited prose), fresh session each run: round-7 brain 4 / 4 answered (471, 473, 473, 544 chars) round-8 brain 2 / 7 answered (five {"error":"no response"}) This looked exactly like a flaky model. It was not — it was mine. Running the two brains side by side on the same prompt is the only reason it was found, and it is the reason the A/B is now part of how this class gets tested. AFTER THE FIX, same protocol: round-7 brain 4 / 4 (473, 473, 473, 544) round-8 brain 4 / 4 (657, 657, 657, 673) THE FIX: remove the [[...]] span and keep BOTH sides, instead of truncating at the marker. An unterminated marker at position 0 is left completely alone — no rule about receipts is worth erasing an answer over. Bounded four-pass loop rather than a conditional exit, because rebinding the counter inside an if-expression is the block-expression shape that miscompiles integer arithmetic under this elc (BUG-PLAINCHAT-1). Verified in the generated C: str_slice(rest, (e + 2), str_len(rest)) <- integer addition, correct el_str_concat(head, tail) <- string concat, correct and zero el_str_concat(<ident>, str_len(...)) sites across all 49 modules. SEAM PROOF (FIX C) rides on the same runs — a real two-search cited answer, inspected byte by byte, in BOTH failure directions: missing separator (the round-7 "to.Good", bytes 77 2e 47): 0 hits. Sentence boundaries measure 2e 20 4d — "." SPACE "M". over-separation (a cited sentence shattered across paragraphs): 0 hits. The answer is one continuous paragraph with its sentences intact, which is the direction a blanket separator would have broken. BUILT: sha256 77115f2733e794c5bc4ad1f55b1acaf658f8f4a91cccd423a2d633d94a726cbc Refs neuron#109 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>ROOT CAUSE (round 9; live-repro'd 5/5 this morning, both faces stub-proven by the prompt-matrix gate). json_get is a first-substring-match scanner (strstr for '"key":', el_runtime.c). bridge_save serialized the RAW messages array BEFORE the tool_use_id scalar, so agentic_resume's json_get(blob, 'tool_use_id') returned the FIRST '"tool_use_id":' occurrence inside the replayed conversation, not the saved field. The resume guard then preferred that misread over the client's correct call_id (its two branches both reduced to saved_use_id), attached the tool_result to the wrong id, and Anthropic 400'd the resume ('unexpected tool_use_id found in tool_result blocks'), surfaced as {"error":"llm unavailable"}. ONE MISREAD, TWO FACES — whichever block owns the first tool_use_id in the array: FACE 1 (search-then-bridge, the Key West killer): the first occurrence is the first web_search_tool_result's srvtoolu_… id — every agentic turn that ran server-side web_search and then bridged on a client tool died on approval, deterministically (messages.2.content.0 … srvtoolu_…). The write itself had already succeeded; only the resume died. FACE 2 (multi-cycle missions): with no search, the first occurrence is ROUND 0's tool_result block — so every LATER approve/resume cycle replayed the round-0 client id (stale-resume-id), killing multi-file missions after ~2 files. And the shape that PASSES on round 8 confirms the mechanism: a single-cycle bridge with no prior tool round has no 'tool_use_id' substring in its messages at all (tool_use blocks carry 'id'), so the scan fell through to the blob's own field and resumed correctly. The server_tool_use ↔ web_search_tool_result pairs themselves replay intact — the defect was a cross-field misread of the blob, the same first-match-scanner class as BUG-6 (approve 'content' matched inside tool_input, 2026-07-17) and round 8's citation-block fix. THE FIX, the pattern not the spot: 1. bridge_save writes every json_safe'd scalar BEFORE both raw fields (an escaped value cannot contain a bare '"key":' byte pattern, so first-match always lands on the blob's own fields), and tools_raw (our fixed schema) before messages_raw (arbitrary conversation), so the raw extractions cannot first-match into model-controlled bytes either. Field order documented as load-bearing. 2. agentic_resume now honors the client's echoed call_id when present — the value with clean provenance (minted from pend_tool_id, never blob-round-tripped) — falling back to the saved id only when the client omits it. Each approve cycle therefore binds to ITS OWN round's id (kills FACE 2 even against a blob written by a pre-fix binary), and an omitted call_id still resumes on the saved id, which the reordered blob now reads correctly. Pattern sweep: the legacy synthetic blob (sessions.el handle_session_approve) embeds only json_safe'd fields — no raw hazard, untouched. No other json_get read of any container that embeds raw conversation JSON before the read field. PROOF: prompt-matrix gate 24/32 RED on the round-8 brain (fails exactly the two resume classes, named) -> 32/32 GREEN on this build; live-key Key West tracer 3/3 consecutive full round-trips (bridge -> approve-as-the-app -> real completion, file on disk), plain-chat and weather-only controls PASS; unpatched round-8 brain and a same-toolchain unpatched baseline build both still fail the identical sequence with the identical srvtoolu 400 (the test discriminates, and the only variable between failing and passing builds is this diff). Refs neuron#109 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>