Files

T

Tim Lingo de65991807 feat(engine): tools + agentic loop on the OpenAI wire, and two chat-breaking fixes found proving it

Teaches the OpenAI-format lane (Groq/OpenAI/Grok/Gemini/Ollama) to offer tools,
execute them, and loop — the capability that until now existed only on the
Anthropic wire. The tool-execution, consent, bridge and run-progress machinery is
reused unchanged; only the wire dialect is new.

Two pre-existing defects were found while proving it, and are fixed here because
both silently break chat:

1. PROVIDER WIRING NEVER CONNECTED. The launcher exports SOUL_LLM_PROVIDER /
   SOUL_LLM_BASE_URL and puts the provider key in ANTHROPIC_API_KEY + SOUL_API_KEY;
   the engine's provider fork read only NEURON_LLM_0_*, which nothing sets in a
   customer build. So use_openai was ALWAYS false: every non-Anthropic user's turns
   went to api.anthropic.com carrying, say, a Groq key, and came back
   "llm unavailable". Proven side-by-side against the pinned round-9 brain
   (sha256 15cf7d1b…): identical env, shipped brain = "llm unavailable" both chat
   modes with ZERO calls to the configured endpoint; this build = a real answer,
   with the probe logging POST /v1/chat/completions and Bearer <provider key>.
   Fixed brain-side only (env fallbacks) — no app or launcher change needed.

2. TRUNCATION SPLITS UTF-8 CHARACTERS. The session preload cuts recalled memory at
   fixed BYTE lengths (continuity snippet 350; session_preload_bullets per bullet).
   A cut landing inside a multi-byte character leaves a dangling lead byte in the
   SYSTEM PROMPT, making the whole request body invalid UTF-8 — providers reject it
   and the user sees an unexplained failure. Captured from a real body: 18,710 bytes,
   decode fails at 18,248 on 'e2', a box-drawing rule (U+2500 = E2 94 80) sliced in
   half. Trigger is ordinary content — em dash, curly quote, accented name, emoji,
   table border — and it gets MORE likely as memory grows. Shared code: this hit the
   Anthropic wire too. Fixed with utf8_safe_slice() applied at BOTH cut sites.

WHAT IS IN THE PORT
- llm_base_url / llm_wire_format / agentic_api_key: fall back to the launcher's own
  SOUL_LLM_* names; anthropic deliberately still returns "" so its native path is
  untouched (endpoint configurability remains neuron#62).
- openai_tools_json(): Anthropic tool schema -> OpenAI function schema; entries with
  no input_schema (Anthropic's server-side web_search) are skipped — they cannot
  execute on this wire.
- agentic_tools_no_web(): the standard set minus that server tool.
- openai_agentic_loop(): forked rather than parameterised, so agentic_loop — which
  carries every round-7/8/9 fix — is provably untouched. Same envelopes, same state
  keys, same consent policy (ask_all / escalate / builtin / always-allow), same
  client-bridge contract, same run-progress ledger, same 12-iteration cap.
- ADR-0005 mirrored on this wire: parallel_tool_calls:false is sent explicitly, and
  if a provider ignores it we honour the FIRST call and echo only that one, so the
  conversation we send is never self-contradictory. The drop is logged loudly.
- The assistant turn echoes the provider's own content bytes (json_get_raw), so a
  JSON null stays null and nothing is lost to a decode/re-encode round trip.
- Tool results are embedded already-escaped (dispatch_tool json_safe's them);
  truncation trims a dangling escape so a cut can't invalidate the body.
- bridge_save() gains a "wire" scalar and agentic_resume branches on it, so a
  suspended turn resumes on the wire it suspended on. Legacy blobs (no field) resume
  as anthropic. The field is read from the blob's SCALAR HEAD only — an unbounded
  first-match scan would run on into messages_raw, which is model-controlled, and
  that is exactly the round-9 resume defect. Pinned by a test.
- Three fork sites: handle_chat_agentic, handle_dharma_room_turn_agentic,
  agentic_resume. Tool assembly is computed once per lane at both entry points
  (it makes an HTTP call to the connector bridge; it was being paid for twice).

TOOLING THAT DID NOT EXIST
- tests/run-el-test.sh — engine tests were never runnable: elc is a compiler, it
  emits C and exits. This emits the test to C, compiles soul.c with main renamed
  away, links the rest + the repo-pinned runtime, and runs it. It also COMPUTES THE
  VERDICT, because every counted test file's "N passed, M failed" summary is a
  permanent 0/0 — the counters increment inside if BLOCKS, which El scoping
  discards (9 files; real fix filed as neuron#116). Proven to discriminate with a
  deliberately-broken assertion.
- tests/gate-openai/ — deterministic OpenAI-dialect provider stub + scenarios +
  driver + hostile modes, and a strict request validator that rejects any
  Anthropic-shaped field so dialect leakage fails loudly.

VERIFICATION (rungs named)
- E2E-VERIFIED against a LIVE provider (Anthropic's OpenAI-compatible endpoint,
  confirmed live): real answer; a tool call whose out-of-root path was DENIED by the
  guard, after which the model refused to claim success ("I won't tell you I did it,
  because I didn't"); then a valid path -> file physically on disk with exact content,
  honest reply, ledger with per-round entries + {done:true}.
- Deterministic lane gate: 11/12 in both consent configurations (bridge + local);
  hostile providers produce no hang and no fabricated answer; the 12-iteration cap
  trips with its honest message. The one FAIL is oa-tools-off and is NOT this port —
  see "Known, not fixed here".
- ANTHROPIC LANE UNCHANGED: gate9 32/32 on this build and on the pinned round-9
  brain; request bytes differ only within the noise band that two runs of the
  UNMODIFIED brain also produce (proven with a baseline-vs-baseline control), and
  the preload sections — the shared code touched here — are byte-identical.
  The rig discriminates: the round-8 brain scores 24/32 on it.
- verify-soul-contract.sh: PASS (27/27 routes, no hard-deletes).
- Unit: test_bridge_serialization 36/36 (incl. 8 new wire/field-order assertions),
  test_utf8_slice 18/18, test_agentic_tools 18 PASS / 0 FAIL / 3 documented skips.

KNOWN, NOT FIXED HERE (deliberate)
- Tools:Off on an OpenAI provider still fails: the non-agentic path goes through the
  el-runtime provider chain, which appends /v1/chat/completions to a base URL that
  already ends in /v1 -> /v1/v1/... 404. Runtime/plain-chat territory, untouched
  mid-beta. Note openai_chat_complete() has zero callers — that lane is served
  entirely by the runtime chain.
- The 12-iteration cap does not bound a chain of BRIDGED tools (iteration is
  per-invocation and resume starts fresh). Parity with the Anthropic lane.
- run_progress resets on each resume, so a client rendering cumulative steps across a
  consent pause sees earlier legs vanish. Parity with the Anthropic lane.
- verify-soul-contract.sh needs bash >= 4; under macOS's stock bash 3.2 it dies
  instantly with a FALSE red ("local: -n: invalid option").
- Groq-specific live E2E not run: no Groq key exists on this machine.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-06 16:41:18 -05:00

5.7 KiB

Raw Blame History

gate-openai — deterministic OpenAI-dialect provider stub

Staging home for the soul-openai-tools-v2 gate scaffolding (docs/specs/SPEC-soul-openai-tools-v2-2026-08-06.md, test-plan rung 1: "stub first — discriminates before El code exists"). Sibling of gate9's Anthropic stub (_wt-beta-round9/scripts/gate9/stub-llm.py): same scenario mechanism, opposite wire dialect. Stdlib Python only, 127.0.0.1 only, refuses ports 7770/7779/17779. Run ./selftest.sh — exit 0 is green.

Files

File	Role
`stub-openai.py`	HTTP server: `POST /v1/chat/completions` (OpenAI dialect), scenario-scripted responses, request validation, ground-truth JSONL log, hostile modes via `--mode`
`scenarios-openai.json`	Scenario contract: scripts + markers + per-class/per-step request assertions
`selftest.sh`	curl-driven proof of every scenario, every rejection, all hostile modes (58 checks)

What each scenario proves (when the brain drives it)

Class	Proves
`oa-plain`	finish_reason `stop` ends the loop; tools + `tool_choice` + `parallel_tool_calls:false` were offered on the wire
`oa-tools-off`	the chat-only lane sends NO tools (offering them there is a 400)
`oa-single-tool`	full round-trip: `tool_calls` parsed, assistant echo + `role:"tool"` turn with matching `tool_call_id` sent back, final text reached
`oa-torture`	`function.arguments` (JSON-encoded string with nested quotes, backslashes, newlines, tabs, unicode) survives exactly ONE decode — the stub recomputes the issued payload from the script and 400s on any drift (`gate_echo_mismatch`, the spec §6 two-escaper trap)
`oa-parallel`	two `tool_calls` in one response: the brain either answers both (paired correctly) or rejects cleanly — an unpaired echo is a 400
`oa-mission`	multi-round loop continuation; step index = assistant-message count, so resume threads index correctly by construction
`oa-api-error`	provider errors 400/429/500/503 in the OpenAI error envelope surface honestly, no retry storm

Universal (every request, any scenario): Anthropic dialect leakage fails loudly with 400 — anthropic-version header, top-level system / stop_sequences / max_tokens_to_sample, input_schema inside tools, Anthropic content blocks (tool_use/tool_result/...). Tools must be {type:"function", function:{name, description, parameters}}, unique names; echoed arguments must be a JSON-encoded STRING, never a decoded object.

Hostile modes (`--mode`, same file)

Mode	Behavior	Brain invariant under test
`black-hole`	reads the request, never responds	HTTP timeout exists and surfaces; no silent hang
`mid-body-drop`	200 headers, half a JSON body, socket abort	truncated body = clean error, never a half-parsed reply shown as real
`tool-pending-forever`	every request gets a fresh `tool_calls` response, forever	the loop's iteration cap trips (`max_loop_iterations: 16` in the contract); count actual round-trips via `GET /gate/stats` (`chat_hits`)

How the brain-side gate consumes this

Start: stub-openai.py --port P --scenarios scenarios-openai.json --log run.jsonl
Point the brain at it: NEURON_LLM_0_URL=http://127.0.0.1:P + NEURON_LLM_0_FORMAT=openai (spec step 0 must verify these actually export at runtime), scratch profile, free soul port.
Send each phrasing's prompt (the marker selects the script); assert the brain's claims (tools_used, reply, ledger) against the stub's JSONL log — truth, not narration — plus files on disk for write_file scenarios.
Any stub 400 = the brain sent a malformed/leaked request; the gate fails with the stub's reason string.
Re-run gate9's Anthropic matrix unchanged = proof the Anthropic lane is byte-untouched.

Reconciliation into gate9 (app repo) — AFTER round 9 merges

This dir is staging only; the merge is mechanical by design:

stub-openai.py + scenarios-openai.json move to scripts/gate9/ alongside stub-llm.py + scenarios.json (shared conventions: marker matching, assistant-count step indexing, --port/--scenarios/--log, JSONL fields seq/ts/kind/scenario_class/phrasing/step/validation/ delivered/http_status, prod-port refusal, benign background responses, GATE-SCRIPT-EXHAUSTED overrun, {N}/{NN} repeat expansion).
prompt-matrix-gate.sh gains a dialect axis (anthropic|openai) choosing stub + scenario file; matrix-asserts.py reads the same log shape.
The --mode hostile flags here are PROVIDER-side (brain↔LLM boundary); gate9's hostile/ servers are SOUL-side (app↔brain boundary). They are complementary, not duplicates — both stay.

Open questions for the port author (stub asserts a position; confirm or change)

parallel_tool_calls must be explicitly false on every tool-bearing request (ADR-0005 pin). If the builder omits it instead, relax defaults.expect_request.parallel_tool_calls to null.
tool_choice must be present ("auto" expected). If the brain relies on the provider default, drop require_tool_choice.
Tool-result content is asserted only to be a string; if the brain sends structured JSON-in-string (like {"ok":true,...}), no change needed.
Groq compatibility: Groq's OpenAI-compat endpoint rejects some optional fields; whatever field set the brain settles on for live Groq E2E must be mirrored here so the deterministic gate and the live lane assert the SAME request shape.
The stub treats a role:"tool" turn answering an already-answered id as 400; if the resume path can legitimately replay tool results, that rule needs a resume-aware carve-out (gate9's Anthropic stub faced the same issue — see its PASS 1 comment).

5.7 KiB Raw Blame History