Tim Lingo
de65991807
Teaches the OpenAI-format lane (Groq/OpenAI/Grok/Gemini/Ollama) to offer tools,
execute them, and loop — the capability that until now existed only on the
Anthropic wire. The tool-execution, consent, bridge and run-progress machinery is
reused unchanged; only the wire dialect is new.
Two pre-existing defects were found while proving it, and are fixed here because
both silently break chat:
1. PROVIDER WIRING NEVER CONNECTED. The launcher exports SOUL_LLM_PROVIDER /
SOUL_LLM_BASE_URL and puts the provider key in ANTHROPIC_API_KEY + SOUL_API_KEY;
the engine's provider fork read only NEURON_LLM_0_*, which nothing sets in a
customer build. So use_openai was ALWAYS false: every non-Anthropic user's turns
went to api.anthropic.com carrying, say, a Groq key, and came back
"llm unavailable". Proven side-by-side against the pinned round-9 brain
(sha256 15cf7d1b…): identical env, shipped brain = "llm unavailable" both chat
modes with ZERO calls to the configured endpoint; this build = a real answer,
with the probe logging POST /v1/chat/completions and Bearer <provider key>.
Fixed brain-side only (env fallbacks) — no app or launcher change needed.
2. TRUNCATION SPLITS UTF-8 CHARACTERS. The session preload cuts recalled memory at
fixed BYTE lengths (continuity snippet 350; session_preload_bullets per bullet).
A cut landing inside a multi-byte character leaves a dangling lead byte in the
SYSTEM PROMPT, making the whole request body invalid UTF-8 — providers reject it
and the user sees an unexplained failure. Captured from a real body: 18,710 bytes,
decode fails at 18,248 on 'e2', a box-drawing rule (U+2500 = E2 94 80) sliced in
half. Trigger is ordinary content — em dash, curly quote, accented name, emoji,
table border — and it gets MORE likely as memory grows. Shared code: this hit the
Anthropic wire too. Fixed with utf8_safe_slice() applied at BOTH cut sites.
WHAT IS IN THE PORT
- llm_base_url / llm_wire_format / agentic_api_key: fall back to the launcher's own
SOUL_LLM_* names; anthropic deliberately still returns "" so its native path is
untouched (endpoint configurability remains neuron#62).
- openai_tools_json(): Anthropic tool schema -> OpenAI function schema; entries with
no input_schema (Anthropic's server-side web_search) are skipped — they cannot
execute on this wire.
- agentic_tools_no_web(): the standard set minus that server tool.
- openai_agentic_loop(): forked rather than parameterised, so agentic_loop — which
carries every round-7/8/9 fix — is provably untouched. Same envelopes, same state
keys, same consent policy (ask_all / escalate / builtin / always-allow), same
client-bridge contract, same run-progress ledger, same 12-iteration cap.
- ADR-0005 mirrored on this wire: parallel_tool_calls:false is sent explicitly, and
if a provider ignores it we honour the FIRST call and echo only that one, so the
conversation we send is never self-contradictory. The drop is logged loudly.
- The assistant turn echoes the provider's own content bytes (json_get_raw), so a
JSON null stays null and nothing is lost to a decode/re-encode round trip.
- Tool results are embedded already-escaped (dispatch_tool json_safe's them);
truncation trims a dangling escape so a cut can't invalidate the body.
- bridge_save() gains a "wire" scalar and agentic_resume branches on it, so a
suspended turn resumes on the wire it suspended on. Legacy blobs (no field) resume
as anthropic. The field is read from the blob's SCALAR HEAD only — an unbounded
first-match scan would run on into messages_raw, which is model-controlled, and
that is exactly the round-9 resume defect. Pinned by a test.
- Three fork sites: handle_chat_agentic, handle_dharma_room_turn_agentic,
agentic_resume. Tool assembly is computed once per lane at both entry points
(it makes an HTTP call to the connector bridge; it was being paid for twice).
TOOLING THAT DID NOT EXIST
- tests/run-el-test.sh — engine tests were never runnable: elc is a compiler, it
emits C and exits. This emits the test to C, compiles soul.c with main renamed
away, links the rest + the repo-pinned runtime, and runs it. It also COMPUTES THE
VERDICT, because every counted test file's "N passed, M failed" summary is a
permanent 0/0 — the counters increment inside if BLOCKS, which El scoping
discards (9 files; real fix filed as neuron#116). Proven to discriminate with a
deliberately-broken assertion.
- tests/gate-openai/ — deterministic OpenAI-dialect provider stub + scenarios +
driver + hostile modes, and a strict request validator that rejects any
Anthropic-shaped field so dialect leakage fails loudly.
VERIFICATION (rungs named)
- E2E-VERIFIED against a LIVE provider (Anthropic's OpenAI-compatible endpoint,
confirmed live): real answer; a tool call whose out-of-root path was DENIED by the
guard, after which the model refused to claim success ("I won't tell you I did it,
because I didn't"); then a valid path -> file physically on disk with exact content,
honest reply, ledger with per-round entries + {done:true}.
- Deterministic lane gate: 11/12 in both consent configurations (bridge + local);
hostile providers produce no hang and no fabricated answer; the 12-iteration cap
trips with its honest message. The one FAIL is oa-tools-off and is NOT this port —
see "Known, not fixed here".
- ANTHROPIC LANE UNCHANGED: gate9 32/32 on this build and on the pinned round-9
brain; request bytes differ only within the noise band that two runs of the
UNMODIFIED brain also produce (proven with a baseline-vs-baseline control), and
the preload sections — the shared code touched here — are byte-identical.
The rig discriminates: the round-8 brain scores 24/32 on it.
- verify-soul-contract.sh: PASS (27/27 routes, no hard-deletes).
- Unit: test_bridge_serialization 36/36 (incl. 8 new wire/field-order assertions),
test_utf8_slice 18/18, test_agentic_tools 18 PASS / 0 FAIL / 3 documented skips.
KNOWN, NOT FIXED HERE (deliberate)
- Tools:Off on an OpenAI provider still fails: the non-agentic path goes through the
el-runtime provider chain, which appends /v1/chat/completions to a base URL that
already ends in /v1 -> /v1/v1/... 404. Runtime/plain-chat territory, untouched
mid-beta. Note openai_chat_complete() has zero callers — that lane is served
entirely by the runtime chain.
- The 12-iteration cap does not bound a chain of BRIDGED tools (iteration is
per-invocation and resume starts fresh). Parity with the Anthropic lane.
- run_progress resets on each resume, so a client rendering cumulative steps across a
consent pause sees earlier legs vanish. Parity with the Anthropic lane.
- verify-soul-contract.sh needs bash >= 4; under macOS's stock bash 3.2 it dies
instantly with a FALSE red ("local: -n: invalid option").
- Groq-specific live E2E not run: no Groq key exists on this machine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
91 lines
5.7 KiB
Markdown
91 lines
5.7 KiB
Markdown
# gate-openai — deterministic OpenAI-dialect provider stub
|
|
|
|
Staging home for the **soul-openai-tools-v2** gate scaffolding
|
|
(`docs/specs/SPEC-soul-openai-tools-v2-2026-08-06.md`, test-plan rung 1:
|
|
"stub first — discriminates before El code exists"). Sibling of gate9's
|
|
Anthropic stub (`_wt-beta-round9/scripts/gate9/stub-llm.py`): same scenario
|
|
mechanism, opposite wire dialect. Stdlib Python only, 127.0.0.1 only,
|
|
refuses ports 7770/7779/17779. Run `./selftest.sh` — exit 0 is green.
|
|
|
|
## Files
|
|
|
|
| File | Role |
|
|
|---|---|
|
|
| `stub-openai.py` | HTTP server: `POST /v1/chat/completions` (OpenAI dialect), scenario-scripted responses, request validation, ground-truth JSONL log, hostile modes via `--mode` |
|
|
| `scenarios-openai.json` | Scenario contract: scripts + markers + per-class/per-step request assertions |
|
|
| `selftest.sh` | curl-driven proof of every scenario, every rejection, all hostile modes (58 checks) |
|
|
|
|
## What each scenario proves (when the brain drives it)
|
|
|
|
| Class | Proves |
|
|
|---|---|
|
|
| `oa-plain` | finish_reason `stop` ends the loop; tools + `tool_choice` + `parallel_tool_calls:false` were offered on the wire |
|
|
| `oa-tools-off` | the chat-only lane sends NO tools (offering them there is a 400) |
|
|
| `oa-single-tool` | full round-trip: `tool_calls` parsed, assistant echo + `role:"tool"` turn with matching `tool_call_id` sent back, final text reached |
|
|
| `oa-torture` | `function.arguments` (JSON-encoded string with nested quotes, backslashes, newlines, tabs, unicode) survives exactly ONE decode — the stub recomputes the issued payload from the script and 400s on any drift (`gate_echo_mismatch`, the spec §6 two-escaper trap) |
|
|
| `oa-parallel` | two `tool_calls` in one response: the brain either answers both (paired correctly) or rejects cleanly — an unpaired echo is a 400 |
|
|
| `oa-mission` | multi-round loop continuation; step index = assistant-message count, so resume threads index correctly by construction |
|
|
| `oa-api-error` | provider errors 400/429/500/503 in the OpenAI error envelope surface honestly, no retry storm |
|
|
|
|
Universal (every request, any scenario): Anthropic dialect leakage fails
|
|
loudly with 400 — `anthropic-version` header, top-level `system` /
|
|
`stop_sequences` / `max_tokens_to_sample`, `input_schema` inside tools,
|
|
Anthropic content blocks (`tool_use`/`tool_result`/...). Tools must be
|
|
`{type:"function", function:{name, description, parameters}}`, unique names;
|
|
echoed `arguments` must be a JSON-encoded STRING, never a decoded object.
|
|
|
|
## Hostile modes (`--mode`, same file)
|
|
|
|
| Mode | Behavior | Brain invariant under test |
|
|
|---|---|---|
|
|
| `black-hole` | reads the request, never responds | HTTP timeout exists and surfaces; no silent hang |
|
|
| `mid-body-drop` | 200 headers, half a JSON body, socket abort | truncated body = clean error, never a half-parsed reply shown as real |
|
|
| `tool-pending-forever` | every request gets a fresh `tool_calls` response, forever | the loop's iteration cap trips (`max_loop_iterations: 16` in the contract); count actual round-trips via `GET /gate/stats` (`chat_hits`) |
|
|
|
|
## How the brain-side gate consumes this
|
|
|
|
1. Start: `stub-openai.py --port P --scenarios scenarios-openai.json --log run.jsonl`
|
|
2. Point the brain at it: `NEURON_LLM_0_URL=http://127.0.0.1:P` +
|
|
`NEURON_LLM_0_FORMAT=openai` (spec step 0 must verify these actually
|
|
export at runtime), scratch profile, free soul port.
|
|
3. Send each phrasing's `prompt` (the marker selects the script); assert the
|
|
brain's claims (`tools_used`, reply, ledger) against the stub's JSONL log
|
|
— truth, not narration — plus files on disk for write_file scenarios.
|
|
4. Any stub 400 = the brain sent a malformed/leaked request; the gate fails
|
|
with the stub's reason string.
|
|
5. Re-run gate9's Anthropic matrix unchanged = proof the Anthropic lane is
|
|
byte-untouched.
|
|
|
|
## Reconciliation into gate9 (app repo) — AFTER round 9 merges
|
|
|
|
This dir is staging only; the merge is mechanical by design:
|
|
- `stub-openai.py` + `scenarios-openai.json` move to `scripts/gate9/`
|
|
alongside `stub-llm.py` + `scenarios.json` (shared conventions: marker
|
|
matching, assistant-count step indexing, `--port/--scenarios/--log`,
|
|
JSONL fields `seq/ts/kind/scenario_class/phrasing/step/validation/
|
|
delivered/http_status`, prod-port refusal, benign background responses,
|
|
`GATE-SCRIPT-EXHAUSTED` overrun, `{N}/{NN}` repeat expansion).
|
|
- `prompt-matrix-gate.sh` gains a dialect axis (anthropic|openai) choosing
|
|
stub + scenario file; `matrix-asserts.py` reads the same log shape.
|
|
- The `--mode` hostile flags here are PROVIDER-side (brain↔LLM boundary);
|
|
gate9's `hostile/` servers are SOUL-side (app↔brain boundary). They are
|
|
complementary, not duplicates — both stay.
|
|
|
|
## Open questions for the port author (stub asserts a position; confirm or change)
|
|
|
|
1. `parallel_tool_calls` must be **explicitly false** on every tool-bearing
|
|
request (ADR-0005 pin). If the builder omits it instead, relax
|
|
`defaults.expect_request.parallel_tool_calls` to `null`.
|
|
2. `tool_choice` must be present (`"auto"` expected). If the brain relies on
|
|
the provider default, drop `require_tool_choice`.
|
|
3. Tool-result `content` is asserted only to be a string; if the brain sends
|
|
structured JSON-in-string (like `{"ok":true,...}`), no change needed.
|
|
4. Groq compatibility: Groq's OpenAI-compat endpoint rejects some optional
|
|
fields; whatever field set the brain settles on for live Groq E2E must be
|
|
mirrored here so the deterministic gate and the live lane assert the SAME
|
|
request shape.
|
|
5. The stub treats a `role:"tool"` turn answering an already-answered id as
|
|
400; if the resume path can legitimately replay tool results, that rule
|
|
needs a resume-aware carve-out (gate9's Anthropic stub faced the same
|
|
issue — see its PASS 1 comment).
|