Compare commits
11 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Neuron
|
635453b936 | ||
|
Neuron
|
315b2eff00 | ||
|
Neuron
|
cf41d12d22 | ||
 tim.lingo
|
18714e6142 | ||
|
tim.lingo
|
4936099c39 | ||
|
tim.lingo
|
f1471763f5 | ||
|
tim.lingo
|
5850793b67 | ||
|
tim.lingo
|
fc1745c652 | ||
|
Tim Lingo
|
43d0449904 | ||
|
Tim Lingo
|
b842e82f77 | ||
|
Tim Lingo
|
98ccbd4704 |
-139
@@ -1,139 +0,0 @@
|
||||
# PORT-NOTES — openai tools port working state (2026-08-06, session handoff-safe)
|
||||
|
||||
Spec: `docs/specs/SPEC-soul-openai-tools-v2-2026-08-06.md` (Tim-approved 2026-08-06). Tasks #1-5
|
||||
tracked in-session (1 ✓ wiring verdict, 2 ✓ stub rig, 3 in-progress = THIS, 4-5 pending).
|
||||
Worktree: HERE (`_wt-openai-tools`, branch `feat/soul-openai-tools-v2` @ dba755d). Round-9 trees
|
||||
READ-ONLY. Nothing committed yet.
|
||||
|
||||
## Step-0 verdict (evidence in journal note ncli-653ba964dd76)
|
||||
Shipped app never wires the v1 lane: launcher exports `SOUL_LLM_MODEL/PROVIDER/BASE_URL` +
|
||||
`ANTHROPIC_API_KEY`+`SOUL_API_KEY` (= Keychain key for WHATEVER provider; installer/macos/
|
||||
neuron-daemons.sh:288-300 on hotfix/beta-round9); brain reads only SOUL_LLM_MODEL (chat.el:8) and
|
||||
NEURON_LLM_0_* (chat.el:1768-1794) which nothing sets. `/api/config` PATCH ignores llm_* fields
|
||||
(studio.el:36 handle_config: POST-only, reads model/provider/api_key only).
|
||||
**Bridge = brain-side ONLY (zero app-repo edits, zero round-9 collision):**
|
||||
- `llm_base_url()`: NEURON_LLM_0_URL → fallback SOUL_LLM_BASE_URL when SOUL_LLM_PROVIDER ∉ {"","anthropic"}
|
||||
- `llm_wire_format()`: NEURON_LLM_0_FORMAT → fallback derive from SOUL_LLM_PROVIDER (openai/grok/gemini/groq/ollama → "openai"; else "anthropic")
|
||||
- `agentic_api_key()`: already works (ANTHROPIC_API_KEY carries the provider key); add NEURON_LLM_0_KEY → SOUL_API_KEY fallback.
|
||||
|
||||
## Design pins (stub asserts these — stub is green 58/58, tests/gate-openai/)
|
||||
- Request MUST send `"tool_choice":"auto"` (string) + `"parallel_tool_calls":false` explicitly.
|
||||
- `arguments` in tool_calls = JSON-ENCODED STRING; decode ONCE via json_get → feed dispatch_tool
|
||||
verbatim. Stub's echo-mismatch check catches double-encode/decode (two-escaper trap).
|
||||
- Assistant echo turn: `{"role":"assistant","content":null,"tool_calls":[...]}` VERBATIM from response.
|
||||
- Feedback: `{"role":"tool","tool_call_id":"<id>","content":"<result string>"}`.
|
||||
- Resume must NOT re-answer an answered id (stub 400s on repeat tool_call_id).
|
||||
- Parallel tool_calls in a response: take FIRST only + log skip (mirror ADR-0005 stopgap); stub
|
||||
scenario `parallel` proves behavior.
|
||||
- No tools in request when tools array empty/absent turns (boot probes) — stub defaults tolerate.
|
||||
|
||||
## el idioms confirmed (from openai_chat_complete :1808-1854 + agentic_loop :2751-2838)
|
||||
- JSON: `json_get(s,k)` decoded string · `json_get_raw(s,k)` raw subtree · `json_array_len` ·
|
||||
`json_array_get(arr,i)` · build by string concat + `json_escape()` (:1797, OpenAI-lane escaper).
|
||||
- HTTP: `let h: Map = {}` + `map_set(h,k,v)` + `http_post_with_headers(url, body, h)`;
|
||||
Bearer auth via `Authorization` header when key non-empty (:1825-1830).
|
||||
- Loop-carried vars must be top-level locals in the fn, mutated as if-expressions at while-body
|
||||
top level (see :2760-2791 pattern + comment :2903-2904 region).
|
||||
- Error shape: `str_starts_with(raw,"{\"error\"") || str_contains(raw,"\"error\":")` → return
|
||||
`{"error":"llm unavailable","reply":""}` (:1835-1838).
|
||||
|
||||
## Remaining read map (before writing the fork)
|
||||
- chat.el 2840-3200: block walk (2923-3000), policy gate (3009-3023: classify_tool_risk /
|
||||
is_builtin_tool / ask_all / tool_auto_approved → needs_bridge), dispatch_tool call (3025),
|
||||
tool_result feedback (3031, 3067-3072), run-progress ledger append (3078-3087), bridge_save
|
||||
(3182), loop end + done envelope (~3100-3200).
|
||||
- agentic_resume 3227-3293 (hardcoded Anthropic headers to make wire-aware; blob gets `wire` field,
|
||||
legacy default anthropic) · handle_tool_result 3293+ · dharma fork site 3465 (calls agentic_loop
|
||||
direct, no use_openai check today).
|
||||
|
||||
## Write plan (order)
|
||||
1. Env fallbacks (edit llm_base_url/llm_wire_format/agentic_api_key) — small, first, testable alone.
|
||||
2. `openai_tools_json(anthropic_tools: String) -> String` converter (walk array; per entry build
|
||||
{"type":"function","function":{name,description,parameters:input_schema-raw}}).
|
||||
3. `openai_agentic_loop(...)` fork: same signature as agentic_loop minus Anthropic-only params;
|
||||
INCLUDE run-progress ledger + tools_log + iteration cap 12; NO container_id/ws_drift/web_search
|
||||
(out of scope; strip web_search entry from tools via agentic_tools_literal()+connector merge,
|
||||
NOT _with_web()).
|
||||
4. Fork sites ×3: handle_chat_agentic :2695-2700 (route agentic to new loop when use_openai);
|
||||
dharma :3465; agentic_resume wire-branch.
|
||||
5. `chat.elh` extern decls. 6. Compile (recipe: dist/ + elc/elb per neuron-soul-build-deploy memory;
|
||||
round-9 tree soul.c regen'd 08-06 proves toolchain live). 7. Gate: stub selftest recipe in
|
||||
tests/gate-openai/README.md. 8. Anthropic-lane regression via gate9 (READ-ONLY consume from
|
||||
_wt-beta-round9). 9. Live Groq E2E (scratch profile, free port, key via Keychain read-only).
|
||||
|
||||
## BUILD RECIPE — CORRECTED 2026-08-06 (the June memory is STALE for August code)
|
||||
`~/el-sdk/el_runtime.c` (Jun 15) is MISSING builtins the Aug engine calls (`engram_wm_count`,
|
||||
`engram_wm_top_json`, `http_delete_json`, `http_serve_async`) → link fails with
|
||||
"symbol(s) not found for architecture arm64". Use the REPO-PINNED runtime:
|
||||
```
|
||||
mkdir -p <scratch>
|
||||
elb --elc=$HOME/el-sdk/elc --runtime=vendor/el-runtime/v1.0.0-20260501 --out=<scratch>/
|
||||
# "elb: link failed" at the end is EXPECTED and harmless — the per-module .c files are produced
|
||||
cc -std=c11 -O1 -DHAVE_CURL -rdynamic \
|
||||
-I vendor/el-runtime/v1.0.0-20260501 -I <scratch> -I /opt/homebrew/opt/openssl@3/include \
|
||||
-L /opt/homebrew/opt/openssl@3/lib \
|
||||
-include dist/elp-c-decls.h -Wno-error=implicit-function-declaration \
|
||||
-o <scratch>/soul <scratch>/*.c vendor/el-runtime/v1.0.0-20260501/el_runtime.c \
|
||||
-lssl -lcrypto -lcurl -lpthread -lm
|
||||
```
|
||||
Source: `_engine-plainchat-20260805/README.md:396-412`. Verified today: 0 errors, 887,296 B.
|
||||
`elb` ALSO rewrites every `*.elh` in the tree (cosmetic em-dash→hyphen in the auto-gen banner,
|
||||
plus true-ups) and drops a stray `soul..elh` — `git restore` the unrelated ones and delete the
|
||||
stray before staging, or the diff drowns in noise.
|
||||
|
||||
## SELF-REVIEW FIX LIST (found by reading my own diff, 2026-08-06 — apply in ONE batch, then rebuild once)
|
||||
- **F3 (CORRECTNESS, do first):** the assistant echo currently replays the provider's FULL
|
||||
`tool_calls` array (`tc_arr`) while the loop answers only the FIRST call. If a provider ignores
|
||||
`parallel_tool_calls:false`, the next request carries an assistant turn with N tool_calls and
|
||||
only ONE `role:"tool"` response → most OpenAI-format providers 400 ("missing tool response for
|
||||
id X") and the run dies. This is the same class as ADR-0005's Anthropic failure, but here it is
|
||||
cheap to close: echo ONLY the honored call (`"[" + tc0 + "]"`), so the conversation we send is
|
||||
self-consistent and the dropped call never existed from the model's view. The DRIFT log line
|
||||
stays (honest accounting of what we dropped).
|
||||
- **F4 (efficiency/latency):** `handle_chat_agentic` computes `agentic_tools_all()` at ~:2681
|
||||
BEFORE the fork, then the OpenAI branch computes `agentic_tools_no_web()` again — two
|
||||
`connector_tools_json()` calls per turn, each an HTTP round-trip to the connector bridge on
|
||||
:7771 (two timeout exposures). Fix: compute the tools array ONCE, per lane, after `use_openai`
|
||||
is known (check no other use of `tools_json` sits between :2681 and the fork before moving it).
|
||||
Note: `openai_tools_json()` already skips any entry with no `input_schema`, so Anthropic's
|
||||
server-side `web_search` entry is auto-dropped even if the full array is passed —
|
||||
`agentic_tools_no_web()` is kept for EXPLICITNESS, not necessity.
|
||||
- **F1 (debuggability):** the "no choices in response" branch logs a generic string and discards
|
||||
the body. Log the response head (as the `is_error` branch does) — a provider that returns 200
|
||||
with an unexpected shape is otherwise undiagnosable from the log.
|
||||
- **OPEN QUESTION (evidence pending from the gate):** the tool-result feedback turn escapes with
|
||||
`json_escape()` (this lane's escaper) rather than `json_safe()` (used everywhere else). The
|
||||
Anthropic lane escapes that field with NEITHER, which is a latent defect on that side. If the
|
||||
torture scenario shows any escaping loss, switch to `json_safe` and note the Anthropic-side
|
||||
finding for Will.
|
||||
|
||||
## TEST HARNESS — built 2026-08-06 (Task 4 side-work, reusable by anyone)
|
||||
- `tests/run-el-test.sh <tests/test_x.el> | --all` — the engine tests were NEVER runnable
|
||||
before this (`elc` is a compiler: emits C to stdout and exits). It emits the test to C,
|
||||
compiles `soul.c` separately with `main` renamed away (soul.c owns the daemon's real main
|
||||
but also defines `layered_cycle` et al.), links the remaining modules + the repo-pinned
|
||||
runtime, and executes. Modules cached under `/tmp/el-test-<worktree>/`; `REBUILD=1` forces.
|
||||
- **The runner computes the verdict itself** because the test FILES cannot: all 9 counted
|
||||
test files do `let pass_count = pass_count + 1` inside an if BLOCK, which El scoping
|
||||
discards, so every summary line reads `0 passed, 0 failed` forever. Per-assertion
|
||||
`PASS:`/`FAIL:` lines ARE reliable; the runner counts those, exits non-zero on any FAIL
|
||||
or on zero assertions, and was proven to discriminate with a negative control (broken
|
||||
assertion → 31 passed / 1 failed / exit 1). Real in-file fix filed: **neuron#116**.
|
||||
- `tests/test_bridge_serialization.el`: 4 `bridge_save` calls updated for the new `wire`
|
||||
argument, plus **Section 9** (8 new assertions) covering wire round-trip both ways, the
|
||||
legacy no-wire blob (resumes as anthropic), and a FIELD-ORDER decoy guard — a fake
|
||||
`"wire":"anthropic"` planted inside `messages_raw` must not beat the blob's own scalar.
|
||||
That decoy is the round-9 first-match-scanner bug class, now pinned by a test. **32/32 green.**
|
||||
|
||||
## MEMORY-SAVE CAVEAT RESOLVED 2026-08-06
|
||||
Earlier saves this session reported `-> OUTBOX only (real mind unreachable or read-back
|
||||
failed)`. That was a **read-back verifier false negative, not data loss** — a direct
|
||||
`POST :7770/api/neuron/recall` returns those notes from the live mind verbatim. Another
|
||||
terminal was fixing exactly this (multi-word read-back probe) the same afternoon. Do NOT
|
||||
re-save on an OUTBOX report without first querying the mind directly, or you duplicate nodes.
|
||||
|
||||
## Standing cautions
|
||||
- PERSIST OFF on the real mind this boot (neuron#98/#92): journal saves only, ferry later. MCP link
|
||||
down this terminal; use neuron_remember.py / neuron_recall.py.
|
||||
- Aug-16: Groq retires llama-3.3-70b-versatile (separate P0, Tim's call, catalog swap).
|
||||
- Never bind 7770/7779/17779; never touch ~/.neuron; round-9 worktrees read-only.
|
||||
@@ -1408,7 +1408,7 @@ fn session_preload_bullets(nodes: String, max_bullets: Int, snip_len: Int) -> St
|
||||
while i < limit {
|
||||
let node: String = json_array_get(nodes, i)
|
||||
let content: String = json_get(node, "content")
|
||||
let snip: String = utf8_safe_slice(content, snip_len)
|
||||
let snip: String = if str_len(content) > snip_len { str_slice(content, 0, snip_len) } else { content }
|
||||
let bullets = if str_eq(snip, "") {
|
||||
bullets
|
||||
} else {
|
||||
@@ -1770,14 +1770,7 @@ fn agentic_api_key() -> String {
|
||||
if !str_eq(k1, "") {
|
||||
return k1
|
||||
}
|
||||
let k2: String = env("NEURON_LLM_0_KEY")
|
||||
if !str_eq(k2, "") {
|
||||
return k2
|
||||
}
|
||||
// Step-0 bridge (2026-08-06): the shipped launcher also exports the Keychain key as
|
||||
// SOUL_API_KEY (neuron-daemons.sh). Honor it so a provider key configured through the
|
||||
// app reaches this lane without any launcher change.
|
||||
return env("SOUL_API_KEY")
|
||||
return env("NEURON_LLM_0_KEY")
|
||||
}
|
||||
|
||||
// ── OpenAI-compatible providers (Ollama / OpenAI / Grok / Gemini) ──────────────────────────────
|
||||
@@ -1785,42 +1778,19 @@ fn agentic_api_key() -> String {
|
||||
// OpenAI-compatible wire format (NEURON_LLM_0_FORMAT=openai) with a configured base URL
|
||||
// (NEURON_LLM_0_URL, e.g. http://localhost:11434/v1 for local Ollama), basic chat turns are served
|
||||
// here instead of the Anthropic agentic loop.
|
||||
// v2 SCOPE (2026-08-06, SPEC-soul-openai-tools-v2): tools + the agentic loop now run on
|
||||
// this wire too (openai_agentic_loop below). Plain completion (openai_chat_complete)
|
||||
// remains for non-agentic turns. Still ADDITIVE: the Anthropic path is untouched.
|
||||
// v1 SCOPE: plain chat completion only — NO tools / agentic loop yet (that is a follow-up port).
|
||||
// This block is ADDITIVE: the Anthropic path is untouched and stays the default.
|
||||
|
||||
fn llm_base_url() -> String {
|
||||
let u: String = env("NEURON_LLM_0_URL")
|
||||
if !str_eq(u, "") {
|
||||
return u
|
||||
}
|
||||
// Step-0 bridge (2026-08-06): the shipped launcher exports SOUL_LLM_BASE_URL +
|
||||
// SOUL_LLM_PROVIDER (installer/macos/neuron-daemons.sh:288-300) and nothing in a
|
||||
// customer build exports the NEURON_LLM_0_* names — so this lane was unreachable
|
||||
// outside test harnesses. Honor the launcher's names as a fallback. Anthropic
|
||||
// deliberately returns "" here: its native path stays hardcoded (endpoint
|
||||
// configurability is neuron#62, out of scope).
|
||||
let p: String = env("SOUL_LLM_PROVIDER")
|
||||
if str_eq(p, "") || str_eq(p, "anthropic") {
|
||||
return ""
|
||||
}
|
||||
return env("SOUL_LLM_BASE_URL")
|
||||
return env("NEURON_LLM_0_URL")
|
||||
}
|
||||
|
||||
fn llm_wire_format() -> String {
|
||||
let f: String = env("NEURON_LLM_0_FORMAT")
|
||||
if !str_eq(f, "") {
|
||||
return f
|
||||
if str_eq(f, "") {
|
||||
return "anthropic"
|
||||
}
|
||||
// Step-0 bridge (2026-08-06): derive the wire format from the launcher's provider
|
||||
// name when the explicit format is unset. Every non-Anthropic provider in the app's
|
||||
// catalog speaks the OpenAI-compatible format (ProviderKeys.kt: llmFormat="openai"
|
||||
// for openai/grok/gemini/groq/ollama).
|
||||
let p: String = env("SOUL_LLM_PROVIDER")
|
||||
if str_eq(p, "openai") || str_eq(p, "grok") || str_eq(p, "gemini") || str_eq(p, "groq") || str_eq(p, "ollama") {
|
||||
return "openai"
|
||||
}
|
||||
return "anthropic"
|
||||
return f
|
||||
}
|
||||
|
||||
// Escape a decoded string so it can be embedded back into a JSON string literal.
|
||||
@@ -1883,354 +1853,6 @@ fn openai_chat_complete(model: String, base_url: String, api_key: String, safe_s
|
||||
return "{\"reply\":\"" + json_escape(content) + "\",\"tools_used\":[]}"
|
||||
}
|
||||
|
||||
// ══ OpenAI-format TOOLS PORT (v2, 2026-08-06, SPEC-soul-openai-tools-v2) ═══════════════
|
||||
// The agentic loop for OpenAI-compatible providers (Groq/OpenAI/Grok/Gemini/Ollama).
|
||||
// The tool-execution, consent, bridge and run-progress machinery is the SAME wire-agnostic
|
||||
// layer agentic_loop uses (dispatch_tool, classify_tool_risk, is_builtin_tool, bridge_save,
|
||||
// handle_tool_result) — only the wire dialect differs. ADR-0005's single-tool constraint is
|
||||
// mirrored on this wire as parallel_tool_calls:false; a provider that ignores it gets its
|
||||
// first call honored and the rest dropped LOUDLY. Anthropic's server-side web_search has no
|
||||
// analogue here, so this lane's tool set comes from agentic_tools_no_web() and "sources"
|
||||
// is always empty — an honest degradation, disclosed in the spec, not a bug.
|
||||
|
||||
// Convert an Anthropic-shape tools array ({"name","description","input_schema"}) to the
|
||||
// OpenAI shape ({"type":"function","function":{"name","description","parameters"}}).
|
||||
// Entries without an input_schema (Anthropic server tools like web_search) are skipped —
|
||||
// they cannot execute on this wire.
|
||||
fn openai_tools_json(tools_anthropic: String) -> String {
|
||||
let out: String = ""
|
||||
let i: Int = 0
|
||||
let n: Int = json_array_len(tools_anthropic)
|
||||
while i < n {
|
||||
let entry: String = json_array_get(tools_anthropic, i)
|
||||
let name: String = json_get(entry, "name")
|
||||
let desc: String = json_get(entry, "description")
|
||||
let schema: String = json_get_raw(entry, "input_schema")
|
||||
let keep: Bool = !str_eq(name, "") && !str_eq(schema, "")
|
||||
let piece: String = if keep {
|
||||
"{\"type\":\"function\",\"function\":{\"name\":\"" + json_escape(name) + "\""
|
||||
+ ",\"description\":\"" + json_escape(desc) + "\""
|
||||
+ ",\"parameters\":" + schema + "}}"
|
||||
} else { "" }
|
||||
let out = if keep {
|
||||
if str_eq(out, "") { piece } else { out + "," + piece }
|
||||
} else { out }
|
||||
let i = i + 1
|
||||
}
|
||||
return "[" + out + "]"
|
||||
}
|
||||
|
||||
// utf8_safe_slice — str_slice with the guarantee that it never splits a character.
|
||||
//
|
||||
// str_slice and str_len count BYTES. Every fixed-length content cut in this file
|
||||
// therefore risks landing inside a multi-byte UTF-8 character and leaving a dangling
|
||||
// lead byte, which makes the ENTIRE request body invalid UTF-8 — providers reject it
|
||||
// and the user gets an unexplained failure. Found live 2026-08-06 in the session
|
||||
// preload: a recalled memory containing box-drawing rules (U+2500 = E2 94 80) was cut
|
||||
// at 350 bytes mid-character, and every turn on that session died. Ordinary content
|
||||
// triggers it — an em dash, a curly quote, an accented name, an emoji — and it gets
|
||||
// MORE likely as a user's memory grows.
|
||||
//
|
||||
// Walk back from the cut over UTF-8 continuation bytes (0x80-0xBF) to the lead byte,
|
||||
// and keep the character only if all of its bytes survived the cut.
|
||||
fn utf8_safe_slice(s: String, n: Int) -> String {
|
||||
if str_len(s) <= n { return s }
|
||||
let cut: String = str_slice(s, 0, n)
|
||||
let total: Int = str_len(cut)
|
||||
let i: Int = total - 1
|
||||
let keep: Int = total
|
||||
let scanning: Bool = true
|
||||
let steps: Int = 0
|
||||
// A UTF-8 character is at most 4 bytes, so at most 4 steps are ever needed.
|
||||
while scanning && steps < 4 && i >= 0 {
|
||||
let c: Int = str_char_code(cut, i)
|
||||
let is_ascii: Bool = c < 128
|
||||
let is_lead: Bool = c >= 192
|
||||
// Expected length declared by the lead byte: 0xF0+ = 4, 0xE0+ = 3, else 2.
|
||||
let need: Int = if c >= 240 { 4 } else { if c >= 224 { 3 } else { 2 } }
|
||||
let have: Int = total - i
|
||||
let keep = if is_ascii { total } else {
|
||||
if is_lead { if have == need { total } else { i } } else { keep }
|
||||
}
|
||||
let scanning = if is_ascii || is_lead { false } else { true }
|
||||
let i = i - 1
|
||||
let steps = steps + 1
|
||||
}
|
||||
return str_slice(cut, 0, keep)
|
||||
}
|
||||
|
||||
// A tool result arrives already json_safe'd from dispatch_tool, so it is embedded into
|
||||
// the wire message RAW (escaping it a second time is what made the model read literal
|
||||
// backslashes). But it is also TRUNCATED at a fixed byte count, and a cut can land in the
|
||||
// middle of an escape pair — leaving a dangling backslash that makes the enclosing JSON
|
||||
// string invalid and 400s the whole turn. Trim any trailing backslash run so the cut is
|
||||
// always on a clean boundary. (The Anthropic lane truncates the same way and has the same
|
||||
// latent exposure; not changed here, flagged in the PR.)
|
||||
fn json_trim_dangling_escape(s: String) -> String {
|
||||
let out: String = s
|
||||
while str_ends_with(out, "\\") {
|
||||
let out = str_slice(out, 0, str_len(out) - 1)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// The standard agentic tool set WITHOUT Anthropic's native web_search entry: built-ins +
|
||||
// every connector tool. Same merge as agentic_tools_all(), minus the server-tool tail.
|
||||
fn agentic_tools_no_web() -> String {
|
||||
let base: String = agentic_tools_literal()
|
||||
let conn: String = connector_tools_json()
|
||||
let base_inner: String = str_slice(base, 1, str_len(base) - 1)
|
||||
let conn_inner: String = str_slice(conn, 1, str_len(conn) - 1)
|
||||
let merged: String = if str_eq(conn_inner, "") {
|
||||
base_inner
|
||||
} else {
|
||||
base_inner + "," + conn_inner
|
||||
}
|
||||
return "[" + strip_client_web_search(merged) + "]"
|
||||
}
|
||||
|
||||
// openai_agentic_loop — the resumable agentic turn on the OpenAI wire. Same two envelopes
|
||||
// as agentic_loop (done / tool_pending), same client-bridge contract, same state keys.
|
||||
// [tools_json] arrives ANTHROPIC-shaped (the bridge blob stays wire-uniform); it is
|
||||
// converted once here. The system prompt travels as the first message (no top-level
|
||||
// "system" on this wire).
|
||||
fn openai_agentic_loop(session_id: String, model: String, safe_sys: String, tools_json: String, messages_in: String, tools_log_in: String) -> String {
|
||||
let api_url: String = llm_base_url() + "/chat/completions"
|
||||
let api_key: String = agentic_api_key()
|
||||
let h: Map = {}
|
||||
map_set(h, "content-type", "application/json")
|
||||
if !str_eq(api_key, "") {
|
||||
map_set(h, "Authorization", "Bearer " + api_key)
|
||||
}
|
||||
let ask_all: Bool = !str_eq(session_id, "") && str_eq(state_get("require_approval_" + session_id), "true")
|
||||
let tools_oai: String = openai_tools_json(tools_json)
|
||||
let has_tools: Bool = json_array_len(tools_oai) > 0
|
||||
|
||||
let messages: String = messages_in
|
||||
let final_text: String = ""
|
||||
let tools_log: String = tools_log_in
|
||||
let iteration: Int = 0
|
||||
let keep_going: Bool = true
|
||||
|
||||
// Suspension state — top level so it escapes the while body (El scope rule).
|
||||
let pending: Bool = false
|
||||
let pend_tool_id: String = ""
|
||||
let pend_tool_name: String = ""
|
||||
let pend_tool_input: String = ""
|
||||
let pend_tool_tier: String = ""
|
||||
let pend_narration: String = ""
|
||||
|
||||
if !str_eq(session_id, "") {
|
||||
state_set("run_progress_" + session_id, "")
|
||||
}
|
||||
|
||||
while keep_going && iteration < 12 {
|
||||
let inner_msgs: String = str_slice(messages, 1, str_len(messages) - 1)
|
||||
let all_msgs: String = if str_eq(inner_msgs, "") {
|
||||
"[{\"role\":\"system\",\"content\":\"" + safe_sys + "\"}]"
|
||||
} else {
|
||||
"[{\"role\":\"system\",\"content\":\"" + safe_sys + "\"}," + inner_msgs + "]"
|
||||
}
|
||||
// tools + the ADR-0005 mirror travel only when there are tools to offer: an empty
|
||||
// tools array is a 400 on real OpenAI-format providers.
|
||||
let tool_frag: String = if has_tools {
|
||||
",\"tools\":" + tools_oai + ",\"tool_choice\":\"auto\",\"parallel_tool_calls\":false"
|
||||
} else { "" }
|
||||
let req_body: String = "{\"model\":\"" + model + "\""
|
||||
+ ",\"max_tokens\":16384"
|
||||
+ tool_frag
|
||||
+ ",\"messages\":" + all_msgs
|
||||
+ "}"
|
||||
|
||||
let raw_resp: String = http_post_with_headers(api_url, req_body, h)
|
||||
// OpenAI-format errors arrive as a top-level {"error":{...}} object. Content
|
||||
// strings inside a valid response are JSON-escaped, so a top-level match cannot
|
||||
// false-positive on reply text.
|
||||
let is_error: Bool = str_eq(raw_resp, "") || str_starts_with(raw_resp, "{\"error\"")
|
||||
if is_error {
|
||||
let err_head: String = if str_len(raw_resp) > 220 { str_slice(raw_resp, 0, 220) } else { raw_resp }
|
||||
println("[soul] llm error (openai lane): " + err_head)
|
||||
return "{\"error\":\"llm unavailable\",\"reply\":\"\"}"
|
||||
}
|
||||
|
||||
let choices: String = json_get_raw(raw_resp, "choices")
|
||||
let eff_choices: String = if str_eq(choices, "") { "[]" } else { choices }
|
||||
if json_array_len(eff_choices) < 1 {
|
||||
// Log the body head, as the error branch does. A provider that answers 200
|
||||
// with an unexpected shape is otherwise undiagnosable from the log alone.
|
||||
let noc_head: String = if str_len(raw_resp) > 220 { str_slice(raw_resp, 0, 220) } else { raw_resp }
|
||||
println("[soul] llm error (openai lane): no choices in response: " + noc_head)
|
||||
return "{\"error\":\"llm unavailable\",\"reply\":\"\"}"
|
||||
}
|
||||
let first: String = json_array_get(eff_choices, 0)
|
||||
let message_o: String = json_get_raw(first, "message")
|
||||
let finish: String = json_get(first, "finish_reason")
|
||||
// Content, read TWO ways on purpose.
|
||||
// content_raw — the provider's own bytes: `null` on a pure tool-call turn, or a
|
||||
// quoted, already-escaped string. This is what goes back on the wire, verbatim.
|
||||
// text_out — the DECODED text, for narration, the ledger and the final reply.
|
||||
// json_get decodes, and a JSON null decodes to the 4-char string "null", which is
|
||||
// never "" — so without the raw check a pure tool-call turn produced the literal
|
||||
// word "null" as the assistant's narration, in the run-progress ledger, and inside
|
||||
// the tool_pending envelope, and echoed `"content":"null"` instead of `content:null`.
|
||||
let content_raw: String = json_get_raw(message_o, "content")
|
||||
let is_null_content: Bool = str_eq(content_raw, "null") || str_eq(content_raw, "")
|
||||
let text_out: String = if is_null_content { "" } else { json_get(message_o, "content") }
|
||||
let tc_raw: String = json_get_raw(message_o, "tool_calls")
|
||||
let tc_arr: String = if str_eq(tc_raw, "") || str_eq(tc_raw, "null") { "[]" } else { tc_raw }
|
||||
let tc_n: Int = json_array_len(tc_arr)
|
||||
let has_tool: Bool = tc_n > 0
|
||||
|
||||
// ADR-0005 mirror: we ask for one call per round; a provider that returns
|
||||
// several anyway gets the FIRST honored and the drop logged loudly.
|
||||
if tc_n > 1 {
|
||||
println("[soul] DRIFT: provider returned " + int_to_str(tc_n) + " parallel tool_calls despite parallel_tool_calls:false - keeping the first only (ADR-0005 mirror)")
|
||||
}
|
||||
// Unknown finish reasons (future API drift): log loudly, never treat an
|
||||
// unrecognised terminal state as a completed answer silently.
|
||||
if !str_eq(finish, "stop") && !str_eq(finish, "tool_calls") && !str_eq(finish, "length") && !str_eq(finish, "") {
|
||||
println("[soul] DRIFT: unknown finish_reason from API: " + finish)
|
||||
}
|
||||
|
||||
let tc0: String = if has_tool { json_array_get(tc_arr, 0) } else { "" }
|
||||
let tool_id: String = if has_tool { json_get(tc0, "id") } else { "" }
|
||||
let tc_fn: String = if has_tool { json_get_raw(tc0, "function") } else { "" }
|
||||
let tool_name: String = if has_tool { json_get(tc_fn, "name") } else { "" }
|
||||
// arguments is a JSON-ENCODED STRING on this wire; json_get decodes it exactly
|
||||
// once, yielding the raw object text dispatch_tool expects. Decoding again — or
|
||||
// re-encoding before dispatch — is the two-escaper trap the gate's echo-mismatch
|
||||
// check exists to catch.
|
||||
let tool_input_raw: String = if has_tool { json_get(tc_fn, "arguments") } else { "" }
|
||||
let tool_input: String = if str_eq(tool_input_raw, "") { "{}" } else { tool_input_raw }
|
||||
|
||||
let is_tool_turn: Bool = has_tool
|
||||
|
||||
// Consent policy — IDENTICAL to the Anthropic lane: ask_all bridges everything,
|
||||
// escalate always bridges, non-builtins bridge unless "always allow" granted.
|
||||
let always_key: String = "always_allow_" + session_id
|
||||
let always_list: String = if !str_eq(session_id, "") { state_get(always_key) } else { "" }
|
||||
let is_always_allowed: Bool = !str_eq(tool_name, "") && !str_eq(always_list, "") && str_contains(always_list, tool_name)
|
||||
let risk_tier: String = if is_tool_turn { classify_tool_risk(tool_name, tool_input) } else { "" }
|
||||
let needs_bridge: Bool = is_tool_turn && (ask_all || str_eq(risk_tier, "escalate") || (!is_builtin_tool(tool_name) && !is_always_allowed))
|
||||
|
||||
let tool_result_raw: String = if is_tool_turn && !needs_bridge { dispatch_tool(tool_name, tool_input) } else { "" }
|
||||
let tool_result: String = if str_len(tool_result_raw) > 6000 {
|
||||
json_trim_dangling_escape(str_slice(tool_result_raw, 0, 6000)) + "...[truncated]"
|
||||
} else { tool_result_raw }
|
||||
|
||||
let tool_quoted: String = "\"" + tool_name + "\""
|
||||
let tools_log = if is_tool_turn {
|
||||
if str_eq(tools_log, "") { tool_quoted } else { tools_log + "," + tool_quoted }
|
||||
} else { tools_log }
|
||||
|
||||
// The assistant turn echoed with its tool_calls array VERBATIM (raw), so the
|
||||
// tool_call_id pairing stays valid on the wire and across a bridge resume.
|
||||
// Echo the provider's content BYTES, never a re-escaped round-trip. Decoding and
|
||||
// re-encoding is where fidelity is lost: json_escape/json_safe both handle only
|
||||
// \\ " \n \r, so any other control character the model emits (a tab, say) would go
|
||||
// back out raw and make the next request body invalid JSON — a provider 400 that
|
||||
// looks like a random failure. The Anthropic lane never had this exposure because
|
||||
// it echoes the response's content array untouched; this now matches it.
|
||||
let content_frag: String = if is_null_content { "null" } else { content_raw }
|
||||
// Echo ONLY the call we actually honor — never the provider's full array.
|
||||
// The loop can assemble exactly one tool response per round, so replaying N
|
||||
// tool_calls while answering one leaves the conversation self-contradictory and
|
||||
// every OpenAI-format provider 400s on the next request ("no tool response for
|
||||
// id X"). That is precisely the failure ADR-0005 documents on the Anthropic wire,
|
||||
// where the block walk keeps the first tool_use and the rest die without a
|
||||
// tool_result. Here it costs one slice to close: the dropped calls simply never
|
||||
// existed from the model's point of view, and the DRIFT line above keeps the
|
||||
// accounting honest about what we discarded.
|
||||
let assist_turn: String = if has_tool {
|
||||
"{\"role\":\"assistant\",\"content\":" + content_frag + ",\"tool_calls\":[" + tc0 + "]}"
|
||||
} else {
|
||||
"{\"role\":\"assistant\",\"content\":" + content_frag + "}"
|
||||
}
|
||||
let inner_now: String = str_slice(messages, 1, str_len(messages) - 1)
|
||||
let messages_with_assistant: String = "[" + inner_now + "," + assist_turn + "]"
|
||||
|
||||
// Local built-in tool turn: append assistant echo + role:"tool" result, loop on.
|
||||
let local_continue: Bool = is_tool_turn && !needs_bridge
|
||||
let messages = if local_continue {
|
||||
let inner2: String = str_slice(messages_with_assistant, 1, str_len(messages_with_assistant) - 1)
|
||||
"[" + inner2 + ",{\"role\":\"tool\",\"tool_call_id\":\"" + tool_id + "\",\"content\":\"" + tool_result + "\"}]"
|
||||
} else { messages }
|
||||
|
||||
// Live run-progress ledger — same key, same shape, same poller as the Anthropic
|
||||
// lane; a forked loop that omitted this would silently kill live step rendering.
|
||||
if !str_eq(session_id, "") {
|
||||
let prog_key: String = "run_progress_" + session_id
|
||||
let prog_prev: String = state_get(prog_key)
|
||||
let prog_snip: String = if str_len(text_out) > 280 { str_slice(text_out, 0, 280) } else { text_out }
|
||||
let prog_entry: String = "{\"i\":" + int_to_str(iteration)
|
||||
+ ",\"t\":\"" + json_safe(prog_snip) + "\""
|
||||
+ ",\"tool\":\"" + json_safe(tool_name) + "\"}"
|
||||
let prog_next: String = if str_eq(prog_prev, "") { prog_entry } else { prog_prev + "," + prog_entry }
|
||||
state_set(prog_key, prog_next)
|
||||
}
|
||||
|
||||
// Bridge turn: persist the continuation (wire-tagged) and stop the loop.
|
||||
let pending = if needs_bridge { true } else { pending }
|
||||
let pend_tool_id = if needs_bridge { tool_id } else { pend_tool_id }
|
||||
let pend_tool_name = if needs_bridge { tool_name } else { pend_tool_name }
|
||||
let pend_tool_input = if needs_bridge { tool_input } else { pend_tool_input }
|
||||
let pend_tool_tier = if needs_bridge { risk_tier } else { pend_tool_tier }
|
||||
let pend_narration = if needs_bridge { text_out } else { pend_narration }
|
||||
if needs_bridge {
|
||||
bridge_save(session_id, model, safe_sys, tools_json, messages_with_assistant, tools_log, tool_id, "openai")
|
||||
}
|
||||
|
||||
// Text accumulation: rounds are separated by tool executions, so the resume seam
|
||||
// is unconditionally a boundary (same rule as the Anthropic loop's seam 2).
|
||||
let final_text = if !is_tool_turn {
|
||||
final_text + text_join_sep(final_text, text_out, true) + text_out
|
||||
} else { final_text }
|
||||
// Output cap hit mid-action (finish_reason "length" with a tool call pending).
|
||||
let final_text = if str_eq(finish, "length") && has_tool {
|
||||
final_text + "\n\n[Output limit reached mid-action - the last planned action did not run. Ask me to continue to finish it.]"
|
||||
} else { final_text }
|
||||
let keep_going = if local_continue { keep_going } else { false }
|
||||
let iteration = iteration + 1
|
||||
}
|
||||
|
||||
if pending {
|
||||
let safe_in: String = if str_eq(pend_tool_input, "") { "{}" } else { pend_tool_input }
|
||||
let tools_arr: String = if str_eq(tools_log, "") { "[]" } else { "[" + tools_log + "]" }
|
||||
return "{\"tool_pending\":true"
|
||||
+ ",\"session_id\":\"" + session_id + "\""
|
||||
+ ",\"call_id\":\"" + pend_tool_id + "\""
|
||||
+ ",\"tool_name\":\"" + pend_tool_name + "\""
|
||||
+ ",\"tool_input\":" + safe_in
|
||||
+ ",\"risk_tier\":\"" + pend_tool_tier + "\""
|
||||
+ ",\"narration\":\"" + json_safe(pend_narration) + "\""
|
||||
+ ",\"model\":\"" + model + "\""
|
||||
+ ",\"agentic\":true"
|
||||
+ ",\"sources\":\"\""
|
||||
+ ",\"tools_used\":" + tools_arr + "}"
|
||||
}
|
||||
|
||||
let final_text = receipt_strip(final_text)
|
||||
if str_eq(final_text, "") {
|
||||
let hit_cap: Bool = iteration >= 12
|
||||
let err_msg: String = if hit_cap {
|
||||
"agentic loop hit the 12-iteration cap without producing a final reply - task may be too complex or a tool call is looping"
|
||||
} else {
|
||||
"no response"
|
||||
}
|
||||
return "{\"error\":\"" + err_msg + "\",\"reply\":\"\",\"iterations\":" + int_to_str(iteration) + "}"
|
||||
}
|
||||
|
||||
let safe_text: String = json_safe(final_text)
|
||||
let tools_arr: String = if str_eq(tools_log, "") { "[]" } else { "[" + tools_log + "]" }
|
||||
if !str_eq(session_id, "") {
|
||||
let done_key: String = "run_progress_" + session_id
|
||||
let done_prev: String = state_get(done_key)
|
||||
let done_next: String = if str_eq(done_prev, "") { "{\"done\":true}" } else { done_prev + ",{\"done\":true}" }
|
||||
state_set(done_key, done_next)
|
||||
}
|
||||
return "{\"reply\":\"" + safe_text + "\",\"model\":\"" + model + "\",\"agentic\":true,\"tools_used\":" + tools_arr + ",\"sources\":\"\",\"iterations\":" + int_to_str(iteration) + "}"
|
||||
}
|
||||
|
||||
fn agentic_tools_literal() -> String {
|
||||
return "[" +
|
||||
"{\"name\":\"read_file\",\"description\":\"Read contents of a file from disk.\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"path\":{\"type\":\"string\",\"description\":\"Absolute file path\"}},\"required\":[\"path\"]}}," +
|
||||
@@ -3014,7 +2636,7 @@ fn handle_chat_agentic(body: String) -> String {
|
||||
let ag_continuity_snip: String = if ag_continuity_ok {
|
||||
let acn0: String = json_array_get(ag_continuity_nodes, 0)
|
||||
let acc: String = json_get(acn0, "content")
|
||||
utf8_safe_slice(acc, 350)
|
||||
if str_len(acc) > 350 { str_slice(acc, 0, 350) } else { acc }
|
||||
} else { "" }
|
||||
let ag_profile_bullets: String = session_preload_bullets(ag_profile_nodes2, 8, 350)
|
||||
let ag_work_bullets: String = session_preload_bullets(ag_work_nodes2, 6, 350)
|
||||
@@ -3045,13 +2667,7 @@ fn handle_chat_agentic(body: String) -> String {
|
||||
" + ctx + ag_session_preload + receipt_rule()
|
||||
|
||||
let api_key: String = agentic_api_key()
|
||||
// Assemble the tool set ONCE, for the lane this turn will actually take. Both
|
||||
// builders call connector_tools_json(), which is an HTTP round-trip to the
|
||||
// connectors bridge on :7771 — computing both would pay that cost, and its timeout
|
||||
// exposure, twice per turn. The OpenAI lane drops Anthropic's server-side
|
||||
// web_search (it has no analogue on that wire and cannot execute there).
|
||||
let tools_lane_openai: Bool = !str_eq(llm_base_url(), "") && str_eq(llm_wire_format(), "openai")
|
||||
let tools_json: String = if tools_lane_openai { agentic_tools_no_web() } else { agentic_tools_all() }
|
||||
let tools_json: String = agentic_tools_all()
|
||||
let safe_msg: String = json_safe(message)
|
||||
let safe_sys: String = json_safe(system)
|
||||
|
||||
@@ -3092,12 +2708,11 @@ fn handle_chat_agentic(body: String) -> String {
|
||||
// for the rest of the run. Absent/false = behavior identical to before this fix.
|
||||
let req_ask_all: String = json_get(body, "require_approval")
|
||||
state_set("require_approval_" + session_id, if str_eq(req_ask_all, "true") { "true" } else { "" })
|
||||
// Provider fork (v2 port, 2026-08-06): OpenAI-compatible providers now take their own
|
||||
// AGENTIC loop — same tools (minus Anthropic-server web_search), same consent policy,
|
||||
// same bridge contract. The Anthropic native path stays the default and is untouched.
|
||||
let use_openai: Bool = tools_lane_openai
|
||||
// Provider fork: OpenAI-compatible providers (Ollama/OpenAI/Grok/Gemini) take the plain-completion
|
||||
// path (v1, no tools); everything else stays on the Anthropic agentic loop (the default).
|
||||
let use_openai: Bool = !str_eq(llm_base_url(), "") && str_eq(llm_wire_format(), "openai")
|
||||
let result: String = if use_openai {
|
||||
openai_agentic_loop(session_id, model, safe_sys, tools_json, messages, "")
|
||||
openai_chat_complete(model, llm_base_url(), agentic_api_key(), safe_sys, messages)
|
||||
} else {
|
||||
agentic_loop(session_id, model, safe_sys, tools_json, messages, h, "")
|
||||
}
|
||||
@@ -3240,6 +2855,30 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
|
||||
+ ",\"messages\":" + messages
|
||||
+ "}"
|
||||
|
||||
// ── ROUND-START MARKER (2026-08-06, round 9.1 D2 / ADR 0006 item 2) ──────────
|
||||
// The ledger below only ever appended AFTER a round returned, so a healthy
|
||||
// first leg produced ZERO progress by construction. Since server-side
|
||||
// web_search moved inside the outbound call (2026-08-04) that leg measures
|
||||
// 84-117 s, and the client had no way to tell "working" from "dead" — which is
|
||||
// how a 25 s client-side watchdog came to kill a healthy mission.
|
||||
//
|
||||
// Only this loop knows a round has started, so only this loop can say so. One
|
||||
// entry, written BEFORE the call goes out, using the ledger and the wire shape
|
||||
// that already exist: the app has handled tool == "__working__" as an
|
||||
// Activity-only life signal since 2026-07-13 (ChatView.kt:1148) and never
|
||||
// received one. Narration is deliberately empty - the marker means "a round
|
||||
// started", nothing more, and the client renders it as a heartbeat, not prose.
|
||||
//
|
||||
// This is a strict subset of WS3 item 3 (push/poll progress). It builds none of
|
||||
// WS3's run registry: no new state key, no new route, no new lifecycle.
|
||||
if !str_eq(session_id, "") {
|
||||
let start_key: String = "run_progress_" + session_id
|
||||
let start_prev: String = state_get(start_key)
|
||||
let start_entry: String = "{\"i\":" + int_to_str(iteration) + ",\"t\":\"\",\"tool\":\"__working__\"}"
|
||||
let start_next: String = if str_eq(start_prev, "") { start_entry } else { start_prev + "," + start_entry }
|
||||
state_set(start_key, start_next)
|
||||
}
|
||||
|
||||
let raw_resp: String = http_post_with_headers(api_url, req_body, h)
|
||||
|
||||
let is_error: Bool = str_starts_with(raw_resp, "{\"error\"")
|
||||
@@ -3500,7 +3139,7 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
|
||||
// client's tool_result block. messages_with_assistant is only meaningful when a
|
||||
// tool was requested, so guard on needs_bridge before persisting.
|
||||
if needs_bridge {
|
||||
bridge_save(session_id, model, safe_sys, tools_json, messages_with_assistant, tools_log, pend_tool_id, "anthropic")
|
||||
bridge_save(session_id, model, safe_sys, tools_json, messages_with_assistant, tools_log, pend_tool_id)
|
||||
}
|
||||
|
||||
// ACCUMULATE across pause/resume cycles instead of overwriting. A resumed turn
|
||||
@@ -3582,7 +3221,7 @@ fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json:
|
||||
// single JSON blob in soul state so agentic_resume can rebuild the exact loop. The
|
||||
// stored `messages` already includes the assistant turn that requested the tool, so
|
||||
// resume just appends the client's tool_result for `tool_use_id`.
|
||||
fn bridge_save(session_id: String, model: String, safe_sys: String, tools_json: String, messages: String, tools_log: String, tool_use_id: String, wire: String) -> Bool {
|
||||
fn bridge_save(session_id: String, model: String, safe_sys: String, tools_json: String, messages: String, tools_log: String, tool_use_id: String) -> Bool {
|
||||
// Guard: empty messages or tools_json would produce syntactically invalid JSON.
|
||||
// Return false so the caller detects the failure rather than writing a corrupt
|
||||
// blob that agentic_resume would later resume with no context.
|
||||
@@ -3612,13 +3251,10 @@ fn bridge_save(session_id: String, model: String, safe_sys: String, tools_json:
|
||||
// messages_raw — arbitrary model/user content — so neither raw extraction can
|
||||
// first-match into model-controlled bytes either. Do not reorder; do not add a
|
||||
// field after messages_raw.
|
||||
// "wire" (v2 port, 2026-08-06) is a json_safe'd SCALAR and therefore sits with the
|
||||
// other scalars BEFORE both raw fields, per the field-order rule above.
|
||||
let blob: String = "{\"model\":\"" + json_safe(model) + "\""
|
||||
+ ",\"safe_sys\":\"" + json_safe(safe_sys) + "\""
|
||||
+ ",\"tools_log\":\"" + json_safe(tools_log) + "\""
|
||||
+ ",\"tool_use_id\":\"" + json_safe(tool_use_id) + "\""
|
||||
+ ",\"wire\":\"" + json_safe(wire) + "\""
|
||||
+ ",\"tools_raw\":" + tools_json
|
||||
+ ",\"messages_raw\":" + messages + "}"
|
||||
state_set("mcp_bridge:" + session_id, blob)
|
||||
@@ -3674,52 +3310,14 @@ fn agentic_resume(session_id: String, tool_use_id: String, content: String) -> S
|
||||
str_slice(content, 0, 6000) + "...[truncated]"
|
||||
} else { content }
|
||||
let safe_result: String = json_safe(trimmed)
|
||||
let tool_msg: String = "{\"type\":\"tool_result\",\"tool_use_id\":\"" + eff_use_id + "\",\"content\":\"" + safe_result + "\"}"
|
||||
|
||||
let inner: String = str_slice(messages, 1, str_len(messages) - 1)
|
||||
let resumed_messages: String = "[" + inner + ",{\"role\":\"user\",\"content\":[" + tool_msg + "]}]"
|
||||
|
||||
// One-shot: clear the saved turn so a session_id can't be replayed.
|
||||
state_set("mcp_bridge:" + session_id, "")
|
||||
|
||||
// Wire-aware resume (v2 port, 2026-08-06): blobs written since the port carry a
|
||||
// "wire" scalar ("anthropic" | "openai") among the scalar fields, where first-match
|
||||
// scanning is safe (see bridge_save's field-order rule). A blob with no wire field
|
||||
// is a legacy pre-port suspension — always Anthropic. On the OpenAI wire the
|
||||
// client's result goes back as a role:"tool" turn keyed by tool_call_id, and a
|
||||
// result for an already-answered id must never be re-sent (the resumed messages
|
||||
// end at the assistant echo, so appending exactly one tool turn preserves that).
|
||||
// Read "wire" from the blob's SCALAR HEAD ONLY — never the whole blob.
|
||||
//
|
||||
// json_get is a first-substring-match scanner. On a blob written by this binary the
|
||||
// scalar sits ahead of the raw fields and wins, but on a LEGACY blob (suspended
|
||||
// before this field existed) there is no match up front, so the scan runs on into
|
||||
// messages_raw — model- and user-controlled bytes. A conversation that merely
|
||||
// CONTAINS the literal "wire":"openai" would then misroute the resume onto the wrong
|
||||
// loop and kill the run. That is exactly the round-9 defect (json_get(blob,
|
||||
// "tool_use_id") matching a web_search_tool_result id inside the replayed
|
||||
// conversation), and the fix is the same shape: bound the search.
|
||||
//
|
||||
// bridge_save guarantees every json_safe'd scalar precedes the bulk fields, so
|
||||
// truncating at the earliest bulk key makes this deterministic — the decoy is not
|
||||
// even inside the string we search. Both the current keys (tools_raw/messages_raw)
|
||||
// and the pre-round-9 legacy ones (tools_json/messages) are covered.
|
||||
let i_traw: Int = str_index_of(blob, ",\"tools_raw\":")
|
||||
let i_tjson: Int = str_index_of(blob, ",\"tools_json\":")
|
||||
let i_mraw: Int = str_index_of(blob, ",\"messages_raw\":")
|
||||
let i_msgs: Int = str_index_of(blob, ",\"messages\":")
|
||||
let cut1: Int = if i_traw > 0 { i_traw } else { str_len(blob) }
|
||||
let cut2: Int = if i_tjson > 0 && i_tjson < cut1 { i_tjson } else { cut1 }
|
||||
let cut3: Int = if i_mraw > 0 && i_mraw < cut2 { i_mraw } else { cut2 }
|
||||
let cut: Int = if i_msgs > 0 && i_msgs < cut3 { i_msgs } else { cut3 }
|
||||
let blob_head: String = str_slice(blob, 0, cut)
|
||||
let wire: String = json_get(blob_head, "wire")
|
||||
if str_eq(wire, "openai") {
|
||||
let tool_msg_o: String = "{\"role\":\"tool\",\"tool_call_id\":\"" + eff_use_id + "\",\"content\":\"" + safe_result + "\"}"
|
||||
let resumed_o: String = "[" + inner + "," + tool_msg_o + "]"
|
||||
return openai_agentic_loop(session_id, model, safe_sys, tools_json, resumed_o, tools_log)
|
||||
}
|
||||
|
||||
let tool_msg: String = "{\"type\":\"tool_result\",\"tool_use_id\":\"" + eff_use_id + "\",\"content\":\"" + safe_result + "\"}"
|
||||
let resumed_messages: String = "[" + inner + ",{\"role\":\"user\",\"content\":[" + tool_msg + "]}]"
|
||||
|
||||
let api_key: String = agentic_api_key()
|
||||
let h: Map = {}
|
||||
map_set(h, "x-api-key", api_key)
|
||||
@@ -3895,11 +3493,7 @@ fn handle_dharma_room_turn_agentic(body: String) -> String {
|
||||
// Hard Bell: pre-LLM safety evaluation on agentic dharma room turns.
|
||||
let system = safety_augment_system(system, transcript)
|
||||
|
||||
// One assembly, for the lane this turn takes (see the same note in handle_chat_agentic:
|
||||
// both builders hit the connectors bridge over HTTP, so computing both doubles the cost
|
||||
// and the timeout exposure).
|
||||
let use_openai_d: Bool = !str_eq(llm_base_url(), "") && str_eq(llm_wire_format(), "openai")
|
||||
let tools_json: String = if use_openai_d { agentic_tools_no_web() } else { agentic_tools_all() }
|
||||
let tools_json: String = agentic_tools_all()
|
||||
let safe_transcript: String = json_safe(transcript)
|
||||
let safe_sys: String = json_safe(system)
|
||||
let messages: String = "[{\"role\":\"user\",\"content\":\"" + safe_transcript + "\"}]"
|
||||
@@ -3910,14 +3504,7 @@ fn handle_dharma_room_turn_agentic(body: String) -> String {
|
||||
|
||||
// Use dharma-prefixed session_id so bridge suspension works correctly per room.
|
||||
let session_id: String = if str_eq(room_id, "") { "dharma:" + next_bridge_id() } else { "dharma:" + room_id }
|
||||
// Provider fork (v2 port, 2026-08-06): same routing rule as handle_chat_agentic.
|
||||
// The Hard Bell augmentation above is baked into safe_sys BEFORE the fork, so the
|
||||
// safety pass is identical on both wires.
|
||||
let loop_result: String = if use_openai_d {
|
||||
openai_agentic_loop(session_id, model, safe_sys, tools_json, messages, "")
|
||||
} else {
|
||||
agentic_loop(session_id, model, safe_sys, tools_json, messages, h, "")
|
||||
}
|
||||
let loop_result: String = agentic_loop(session_id, model, safe_sys, tools_json, messages, h, "")
|
||||
|
||||
let result_error: String = json_get(loop_result, "error")
|
||||
if !str_eq(result_error, "") {
|
||||
|
||||
@@ -53,11 +53,6 @@ extern fn llm_base_url() -> String
|
||||
extern fn llm_wire_format() -> String
|
||||
extern fn json_escape(s: String) -> String
|
||||
extern fn openai_chat_complete(model: String, base_url: String, api_key: String, safe_sys: String, messages_json: String) -> String
|
||||
extern fn openai_tools_json(tools_anthropic: String) -> String
|
||||
extern fn utf8_safe_slice(s: String, n: Int) -> String
|
||||
extern fn json_trim_dangling_escape(s: String) -> String
|
||||
extern fn agentic_tools_no_web() -> String
|
||||
extern fn openai_agentic_loop(session_id: String, model: String, safe_sys: String, tools_json: String, messages_in: String, tools_log_in: String) -> String
|
||||
extern fn agentic_tools_literal() -> String
|
||||
extern fn web_search_tool_json() -> String
|
||||
extern fn strip_client_web_search(tools_inner: String) -> String
|
||||
@@ -80,7 +75,7 @@ extern fn next_bridge_id() -> String
|
||||
extern fn handle_chat_plan(body: String) -> String
|
||||
extern fn handle_chat_agentic(body: String) -> String
|
||||
extern fn agentic_loop(session_id: String, model: String, safe_sys: String, tools_json: String, messages_in: String, h: Map, tools_log_in: String) -> String
|
||||
extern fn bridge_save(session_id: String, model: String, safe_sys: String, tools_json: String, messages: String, tools_log: String, tool_use_id: String, wire: String) -> Bool
|
||||
extern fn bridge_save(session_id: String, model: String, safe_sys: String, tools_json: String, messages: String, tools_log: String, tool_use_id: String) -> Bool
|
||||
extern fn agentic_resume(session_id: String, tool_use_id: String, content: String) -> String
|
||||
extern fn handle_tool_result(session_id: String, body: String) -> String
|
||||
extern fn handle_chat_as_soul(body: String) -> String
|
||||
|
||||
+1
-6
@@ -141,7 +141,7 @@ el_val_t awareness_run(void);
|
||||
el_val_t axon_get(el_val_t path);
|
||||
el_val_t axon_post(el_val_t path, el_val_t body);
|
||||
el_val_t bounded_persona_floor(void);
|
||||
el_val_t bridge_save(el_val_t session_id, el_val_t model, el_val_t safe_sys, el_val_t tools_json, el_val_t messages, el_val_t tools_log, el_val_t tool_use_id, el_val_t wire);
|
||||
el_val_t bridge_save(el_val_t session_id, el_val_t model, el_val_t safe_sys, el_val_t tools_json, el_val_t messages, el_val_t tools_log, el_val_t tool_use_id);
|
||||
el_val_t build_form_from_json(el_val_t semantic_form_json, el_val_t lang_code);
|
||||
el_val_t build_np(el_val_t referent, el_val_t slots);
|
||||
el_val_t build_pp(el_val_t loc);
|
||||
@@ -875,11 +875,6 @@ el_val_t non_weak_past(el_val_t stem, el_val_t slot);
|
||||
el_val_t non_weak_present(el_val_t stem, el_val_t slot);
|
||||
el_val_t one_cycle(void);
|
||||
el_val_t openai_chat_complete(el_val_t model, el_val_t base_url, el_val_t api_key, el_val_t safe_sys, el_val_t messages_json);
|
||||
el_val_t openai_tools_json(el_val_t tools_anthropic);
|
||||
el_val_t json_trim_dangling_escape(el_val_t s);
|
||||
el_val_t utf8_safe_slice(el_val_t s, el_val_t n);
|
||||
el_val_t agentic_tools_no_web(void);
|
||||
el_val_t openai_agentic_loop(el_val_t session_id, el_val_t model, el_val_t safe_sys, el_val_t tools_json, el_val_t messages_in, el_val_t tools_log_in);
|
||||
el_val_t parse_float_x100(el_val_t s);
|
||||
el_val_t path_within_root(el_val_t path, el_val_t root);
|
||||
el_val_t peo_ah_past(el_val_t slot);
|
||||
|
||||
@@ -1,90 +0,0 @@
|
||||
# gate-openai — deterministic OpenAI-dialect provider stub
|
||||
|
||||
Staging home for the **soul-openai-tools-v2** gate scaffolding
|
||||
(`docs/specs/SPEC-soul-openai-tools-v2-2026-08-06.md`, test-plan rung 1:
|
||||
"stub first — discriminates before El code exists"). Sibling of gate9's
|
||||
Anthropic stub (`_wt-beta-round9/scripts/gate9/stub-llm.py`): same scenario
|
||||
mechanism, opposite wire dialect. Stdlib Python only, 127.0.0.1 only,
|
||||
refuses ports 7770/7779/17779. Run `./selftest.sh` — exit 0 is green.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `stub-openai.py` | HTTP server: `POST /v1/chat/completions` (OpenAI dialect), scenario-scripted responses, request validation, ground-truth JSONL log, hostile modes via `--mode` |
|
||||
| `scenarios-openai.json` | Scenario contract: scripts + markers + per-class/per-step request assertions |
|
||||
| `selftest.sh` | curl-driven proof of every scenario, every rejection, all hostile modes (58 checks) |
|
||||
|
||||
## What each scenario proves (when the brain drives it)
|
||||
|
||||
| Class | Proves |
|
||||
|---|---|
|
||||
| `oa-plain` | finish_reason `stop` ends the loop; tools + `tool_choice` + `parallel_tool_calls:false` were offered on the wire |
|
||||
| `oa-tools-off` | the chat-only lane sends NO tools (offering them there is a 400) |
|
||||
| `oa-single-tool` | full round-trip: `tool_calls` parsed, assistant echo + `role:"tool"` turn with matching `tool_call_id` sent back, final text reached |
|
||||
| `oa-torture` | `function.arguments` (JSON-encoded string with nested quotes, backslashes, newlines, tabs, unicode) survives exactly ONE decode — the stub recomputes the issued payload from the script and 400s on any drift (`gate_echo_mismatch`, the spec §6 two-escaper trap) |
|
||||
| `oa-parallel` | two `tool_calls` in one response: the brain either answers both (paired correctly) or rejects cleanly — an unpaired echo is a 400 |
|
||||
| `oa-mission` | multi-round loop continuation; step index = assistant-message count, so resume threads index correctly by construction |
|
||||
| `oa-api-error` | provider errors 400/429/500/503 in the OpenAI error envelope surface honestly, no retry storm |
|
||||
|
||||
Universal (every request, any scenario): Anthropic dialect leakage fails
|
||||
loudly with 400 — `anthropic-version` header, top-level `system` /
|
||||
`stop_sequences` / `max_tokens_to_sample`, `input_schema` inside tools,
|
||||
Anthropic content blocks (`tool_use`/`tool_result`/...). Tools must be
|
||||
`{type:"function", function:{name, description, parameters}}`, unique names;
|
||||
echoed `arguments` must be a JSON-encoded STRING, never a decoded object.
|
||||
|
||||
## Hostile modes (`--mode`, same file)
|
||||
|
||||
| Mode | Behavior | Brain invariant under test |
|
||||
|---|---|---|
|
||||
| `black-hole` | reads the request, never responds | HTTP timeout exists and surfaces; no silent hang |
|
||||
| `mid-body-drop` | 200 headers, half a JSON body, socket abort | truncated body = clean error, never a half-parsed reply shown as real |
|
||||
| `tool-pending-forever` | every request gets a fresh `tool_calls` response, forever | the loop's iteration cap trips (`max_loop_iterations: 16` in the contract); count actual round-trips via `GET /gate/stats` (`chat_hits`) |
|
||||
|
||||
## How the brain-side gate consumes this
|
||||
|
||||
1. Start: `stub-openai.py --port P --scenarios scenarios-openai.json --log run.jsonl`
|
||||
2. Point the brain at it: `NEURON_LLM_0_URL=http://127.0.0.1:P` +
|
||||
`NEURON_LLM_0_FORMAT=openai` (spec step 0 must verify these actually
|
||||
export at runtime), scratch profile, free soul port.
|
||||
3. Send each phrasing's `prompt` (the marker selects the script); assert the
|
||||
brain's claims (`tools_used`, reply, ledger) against the stub's JSONL log
|
||||
— truth, not narration — plus files on disk for write_file scenarios.
|
||||
4. Any stub 400 = the brain sent a malformed/leaked request; the gate fails
|
||||
with the stub's reason string.
|
||||
5. Re-run gate9's Anthropic matrix unchanged = proof the Anthropic lane is
|
||||
byte-untouched.
|
||||
|
||||
## Reconciliation into gate9 (app repo) — AFTER round 9 merges
|
||||
|
||||
This dir is staging only; the merge is mechanical by design:
|
||||
- `stub-openai.py` + `scenarios-openai.json` move to `scripts/gate9/`
|
||||
alongside `stub-llm.py` + `scenarios.json` (shared conventions: marker
|
||||
matching, assistant-count step indexing, `--port/--scenarios/--log`,
|
||||
JSONL fields `seq/ts/kind/scenario_class/phrasing/step/validation/
|
||||
delivered/http_status`, prod-port refusal, benign background responses,
|
||||
`GATE-SCRIPT-EXHAUSTED` overrun, `{N}/{NN}` repeat expansion).
|
||||
- `prompt-matrix-gate.sh` gains a dialect axis (anthropic|openai) choosing
|
||||
stub + scenario file; `matrix-asserts.py` reads the same log shape.
|
||||
- The `--mode` hostile flags here are PROVIDER-side (brain↔LLM boundary);
|
||||
gate9's `hostile/` servers are SOUL-side (app↔brain boundary). They are
|
||||
complementary, not duplicates — both stay.
|
||||
|
||||
## Open questions for the port author (stub asserts a position; confirm or change)
|
||||
|
||||
1. `parallel_tool_calls` must be **explicitly false** on every tool-bearing
|
||||
request (ADR-0005 pin). If the builder omits it instead, relax
|
||||
`defaults.expect_request.parallel_tool_calls` to `null`.
|
||||
2. `tool_choice` must be present (`"auto"` expected). If the brain relies on
|
||||
the provider default, drop `require_tool_choice`.
|
||||
3. Tool-result `content` is asserted only to be a string; if the brain sends
|
||||
structured JSON-in-string (like `{"ok":true,...}`), no change needed.
|
||||
4. Groq compatibility: Groq's OpenAI-compat endpoint rejects some optional
|
||||
fields; whatever field set the brain settles on for live Groq E2E must be
|
||||
mirrored here so the deterministic gate and the live lane assert the SAME
|
||||
request shape.
|
||||
5. The stub treats a `role:"tool"` turn answering an already-answered id as
|
||||
400; if the resume path can legitimately replay tool results, that rule
|
||||
needs a resume-aware carve-out (gate9's Anthropic stub faced the same
|
||||
issue — see its PASS 1 comment).
|
||||
@@ -1,559 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# run-lane-gate.sh — brain-side driver for the OpenAI-dialect gate.
|
||||
#
|
||||
# Drives the REAL soul binary against stub-openai.py for every class and every
|
||||
# phrasing in scenarios-openai.json, plus the three hostile provider modes, and
|
||||
# asserts the brain's claims against the stub's ground-truth JSONL (truth, not
|
||||
# narration) and against files on disk.
|
||||
#
|
||||
# SAFETY (hard rules, enforced below):
|
||||
# - never binds 7770 / 7779 / 17779 - only 7891-7894
|
||||
# - never reads or writes ~/.neuron - HOME is redirected to a scratch dir
|
||||
# - every process started here is killed on exit (trap) and proven with lsof
|
||||
#
|
||||
# The soul runs under `script -q /dev/null` so its stdout is a pty: El's
|
||||
# println() uses puts(), which is FULLY buffered to a file, and the process is
|
||||
# killed without flushing — the DRIFT lines would be invisible otherwise.
|
||||
#
|
||||
# Usage: ./run-lane-gate.sh [all|bridge|local|toolsoff|hostile]
|
||||
# bridge = consent round-trip config (no workspace root -> write_file is
|
||||
# "escalate" -> the loop suspends and the CLIENT executes the tool)
|
||||
# local = workspace-root config (write_file is "reversible" + builtin ->
|
||||
# the loop executes the tool in-process and runs to completion)
|
||||
# toolsoff = supplementary: non-agentic lane against a base URL WITHOUT the
|
||||
# /v1 suffix (the el-runtime provider chain appends
|
||||
# /v1/chat/completions itself, unlike chat.el which appends only
|
||||
# /chat/completions)
|
||||
# hostile = black-hole / mid-body-drop / tool-pending-forever
|
||||
#
|
||||
# Env overrides: SOUL_BIN, STUB_PORT, SOUL_PORT, SOUL_PORT_B, RUN_ROOT
|
||||
set -uo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
PHASES="${1:-all}"
|
||||
|
||||
SOUL_BIN="${SOUL_BIN:-/tmp/soul-oai2/soul-openai-tools}"
|
||||
STUB_PORT="${STUB_PORT:-7891}"
|
||||
SOUL_PORT="${SOUL_PORT:-7892}"
|
||||
SOUL_PORT_B="${SOUL_PORT_B:-7893}"
|
||||
RUN_ROOT="${RUN_ROOT:-/tmp/oa-lane-gate}"
|
||||
STAMP="$(date +%Y%m%d-%H%M%S)"
|
||||
RUN="$RUN_ROOT/$STAMP"
|
||||
|
||||
for p in "$STUB_PORT" "$SOUL_PORT" "$SOUL_PORT_B"; do
|
||||
case "$p" in
|
||||
7770|7779|17779) echo "FATAL: refusing production Neuron port $p"; exit 2;;
|
||||
789[1-4]) ;;
|
||||
*) echo "FATAL: port $p outside the allowed 7891-7894 range"; exit 2;;
|
||||
esac
|
||||
done
|
||||
[ -x "$SOUL_BIN" ] || { echo "FATAL: soul binary not found/executable: $SOUL_BIN"; exit 2; }
|
||||
|
||||
mkdir -p "$RUN/home" "$RUN/ws-bridge" "$RUN/ws-local" "$RUN/ws-off" "$RUN/engram"
|
||||
echo '{"nodes":[],"edges":[]}' > "$RUN/engram/snapshot.json"
|
||||
DRV="$RUN/drv.py"
|
||||
|
||||
STUB_PID=""; SOUL_PID=""
|
||||
cleanup() {
|
||||
[ -n "$SOUL_PID" ] && kill "$SOUL_PID" 2>/dev/null
|
||||
pkill -f "$SOUL_BIN" 2>/dev/null
|
||||
[ -n "$STUB_PID" ] && kill "$STUB_PID" 2>/dev/null
|
||||
sleep 0.4
|
||||
[ -n "$SOUL_PID" ] && kill -9 "$SOUL_PID" 2>/dev/null
|
||||
[ -n "$STUB_PID" ] && kill -9 "$STUB_PID" 2>/dev/null
|
||||
return 0
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
start_stub() { # $1 = mode, $2 = log path
|
||||
local mode="$1" log="$2" args=""
|
||||
[ "$mode" = "normal" ] && args="--scenarios $HERE/scenarios-openai.json"
|
||||
# shellcheck disable=SC2086
|
||||
python3 "$HERE/stub-openai.py" --port "$STUB_PORT" --mode "$mode" --log "$log" $args \
|
||||
> "$RUN/stub-$mode.out" 2>&1 &
|
||||
STUB_PID=$!
|
||||
for _ in $(seq 1 50); do
|
||||
curl -sf "http://127.0.0.1:$STUB_PORT/gate/health" >/dev/null 2>&1 && return 0
|
||||
sleep 0.2
|
||||
done
|
||||
echo "FATAL: stub did not come up on $STUB_PORT"; cat "$RUN/stub-$mode.out"; exit 3
|
||||
}
|
||||
stop_stub() { [ -n "$STUB_PID" ] && kill "$STUB_PID" 2>/dev/null; sleep 0.3; STUB_PID=""; }
|
||||
|
||||
start_soul() { # $1 = port, $2 = base url, $3 = soul log, $4 = agent root ("" = none)
|
||||
local port="$1" base="$2" log="$3" root="$4"
|
||||
script -q /dev/null \
|
||||
env -u ANTHROPIC_API_KEY -u SOUL_API_KEY -u ENGRAM_URL -u ENGRAM_API_KEY \
|
||||
-u NEURON_API_URL -u NEURON_TOKEN -u SOUL_LLM_PROVIDER -u SOUL_LLM_BASE_URL \
|
||||
-u NEURON_LLM_1_URL -u NEURON_LLM_1_KEY -u SOUL_IDENTITY \
|
||||
HOME="$RUN/home" PATH="$PATH" \
|
||||
NEURON_PORT="$port" EL_HTTP_BIND_HOST=127.0.0.1 \
|
||||
SOUL_ENGRAM_PATH="$RUN/engram/snapshot.json" \
|
||||
SOUL_CGI_ID=ntn-test SOUL_PERSONA_NAME=Neuron \
|
||||
NEURON_LLM_0_URL="$base" NEURON_LLM_0_FORMAT=openai NEURON_LLM_0_KEY=gate-test-key \
|
||||
${root:+NEURON_AGENT_ROOT="$root"} \
|
||||
"$SOUL_BIN" > "$log" 2>&1 &
|
||||
SOUL_PID=$!
|
||||
for _ in $(seq 1 100); do
|
||||
curl -sf "http://127.0.0.1:$port/health" >/dev/null 2>&1 && return 0
|
||||
sleep 0.2
|
||||
done
|
||||
echo "FATAL: soul did not come up on $port"; tail -20 "$log"; exit 3
|
||||
}
|
||||
stop_soul() {
|
||||
[ -n "$SOUL_PID" ] && kill "$SOUL_PID" 2>/dev/null
|
||||
pkill -f "$SOUL_BIN" 2>/dev/null
|
||||
sleep 0.6; SOUL_PID=""
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ driver ----
|
||||
cat > "$DRV" <<'PYEOF'
|
||||
import json, os, sys, time, threading, urllib.request, urllib.error
|
||||
|
||||
CFG = json.load(open(sys.argv[1]))
|
||||
SOUL = "http://127.0.0.1:%d" % CFG["soul_port"]
|
||||
STUB = "http://127.0.0.1:%d" % CFG["stub_port"]
|
||||
WS = CFG["workspace"]
|
||||
MODE = CFG["mode"] # bridge | local | toolsoff
|
||||
SCEN = json.load(open(CFG["scenarios"]))
|
||||
STUBLOG = CFG["stub_log"]
|
||||
SOULLOG = CFG["soul_log"]
|
||||
ONLY = CFG.get("classes") or list(SCEN["classes"].keys())
|
||||
MAXHOPS = CFG.get("max_hops", 15)
|
||||
OUT = CFG["out"]
|
||||
# the chat-only class must be driven on the NON-agentic door: the agentic door
|
||||
# always advertises tools, which is a 400 on that scenario by contract.
|
||||
NON_AGENTIC = {"oa-tools-off"}
|
||||
|
||||
def http(method, url, obj=None, timeout=300):
|
||||
data = None if obj is None else json.dumps(obj).encode()
|
||||
req = urllib.request.Request(url, data=data, method=method,
|
||||
headers={"Content-Type": "application/json"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
body = r.read().decode("utf-8", "replace")
|
||||
st = r.status
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", "replace"); st = e.code
|
||||
except Exception as e:
|
||||
return -1, "TRANSPORT-ERROR: %r" % (e,), None
|
||||
try:
|
||||
return st, body, json.loads(body)
|
||||
except ValueError:
|
||||
return st, body, None
|
||||
|
||||
def fsize(p):
|
||||
return os.path.getsize(p) if os.path.exists(p) else 0
|
||||
|
||||
def tail_from(path, off):
|
||||
if not os.path.exists(path):
|
||||
return "", off
|
||||
with open(path, "rb") as f:
|
||||
f.seek(off); chunk = f.read(); return chunk.decode("utf-8", "replace"), f.tell()
|
||||
|
||||
def stub_since(off):
|
||||
"""Exact correlation: only the JSONL bytes appended during this phrasing."""
|
||||
txt, noff = tail_from(STUBLOG, off)
|
||||
recs = []
|
||||
for line in txt.splitlines():
|
||||
line = line.strip()
|
||||
if line:
|
||||
try: recs.append(json.loads(line))
|
||||
except ValueError: pass
|
||||
return recs, noff
|
||||
|
||||
def perform(name, ti):
|
||||
"""Execute the bridged tool for real, like the desktop client would."""
|
||||
if name in ("write_file", "edit_file"):
|
||||
p = ti.get("path", "")
|
||||
dest = p if os.path.isabs(p) else os.path.join(WS, p)
|
||||
os.makedirs(os.path.dirname(dest) or WS, exist_ok=True)
|
||||
body = ti.get("content", "")
|
||||
with open(dest, "w") as f:
|
||||
f.write(body)
|
||||
return "wrote %s (%d bytes)" % (p, len(body.encode()))
|
||||
return "ok"
|
||||
|
||||
class Poller(threading.Thread):
|
||||
def __init__(self, sid):
|
||||
super().__init__(daemon=True); self.sid = sid; self.snaps = []; self.stop = False
|
||||
def run(self):
|
||||
while not self.stop:
|
||||
st, body, js = http("GET", SOUL + "/api/run-progress/" + self.sid, timeout=60)
|
||||
if js and js.get("progress"):
|
||||
if not self.snaps or self.snaps[-1] != js["progress"]:
|
||||
self.snaps.append(js["progress"])
|
||||
time.sleep(0.1)
|
||||
|
||||
def progress(sid):
|
||||
_, _, pj = http("GET", SOUL + "/api/run-progress/" + sid, timeout=30)
|
||||
return (pj or {}).get("progress")
|
||||
|
||||
def run_phrasing(cname, ph):
|
||||
st, body, js = http("POST", SOUL + "/api/sessions", {"title": ph["id"]}, timeout=60)
|
||||
sid = (js or {}).get("id", "")
|
||||
rec = {"class": cname, "phrasing": ph["id"], "session_id": sid, "legs": [],
|
||||
"pendings": [], "progress_during": [], "progress_per_leg": [],
|
||||
"progress_final": None, "soul_log": "", "stub": [], "http": [],
|
||||
"agentic": cname not in NON_AGENTIC}
|
||||
if not sid:
|
||||
rec["fatal"] = "session create failed: %s %s" % (st, body[:300]); return rec
|
||||
soff = fsize(SOULLOG); loff = fsize(STUBLOG)
|
||||
t0 = time.time()
|
||||
pol = Poller(sid); pol.start()
|
||||
payload = {"message": ph["prompt"], "session_id": sid, "workspace_root": WS,
|
||||
"agentic": rec["agentic"]}
|
||||
if MODE == "local":
|
||||
payload["agent_workspace_root"] = WS
|
||||
st, body, js = http("POST", SOUL + "/api/chat", payload, timeout=CFG.get("chat_timeout", 240))
|
||||
rec["http"].append(st)
|
||||
rec["legs"].append(js if js is not None else body[:600])
|
||||
rec["progress_per_leg"].append(progress(sid))
|
||||
hops = 0
|
||||
while isinstance(js, dict) and js.get("tool_pending") and hops < MAXHOPS:
|
||||
rec["pendings"].append({"call_id": js.get("call_id"), "tool_name": js.get("tool_name"),
|
||||
"tool_input": js.get("tool_input"), "risk_tier": js.get("risk_tier"),
|
||||
"narration": js.get("narration"), "tools_used": js.get("tools_used")})
|
||||
try:
|
||||
eff = perform(js.get("tool_name", ""), js.get("tool_input") or {})
|
||||
except Exception as e:
|
||||
eff = "client error: %r" % (e,)
|
||||
st, body, js = http("POST", SOUL + "/api/sessions/%s/tool_result" % sid,
|
||||
{"call_id": js.get("call_id"), "content": eff},
|
||||
timeout=CFG.get("chat_timeout", 240))
|
||||
rec["http"].append(st)
|
||||
rec["legs"].append(js if js is not None else body[:600])
|
||||
rec["progress_per_leg"].append(progress(sid))
|
||||
hops += 1
|
||||
pol.stop = True; time.sleep(0.3)
|
||||
t1 = time.time()
|
||||
rec["elapsed"] = round(t1 - t0, 2)
|
||||
rec["progress_during"] = pol.snaps
|
||||
rec["progress_final"] = progress(sid)
|
||||
rec["soul_log"], _ = tail_from(SOULLOG, soff)
|
||||
rec["stub"], _ = stub_since(loff)
|
||||
rec["hops"] = hops
|
||||
return rec
|
||||
|
||||
# ------------------------------------------------------------- assertions ----
|
||||
def expected_calls(cname, first_only=False):
|
||||
out = []
|
||||
for step in SCEN["classes"][cname]["script"]:
|
||||
for k, call in enumerate(step.get("tool_calls") or []):
|
||||
if first_only and k > 0:
|
||||
continue
|
||||
out.append((call["name"], call["arguments"]))
|
||||
return out
|
||||
|
||||
def final_text(cname):
|
||||
for step in reversed(SCEN["classes"][cname]["script"]):
|
||||
if step.get("text") and not step.get("tool_calls"):
|
||||
return step["text"]
|
||||
return None
|
||||
|
||||
def judge(rec):
|
||||
cname = rec["class"]; ok = []; bad = []
|
||||
last = rec["legs"][-1] if rec["legs"] else None
|
||||
reply = last.get("reply") if isinstance(last, dict) else None
|
||||
err = last.get("error") if isinstance(last, dict) else None
|
||||
tools_used = last.get("tools_used") if isinstance(last, dict) else None
|
||||
stub = rec["stub"]
|
||||
scen_recs = [r for r in stub if r.get("kind") == "scenario"]
|
||||
rejects = [r for r in stub if r.get("validation") != "ok"]
|
||||
bg = [r for r in stub if r.get("kind") in ("wrong_path", "background")]
|
||||
|
||||
def wire_clean():
|
||||
if rejects:
|
||||
for r in rejects:
|
||||
bad.append("stub REJECTED a request: [%s] %s"
|
||||
% (r.get("validation"), r.get("validation_detail")))
|
||||
else:
|
||||
ok.append("stub ground truth: validation \"ok\" on all %d scenario leg(s), no "
|
||||
"gate_echo_mismatch / gate_tool_call_shape / dialect-leak 400s"
|
||||
% len(scen_recs))
|
||||
if bg:
|
||||
ok.append("NOTE background non-scenario request(s) in this window: %s"
|
||||
% [(r.get("kind"), r.get("path"), r.get("http_status")) for r in bg])
|
||||
|
||||
if cname == "oa-plain":
|
||||
wire_clean()
|
||||
want = final_text(cname)
|
||||
if reply == want: ok.append("final reply == scripted final text (byte-exact)")
|
||||
else: bad.append("final reply mismatch:\n WANT: %r\n GOT : %r" % (want, reply))
|
||||
if tools_used == []: ok.append("tools_used == [] (no tool ran)")
|
||||
else: bad.append("tools_used expected [] got %r" % (tools_used,))
|
||||
if reply and ('"tool_calls"' in reply or '"function"' in reply or '"tool_use"' in reply):
|
||||
bad.append("tool-call JSON leaked into the reply text")
|
||||
else: ok.append("no tool-call JSON anywhere in the reply")
|
||||
|
||||
elif cname in ("oa-single-tool", "oa-torture", "oa-mission"):
|
||||
wire_clean()
|
||||
want = final_text(cname)
|
||||
if reply == want: ok.append("final reply == scripted final text (byte-exact)")
|
||||
else: bad.append("final reply mismatch:\n WANT: %r\n GOT : %r" % (want, reply))
|
||||
exp = expected_calls(cname)
|
||||
wantnames = [n for n, _ in exp]
|
||||
if tools_used == wantnames:
|
||||
ok.append("tools_used == %r (carried across %d suspension(s))" % (wantnames, rec["hops"]))
|
||||
else:
|
||||
bad.append("tools_used expected %r got %r" % (wantnames, tools_used))
|
||||
for name, args in exp:
|
||||
p = args.get("path"); c = args.get("content")
|
||||
dest = os.path.join(WS, p)
|
||||
if not os.path.exists(dest):
|
||||
bad.append("expected file missing on disk: %s" % dest); continue
|
||||
got = open(dest, "rb").read()
|
||||
if got == c.encode():
|
||||
ok.append("%s on disk is byte-for-byte the issued payload (%d bytes)" % (p, len(got)))
|
||||
else:
|
||||
bad.append("%s content differs\n WANT %r\n GOT %r"
|
||||
% (p, c[:300], got[:300].decode("utf-8", "replace")))
|
||||
if MODE == "bridge":
|
||||
for pend, (name, args) in zip(rec["pendings"], exp):
|
||||
if pend["tool_input"] == args:
|
||||
ok.append("tool_input for %s survived exactly ONE decode (deep-equal to the "
|
||||
"issued arguments; no double-escaping)" % name)
|
||||
else:
|
||||
bad.append("tool_input != issued arguments for %s\n WANT %r\n GOT %r"
|
||||
% (name, args, pend["tool_input"]))
|
||||
if rec["pendings"] and all(p["risk_tier"] == "escalate" for p in rec["pendings"]):
|
||||
ok.append("every write_file classified \"escalate\" and bridged for consent")
|
||||
|
||||
elif cname == "oa-parallel":
|
||||
drift = [l.strip() for l in rec["soul_log"].splitlines() if "DRIFT: provider returned" in l]
|
||||
if drift: ok.append("soul log: " + drift[0])
|
||||
else: bad.append("no 'DRIFT: provider returned N parallel tool_calls' line in the soul log")
|
||||
delivered = [r for r in stub if r.get("delivered", {}).get("tool_calls")]
|
||||
if delivered and len(delivered[0]["delivered"]["tool_calls"]) == 2:
|
||||
ok.append("stub delivered 2 parallel tool_calls in one response (ground truth)")
|
||||
if MODE == "bridge":
|
||||
if len(rec["pendings"]) == 1:
|
||||
ok.append("exactly ONE call honored: %s" % rec["pendings"][0]["call_id"])
|
||||
else:
|
||||
bad.append("expected exactly 1 honored call, got %d" % len(rec["pendings"]))
|
||||
pairing = [r for r in rejects if "gate_pairing" in str(r.get("validation_detail")) or
|
||||
"tool_calls at end of thread" in str(r.get("validation_detail")) or
|
||||
"not fully answered" in str(r.get("validation_detail"))]
|
||||
for r in pairing:
|
||||
ok.append("EXPECTED-BY-CONTRACT stub 400 on the unpaired echo: %s"
|
||||
% r.get("validation_detail"))
|
||||
other = [r for r in rejects if r not in pairing]
|
||||
for r in other:
|
||||
bad.append("unexpected stub rejection: [%s] %s"
|
||||
% (r.get("validation"), r.get("validation_detail")))
|
||||
if err and not reply:
|
||||
ok.append("honest error envelope after the 400 (no fabricated answer): %r" % err)
|
||||
elif reply == final_text(cname):
|
||||
ok.append("final reply == scripted final text (both calls paired)")
|
||||
else:
|
||||
bad.append("neither an honest error nor the scripted final text: %r" % (last,))
|
||||
|
||||
elif cname == "oa-api-error":
|
||||
if err and not reply:
|
||||
ok.append("honest error envelope: error=%r reply=%r" % (err, reply))
|
||||
else:
|
||||
bad.append("expected an error envelope with an empty reply, got %r" % (last,))
|
||||
delivered = [r["delivered"].get("api_error") for r in stub if r.get("delivered")]
|
||||
ok.append("stub delivered api_error status(es): %r" % [d for d in delivered if d])
|
||||
n = len([r for r in stub if r.get("kind") == "scenario"])
|
||||
ok.append("provider hit %d time(s) - no retry storm" % n)
|
||||
if reply:
|
||||
bad.append("FABRICATED ANSWER: reply non-empty on a provider error")
|
||||
|
||||
elif cname == "oa-tools-off":
|
||||
ok.append("stub records for this phrasing: %r"
|
||||
% [{k: r.get(k) for k in ("kind", "path", "validation", "http_status")} for r in stub])
|
||||
wrong = [r for r in stub if r.get("kind") == "wrong_path"]
|
||||
matched = [r for r in stub if r.get("scenario_class") == cname]
|
||||
if matched and not rejects:
|
||||
ok.append("chat-only request reached /v1/chat/completions with NO tools offered")
|
||||
want = final_text(cname)
|
||||
if reply == want: ok.append("final reply == scripted final text (byte-exact)")
|
||||
else: bad.append("final reply mismatch:\n WANT: %r\n GOT : %r" % (want, reply))
|
||||
if reply and ('"tool_calls"' in reply or '"function"' in reply):
|
||||
bad.append("tool-call JSON leaked into the reply text")
|
||||
else: ok.append("no tool-call JSON in the reply")
|
||||
elif wrong:
|
||||
bad.append("the non-agentic lane never reached the provider endpoint: stub saw "
|
||||
"%s -> %s (the el-runtime provider chain appends /v1/chat/completions "
|
||||
"to NEURON_LLM_0_URL, chat.el appends only /chat/completions)"
|
||||
% (wrong[0]["path"], wrong[0]["http_status"]))
|
||||
elif not stub:
|
||||
bad.append("no request reached the stub at all")
|
||||
else:
|
||||
for r in rejects:
|
||||
bad.append("stub REJECTED: [%s] %s" % (r.get("validation"), r.get("validation_detail")))
|
||||
return ok, bad
|
||||
|
||||
def main():
|
||||
results = []
|
||||
for cname in ONLY:
|
||||
for ph in SCEN["classes"][cname]["phrasings"]:
|
||||
rec = run_phrasing(cname, ph)
|
||||
ok, bad = judge(rec)
|
||||
rec["ok"] = ok; rec["bad"] = bad
|
||||
rec["verdict"] = "FAIL" if bad else "PASS"
|
||||
results.append(rec)
|
||||
print("=" * 78)
|
||||
print("[%s] %s / %s (%.2fs, %d bridge hop(s), agentic=%s, mode=%s)"
|
||||
% (rec["verdict"], cname, ph["id"], rec.get("elapsed", 0),
|
||||
rec.get("hops", 0), rec["agentic"], MODE))
|
||||
for l in ok: print(" ok " + l.replace("\n", "\n "))
|
||||
for l in bad: print(" FAIL " + l.replace("\n", "\n "))
|
||||
for i, leg in enumerate(rec["legs"]):
|
||||
print(" leg%d envelope: %s" % (i, json.dumps(leg)[:430]))
|
||||
for i, pr in enumerate(rec["progress_per_leg"]):
|
||||
print(" run-progress after leg%d: %s" % (i, json.dumps(pr)[:380]))
|
||||
if rec["progress_during"]:
|
||||
print(" run-progress polled DURING (%d distinct snapshot(s)), last: %s"
|
||||
% (len(rec["progress_during"]), json.dumps(rec["progress_during"][-1])[:300]))
|
||||
for r in rec["stub"]:
|
||||
print(" stub: kind=%s class=%s phrasing=%s step=%s validation=%s%s delivered=%s http=%s"
|
||||
% (r.get("kind"), r.get("scenario_class"), r.get("phrasing"), r.get("step"),
|
||||
r.get("validation"),
|
||||
("(" + str(r.get("validation_detail")) + ")") if r.get("validation_detail") else "",
|
||||
json.dumps(r.get("delivered")), r.get("http_status")))
|
||||
if rec["soul_log"].strip():
|
||||
for l in rec["soul_log"].splitlines():
|
||||
if l.strip(): print(" soul: " + l.strip())
|
||||
json.dump(results, open(OUT, "w"), indent=1)
|
||||
npass = sum(1 for r in results if r["verdict"] == "PASS")
|
||||
print("=" * 78)
|
||||
print("PHASE %s: %d/%d PASS" % (MODE, npass, len(results)))
|
||||
for r in results:
|
||||
print(" %-6s %-16s %s" % (r["verdict"], r["class"], r["phrasing"]))
|
||||
return 0 if npass == len(results) else 1
|
||||
|
||||
sys.exit(main())
|
||||
PYEOF
|
||||
|
||||
# ------------------------------------------------------------- hostile drv ---
|
||||
cat > "$RUN/hostile.py" <<'PYEOF'
|
||||
import json, os, sys, time, urllib.request, urllib.error
|
||||
|
||||
CFG = json.load(open(sys.argv[1]))
|
||||
SOUL = "http://127.0.0.1:%d" % CFG["soul_port"]
|
||||
STUB = "http://127.0.0.1:%d" % CFG["stub_port"]
|
||||
|
||||
def http(method, url, obj=None, timeout=400):
|
||||
data = None if obj is None else json.dumps(obj).encode()
|
||||
req = urllib.request.Request(url, data=data, method=method,
|
||||
headers={"Content-Type": "application/json"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
b = r.read().decode("utf-8", "replace"); st = r.status
|
||||
except urllib.error.HTTPError as e:
|
||||
b = e.read().decode("utf-8", "replace"); st = e.code
|
||||
except Exception as e:
|
||||
return -1, "TRANSPORT-ERROR: %r" % (e,), None
|
||||
try:
|
||||
return st, b, json.loads(b)
|
||||
except ValueError:
|
||||
return st, b, None
|
||||
|
||||
mode = CFG["mode"]; wsmode = CFG["ws_mode"]; WS = CFG["workspace"]
|
||||
_, _, js = http("POST", SOUL + "/api/sessions", {"title": "hostile-" + mode}, timeout=60)
|
||||
sid = (js or {}).get("id", "")
|
||||
payload = {"message": "oa-gate plain probe: hostile mode %s" % mode,
|
||||
"agentic": True, "session_id": sid, "workspace_root": WS}
|
||||
if wsmode == "local":
|
||||
payload["agent_workspace_root"] = WS
|
||||
t0 = time.time()
|
||||
st, body, js = http("POST", SOUL + "/api/chat", payload, timeout=CFG.get("timeout", 400))
|
||||
t_first = time.time() - t0
|
||||
legs = [js if js is not None else body[:500]]
|
||||
hops = 0
|
||||
while isinstance(js, dict) and js.get("tool_pending") and hops < CFG.get("max_hops", 14):
|
||||
ti = js.get("tool_input") or {}
|
||||
p = ti.get("path", "x.md")
|
||||
dest = p if os.path.isabs(p) else os.path.join(WS, p)
|
||||
try: open(dest, "w").write(ti.get("content", ""))
|
||||
except Exception: pass
|
||||
st, body, js = http("POST", SOUL + "/api/sessions/%s/tool_result" % sid,
|
||||
{"call_id": js.get("call_id"), "content": "ok"},
|
||||
timeout=CFG.get("timeout", 400))
|
||||
legs.append(js if js is not None else body[:500]); hops += 1
|
||||
el = time.time() - t0
|
||||
_, _, stats = http("GET", STUB + "/gate/stats", timeout=30)
|
||||
_, _, prog = http("GET", SOUL + "/api/run-progress/" + sid, timeout=30)
|
||||
fab = [l for l in legs if isinstance(l, dict) and l.get("reply")]
|
||||
print("HOSTILE %s (ws_mode=%s)" % (mode, wsmode))
|
||||
print(" first /api/chat POST returned after %.2fs; whole chain %.2fs; client bridge hops=%d; "
|
||||
"stub chat_hits=%s" % (t_first, el, hops, (stats or {}).get("chat_hits")))
|
||||
print(" first envelope : " + json.dumps(legs[0])[:430])
|
||||
print(" final envelope : " + json.dumps(legs[-1])[:430])
|
||||
print(" non-empty replies anywhere in the chain (fabrication check): %d" % len(fab))
|
||||
print(" run-progress : " + json.dumps(prog)[:300])
|
||||
json.dump({"mode": mode, "ws_mode": wsmode, "t_first": t_first, "elapsed": el, "hops": hops,
|
||||
"chat_hits": (stats or {}).get("chat_hits"), "legs": legs, "progress": prog},
|
||||
open(CFG["out"], "w"), indent=1)
|
||||
PYEOF
|
||||
|
||||
# ------------------------------------------------------------------ phases ---
|
||||
RC_BRIDGE=0; RC_LOCAL=0; RC_OFF=0
|
||||
run_normal_phase() { # $1 = label, $2 = soul port, $3 = agent root, $4 = ws, $5 = base, $6 = classes json
|
||||
local m="$1" port="$2" root="$3" ws="$4" base="$5" classes="$6"
|
||||
echo; echo "############ PHASE: $m (soul :$port, NEURON_LLM_0_URL=$base) ############"
|
||||
start_stub normal "$RUN/stub-$m.jsonl"
|
||||
start_soul "$port" "$base" "$RUN/soul-$m.log" "$root"
|
||||
cat > "$RUN/cfg-$m.json" <<JSON
|
||||
{"soul_port": $port, "stub_port": $STUB_PORT, "workspace": "$ws", "mode": "$m",
|
||||
"scenarios": "$HERE/scenarios-openai.json", "stub_log": "$RUN/stub-$m.jsonl",
|
||||
"soul_log": "$RUN/soul-$m.log", "out": "$RUN/results-$m.json", "chat_timeout": 240,
|
||||
"classes": $classes}
|
||||
JSON
|
||||
python3 "$DRV" "$RUN/cfg-$m.json"
|
||||
local rc=$?
|
||||
stop_soul; stop_stub
|
||||
return $rc
|
||||
}
|
||||
|
||||
if [ "$PHASES" = "all" ] || [ "$PHASES" = "bridge" ]; then
|
||||
run_normal_phase bridge "$SOUL_PORT" "" "$RUN/ws-bridge" "http://127.0.0.1:$STUB_PORT/v1" null
|
||||
RC_BRIDGE=$?
|
||||
fi
|
||||
if [ "$PHASES" = "all" ] || [ "$PHASES" = "local" ]; then
|
||||
run_normal_phase local "$SOUL_PORT_B" "$RUN/ws-local" "$RUN/ws-local" "http://127.0.0.1:$STUB_PORT/v1" null
|
||||
RC_LOCAL=$?
|
||||
fi
|
||||
if [ "$PHASES" = "all" ] || [ "$PHASES" = "toolsoff" ]; then
|
||||
# supplementary: the el-runtime provider chain appends /v1/chat/completions itself,
|
||||
# so the non-agentic door needs the base WITHOUT the /v1 suffix.
|
||||
run_normal_phase toolsoff "$SOUL_PORT" "" "$RUN/ws-off" "http://127.0.0.1:$STUB_PORT" '["oa-tools-off","oa-plain"]'
|
||||
RC_OFF=$?
|
||||
fi
|
||||
|
||||
if [ "$PHASES" = "all" ] || [ "$PHASES" = "hostile" ]; then
|
||||
echo; echo "############ PHASE: hostile ############"
|
||||
for spec in "black-hole:bridge" "mid-body-drop:bridge" "tool-pending-forever:bridge" "tool-pending-forever:local"; do
|
||||
mode="${spec%%:*}"; wsm="${spec##*:}"
|
||||
echo; echo "---- hostile mode=$mode ws_mode=$wsm ----"
|
||||
start_stub "$mode" "$RUN/stub-$mode-$wsm.jsonl"
|
||||
if [ "$wsm" = "local" ]; then
|
||||
start_soul "$SOUL_PORT" "http://127.0.0.1:$STUB_PORT/v1" "$RUN/soul-$mode-$wsm.log" "$RUN/ws-local"
|
||||
else
|
||||
start_soul "$SOUL_PORT" "http://127.0.0.1:$STUB_PORT/v1" "$RUN/soul-$mode-$wsm.log" ""
|
||||
fi
|
||||
cat > "$RUN/cfg-$mode-$wsm.json" <<JSON
|
||||
{"soul_port": $SOUL_PORT, "stub_port": $STUB_PORT, "mode": "$mode", "ws_mode": "$wsm",
|
||||
"workspace": "$RUN/ws-local", "out": "$RUN/hostile-$mode-$wsm.json", "timeout": 400}
|
||||
JSON
|
||||
python3 "$RUN/hostile.py" "$RUN/cfg-$mode-$wsm.json"
|
||||
echo " soul log (llm/DRIFT/cap lines):"
|
||||
grep -E "DRIFT|llm error|iteration cap|\[llm\]" "$RUN/soul-$mode-$wsm.log" | tail -8 | sed 's/^/ /'
|
||||
stop_soul; stop_stub
|
||||
done
|
||||
fi
|
||||
|
||||
echo; echo "############ CLEANUP ############"
|
||||
cleanup
|
||||
sleep 0.5
|
||||
echo "processes still matching the soul binary:"; pgrep -fl "$SOUL_BIN" || echo " (none)"
|
||||
echo "processes still matching stub-openai.py:"; pgrep -fl "stub-openai.py" || echo " (none)"
|
||||
echo "lsof on 7891-7894 after cleanup:"
|
||||
lsof -nP -iTCP:7891 -iTCP:7892 -iTCP:7893 -iTCP:7894 2>/dev/null || echo " (no listeners - ports free)"
|
||||
echo
|
||||
echo "############ SUMMARY ############"
|
||||
echo "run dir: $RUN"
|
||||
echo "bridge rc=$RC_BRIDGE local rc=$RC_LOCAL toolsoff rc=$RC_OFF (0 = every class PASS)"
|
||||
exit $(( RC_BRIDGE + RC_LOCAL + RC_OFF ))
|
||||
@@ -1,110 +0,0 @@
|
||||
{
|
||||
"_comment": "OpenAI-dialect gate scenario contract (soul-openai-tools-v2). Single source of truth shared by stub-openai.py (scripted provider responses + request assertions), selftest.sh (stub self-verification), and the future brain-side gate driver. Same structure as gate9's scenarios.json: classes -> script + phrasings with markers; scripts are CLASS-level so assertions are behavioral, never pinned to a sentence. expect_request keys: require_tools, require_tool_choice, parallel_tool_calls (expected literal value; null = don't check), forbid_tools. defaults apply to every class unless overridden; steps may override with their own expect_request.",
|
||||
"deadline_secs": 60,
|
||||
"max_loop_iterations": 16,
|
||||
"defaults": {
|
||||
"expect_request": {
|
||||
"require_tools": true,
|
||||
"require_tool_choice": true,
|
||||
"parallel_tool_calls": false
|
||||
}
|
||||
},
|
||||
"classes": {
|
||||
"oa-plain": {
|
||||
"script": [
|
||||
{ "text": "Plain OpenAI-lane answer (gate fixture): the mechanism, the main caveat, and the practical takeaway in three sentences. No tools were needed for this one, and the finish reason on the wire is stop, which the loop must treat as terminal." }
|
||||
],
|
||||
"phrasings": [
|
||||
{ "id": "oa-plain-p1", "marker": "oa-gate plain probe", "prompt": "oa-gate plain probe: explain the fixture topic simply." },
|
||||
{ "id": "oa-plain-p2", "marker": "oa-gate second plain", "prompt": "oa-gate second plain: another phrasing of the plain question." }
|
||||
]
|
||||
},
|
||||
"oa-tools-off": {
|
||||
"expect_request": {
|
||||
"require_tools": false,
|
||||
"forbid_tools": true,
|
||||
"require_tool_choice": false,
|
||||
"parallel_tool_calls": null
|
||||
},
|
||||
"script": [
|
||||
{ "text": "Chat-only OpenAI-lane answer (gate fixture): this lane offered no tools and none were used; the reply is plain text with finish reason stop." }
|
||||
],
|
||||
"phrasings": [
|
||||
{ "id": "oa-tools-off-p1", "marker": "oa-gate tools-off probe", "prompt": "oa-gate tools-off probe: plain chat with no tools offered." }
|
||||
]
|
||||
},
|
||||
"oa-single-tool": {
|
||||
"script": [
|
||||
{ "text": "Step 1: writing the note.",
|
||||
"tool_calls": [
|
||||
{ "name": "write_file",
|
||||
"arguments": { "path": "openai-single-note.md", "content": "# Note (gate fixture, OpenAI lane)\n\nDeterministic single-tool body.\n" } }
|
||||
] },
|
||||
{ "text": "All set - openai-single-note.md is written with the fixture body. Nothing else was needed for this one." }
|
||||
],
|
||||
"phrasings": [
|
||||
{ "id": "oa-single-p1", "marker": "oa-gate single tool note", "prompt": "oa-gate single tool note: save the fixture note to a file." },
|
||||
{ "id": "oa-single-p2", "marker": "oa-gate one file please", "prompt": "oa-gate one file please: write the fixture note file." }
|
||||
]
|
||||
},
|
||||
"oa-torture": {
|
||||
"script": [
|
||||
{ "tool_calls": [
|
||||
{ "name": "write_file",
|
||||
"arguments": { "path": "torture-note.md", "content": "Line 1 has \"double quotes\", 'singles', and a mid-line backslash \\ here.\nLine 2\thas a tab, a literal \\n two-char sequence, and a Windows path C:\\temp\\new.txt.\nLine 3 unicode: naïve café — 日本語 ✓ 🚀\nLine 4 JSON-in-string: {\"k\": \"v\", \"arr\": [1, 2], \"s\": \"nested \\\"deep\\\" quotes\"}\nLine 5 ends with a lone backslash \\" } }
|
||||
] },
|
||||
{ "text": "Torture round-trip complete: the payload with nested quotes, backslashes, newlines, tabs, and unicode survived exactly one encode and one decode." }
|
||||
],
|
||||
"phrasings": [
|
||||
{ "id": "oa-torture-p1", "marker": "oa-gate torture probe", "prompt": "oa-gate torture probe: write the escaping torture file." }
|
||||
]
|
||||
},
|
||||
"oa-parallel": {
|
||||
"script": [
|
||||
{ "text": "Step 1: two writes at once (parallel probe).",
|
||||
"tool_calls": [
|
||||
{ "name": "write_file", "arguments": { "path": "parallel-a.md", "content": "Parallel A (gate fixture).\n" } },
|
||||
{ "name": "write_file", "arguments": { "path": "parallel-b.md", "content": "Parallel B (gate fixture).\n" } }
|
||||
] },
|
||||
{ "text": "Parallel probe complete: both tool results arrived and were paired correctly. A brain that instead rejects the double call must do so cleanly - that outcome is asserted brain-side, not here." }
|
||||
],
|
||||
"phrasings": [
|
||||
{ "id": "oa-parallel-p1", "marker": "oa-gate parallel probe", "prompt": "oa-gate parallel probe: run the two-write parallel case." }
|
||||
]
|
||||
},
|
||||
"oa-mission": {
|
||||
"script": [
|
||||
{ "text": "Step 1: drafting part one.",
|
||||
"tool_calls": [
|
||||
{ "name": "write_file", "arguments": { "path": "mission-part-1.md", "content": "Mission part 1 (gate fixture).\n" } }
|
||||
] },
|
||||
{ "text": "Step 2: drafting part two.",
|
||||
"tool_calls": [
|
||||
{ "name": "write_file", "arguments": { "path": "mission-part-2.md", "content": "Mission part 2 (gate fixture).\n" } }
|
||||
] },
|
||||
{ "text": "Mission complete: mission-part-1.md and mission-part-2.md are written; the loop ran two tool rounds and finished cleanly with finish reason stop." }
|
||||
],
|
||||
"phrasings": [
|
||||
{ "id": "oa-mission-p1", "marker": "oa-gate mission probe", "prompt": "oa-gate mission probe: run the two-round mission." }
|
||||
]
|
||||
},
|
||||
"oa-api-error": {
|
||||
"expect_request": {
|
||||
"require_tools": false,
|
||||
"require_tool_choice": false,
|
||||
"parallel_tool_calls": null
|
||||
},
|
||||
"script": [],
|
||||
"phrasings": [
|
||||
{ "id": "oa-err-400", "marker": "oa-gate error four hundred", "prompt": "oa-gate error four hundred: trigger the injected failure.",
|
||||
"script": [ { "api_error": { "status": 400, "type": "invalid_request_error", "message": "gate-injected 400: request rejected by fixture", "code": "gate_injected" } } ] },
|
||||
{ "id": "oa-err-429", "marker": "oa-gate error rate limit", "prompt": "oa-gate error rate limit: trigger the injected failure.",
|
||||
"script": [ { "api_error": { "status": 429, "type": "rate_limit_error", "message": "gate-injected 429: rate limited by fixture", "code": "rate_limit_exceeded" } } ] },
|
||||
{ "id": "oa-err-500", "marker": "oa-gate error five hundred", "prompt": "oa-gate error five hundred: trigger the injected failure.",
|
||||
"script": [ { "api_error": { "status": 500, "type": "server_error", "message": "gate-injected 500: internal fixture error", "code": "gate_injected" } } ] },
|
||||
{ "id": "oa-err-503", "marker": "oa-gate error unavailable", "prompt": "oa-gate error unavailable: trigger the injected failure.",
|
||||
"script": [ { "api_error": { "status": 503, "type": "server_error", "message": "gate-injected 503: fixture overloaded", "code": "gate_injected" } } ] }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,409 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# selftest.sh - proves stub-openai.py before any brain code exists.
|
||||
# Drives the stub with curl through every scenario (plain, tools-off,
|
||||
# single tool round-trip, escaping torture, parallel double-call,
|
||||
# two-round mission, injected API errors, background, overrun), every
|
||||
# validation rejection (dialect leaks, pairing, echo round-trip, scenario
|
||||
# expectations), and all three hostile modes. Exit 0 = green.
|
||||
set -u
|
||||
cd "$(dirname "$0")" || exit 1
|
||||
PY=python3
|
||||
TMP="$(mktemp -d)"
|
||||
PIDS=()
|
||||
cleanup() {
|
||||
for p in "${PIDS[@]:-}"; do kill -9 "$p" >/dev/null 2>&1; done
|
||||
rm -rf "$TMP"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
PASS=0; FAIL=0
|
||||
ok() { printf 'ok - %s\n' "$1"; PASS=$((PASS+1)); }
|
||||
bad() { printf 'FAIL - %s\n' "$1"; FAIL=$((FAIL+1)); }
|
||||
check() { # check <name> <cmd...> - pass if cmd exits 0; show output on fail
|
||||
local name="$1"; shift
|
||||
local out
|
||||
if out="$("$@" 2>&1)"; then ok "$name"
|
||||
else bad "$name"; [ -n "$out" ] && printf '%s\n' "$out" | sed 's/^/ /' | head -8
|
||||
fi
|
||||
}
|
||||
|
||||
freeport() { "$PY" -c 'import socket;s=socket.socket();s.bind(("127.0.0.1",0));print(s.getsockname()[1]);s.close()'; }
|
||||
waithealth() {
|
||||
local p="$1" i
|
||||
for i in $(seq 1 60); do
|
||||
curl -sf "http://127.0.0.1:$p/gate/health" >/dev/null 2>&1 && return 0
|
||||
sleep 0.1
|
||||
done
|
||||
echo "stub on :$p never became healthy"; return 1
|
||||
}
|
||||
post() { # post <port> <bodyfile> <respfile> [extra curl args...] -> echoes http code
|
||||
local port="$1" body="$2" resp="$3"; shift 3
|
||||
curl -s -o "$resp" -w '%{http_code}' -H 'content-type: application/json' \
|
||||
"$@" --data-binary @"$body" "http://127.0.0.1:$port/v1/chat/completions"
|
||||
}
|
||||
|
||||
# ---- embedded helper: builds OpenAI-dialect bodies, asserts on responses ----
|
||||
cat > "$TMP/helpers.py" <<'PYEOF'
|
||||
import copy, json, sys
|
||||
|
||||
TOOLS = [
|
||||
{"type": "function", "function": {
|
||||
"name": "write_file", "description": "Write content to a file on disk.",
|
||||
"parameters": {"type": "object",
|
||||
"properties": {"path": {"type": "string"},
|
||||
"content": {"type": "string"}},
|
||||
"required": ["path", "content"]}}},
|
||||
{"type": "function", "function": {
|
||||
"name": "read_file", "description": "Read contents of a file from disk.",
|
||||
"parameters": {"type": "object",
|
||||
"properties": {"path": {"type": "string"}},
|
||||
"required": ["path"]}}},
|
||||
]
|
||||
|
||||
def dump(obj, out):
|
||||
json.dump(obj, open(out, "w"), ensure_ascii=False)
|
||||
|
||||
def base(prompt, tools=True):
|
||||
b = {"model": "gate-openai-model", "max_tokens": 1024,
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are Neuron (gate fixture)."},
|
||||
{"role": "user", "content": prompt}]}
|
||||
if tools:
|
||||
b["tools"] = copy.deepcopy(TOOLS)
|
||||
b["tool_choice"] = "auto"
|
||||
b["parallel_tool_calls"] = False
|
||||
return b
|
||||
|
||||
def cmd_plain(out, prompt):
|
||||
dump(base(prompt), out)
|
||||
|
||||
def cmd_notools(out, prompt):
|
||||
dump(base(prompt, tools=False), out)
|
||||
|
||||
def cmd_mut(out, prompt, mutation):
|
||||
b = base(prompt)
|
||||
if mutation == "no-tool-choice":
|
||||
del b["tool_choice"]
|
||||
elif mutation == "ptc-true":
|
||||
b["parallel_tool_calls"] = True
|
||||
elif mutation == "top-system":
|
||||
b["system"] = "You are Neuron."
|
||||
elif mutation == "anth-tools":
|
||||
b["tools"] = [{"name": "write_file", "description": "x",
|
||||
"input_schema": {"type": "object", "properties": {}}}]
|
||||
elif mutation == "anth-block":
|
||||
b["messages"][1] = {"role": "user", "content": [
|
||||
{"type": "tool_result", "tool_use_id": "toolu_x", "content": "hi"},
|
||||
{"type": "text", "text": prompt}]}
|
||||
else:
|
||||
raise SystemExit("unknown mutation " + mutation)
|
||||
dump(b, out)
|
||||
|
||||
def cmd_chain(out, prompt, variant, *resps):
|
||||
"""Build the next leg: echo each response's assistant turn and answer its
|
||||
tool calls. `variant` applies to the LAST response only:
|
||||
ok | no-tool-turn | wrong-id | only-first | double-encode | object-args"""
|
||||
b = base(prompt)
|
||||
for idx, p in enumerate(resps):
|
||||
last = idx == len(resps) - 1
|
||||
msg = json.load(open(p))["choices"][0]["message"]
|
||||
tcs = msg.get("tool_calls")
|
||||
if not tcs:
|
||||
b["messages"].append({"role": "assistant",
|
||||
"content": msg.get("content")})
|
||||
continue
|
||||
v = variant if last else "ok"
|
||||
asst = {"role": "assistant", "content": msg.get("content"),
|
||||
"tool_calls": copy.deepcopy(tcs)}
|
||||
if v == "double-encode":
|
||||
for tc in asst["tool_calls"]:
|
||||
tc["function"]["arguments"] = json.dumps(
|
||||
tc["function"]["arguments"])
|
||||
if v == "object-args":
|
||||
for tc in asst["tool_calls"]:
|
||||
tc["function"]["arguments"] = json.loads(
|
||||
tc["function"]["arguments"])
|
||||
b["messages"].append(asst)
|
||||
if v == "no-tool-turn":
|
||||
continue
|
||||
use = tcs[:1] if v == "only-first" else tcs
|
||||
for tc in use:
|
||||
tid = "call_bogus_123" if v == "wrong-id" else tc["id"]
|
||||
b["messages"].append({"role": "tool", "tool_call_id": tid,
|
||||
"content": "{\"ok\":true,\"bytes\":42}"})
|
||||
dump(b, out)
|
||||
|
||||
def cmd_chk(resp, expr):
|
||||
r = json.load(open(resp))
|
||||
if not eval(expr, {"r": r, "json": json, "len": len, "str": str,
|
||||
"isinstance": isinstance, "any": any, "all": all,
|
||||
"sorted": sorted}):
|
||||
print("assertion failed:", expr)
|
||||
print("resp:", json.dumps(r, ensure_ascii=False)[:400])
|
||||
raise SystemExit(1)
|
||||
|
||||
def cmd_torture(resp, scen):
|
||||
r = json.load(open(resp))
|
||||
tc = r["choices"][0]["message"]["tool_calls"][0]
|
||||
raw = tc["function"]["arguments"]
|
||||
assert isinstance(raw, str), "arguments must be a JSON-encoded string"
|
||||
got = json.loads(raw)
|
||||
exp = json.load(open(scen))["classes"]["oa-torture"]["script"][0]["tool_calls"][0]["arguments"]
|
||||
assert got == exp, "decoded arguments != scripted torture payload"
|
||||
content = got["content"]
|
||||
for needle in ['"', "\\", "\n", "\t", "日本語", "naïve", "🚀"]:
|
||||
assert needle in content, "missing torture needle %r" % needle
|
||||
|
||||
def cmd_notjson(path):
|
||||
data = open(path, "rb").read()
|
||||
assert data, "file empty - no partial body arrived"
|
||||
try:
|
||||
json.loads(data.decode("utf-8", "replace"))
|
||||
except ValueError:
|
||||
return
|
||||
raise SystemExit("partial body unexpectedly parsed as complete JSON")
|
||||
|
||||
def cmd_pending(*paths):
|
||||
ids = []
|
||||
for p in paths:
|
||||
c = json.load(open(p))["choices"][0]
|
||||
assert c["finish_reason"] == "tool_calls", c["finish_reason"]
|
||||
tc = c["message"]["tool_calls"][0]
|
||||
assert tc["function"]["name"] == "write_file"
|
||||
json.loads(tc["function"]["arguments"]) # must decode
|
||||
ids.append(tc["id"])
|
||||
assert len(set(ids)) == len(ids), "call ids not distinct: %r" % ids
|
||||
|
||||
def cmd_logcheck(path):
|
||||
recs = [json.loads(l) for l in open(path) if l.strip()]
|
||||
seqs = [r["seq"] for r in recs]
|
||||
assert seqs == sorted(seqs) and len(set(seqs)) == len(seqs), "seq not monotonic"
|
||||
kinds = {}
|
||||
for r in recs:
|
||||
kinds[r["kind"]] = kinds.get(r["kind"], 0) + 1
|
||||
assert kinds.get("scenario", 0) >= 10, "too few scenario records: %r" % kinds
|
||||
assert kinds.get("background", 0) >= 1, "no background record"
|
||||
assert kinds.get("overrun", 0) >= 1, "no overrun record"
|
||||
rejected = [r for r in recs if r["validation"] == "rejected"]
|
||||
assert len(rejected) >= 10, "too few rejected records: %d" % len(rejected)
|
||||
assert any(r["delivered"].get("tool_calls") == ["write_file"]
|
||||
for r in recs), "no single write_file ground truth"
|
||||
assert any(r["delivered"].get("tool_calls") == ["write_file", "write_file"]
|
||||
for r in recs), "no parallel ground truth"
|
||||
|
||||
def main():
|
||||
fn = globals()["cmd_" + sys.argv[1].replace("-", "_")]
|
||||
fn(*sys.argv[2:])
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
PYEOF
|
||||
mk() { "$PY" "$TMP/helpers.py" "$@"; }
|
||||
|
||||
echo "=== stub-openai selftest ==="
|
||||
|
||||
# ---- normal mode ------------------------------------------------------------
|
||||
PORT="$(freeport)"
|
||||
"$PY" stub-openai.py --port "$PORT" --scenarios scenarios-openai.json \
|
||||
--log "$TMP/req.jsonl" >"$TMP/stub.out" 2>&1 &
|
||||
PIDS+=($!); disown
|
||||
check "stub starts and answers /gate/health" waithealth "$PORT"
|
||||
|
||||
# 1. plain completion
|
||||
mk plain "$TMP/plain.json" "oa-gate plain probe: explain the fixture topic simply."
|
||||
code="$(post "$PORT" "$TMP/plain.json" "$TMP/r_plain.json")"
|
||||
check "plain: HTTP 200" test "$code" = "200"
|
||||
check "plain: chat.completion envelope, finish stop, real content" mk chk "$TMP/r_plain.json" \
|
||||
'r["object"]=="chat.completion" and r["choices"][0]["finish_reason"]=="stop" and isinstance(r["choices"][0]["message"]["content"],str) and len(r["choices"][0]["message"]["content"])>40'
|
||||
|
||||
# 2. tools-off lane (chat-only request accepted, tool-bearing request refused)
|
||||
mk notools "$TMP/toolsoff.json" "oa-gate tools-off probe: plain chat with no tools offered."
|
||||
code="$(post "$PORT" "$TMP/toolsoff.json" "$TMP/r_toolsoff.json")"
|
||||
check "tools-off: chat-only request -> 200" test "$code" = "200"
|
||||
mk plain "$TMP/toolsoff_bad.json" "oa-gate tools-off probe: plain chat with no tools offered."
|
||||
code="$(post "$PORT" "$TMP/toolsoff_bad.json" "$TMP/r_toolsoff_bad.json")"
|
||||
check "tools-off negative: offering tools -> 400 gate_expect" \
|
||||
bash -c "test $code = 400"
|
||||
check "tools-off negative: reason names gate_expect" mk chk "$TMP/r_toolsoff_bad.json" \
|
||||
'r["error"]["code"]=="gate_expect"'
|
||||
|
||||
# 3. dialect-leak rejections (the loud-failure contract)
|
||||
code="$(post "$PORT" "$TMP/plain.json" "$TMP/r_leak_hdr.json" -H 'anthropic-version: 2023-06-01')"
|
||||
check "leak: anthropic-version header -> 400" test "$code" = "400"
|
||||
check "leak: header reason names the leak" mk chk "$TMP/r_leak_hdr.json" \
|
||||
'r["error"]["code"]=="gate_dialect_leak" and "anthropic-version" in r["error"]["message"]'
|
||||
mk mut "$TMP/leak_tools.json" "oa-gate plain probe: explain the fixture topic simply." anth-tools
|
||||
code="$(post "$PORT" "$TMP/leak_tools.json" "$TMP/r_leak_tools.json")"
|
||||
check "leak: input_schema tools -> 400 gate_dialect_leak" bash -c \
|
||||
"test $code = 400"
|
||||
check "leak: input_schema reason" mk chk "$TMP/r_leak_tools.json" \
|
||||
'r["error"]["code"]=="gate_dialect_leak" and "input_schema" in r["error"]["message"]'
|
||||
mk mut "$TMP/leak_sys.json" "oa-gate plain probe: explain the fixture topic simply." top-system
|
||||
code="$(post "$PORT" "$TMP/leak_sys.json" "$TMP/r_leak_sys.json")"
|
||||
check "leak: top-level system -> 400" test "$code" = "400"
|
||||
mk mut "$TMP/leak_block.json" "oa-gate plain probe: explain the fixture topic simply." anth-block
|
||||
code="$(post "$PORT" "$TMP/leak_block.json" "$TMP/r_leak_block.json")"
|
||||
check "leak: Anthropic tool_result content block -> 400" test "$code" = "400"
|
||||
|
||||
# 4. scenario request expectations
|
||||
mk mut "$TMP/no_tc.json" "oa-gate plain probe: explain the fixture topic simply." no-tool-choice
|
||||
code="$(post "$PORT" "$TMP/no_tc.json" "$TMP/r_no_tc.json")"
|
||||
check "expect: missing tool_choice -> 400" test "$code" = "400"
|
||||
mk mut "$TMP/ptc.json" "oa-gate plain probe: explain the fixture topic simply." ptc-true
|
||||
code="$(post "$PORT" "$TMP/ptc.json" "$TMP/r_ptc.json")"
|
||||
check "expect: parallel_tool_calls true -> 400 (ADR-0005 pin)" test "$code" = "400"
|
||||
|
||||
# 5. single tool round-trip
|
||||
ST_PROMPT="oa-gate single tool note: save the fixture note to a file."
|
||||
mk plain "$TMP/st1.json" "$ST_PROMPT"
|
||||
code="$(post "$PORT" "$TMP/st1.json" "$TMP/r_st1.json")"
|
||||
check "single-tool leg1: HTTP 200" test "$code" = "200"
|
||||
check "single-tool leg1: one write_file call, finish tool_calls, string args" mk chk "$TMP/r_st1.json" \
|
||||
'r["choices"][0]["finish_reason"]=="tool_calls" and len(r["choices"][0]["message"]["tool_calls"])==1 and r["choices"][0]["message"]["tool_calls"][0]["type"]=="function" and r["choices"][0]["message"]["tool_calls"][0]["function"]["name"]=="write_file" and isinstance(r["choices"][0]["message"]["tool_calls"][0]["function"]["arguments"],str) and json.loads(r["choices"][0]["message"]["tool_calls"][0]["function"]["arguments"])["path"]=="openai-single-note.md"'
|
||||
mk chain "$TMP/st2.json" "$ST_PROMPT" ok "$TMP/r_st1.json"
|
||||
code="$(post "$PORT" "$TMP/st2.json" "$TMP/r_st2.json")"
|
||||
check "single-tool leg2: echo + tool turn -> 200 final text" test "$code" = "200"
|
||||
check "single-tool leg2: final names the file, finish stop" mk chk "$TMP/r_st2.json" \
|
||||
'r["choices"][0]["finish_reason"]=="stop" and "openai-single-note.md" in r["choices"][0]["message"]["content"]'
|
||||
mk chain "$TMP/st2_no.json" "$ST_PROMPT" no-tool-turn "$TMP/r_st1.json"
|
||||
code="$(post "$PORT" "$TMP/st2_no.json" "$TMP/r_st2_no.json")"
|
||||
check "single-tool negative: echo without tool turn -> 400 gate_pairing" \
|
||||
bash -c "test $code = 400"
|
||||
check "single-tool negative: pairing reason" mk chk "$TMP/r_st2_no.json" \
|
||||
'r["error"]["code"]=="gate_pairing"'
|
||||
mk chain "$TMP/st2_wrong.json" "$ST_PROMPT" wrong-id "$TMP/r_st1.json"
|
||||
code="$(post "$PORT" "$TMP/st2_wrong.json" "$TMP/r_st2_wrong.json")"
|
||||
check "single-tool negative: wrong tool_call_id -> 400" test "$code" = "400"
|
||||
mk chain "$TMP/st2_obj.json" "$ST_PROMPT" object-args "$TMP/r_st1.json"
|
||||
code="$(post "$PORT" "$TMP/st2_obj.json" "$TMP/r_st2_obj.json")"
|
||||
check "single-tool negative: arguments echoed as object -> 400 shape" \
|
||||
bash -c "test $code = 400"
|
||||
check "single-tool negative: shape reason names STRING" mk chk "$TMP/r_st2_obj.json" \
|
||||
'r["error"]["code"]=="gate_tool_call_shape" and "STRING" in r["error"]["message"]'
|
||||
|
||||
# 6. escaping torture (the two-escaper trap, spec section 6)
|
||||
T_PROMPT="oa-gate torture probe: write the escaping torture file."
|
||||
mk plain "$TMP/t1.json" "$T_PROMPT"
|
||||
code="$(post "$PORT" "$TMP/t1.json" "$TMP/r_t1.json")"
|
||||
check "torture leg1: HTTP 200" test "$code" = "200"
|
||||
check "torture leg1: arguments decode to the exact nasty payload" \
|
||||
mk torture "$TMP/r_t1.json" scenarios-openai.json
|
||||
mk chain "$TMP/t2.json" "$T_PROMPT" ok "$TMP/r_t1.json"
|
||||
code="$(post "$PORT" "$TMP/t2.json" "$TMP/r_t2.json")"
|
||||
check "torture leg2: faithful echo -> 200 final" test "$code" = "200"
|
||||
mk chain "$TMP/t2_dbl.json" "$T_PROMPT" double-encode "$TMP/r_t1.json"
|
||||
code="$(post "$PORT" "$TMP/t2_dbl.json" "$TMP/r_t2_dbl.json")"
|
||||
check "torture negative: double-encoded echo -> 400" test "$code" = "400"
|
||||
check "torture negative: reason names the two-escaper trap" mk chk "$TMP/r_t2_dbl.json" \
|
||||
'r["error"]["code"]=="gate_echo_mismatch" and "two-escaper" in r["error"]["message"]'
|
||||
|
||||
# 7. parallel double-call
|
||||
P_PROMPT="oa-gate parallel probe: run the two-write parallel case."
|
||||
mk plain "$TMP/p1.json" "$P_PROMPT"
|
||||
code="$(post "$PORT" "$TMP/p1.json" "$TMP/r_p1.json")"
|
||||
check "parallel leg1: TWO tool_calls, distinct ids" mk chk "$TMP/r_p1.json" \
|
||||
'r["choices"][0]["finish_reason"]=="tool_calls" and len(r["choices"][0]["message"]["tool_calls"])==2 and r["choices"][0]["message"]["tool_calls"][0]["id"]!=r["choices"][0]["message"]["tool_calls"][1]["id"]'
|
||||
mk chain "$TMP/p2.json" "$P_PROMPT" ok "$TMP/r_p1.json"
|
||||
code="$(post "$PORT" "$TMP/p2.json" "$TMP/r_p2.json")"
|
||||
check "parallel leg2: both results -> 200 final" test "$code" = "200"
|
||||
mk chain "$TMP/p2_one.json" "$P_PROMPT" only-first "$TMP/r_p1.json"
|
||||
code="$(post "$PORT" "$TMP/p2_one.json" "$TMP/r_p2_one.json")"
|
||||
check "parallel negative: answering only one call -> 400 pairing" test "$code" = "400"
|
||||
|
||||
# 8. two-round mission (loop continuation + step indexing)
|
||||
M_PROMPT="oa-gate mission probe: run the two-round mission."
|
||||
mk plain "$TMP/m1.json" "$M_PROMPT"
|
||||
code="$(post "$PORT" "$TMP/m1.json" "$TMP/r_m1.json")"
|
||||
check "mission leg1: part-1 tool call" mk chk "$TMP/r_m1.json" \
|
||||
'json.loads(r["choices"][0]["message"]["tool_calls"][0]["function"]["arguments"])["path"]=="mission-part-1.md"'
|
||||
mk chain "$TMP/m2.json" "$M_PROMPT" ok "$TMP/r_m1.json"
|
||||
code="$(post "$PORT" "$TMP/m2.json" "$TMP/r_m2.json")"
|
||||
check "mission leg2: part-2 tool call (step indexed by assistant count)" mk chk "$TMP/r_m2.json" \
|
||||
'r["choices"][0]["finish_reason"]=="tool_calls" and json.loads(r["choices"][0]["message"]["tool_calls"][0]["function"]["arguments"])["path"]=="mission-part-2.md"'
|
||||
mk chain "$TMP/m3.json" "$M_PROMPT" ok "$TMP/r_m1.json" "$TMP/r_m2.json"
|
||||
code="$(post "$PORT" "$TMP/m3.json" "$TMP/r_m3.json")"
|
||||
check "mission leg3: final text, finish stop" mk chk "$TMP/r_m3.json" \
|
||||
'r["choices"][0]["finish_reason"]=="stop" and "Mission complete" in r["choices"][0]["message"]["content"]'
|
||||
mk chain "$TMP/m4.json" "$M_PROMPT" ok "$TMP/r_m1.json" "$TMP/r_m2.json" "$TMP/r_m3.json"
|
||||
code="$(post "$PORT" "$TMP/m4.json" "$TMP/r_m4.json")"
|
||||
check "mission overrun: past-script request -> GATE-SCRIPT-EXHAUSTED" mk chk "$TMP/r_m4.json" \
|
||||
'r["choices"][0]["message"]["content"].startswith("GATE-SCRIPT-EXHAUSTED")'
|
||||
|
||||
# 9. injected API errors (OpenAI error envelope)
|
||||
for want in 400 429 500 503; do
|
||||
case "$want" in
|
||||
400) marker="four hundred";; 429) marker="rate limit";;
|
||||
500) marker="five hundred";; 503) marker="unavailable";;
|
||||
esac
|
||||
mk plain "$TMP/e_$want.json" "oa-gate error $marker: trigger the injected failure."
|
||||
code="$(post "$PORT" "$TMP/e_$want.json" "$TMP/r_e_$want.json")"
|
||||
check "api-error $want: status returned" test "$code" = "$want"
|
||||
check "api-error $want: OpenAI error envelope" mk chk "$TMP/r_e_$want.json" \
|
||||
'isinstance(r["error"]["message"],str) and "gate-injected" in r["error"]["message"] and isinstance(r["error"]["type"],str)'
|
||||
done
|
||||
|
||||
# 10. background (unmatched) request
|
||||
mk plain "$TMP/bg.json" "hello there, just a boot probe with no marker"
|
||||
code="$(post "$PORT" "$TMP/bg.json" "$TMP/r_bg.json")"
|
||||
check "background: unmatched prompt -> benign ok" mk chk "$TMP/r_bg.json" \
|
||||
'r["choices"][0]["message"]["content"]=="ok"'
|
||||
|
||||
# 11. ground-truth log invariants
|
||||
check "ground-truth JSONL log invariants" mk logcheck "$TMP/req.jsonl"
|
||||
|
||||
# 12. production-port refusal
|
||||
rc=0
|
||||
"$PY" stub-openai.py --port 7770 --scenarios scenarios-openai.json \
|
||||
--log "$TMP/never.jsonl" >/dev/null 2>&1 || rc=$?
|
||||
check "refuses production port 7770" test "$rc" -ne 0
|
||||
|
||||
# ---- hostile mode: black-hole ----------------------------------------------
|
||||
BH="$(freeport)"
|
||||
"$PY" stub-openai.py --port "$BH" --log "$TMP/bh.jsonl" --mode black-hole \
|
||||
>/dev/null 2>&1 &
|
||||
PIDS+=($!); disown
|
||||
check "black-hole: healthy" waithealth "$BH"
|
||||
rc=0
|
||||
curl -s -o /dev/null --max-time 3 -H 'content-type: application/json' \
|
||||
--data-binary @"$TMP/plain.json" \
|
||||
"http://127.0.0.1:$BH/v1/chat/completions" || rc=$?
|
||||
check "black-hole: client times out (curl rc 28)" test "$rc" -eq 28
|
||||
check "black-hole: health still answers during the hang" \
|
||||
curl -sf --max-time 2 "http://127.0.0.1:$BH/gate/health"
|
||||
|
||||
# ---- hostile mode: mid-body-drop -------------------------------------------
|
||||
MD="$(freeport)"
|
||||
"$PY" stub-openai.py --port "$MD" --log "$TMP/md.jsonl" --mode mid-body-drop \
|
||||
>/dev/null 2>&1 &
|
||||
PIDS+=($!); disown
|
||||
check "mid-body-drop: healthy" waithealth "$MD"
|
||||
rc=0
|
||||
curl -s --max-time 5 -o "$TMP/half.json" -H 'content-type: application/json' \
|
||||
--data-binary @"$TMP/plain.json" \
|
||||
"http://127.0.0.1:$MD/v1/chat/completions" || rc=$?
|
||||
check "mid-body-drop: transfer fails (curl rc $rc)" test "$rc" -ne 0
|
||||
check "mid-body-drop: partial body is not parseable JSON" mk notjson "$TMP/half.json"
|
||||
|
||||
# ---- hostile mode: tool-pending-forever ------------------------------------
|
||||
TP="$(freeport)"
|
||||
"$PY" stub-openai.py --port "$TP" --log "$TMP/tp.jsonl" \
|
||||
--mode tool-pending-forever >/dev/null 2>&1 &
|
||||
PIDS+=($!); disown
|
||||
check "tool-pending-forever: healthy" waithealth "$TP"
|
||||
for i in 1 2 3; do
|
||||
code="$(post "$TP" "$TMP/plain.json" "$TMP/r_tp$i.json")"
|
||||
check "tool-pending-forever: request $i -> 200" test "$code" = "200"
|
||||
done
|
||||
check "tool-pending-forever: three FRESH tool_calls, distinct ids" \
|
||||
mk pending "$TMP/r_tp1.json" "$TMP/r_tp2.json" "$TMP/r_tp3.json"
|
||||
check "tool-pending-forever: /gate/stats counts 3 chat hits" \
|
||||
bash -c "curl -sf http://127.0.0.1:$TP/gate/stats | grep -q '\"chat_hits\": 3'"
|
||||
|
||||
# ---- summary ----------------------------------------------------------------
|
||||
echo
|
||||
echo "selftest: $PASS passed, $FAIL failed"
|
||||
if [ "$FAIL" -ne 0 ]; then
|
||||
echo "SELFTEST RED"
|
||||
exit 1
|
||||
fi
|
||||
echo "SELFTEST GREEN (stub-openai gate scaffolding verified)"
|
||||
@@ -1,652 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""stub-openai.py - deterministic local stand-in for an OpenAI-format
|
||||
/v1/chat/completions provider, for the soul-openai-tools-v2 gate
|
||||
(docs/specs/SPEC-soul-openai-tools-v2-2026-08-06.md). No API key, no network,
|
||||
no model.
|
||||
|
||||
Sibling of gate9's stub-llm.py (Anthropic dialect, _wt-beta-round9/scripts/
|
||||
gate9/): same scenario mechanism (marker matching, assistant-count step
|
||||
indexing, ground-truth JSONL log, prod-port refusal), different wire.
|
||||
Staging home is tests/gate-openai/ in _wt-openai-tools; folds into
|
||||
scripts/gate9/ after round 9 merges (see README.md).
|
||||
|
||||
WHAT IT DOES
|
||||
* Serves POST /v1/chat/completions on 127.0.0.1 only (OpenAI dialect).
|
||||
* VALIDATES every request - this is the gate's discriminator, built
|
||||
BEFORE the brain-side El code exists so dialect leakage fails loudly:
|
||||
- Anthropic tells are 400 code=gate_dialect_leak: `anthropic-version`
|
||||
header; top-level `system` / `stop_sequences` / `max_tokens_to_sample`
|
||||
/ `anthropic_version`; `input_schema` inside a tool entry; Anthropic
|
||||
content blocks (tool_use / tool_result / server_tool_use / ...).
|
||||
- tools[] must be OpenAI-shaped {type:"function", function:{name,
|
||||
description, parameters}} with unique names -> 400 gate_tools_shape.
|
||||
- assistant tool_calls echoes must be {id, type:"function",
|
||||
function:{name, arguments:<JSON-encoded STRING>}}; a decoded-object
|
||||
`arguments` is a wire bug -> 400 gate_tool_call_shape.
|
||||
- every assistant tool_calls turn must be answered by role:"tool"
|
||||
messages covering EVERY tool_call_id, immediately following;
|
||||
unknown / duplicate / missing ids -> 400 gate_pairing.
|
||||
- echoed `arguments` for gate-issued call ids (call_gate_*) are
|
||||
recomputed from the script and compared after ONE json decode ->
|
||||
400 gate_echo_mismatch. This is the two-escaper-trap discriminator
|
||||
named in the spec's security model (section 6).
|
||||
- scenario-level request expectations from scenarios-openai.json
|
||||
(tools offered, OpenAI-shaped tool_choice, parallel_tool_calls
|
||||
pinned false per ADR-0005) -> 400 gate_expect.
|
||||
* Answers with SCRIPTED responses: plain text (finish_reason "stop"),
|
||||
tool calls (finish_reason "tool_calls", arguments JSON-encoded, incl. a
|
||||
nested-quote/escaping torture payload and a parallel two-call case), and
|
||||
API-error injection (OpenAI error envelope). Scenario is selected by
|
||||
scanning user-message text (newest first) for a registered marker
|
||||
substring; the step index is the number of assistant messages already in
|
||||
the request (stateless replay - resumes index correctly by construction).
|
||||
* Writes a ground-truth JSONL log (--log): one record per request with the
|
||||
validation verdict, matched scenario/step, and exactly which tool calls
|
||||
were delivered. Gate assertions compare the brain's claims against THIS
|
||||
log - truth, not narration.
|
||||
* Unmatched requests (boot probes, awareness chatter) get a benign "ok"
|
||||
text response, logged kind=background, never counted as ground truth.
|
||||
* HOSTILE MODES (--mode) on the same file:
|
||||
black-hole accept + read the request, never respond;
|
||||
mid-body-drop send half a JSON body, then abort the socket;
|
||||
tool-pending-forever every request gets a FRESH tool_call
|
||||
(finish_reason "tool_calls"), forever - tests
|
||||
the agentic loop's iteration cap; count the
|
||||
brain's round-trips via GET /gate/stats.
|
||||
|
||||
usage: stub-openai.py --port P --scenarios scenarios-openai.json \
|
||||
--log requests.jsonl [--mode MODE]
|
||||
Listens on 127.0.0.1 only. Refuses production ports 7770/7779/17779.
|
||||
"""
|
||||
import argparse
|
||||
import itertools
|
||||
import json
|
||||
import socket
|
||||
import struct
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
STATE = {"scenarios": None, "log_path": None, "lock": threading.Lock(),
|
||||
"seq": 0, "mode": "normal", "chat_hits": 0}
|
||||
_PENDING_SEQ = itertools.count(1)
|
||||
|
||||
ANTHROPIC_TOP_KEYS = ("system", "stop_sequences", "max_tokens_to_sample",
|
||||
"anthropic_version")
|
||||
ANTHROPIC_BLOCK_TYPES = {"tool_use", "tool_result", "server_tool_use",
|
||||
"web_search_tool_result", "thinking",
|
||||
"redacted_thinking"}
|
||||
DEFAULT_EXPECT = {"require_tools": True, "require_tool_choice": True,
|
||||
"parallel_tool_calls": False, "forbid_tools": False}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- loading ----
|
||||
def load_scenarios(path):
|
||||
cfg = json.load(open(path))
|
||||
defaults = dict(DEFAULT_EXPECT)
|
||||
defaults.update(cfg.get("defaults", {}).get("expect_request", {}))
|
||||
marker_map = [] # (marker_lower, cname, pid)
|
||||
scripts = {} # cname or cname/pid -> expanded script
|
||||
pid_map = {} # pid -> cname (for call_gate_* id -> script lookup)
|
||||
expects = {} # cname -> merged expect_request
|
||||
for cname, cls in cfg["classes"].items():
|
||||
scripts[cname] = expand_script(cls.get("script", []))
|
||||
exp = dict(defaults)
|
||||
exp.update(cls.get("expect_request", {}))
|
||||
expects[cname] = exp
|
||||
for ph in cls["phrasings"]:
|
||||
if ph.get("script") is not None:
|
||||
scripts[cname + "/" + ph["id"]] = expand_script(ph["script"])
|
||||
marker_map.append((ph["marker"].lower(), cname, ph["id"]))
|
||||
pid_map[ph["id"]] = cname
|
||||
return {"cfg": cfg, "marker_map": marker_map, "scripts": scripts,
|
||||
"pid_map": pid_map, "expects": expects}
|
||||
|
||||
|
||||
def expand_script(script):
|
||||
"""Same repeat-expansion contract as gate9's stub-llm.py ({N}/{NN})."""
|
||||
out = []
|
||||
for step in script:
|
||||
if "repeat" in step:
|
||||
for n in range(1, step["repeat"] + 1):
|
||||
t = {k: v for k, v in step.items() if k != "repeat"}
|
||||
out.append(json.loads(json.dumps(t)
|
||||
.replace("{NN}", "%02d" % n)
|
||||
.replace("{N}", str(n))))
|
||||
else:
|
||||
out.append(step)
|
||||
return out
|
||||
|
||||
|
||||
# ------------------------------------------------------------- validation ----
|
||||
def _rej(message, code):
|
||||
return {"status": 400, "message": message, "code": code}
|
||||
|
||||
|
||||
def validate_dialect(headers, req):
|
||||
"""Universal checks - run on EVERY request, scenario-matched or not.
|
||||
Anything Anthropic-shaped on this lane means the brain's translator
|
||||
leaked; the whole point is that it fails loudly, here, with a reason."""
|
||||
if headers.get("anthropic-version"):
|
||||
return _rej("anthropic-version header on the OpenAI lane: this "
|
||||
"request was built by the Anthropic dialect path",
|
||||
"gate_dialect_leak")
|
||||
for k in ANTHROPIC_TOP_KEYS:
|
||||
if k in req:
|
||||
return _rej("top-level `%s` is Anthropic dialect; the OpenAI "
|
||||
"dialect has no such field (system prompt goes in "
|
||||
"messages[0])" % k, "gate_dialect_leak")
|
||||
tools = req.get("tools")
|
||||
if tools is not None:
|
||||
if not isinstance(tools, list):
|
||||
return _rej("`tools` must be an array", "gate_tools_shape")
|
||||
names = []
|
||||
for i, t in enumerate(tools):
|
||||
if not isinstance(t, dict):
|
||||
return _rej("tools[%d] is not an object" % i,
|
||||
"gate_tools_shape")
|
||||
if "input_schema" in t or (isinstance(t.get("function"), dict)
|
||||
and "input_schema" in t["function"]):
|
||||
return _rej("tools[%d] carries `input_schema` (Anthropic "
|
||||
"dialect); OpenAI dialect wants "
|
||||
"function.parameters" % i, "gate_dialect_leak")
|
||||
if t.get("type") != "function":
|
||||
return _rej("tools[%d].type must be \"function\", got %r"
|
||||
% (i, t.get("type")), "gate_tools_shape")
|
||||
fn = t.get("function")
|
||||
if not isinstance(fn, dict):
|
||||
return _rej("tools[%d].function missing" % i,
|
||||
"gate_tools_shape")
|
||||
if not isinstance(fn.get("name"), str) or not fn["name"]:
|
||||
return _rej("tools[%d].function.name missing/empty" % i,
|
||||
"gate_tools_shape")
|
||||
if not isinstance(fn.get("description"), str) or not fn["description"]:
|
||||
return _rej("tools[%d].function.description missing/empty" % i,
|
||||
"gate_tools_shape")
|
||||
if not isinstance(fn.get("parameters"), dict):
|
||||
return _rej("tools[%d].function.parameters missing (JSON "
|
||||
"Schema object expected)" % i, "gate_tools_shape")
|
||||
names.append(fn["name"])
|
||||
if len(names) != len(set(names)):
|
||||
return _rej("tools: tool names must be unique", "gate_tools_shape")
|
||||
msgs = req.get("messages")
|
||||
if not isinstance(msgs, list) or not msgs:
|
||||
return _rej("`messages` must be a non-empty array",
|
||||
"gate_messages_shape")
|
||||
for i, m in enumerate(msgs):
|
||||
if not isinstance(m, dict):
|
||||
return _rej("messages[%d] is not an object" % i,
|
||||
"gate_messages_shape")
|
||||
c = m.get("content")
|
||||
if isinstance(c, list):
|
||||
for j, b in enumerate(c):
|
||||
if isinstance(b, dict) and b.get("type") in ANTHROPIC_BLOCK_TYPES:
|
||||
return _rej("messages[%d].content[%d] is an Anthropic "
|
||||
"`%s` block; the OpenAI dialect uses "
|
||||
"tool_calls / role:\"tool\" messages"
|
||||
% (i, j, b.get("type")), "gate_dialect_leak")
|
||||
if m.get("role") == "tool":
|
||||
if not isinstance(m.get("tool_call_id"), str) or not m["tool_call_id"]:
|
||||
return _rej("messages[%d]: role \"tool\" requires a "
|
||||
"`tool_call_id`" % i, "gate_messages_shape")
|
||||
if "content" not in m:
|
||||
return _rej("messages[%d]: role \"tool\" requires `content`"
|
||||
% i, "gate_messages_shape")
|
||||
if m.get("role") == "assistant" and m.get("tool_calls") is not None:
|
||||
tcs = m["tool_calls"]
|
||||
if not isinstance(tcs, list) or not tcs:
|
||||
return _rej("messages[%d].tool_calls must be a non-empty "
|
||||
"array" % i, "gate_tool_call_shape")
|
||||
for j, tc in enumerate(tcs):
|
||||
if not isinstance(tc, dict) or tc.get("type") != "function":
|
||||
return _rej("messages[%d].tool_calls[%d].type must be "
|
||||
"\"function\"" % (i, j), "gate_tool_call_shape")
|
||||
if not isinstance(tc.get("id"), str) or not tc["id"]:
|
||||
return _rej("messages[%d].tool_calls[%d].id missing"
|
||||
% (i, j), "gate_tool_call_shape")
|
||||
fn = tc.get("function")
|
||||
if not isinstance(fn, dict) or not isinstance(fn.get("name"), str):
|
||||
return _rej("messages[%d].tool_calls[%d].function.name "
|
||||
"missing" % (i, j), "gate_tool_call_shape")
|
||||
if not isinstance(fn.get("arguments"), str):
|
||||
return _rej("messages[%d].tool_calls[%d].function."
|
||||
"arguments must be a JSON-encoded STRING, "
|
||||
"got %s" % (i, j,
|
||||
type(fn.get("arguments")).__name__),
|
||||
"gate_tool_call_shape")
|
||||
return None
|
||||
|
||||
|
||||
def validate_pairing(msgs):
|
||||
"""OpenAI pairing rule: every assistant tool_calls turn must be followed
|
||||
immediately by role:"tool" messages answering every tool_call_id."""
|
||||
open_ids, open_at = set(), None
|
||||
for i, m in enumerate(msgs):
|
||||
role = m.get("role")
|
||||
if role == "tool":
|
||||
tid = m.get("tool_call_id")
|
||||
if open_at is None:
|
||||
return _rej("messages[%d]: role \"tool\" message with no "
|
||||
"preceding assistant tool_calls turn "
|
||||
"(tool_call_id=%s)" % (i, tid), "gate_pairing")
|
||||
if tid not in open_ids:
|
||||
return _rej("messages[%d]: tool message answers unknown or "
|
||||
"already-answered tool_call_id %s" % (i, tid),
|
||||
"gate_pairing")
|
||||
open_ids.discard(tid)
|
||||
continue
|
||||
if open_ids:
|
||||
return _rej("messages[%d]: assistant tool_calls not fully "
|
||||
"answered before messages[%d]; missing tool "
|
||||
"responses for: %s" % (open_at, i, sorted(open_ids)),
|
||||
"gate_pairing")
|
||||
open_ids, open_at = set(), None
|
||||
if role == "assistant" and m.get("tool_calls"):
|
||||
ids = [tc.get("id") for tc in m["tool_calls"]]
|
||||
open_ids, open_at = set(ids), i
|
||||
if open_ids:
|
||||
return _rej("messages[%d]: assistant tool_calls at end of thread "
|
||||
"without tool responses for: %s"
|
||||
% (open_at, sorted(open_ids)), "gate_pairing")
|
||||
return None
|
||||
|
||||
|
||||
def validate_echo_args(msgs, loaded):
|
||||
"""Ground-truth round-trip check: for every echoed gate-issued call id,
|
||||
recompute the arguments this stub originally sent from the script and
|
||||
require one json decode to reproduce them exactly. Catches the
|
||||
two-escaper trap (spec section 6) deterministically."""
|
||||
if not loaded:
|
||||
return None
|
||||
for i, m in enumerate(msgs):
|
||||
if m.get("role") != "assistant":
|
||||
continue
|
||||
for tc in m.get("tool_calls") or []:
|
||||
tid = tc.get("id", "")
|
||||
if not tid.startswith("call_gate_"):
|
||||
continue
|
||||
rest = tid[len("call_gate_"):]
|
||||
try:
|
||||
pid, s_part, k_part = rest.rsplit("_", 2)
|
||||
step_idx, k = int(s_part[1:]), int(k_part)
|
||||
except (ValueError, IndexError):
|
||||
continue
|
||||
cname = loaded["pid_map"].get(pid)
|
||||
if cname is None:
|
||||
continue
|
||||
script = (loaded["scripts"].get(cname + "/" + pid)
|
||||
or loaded["scripts"].get(cname) or [])
|
||||
if step_idx >= len(script):
|
||||
continue
|
||||
calls = script[step_idx].get("tool_calls") or []
|
||||
if k >= len(calls):
|
||||
continue
|
||||
expected = calls[k]
|
||||
fn = tc.get("function") or {}
|
||||
if fn.get("name") != expected["name"]:
|
||||
return _rej("messages[%d]: echoed tool name %r != issued %r "
|
||||
"for %s" % (i, fn.get("name"), expected["name"],
|
||||
tid), "gate_echo_mismatch")
|
||||
try:
|
||||
got = json.loads(fn.get("arguments", ""))
|
||||
except ValueError:
|
||||
return _rej("messages[%d]: echoed arguments for %s are not "
|
||||
"valid JSON after one decode (truncated or "
|
||||
"half-escaped?)" % (i, tid), "gate_echo_mismatch")
|
||||
if got != expected["arguments"]:
|
||||
hint = (" (decoded to a string, not an object: "
|
||||
"double-encoded - the two-escaper trap)"
|
||||
if isinstance(got, str) else "")
|
||||
return _rej("messages[%d]: echoed arguments for %s do not "
|
||||
"round-trip to the issued payload%s"
|
||||
% (i, tid, hint), "gate_echo_mismatch")
|
||||
return None
|
||||
|
||||
|
||||
def validate_expect(req, exp):
|
||||
"""Scenario-level request expectations (scenarios-openai.json)."""
|
||||
tools = req.get("tools") or []
|
||||
if exp.get("forbid_tools") and tools:
|
||||
return _rej("this scenario is chat-only: no `tools` may be offered "
|
||||
"on it", "gate_expect")
|
||||
if exp.get("require_tools") and not tools:
|
||||
return _rej("scenario expects a `tools` array to be offered (the "
|
||||
"agentic lane must advertise its tools)", "gate_expect")
|
||||
if exp.get("require_tool_choice"):
|
||||
tc = req.get("tool_choice")
|
||||
ok = tc in ("auto", "none", "required") or (
|
||||
isinstance(tc, dict) and tc.get("type") == "function"
|
||||
and isinstance(tc.get("function"), dict)
|
||||
and tc["function"].get("name"))
|
||||
if not ok:
|
||||
return _rej("scenario expects an OpenAI-shaped `tool_choice`, "
|
||||
"got %r" % (tc,), "gate_expect")
|
||||
want_ptc = exp.get("parallel_tool_calls", None)
|
||||
if want_ptc is not None:
|
||||
if "parallel_tool_calls" not in req:
|
||||
return _rej("scenario expects explicit `parallel_tool_calls` "
|
||||
"(ADR-0005: must be pinned false on the wire)",
|
||||
"gate_expect")
|
||||
if req["parallel_tool_calls"] != want_ptc:
|
||||
return _rej("scenario expects parallel_tool_calls=%s, got %s"
|
||||
% (json.dumps(want_ptc),
|
||||
json.dumps(req["parallel_tool_calls"])),
|
||||
"gate_expect")
|
||||
return None
|
||||
|
||||
|
||||
# --------------------------------------------------------- scenario match ----
|
||||
def extract_user_texts_newest_first(msgs):
|
||||
texts = []
|
||||
for m in reversed(msgs):
|
||||
if not isinstance(m, dict) or m.get("role") != "user":
|
||||
continue
|
||||
c = m.get("content")
|
||||
if isinstance(c, str):
|
||||
texts.append(c)
|
||||
elif isinstance(c, list):
|
||||
for b in c:
|
||||
if isinstance(b, dict) and b.get("type") == "text":
|
||||
texts.append(b.get("text", ""))
|
||||
return texts
|
||||
|
||||
|
||||
def match_scenario(loaded, msgs):
|
||||
for text in extract_user_texts_newest_first(msgs):
|
||||
tl = text.lower()
|
||||
for marker, cname, pid in loaded["marker_map"]:
|
||||
if marker in tl:
|
||||
return cname, pid
|
||||
return None, None
|
||||
|
||||
|
||||
# ------------------------------------------------------------- rendering ----
|
||||
def completion_envelope(msg, finish, model, usage=(100, 100)):
|
||||
return {"id": "chatcmpl-gate-" + uuid.uuid4().hex[:12],
|
||||
"object": "chat.completion", "created": int(time.time()),
|
||||
"model": model,
|
||||
"choices": [{"index": 0, "message": msg,
|
||||
"finish_reason": finish, "logprobs": None}],
|
||||
"usage": {"prompt_tokens": usage[0],
|
||||
"completion_tokens": usage[1],
|
||||
"total_tokens": usage[0] + usage[1]}}
|
||||
|
||||
|
||||
def text_completion(text, model):
|
||||
return completion_envelope({"role": "assistant", "content": text},
|
||||
"stop", model, usage=(1, 1))
|
||||
|
||||
|
||||
def pending_body(seq, model):
|
||||
args = json.dumps({"path": "never-%04d.md" % seq,
|
||||
"content": "this run never completes"})
|
||||
msg = {"role": "assistant", "content": None,
|
||||
"tool_calls": [{"id": "call_hostile_pending_%04d" % seq,
|
||||
"type": "function",
|
||||
"function": {"name": "write_file",
|
||||
"arguments": args}}]}
|
||||
return completion_envelope(msg, "tool_calls", model, usage=(1, 1))
|
||||
|
||||
|
||||
def render_step(step, cname, pid, step_idx, model):
|
||||
"""Returns (http_status, body_dict, delivered) - delivered is ground
|
||||
truth for the JSONL log."""
|
||||
delivered = {"tool_calls": [], "finish_reason": None, "api_error": None}
|
||||
if "api_error" in step:
|
||||
e = step["api_error"]
|
||||
delivered["api_error"] = e["status"]
|
||||
return (e["status"],
|
||||
{"error": {"message": e["message"],
|
||||
"type": e.get("type", "server_error"),
|
||||
"param": None, "code": e.get("code")}},
|
||||
delivered)
|
||||
msg = {"role": "assistant"}
|
||||
finish = "stop"
|
||||
if step.get("tool_calls"):
|
||||
tcs = []
|
||||
for k, call in enumerate(step["tool_calls"]):
|
||||
tid = "call_gate_%s_s%d_%d" % (pid, step_idx, k)
|
||||
tcs.append({"id": tid, "type": "function",
|
||||
"function": {"name": call["name"],
|
||||
"arguments": json.dumps(
|
||||
call["arguments"],
|
||||
ensure_ascii=False)}})
|
||||
delivered["tool_calls"].append(call["name"])
|
||||
msg["tool_calls"] = tcs
|
||||
msg["content"] = step.get("text") # null when no narration, like real
|
||||
finish = "tool_calls"
|
||||
else:
|
||||
msg["content"] = step["text"]
|
||||
delivered["finish_reason"] = finish
|
||||
return 200, completion_envelope(msg, finish, model), delivered
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ log ------
|
||||
def log_record(rec):
|
||||
with STATE["lock"]:
|
||||
STATE["seq"] += 1
|
||||
rec["seq"] = STATE["seq"]
|
||||
with open(STATE["log_path"], "a") as f:
|
||||
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- server -----
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
|
||||
def _send_json(self, status, obj):
|
||||
body = json.dumps(obj, ensure_ascii=False).encode("utf-8")
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def _send_error(self, verdict):
|
||||
self._send_json(verdict["status"],
|
||||
{"error": {"message": verdict["message"],
|
||||
"type": "invalid_request_error",
|
||||
"param": None, "code": verdict["code"]}})
|
||||
|
||||
def _drop_mid_body(self):
|
||||
"""Valid 200 headers, half the promised body, then a socket abort
|
||||
(same SO_LINGER teardown as gate9's mid-body-drop-brain.py)."""
|
||||
full = json.dumps(text_completion(
|
||||
"This reply will never finish arriving because the connection "
|
||||
"dies in the middle of the body, which is exactly the point of "
|
||||
"this hostile fixture.", "hostile-mid-drop")).encode("utf-8")
|
||||
half = full[: len(full) // 2]
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(full))) # promises more
|
||||
self.end_headers()
|
||||
self.wfile.write(half)
|
||||
self.wfile.flush()
|
||||
try:
|
||||
self.connection.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER,
|
||||
struct.pack("ii", 1, 0))
|
||||
self.connection.shutdown(socket.SHUT_RDWR)
|
||||
except OSError:
|
||||
pass
|
||||
self.close_connection = True
|
||||
|
||||
def do_GET(self):
|
||||
path = self.path.split("?")[0]
|
||||
if path == "/gate/health":
|
||||
self._send_json(200, {"ok": True, "mode": STATE["mode"]})
|
||||
elif path == "/gate/stats":
|
||||
with STATE["lock"]:
|
||||
self._send_json(200, {"mode": STATE["mode"],
|
||||
"chat_hits": STATE["chat_hits"]})
|
||||
else:
|
||||
self._send_json(404, {"error": {"message": "not found",
|
||||
"type": "invalid_request_error",
|
||||
"param": None,
|
||||
"code": "unknown_route"}})
|
||||
|
||||
def do_POST(self):
|
||||
n = int(self.headers.get("Content-Length") or 0)
|
||||
raw = self.rfile.read(n)
|
||||
mode = STATE["mode"]
|
||||
rec = {"ts": time.time(), "path": self.path, "mode": mode,
|
||||
"kind": "background", "scenario_class": None, "phrasing": None,
|
||||
"step": None, "n_messages": 0, "n_assistant": 0,
|
||||
"validation": "ok", "validation_detail": None,
|
||||
"delivered": {"tool_calls": [], "finish_reason": None,
|
||||
"api_error": None},
|
||||
"http_status": 200}
|
||||
if self.path.split("?")[0] != "/v1/chat/completions":
|
||||
rec.update(kind="wrong_path", http_status=404)
|
||||
log_record(rec)
|
||||
self._send_json(404, {"error": {
|
||||
"message": "no such route: %s" % self.path,
|
||||
"type": "invalid_request_error", "param": None,
|
||||
"code": "unknown_route"}})
|
||||
return
|
||||
with STATE["lock"]:
|
||||
STATE["chat_hits"] += 1
|
||||
|
||||
# ---- hostile modes: behavior first, no validation ----------------
|
||||
if mode == "black-hole":
|
||||
rec.update(kind="hostile", http_status=None)
|
||||
log_record(rec)
|
||||
threading.Event().wait() # hold the socket open forever
|
||||
return
|
||||
if mode == "mid-body-drop":
|
||||
rec.update(kind="hostile", http_status=200)
|
||||
log_record(rec)
|
||||
self._drop_mid_body()
|
||||
return
|
||||
if mode == "tool-pending-forever":
|
||||
seq = next(_PENDING_SEQ)
|
||||
rec.update(kind="hostile",
|
||||
delivered={"tool_calls": ["write_file"],
|
||||
"finish_reason": "tool_calls",
|
||||
"api_error": None})
|
||||
log_record(rec)
|
||||
self._send_json(200, pending_body(seq, "gate-openai-model"))
|
||||
return
|
||||
|
||||
# ---- normal mode -------------------------------------------------
|
||||
try:
|
||||
req = json.loads(raw)
|
||||
except ValueError as exc:
|
||||
# DIAGNOSTIC CAPTURE (2026-08-06): an unparseable body used to be recorded as
|
||||
# a bare "bad_json" with the bytes thrown away, which made an intermittent
|
||||
# failure impossible to root-cause — you cannot fix what you did not keep.
|
||||
# Dump the raw body next to the log, and record exactly where the parser gave
|
||||
# up plus the offending byte, so one occurrence is enough to diagnose.
|
||||
dump_path = "%s.badbody.%s" % (STATE.get("log_path", "/tmp/stub-openai"),
|
||||
rec.get("seq", "x"))
|
||||
try:
|
||||
data = raw if isinstance(raw, (bytes, bytearray)) else str(raw).encode()
|
||||
with open(dump_path, "wb") as fh:
|
||||
fh.write(data)
|
||||
except Exception as dump_exc:
|
||||
dump_path = "(dump failed: %s)" % dump_exc
|
||||
pos = getattr(exc, "pos", None)
|
||||
near = ""
|
||||
byte_repr = ""
|
||||
if isinstance(pos, int):
|
||||
blob = raw if isinstance(raw, (bytes, bytearray)) else str(raw).encode()
|
||||
near = blob[max(0, pos - 60):pos + 60].decode("utf-8", "replace")
|
||||
if 0 <= pos < len(blob):
|
||||
byte_repr = "0x%02x" % blob[pos]
|
||||
rec.update(kind="bad_json", validation="rejected",
|
||||
validation_detail="request body is not valid JSON: %s" % exc,
|
||||
http_status=400, raw_len=len(raw), raw_dump=dump_path,
|
||||
err_pos=pos, err_byte=byte_repr, err_near=near)
|
||||
log_record(rec)
|
||||
self._send_error(_rej("request body is not valid JSON",
|
||||
"bad_json"))
|
||||
return
|
||||
msgs = req.get("messages") or []
|
||||
rec["n_messages"] = len(msgs)
|
||||
rec["n_assistant"] = sum(1 for m in msgs if isinstance(m, dict)
|
||||
and m.get("role") == "assistant")
|
||||
loaded = STATE["scenarios"]
|
||||
cname, pid = match_scenario(loaded, msgs)
|
||||
if cname:
|
||||
rec.update(kind="scenario", scenario_class=cname, phrasing=pid)
|
||||
|
||||
# Wire-level validation runs for EVERY request, scenario or not.
|
||||
verdict = (validate_dialect(self.headers, req)
|
||||
or validate_pairing([m for m in msgs
|
||||
if isinstance(m, dict)])
|
||||
or validate_echo_args(msgs, loaded))
|
||||
if verdict:
|
||||
rec.update(validation="rejected",
|
||||
validation_detail=verdict["message"],
|
||||
http_status=verdict["status"])
|
||||
log_record(rec)
|
||||
self._send_error(verdict)
|
||||
return
|
||||
|
||||
model = req.get("model", "gate-openai-model")
|
||||
if not cname:
|
||||
log_record(rec)
|
||||
self._send_json(200, text_completion("ok", model))
|
||||
return
|
||||
|
||||
script = (loaded["scripts"].get(cname + "/" + pid)
|
||||
or loaded["scripts"][cname])
|
||||
step_idx = rec["n_assistant"]
|
||||
if step_idx >= len(script):
|
||||
rec.update(kind="overrun", step=step_idx)
|
||||
log_record(rec)
|
||||
self._send_json(200, text_completion(
|
||||
"GATE-SCRIPT-EXHAUSTED %s step %d" % (pid, step_idx), model))
|
||||
return
|
||||
|
||||
step = script[step_idx]
|
||||
exp = dict(loaded["expects"][cname])
|
||||
exp.update(step.get("expect_request", {}))
|
||||
verdict = validate_expect(req, exp)
|
||||
if verdict:
|
||||
rec.update(step=step_idx, validation="rejected",
|
||||
validation_detail=verdict["message"],
|
||||
http_status=verdict["status"])
|
||||
log_record(rec)
|
||||
self._send_error(verdict)
|
||||
return
|
||||
|
||||
status, body, delivered = render_step(step, cname, pid, step_idx,
|
||||
model)
|
||||
rec.update(step=step_idx, delivered=delivered, http_status=status)
|
||||
log_record(rec)
|
||||
self._send_json(status, body)
|
||||
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--port", type=int, required=True)
|
||||
ap.add_argument("--scenarios",
|
||||
help="scenarios-openai.json (required in normal mode)")
|
||||
ap.add_argument("--log", required=True)
|
||||
ap.add_argument("--mode", default="normal",
|
||||
choices=["normal", "black-hole", "mid-body-drop",
|
||||
"tool-pending-forever"])
|
||||
args = ap.parse_args()
|
||||
if args.port in (7770, 7779, 17779):
|
||||
raise SystemExit("stub-openai: refusing production Neuron port")
|
||||
if args.mode == "normal" and not args.scenarios:
|
||||
raise SystemExit("stub-openai: --scenarios is required in normal mode")
|
||||
STATE["mode"] = args.mode
|
||||
STATE["scenarios"] = (load_scenarios(args.scenarios)
|
||||
if args.scenarios else None)
|
||||
STATE["log_path"] = args.log
|
||||
open(args.log, "w").close()
|
||||
n_markers = (len(STATE["scenarios"]["marker_map"])
|
||||
if STATE["scenarios"] else 0)
|
||||
print("stub-openai [%s]: 127.0.0.1:%d /v1/chat/completions "
|
||||
"(%d markers registered, log=%s)"
|
||||
% (args.mode, args.port, n_markers, args.log), flush=True)
|
||||
ThreadingHTTPServer(("127.0.0.1", args.port), Handler).serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,148 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# run-el-test.sh — build and RUN one engine test (tests/*.el), printing its assertions.
|
||||
#
|
||||
# WHY THIS EXISTS (2026-08-06): the engine's tests/*.el files were never runnable from the
|
||||
# tree. `elc` is a COMPILER — it emits C to stdout and exits; it does not execute anything.
|
||||
# So "the tests" could only ever be read, not run, and a signature change could silently
|
||||
# break them (exactly what happened when bridge_save gained its `wire` argument). This
|
||||
# script closes that: emit the test to C, link it against the engine modules, execute it.
|
||||
#
|
||||
# HOW IT WORKS
|
||||
# 1. elc <test>.el -> C on stdout (the test file's `main` + prototypes)
|
||||
# 2. elb (once, cached) -> per-module C for the whole engine into a scratch dir
|
||||
# 3. cc test.c + all modules EXCEPT soul.c (soul.c owns the real `main`) + the runtime
|
||||
# 4. run it
|
||||
#
|
||||
# The test C references only the engine functions it actually calls, so there are no
|
||||
# duplicate-symbol collisions with the module objects.
|
||||
#
|
||||
# RUNTIME: the REPO-PINNED vendor/el-runtime (NOT ~/el-sdk/el_runtime.c — that June build
|
||||
# is missing builtins August code calls: engram_wm_count, engram_wm_top_json,
|
||||
# http_delete_json, http_serve_async; linking against it fails with "symbol(s) not found").
|
||||
#
|
||||
# USAGE
|
||||
# tests/run-el-test.sh tests/test_bridge_serialization.el # one test
|
||||
# tests/run-el-test.sh --all # every tests/test_*.el
|
||||
# REBUILD=1 tests/run-el-test.sh ... # force module regeneration
|
||||
#
|
||||
# Tests that need a live API key / running soul (see each file's header) will report their
|
||||
# own skips or failures — this runner does not fake them.
|
||||
set -uo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO_ROOT" || exit 2
|
||||
|
||||
ELC="${ELC:-$HOME/el-sdk/elc}"
|
||||
ELB="${ELB:-$HOME/Development/el-sdk/bin/elb}"
|
||||
RUNTIME_DIR="${RUNTIME_DIR:-$REPO_ROOT/vendor/el-runtime/v1.0.0-20260501}"
|
||||
SCRATCH="${SCRATCH:-/tmp/el-test-$(basename "$REPO_ROOT")}"
|
||||
MODDIR="$SCRATCH/modules"
|
||||
OPENSSL_INC="${OPENSSL_INC:-/opt/homebrew/opt/openssl@3/include}"
|
||||
OPENSSL_LIB="${OPENSSL_LIB:-/opt/homebrew/opt/openssl@3/lib}"
|
||||
|
||||
for req in "$ELC" "$ELB" "$RUNTIME_DIR/el_runtime.c"; do
|
||||
[ -e "$req" ] || { echo "run-el-test: missing required input: $req" >&2; exit 2; }
|
||||
done
|
||||
|
||||
mkdir -p "$MODDIR" || exit 2
|
||||
|
||||
# ── Step 1: engine modules (cached — regeneration is the slow part) ────────────
|
||||
if [ "${REBUILD:-0}" = "1" ] || [ ! -f "$MODDIR/chat.c" ] || [ "chat.el" -nt "$MODDIR/chat.c" ]; then
|
||||
echo "run-el-test: generating engine modules into $MODDIR (this takes ~1-2 min)..."
|
||||
# elb's own final link step fails by design here (it wants to produce a binary named
|
||||
# `neuron` and we only need the per-module .c files it emits first). Ignore its rc.
|
||||
"$ELB" --elc="$ELC" --runtime="$RUNTIME_DIR" --out="$MODDIR/" >"$SCRATCH/elb.log" 2>&1
|
||||
if [ ! -f "$MODDIR/chat.c" ]; then
|
||||
echo "run-el-test: FATAL — elb produced no chat.c; see $SCRATCH/elb.log" >&2
|
||||
tail -5 "$SCRATCH/elb.log" >&2
|
||||
exit 2
|
||||
fi
|
||||
# elb rewrites *.elh in the source tree as a side effect (cosmetic banner churn plus a
|
||||
# stray soul..elh). Say so; the caller decides whether to `git restore` them.
|
||||
echo "run-el-test: NOTE — elb regenerated *.elh in the source tree (cosmetic churn is expected; a stray soul..elh may appear)."
|
||||
fi
|
||||
|
||||
# soul.c is needed for its engine functions (layered_cycle et al.) but it also owns the
|
||||
# daemon's real `main`, which would collide with the test's own. Compile it ONCE to an
|
||||
# object with `main` renamed away, and link that instead of the .c.
|
||||
SOUL_OBJ="$SCRATCH/soul-nomain.o"
|
||||
if [ "${REBUILD:-0}" = "1" ] || [ ! -f "$SOUL_OBJ" ] || [ "$MODDIR/soul.c" -nt "$SOUL_OBJ" ]; then
|
||||
cc -std=c11 -O1 -DHAVE_CURL -Dmain=el_soul_daemon_main_unused \
|
||||
-I "$RUNTIME_DIR" -I "$MODDIR" -I "$OPENSSL_INC" \
|
||||
-include dist/elp-c-decls.h -Wno-error=implicit-function-declaration \
|
||||
-c "$MODDIR/soul.c" -o "$SOUL_OBJ" 2>"$SCRATCH/soul-nomain.err" \
|
||||
|| { echo "run-el-test: FATAL — could not compile soul.c without main" >&2
|
||||
grep -E 'error:' "$SCRATCH/soul-nomain.err" | head -5 >&2; exit 2; }
|
||||
fi
|
||||
|
||||
# Every module except soul.c (linked as the renamed object above) and the stray soul.elh.c.
|
||||
MODS=("$SOUL_OBJ")
|
||||
for f in "$MODDIR"/*.c; do
|
||||
case "$(basename "$f")" in
|
||||
soul.c|soul.elh.c) continue ;;
|
||||
esac
|
||||
MODS+=("$f")
|
||||
done
|
||||
[ "${#MODS[@]}" -gt 1 ] || { echo "run-el-test: no module objects found" >&2; exit 2; }
|
||||
|
||||
run_one() {
|
||||
local test_el="$1"
|
||||
local name; name="$(basename "$test_el" .el)"
|
||||
local cfile="$SCRATCH/$name.c"
|
||||
local bin="$SCRATCH/$name"
|
||||
|
||||
printf '\n══ %s ══\n' "$name"
|
||||
|
||||
if ! "$ELC" "$test_el" >"$cfile" 2>"$SCRATCH/$name.elc.err"; then
|
||||
echo "COMPILE FAILED (elc):"; tail -10 "$SCRATCH/$name.elc.err"; return 1
|
||||
fi
|
||||
[ -s "$cfile" ] || { echo "COMPILE FAILED (elc produced empty C)"; return 1; }
|
||||
|
||||
if ! cc -std=c11 -O1 -DHAVE_CURL -rdynamic \
|
||||
-I "$RUNTIME_DIR" -I "$MODDIR" -I "$OPENSSL_INC" -L "$OPENSSL_LIB" \
|
||||
-include dist/elp-c-decls.h -Wno-error=implicit-function-declaration \
|
||||
-o "$bin" "$cfile" "${MODS[@]}" "$RUNTIME_DIR/el_runtime.c" \
|
||||
-lssl -lcrypto -lcurl -lpthread -lm 2>"$SCRATCH/$name.link.err"; then
|
||||
echo "LINK FAILED:"; grep -E '"_|error:' "$SCRATCH/$name.link.err" | head -10; return 1
|
||||
fi
|
||||
|
||||
# THE RUNNER OWNS THE VERDICT — the test files cannot be trusted to report it.
|
||||
#
|
||||
# Every tests/*.el assert helper does `let pass_count = pass_count + 1` INSIDE an if
|
||||
# BLOCK. El's scope rule (the same one chat.el documents at every while-body mutation:
|
||||
# "mutations inside if *blocks* don't escape scope") means those counters never
|
||||
# increment, so all 9 counted test files print "N passed, M failed" as "0 passed, 0
|
||||
# failed" — forever, whatever actually happened. A summary that can never report a
|
||||
# failure is worth exactly as much as an assertion that can never fail. Logged as a
|
||||
# bug for the real in-file fix; until then the verdict is computed HERE, from the
|
||||
# assert helpers' own per-line output, which IS reliable.
|
||||
local out="$SCRATCH/$name.out"
|
||||
"$bin" 2>&1 | tee "$out"; local rc=${PIPESTATUS[0]}
|
||||
|
||||
# NOTE: `grep -c` prints 0 AND exits 1 when there are no matches, so a `|| echo 0`
|
||||
# fallback appends a SECOND zero and every later integer test breaks on "0\n0".
|
||||
# (Caught by running this script — which is the whole argument for running things.)
|
||||
local n_pass n_fail
|
||||
n_pass=$(grep -c '^ PASS: ' "$out" 2>/dev/null); n_pass=${n_pass:-0}
|
||||
n_fail=$(grep -c '^ FAIL: ' "$out" 2>/dev/null); n_fail=${n_fail:-0}
|
||||
echo "── $name: $n_pass passed, $n_fail failed (counted by the runner, not by the file's dead counters)"
|
||||
if [ "$n_fail" -gt 0 ]; then
|
||||
echo " failing assertions:"; grep '^ FAIL: ' "$out" | sed 's/^/ /'
|
||||
return 1
|
||||
fi
|
||||
if [ "$n_pass" -eq 0 ]; then
|
||||
echo " WARNING: no assertions ran — treating as FAILURE (a test that asserts nothing is not a passing test)"
|
||||
return 1
|
||||
fi
|
||||
[ $rc -eq 0 ] || { echo " (test binary exited rc=$rc)"; return 1; }
|
||||
return 0
|
||||
}
|
||||
|
||||
rc_all=0
|
||||
if [ "${1:-}" = "--all" ]; then
|
||||
for t in tests/test_*.el; do run_one "$t" || rc_all=1; done
|
||||
else
|
||||
[ $# -ge 1 ] || { echo "usage: tests/run-el-test.sh <tests/test_x.el> | --all" >&2; exit 2; }
|
||||
for t in "$@"; do run_one "$t" || rc_all=1; done
|
||||
fi
|
||||
exit $rc_all
|
||||
@@ -93,7 +93,7 @@ println("1. bridge_save — empty messages guard")
|
||||
let sid1: String = "test-session-empty-messages"
|
||||
state_set("mcp_bridge:" + sid1, "")
|
||||
|
||||
let save1_ok: Bool = bridge_save(sid1, "claude-sonnet-4-5", "sys", "[]", "", "", "call-1", "anthropic")
|
||||
let save1_ok: Bool = bridge_save(sid1, "claude-sonnet-4-5", "sys", "[]", "", "", "call-1")
|
||||
assert_false("empty messages -> bridge_save returns false", save1_ok)
|
||||
|
||||
let saved1: String = state_get("mcp_bridge:" + sid1)
|
||||
@@ -107,7 +107,7 @@ println("2. bridge_save — empty tools_json guard")
|
||||
let sid2: String = "test-session-empty-tools"
|
||||
state_set("mcp_bridge:" + sid2, "")
|
||||
|
||||
let save2_ok: Bool = bridge_save(sid2, "claude-sonnet-4-5", "sys", "", "[{\"role\":\"user\",\"content\":\"hi\"}]", "", "call-2", "anthropic")
|
||||
let save2_ok: Bool = bridge_save(sid2, "claude-sonnet-4-5", "sys", "", "[{\"role\":\"user\",\"content\":\"hi\"}]", "", "call-2")
|
||||
assert_false("empty tools_json -> bridge_save returns false", save2_ok)
|
||||
|
||||
let saved2: String = state_get("mcp_bridge:" + sid2)
|
||||
@@ -126,7 +126,7 @@ state_set("mcp_bridge:" + sid3, "")
|
||||
|
||||
let msgs3: String = "[{\"role\":\"user\",\"content\":\"hello\"}]"
|
||||
let tools3: String = "[{\"name\":\"read_file\"}]"
|
||||
let save3_ok: Bool = bridge_save(sid3, "claude-sonnet-4-5", "You are a helper.", tools3, msgs3, "read_file", "toolu_abc", "anthropic")
|
||||
let save3_ok: Bool = bridge_save(sid3, "claude-sonnet-4-5", "You are a helper.", tools3, msgs3, "read_file", "toolu_abc")
|
||||
assert_true("valid args -> bridge_save returns true", save3_ok)
|
||||
|
||||
let blob3: String = state_get("mcp_bridge:" + sid3)
|
||||
@@ -243,7 +243,7 @@ state_set("mcp_bridge:" + sid8, "")
|
||||
let special_id: String = "toolu_test\"quoted\""
|
||||
let msgs8: String = "[{\"role\":\"user\",\"content\":\"hi\"}]"
|
||||
let tools8: String = "[{\"name\":\"read_file\"}]"
|
||||
let save8_ok: Bool = bridge_save(sid8, "claude-sonnet-4-5", "sys", tools8, msgs8, "", special_id, "anthropic")
|
||||
let save8_ok: Bool = bridge_save(sid8, "claude-sonnet-4-5", "sys", tools8, msgs8, "", special_id)
|
||||
assert_true("special chars in tool_use_id -> bridge_save returns true", save8_ok)
|
||||
|
||||
let blob8: String = state_get("mcp_bridge:" + sid8)
|
||||
@@ -251,111 +251,6 @@ let blob8: String = state_get("mcp_bridge:" + sid8)
|
||||
let retrieved_id: String = json_get(blob8, "tool_use_id")
|
||||
assert_eq("tool_use_id with quotes round-trips via json_safe", retrieved_id, special_id)
|
||||
|
||||
// ── Section 9: the "wire" field (OpenAI-tools port, 2026-08-06) ───────────────
|
||||
//
|
||||
// A suspended turn must resume on the SAME wire format it suspended on: an OpenAI-lane
|
||||
// bridge answered with an Anthropic-shaped tool_result (or vice versa) is a dead run.
|
||||
// bridge_save therefore stamps the blob with "wire", and agentic_resume branches on it.
|
||||
//
|
||||
// §9c is the important one. json_get is a first-substring-match scanner, so any key that
|
||||
// appears inside the UNESCAPED conversation embedded in messages_raw can be matched
|
||||
// instead of the blob's own field — that exact class of bug produced the round-9 resume
|
||||
// failure (json_get(blob,"tool_use_id") matching a web_search_tool_result's id inside the
|
||||
// replayed conversation). "wire" is written as a json_safe'd SCALAR ahead of both raw
|
||||
// fields precisely so a decoy in model-controlled bytes can never win. This test plants
|
||||
// that decoy on purpose. If someone later moves the field after messages_raw, this fails.
|
||||
|
||||
println("")
|
||||
println("9. bridge_save — wire tagging and its field-order guarantee")
|
||||
|
||||
// 9a. an OpenAI-lane suspension round-trips as "openai"
|
||||
let sid9: String = "test-session-wire-openai"
|
||||
state_set("mcp_bridge:" + sid9, "")
|
||||
let msgs9: String = "[{\"role\":\"user\",\"content\":\"hi\"}]"
|
||||
let tools9: String = "[{\"name\":\"read_file\"}]"
|
||||
let save9_ok: Bool = bridge_save(sid9, "llama-3.3-70b-versatile", "sys", tools9, msgs9, "", "call_abc", "openai")
|
||||
assert_true("openai wire -> bridge_save returns true", save9_ok)
|
||||
let blob9: String = state_get("mcp_bridge:" + sid9)
|
||||
assert_eq("wire round-trips as openai", json_get(blob9, "wire"), "openai")
|
||||
|
||||
// 9b. an Anthropic-lane suspension round-trips as "anthropic"
|
||||
let sid9b: String = "test-session-wire-anthropic"
|
||||
state_set("mcp_bridge:" + sid9b, "")
|
||||
let save9b_ok: Bool = bridge_save(sid9b, "claude-sonnet-4-5", "sys", tools9, msgs9, "", "toolu_abc", "anthropic")
|
||||
assert_true("anthropic wire -> bridge_save returns true", save9b_ok)
|
||||
let blob9b: String = state_get("mcp_bridge:" + sid9b)
|
||||
assert_eq("wire round-trips as anthropic", json_get(blob9b, "wire"), "anthropic")
|
||||
|
||||
// 9c. FIELD-ORDER GUARD: a decoy "wire" inside the conversation must NOT be matched.
|
||||
let sid9c: String = "test-session-wire-decoy"
|
||||
state_set("mcp_bridge:" + sid9c, "")
|
||||
let msgs9c: String = "[{\"role\":\"user\",\"content\":\"please save this literal text: \\\"wire\\\":\\\"anthropic\\\" end\"}]"
|
||||
let save9c_ok: Bool = bridge_save(sid9c, "llama-3.3-70b-versatile", "sys", tools9, msgs9c, "", "call_decoy", "openai")
|
||||
assert_true("decoy conversation -> bridge_save returns true", save9c_ok)
|
||||
let blob9c: String = state_get("mcp_bridge:" + sid9c)
|
||||
assert_eq("blob's own wire wins over a decoy planted in messages_raw", json_get(blob9c, "wire"), "openai")
|
||||
|
||||
// 9d. LEGACY blob (written before the port) has no wire field: json_get yields "",
|
||||
// which agentic_resume treats as the Anthropic path — old suspensions still resume.
|
||||
let sid9d: String = "test-session-wire-legacy"
|
||||
let legacy_blob: String = "{\"model\":\"claude-sonnet-4-5\",\"safe_sys\":\"sys\",\"tools_log\":\"\""
|
||||
+ ",\"tool_use_id\":\"toolu_legacy\",\"tools_raw\":[{\"name\":\"read_file\"}]"
|
||||
+ ",\"messages_raw\":[{\"role\":\"user\",\"content\":\"hi\"}]}"
|
||||
state_set("mcp_bridge:" + sid9d, legacy_blob)
|
||||
let blob9d: String = state_get("mcp_bridge:" + sid9d)
|
||||
assert_eq("legacy blob has no wire field -> empty (resumes as anthropic)", json_get(blob9d, "wire"), "")
|
||||
assert_eq("legacy blob still reads its tool_use_id", json_get(blob9d, "tool_use_id"), "toolu_legacy")
|
||||
|
||||
// 9e. THE HARDER DECOY: a LEGACY blob (no wire field of its own) that carries the bytes
|
||||
// of a wire tag deeper inside, where an unbounded first-match scan would find it and
|
||||
// misroute the resume onto the wrong loop — the round-9 defect class exactly.
|
||||
//
|
||||
// WHAT IS AND IS NOT REACHABLE (measured here, not assumed — an earlier version of this
|
||||
// test asserted the wrong thing and was corrected by running it):
|
||||
// * NOT reachable from ordinary conversation TEXT. Any quote a user or model writes is
|
||||
// backslash-escaped when it is serialized into the blob, so prose containing
|
||||
// "wire":"openai" is stored as \"wire\":\"openai\" and does not match a scan for the
|
||||
// unescaped key. §9f pins that.
|
||||
// * REACHABLE from STRUCTURAL keys, which are embedded raw. Conversation and tool
|
||||
// objects keep real quotes — that is precisely how round 9's scan found a
|
||||
// web_search_tool_result's tool_use_id. A connector-supplied tool schema or a future
|
||||
// message field literally named "wire" would be found the same way.
|
||||
// The bound removes the whole class rather than reasoning about which keys exist today.
|
||||
let sid9e: String = "test-session-wire-legacy-decoy"
|
||||
let decoy_blob: String = "{\"model\":\"claude-sonnet-4-5\",\"safe_sys\":\"sys\",\"tools_log\":\"\""
|
||||
+ ",\"tool_use_id\":\"toolu_legacy\""
|
||||
+ ",\"tools_raw\":[{\"name\":\"read_file\",\"wire\":\"openai\"}]"
|
||||
+ ",\"messages_raw\":[{\"role\":\"user\",\"content\":\"hi\"}]}"
|
||||
state_set("mcp_bridge:" + sid9e, decoy_blob)
|
||||
let blob9e: String = state_get("mcp_bridge:" + sid9e)
|
||||
|
||||
// Unbounded read (what NOT to do) — proves the hazard this guard exists for is real.
|
||||
assert_eq("unbounded scan DOES find a structural decoy (why the bound is needed)", json_get(blob9e, "wire"), "openai")
|
||||
|
||||
// Bounded read — the same computation agentic_resume performs.
|
||||
let d_traw: Int = str_index_of(blob9e, ",\"tools_raw\":")
|
||||
let d_tjson: Int = str_index_of(blob9e, ",\"tools_json\":")
|
||||
let d_mraw: Int = str_index_of(blob9e, ",\"messages_raw\":")
|
||||
let d_msgs: Int = str_index_of(blob9e, ",\"messages\":")
|
||||
let dcut1: Int = if d_traw > 0 { d_traw } else { str_len(blob9e) }
|
||||
let dcut2: Int = if d_tjson > 0 && d_tjson < dcut1 { d_tjson } else { dcut1 }
|
||||
let dcut3: Int = if d_mraw > 0 && d_mraw < dcut2 { d_mraw } else { dcut2 }
|
||||
let dcut: Int = if d_msgs > 0 && d_msgs < dcut3 { d_msgs } else { dcut3 }
|
||||
let head9e: String = str_slice(blob9e, 0, dcut)
|
||||
assert_eq("bounded scan ignores the decoy -> legacy blob resumes as anthropic", json_get(head9e, "wire"), "")
|
||||
assert_not_contains("scalar head excludes the bulk fields entirely", head9e, "read_file")
|
||||
|
||||
// 9f. Escaping bounds the severity: prose CANNOT inject a scalar-looking key, because
|
||||
// its quotes are escaped on the way in. Documented as a measured fact, so nobody has to
|
||||
// re-derive it the next time this question comes up.
|
||||
let sid9f: String = "test-session-wire-prose"
|
||||
let prose_blob: String = "{\"model\":\"claude-sonnet-4-5\",\"safe_sys\":\"sys\",\"tools_log\":\"\""
|
||||
+ ",\"tool_use_id\":\"toolu_legacy\",\"tools_raw\":[{\"name\":\"read_file\"}]"
|
||||
+ ",\"messages_raw\":[{\"role\":\"user\",\"content\":\"remember this: \\\"wire\\\":\\\"openai\\\"\"}]}"
|
||||
state_set("mcp_bridge:" + sid9f, prose_blob)
|
||||
let blob9f: String = state_get("mcp_bridge:" + sid9f)
|
||||
assert_eq("escaped prose cannot spoof the key even unbounded (severity bound)", json_get(blob9f, "wire"), "")
|
||||
|
||||
// ── Summary ────────────────────────────────────────────────────────────────────
|
||||
|
||||
println("")
|
||||
|
||||
@@ -1,92 +0,0 @@
|
||||
// tests/test_utf8_slice.el
|
||||
//
|
||||
// Guards utf8_safe_slice(), the fix for a live defect found 2026-08-06:
|
||||
//
|
||||
// The session preload cuts recalled memory content at a fixed length
|
||||
// (chat.el: `if str_len(acc) > 350 { str_slice(acc, 0, 350) }` and
|
||||
// session_preload_bullets' identical per-bullet cut). str_slice and str_len count
|
||||
// BYTES, so any cut landing inside a multi-byte UTF-8 character leaves a dangling
|
||||
// lead byte in the system prompt — and the whole request body is then invalid UTF-8.
|
||||
// Providers reject it outright, so the user sees "AI unavailable" with no clue why,
|
||||
// on both wire formats. Caught by an OpenAI-lane gate whose stub decodes strictly;
|
||||
// reproduced from a real memory whose content contained box-drawing rules (E2 94 80).
|
||||
//
|
||||
// Trigger is ordinary content: an em dash, a curly quote, an accented name, a table
|
||||
// border, an emoji — anything non-ASCII sitting on the cut boundary. It gets MORE
|
||||
// likely as a user's memory grows, which is the opposite of what should happen.
|
||||
//
|
||||
// §1 also pins the semantics this fix depends on: that str_char_code returns the
|
||||
// BYTE value at a byte index (not a decoded code point). If a future runtime changes
|
||||
// that, these assertions fail loudly instead of the truncation silently rotting.
|
||||
|
||||
import "../chat.el"
|
||||
|
||||
let pass_count: Int = 0
|
||||
let fail_count: Int = 0
|
||||
|
||||
fn assert_eq(label: String, got: String, expected: String) -> Void {
|
||||
if str_eq(got, expected) {
|
||||
let pass_count = pass_count + 1
|
||||
println(" PASS: " + label)
|
||||
} else {
|
||||
let fail_count = fail_count + 1
|
||||
println(" FAIL: " + label)
|
||||
println(" got: " + got)
|
||||
println(" expected: " + expected)
|
||||
}
|
||||
}
|
||||
|
||||
fn assert_eq_int(label: String, got: Int, expected: Int) -> Void {
|
||||
assert_eq(label, int_to_str(got), int_to_str(expected))
|
||||
}
|
||||
|
||||
println("")
|
||||
println("1. runtime semantics this fix relies on")
|
||||
|
||||
// "─" is U+2500 = E2 94 80 (three bytes). If str_len counts bytes, len("─") is 3.
|
||||
let dash: String = "─"
|
||||
assert_eq_int("str_len counts BYTES (one box-drawing char = 3)", str_len(dash), 3)
|
||||
assert_eq_int("str_char_code returns the BYTE value (lead byte of U+2500 = 0xE2 = 226)", str_char_code(dash, 0), 226)
|
||||
assert_eq_int("str_char_code second byte = 0x94 = 148", str_char_code(dash, 1), 148)
|
||||
assert_eq_int("str_char_code third byte = 0x80 = 128", str_char_code(dash, 2), 128)
|
||||
|
||||
println("")
|
||||
println("2. utf8_safe_slice — never leaves a partial character")
|
||||
|
||||
// Pure ASCII: behaves exactly like str_slice.
|
||||
assert_eq("ascii under the limit is untouched", utf8_safe_slice("hello", 10), "hello")
|
||||
assert_eq("ascii over the limit cuts exactly", utf8_safe_slice("hello world", 5), "hello")
|
||||
|
||||
// A cut landing INSIDE a 3-byte character must drop that character entirely.
|
||||
// "ab─cd": bytes a b E2 94 80 c d. Cutting at 3 or 4 lands mid-dash.
|
||||
let mixed: String = "ab─cd"
|
||||
assert_eq_int("fixture is 7 bytes (2 ascii + 3 + 2 ascii)", str_len(mixed), 7)
|
||||
assert_eq("cut inside the char (n=3) drops the partial char", utf8_safe_slice(mixed, 3), "ab")
|
||||
assert_eq("cut inside the char (n=4) drops the partial char", utf8_safe_slice(mixed, 4), "ab")
|
||||
// A cut landing exactly AFTER a complete character keeps it.
|
||||
assert_eq("cut on the char boundary (n=5) keeps the whole char", utf8_safe_slice(mixed, 5), "ab─")
|
||||
|
||||
// 2-byte character (é = C3 A9) and 4-byte character (😀 = F0 9F 98 80).
|
||||
let acc: String = "xé"
|
||||
assert_eq("cut inside a 2-byte char drops it", utf8_safe_slice(acc, 2), "x")
|
||||
assert_eq("cut after a 2-byte char keeps it", utf8_safe_slice(acc, 3), "xé")
|
||||
let emo: String = "x😀"
|
||||
assert_eq("cut inside a 4-byte char drops it (n=3)", utf8_safe_slice(emo, 3), "x")
|
||||
assert_eq("cut inside a 4-byte char drops it (n=4)", utf8_safe_slice(emo, 4), "x")
|
||||
assert_eq("cut after a 4-byte char keeps it", utf8_safe_slice(emo, 5), "x😀")
|
||||
|
||||
println("")
|
||||
println("3. the real-world shape that produced the bug")
|
||||
|
||||
// A run of box-drawing rules, cut mid-character — the exact captured failure.
|
||||
let rules: String = "──────"
|
||||
assert_eq_int("six box rules = 18 bytes", str_len(rules), 18)
|
||||
// n=16 lands one byte into the sixth character.
|
||||
let cut16: String = utf8_safe_slice(rules, 16)
|
||||
assert_eq_int("cut at 16 backs off to a clean 15-byte boundary", str_len(cut16), 15)
|
||||
// Every byte of the result must belong to a complete character: the last byte of a
|
||||
// well-formed run of these is always 0x80, and 15 is divisible by 3.
|
||||
assert_eq_int("result ends on a complete char (last byte 0x80)", str_char_code(cut16, 14), 128)
|
||||
|
||||
println("")
|
||||
println("test_utf8_slice.el: " + int_to_str(pass_count) + " passed, " + int_to_str(fail_count) + " failed")
|
||||
@@ -0,0 +1,147 @@
|
||||
# Retrieval eval harness
|
||||
|
||||
Measures Neuron's memory retrieval so a change can be shown to help before it is
|
||||
believed to help. Nothing else on the memory roadmap should ship without a run
|
||||
through this.
|
||||
|
||||
```
|
||||
tools/retrieval-eval/run_comparison.sh --baseline main --candidate <branch>
|
||||
```
|
||||
|
||||
That builds a soul from each ref, boots each in isolation on a fixed corpus,
|
||||
runs the gold set three times per ref, and prints a table plus a verdict that
|
||||
refuses to call a difference real if it is inside the noise band.
|
||||
|
||||
## What was reused
|
||||
|
||||
This is not a new idea, it is the missing third of an existing one.
|
||||
|
||||
| Prior work | What it gave | What was missing |
|
||||
|---|---|---|
|
||||
| `docs/research/graphrag_eval/` (`collect.py`, `score.py`, 2026-06-08) | The three-retriever comparison that produced the numbers everyone quotes: substring 1.7% P@5, graph 21.7%, BM25 55%. Per-query relevant-id scoring, fixed-denominator precision@5, unique-relevant analysis. | 13 hand-written queries, judged by an LLM after the fact; measured the *live* soul on the *live* engram. |
|
||||
| `docs/research-archive/p0-prototypes/eval_pinned_40q_20260715.py` | The pinned-query discipline: ground truth committed as regexes so every run judges alike, plus a `--check` winnability gate. 40 queries in 5 bands including a deliberate paraphrase-hard band. | Scored offline replicas of substring/BM25 — it never ran the real retrieval path. |
|
||||
| `docs/research-archive/p0-prototypes/stage0_eval_20260714.py` | The `hit@5` metric and the substring/BM25 reference implementations. | Same: offline only. |
|
||||
| `scripts/verify-soul-contract.sh` | The isolation recipe, verbatim: throwaway port, throwaway `HOME`, `SOUL_ENGRAM_PATH`, and the non-obvious `SOUL_ISE_URL` pin that stops an "isolated" soul silently syncing the operator's live brain. | It is a contract gate, not a measurement. |
|
||||
| `_engine-liveness-91/gen-soul-amalgam.sh` + `.gitea/workflows/ci.yaml` | The build recipe (`elc --target=c` with every `.elh` on the import chain removed) and CI's exact compile flags. | — |
|
||||
|
||||
**Reused directly:** the isolation recipe, the build recipe, fixed-denominator
|
||||
precision@5, the pinned-ground-truth and winnability ideas.
|
||||
**New here:** ids rather than regexes as ground truth, an associative category
|
||||
derived from real graph edges, a superseded/contradicted category scored on
|
||||
ranking, a machine-checked zero-lexical-overlap guarantee on paraphrases,
|
||||
paired significance testing, and — the point — measurement against the **real
|
||||
compiled soul** rather than an offline replica of one leg of it.
|
||||
|
||||
## Design fit
|
||||
|
||||
The thing under measurement is Will's designed retrieval: spreading activation
|
||||
over the weighted directed graph, four-factor multiplicative scoring (parent
|
||||
strength x edge weight x target salience x query/target cosine). A Python
|
||||
re-implementation would measure my reading of the design. So the harness
|
||||
compiles the actual `soul.el` amalgam and asks it over HTTP on
|
||||
`/api/neuron/recall`, exactly as the MCP wrapper and the app do.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Does |
|
||||
|---|---|
|
||||
| `build_gold_set.py` | Derives and **validates** the gold set from the corpus. `--check` re-validates and exits non-zero if a query became unwinnable or a paraphrase leaked a word. |
|
||||
| `gold_set.json` | 38 queries. Every one carries a `derivation` string. |
|
||||
| `run_eval.py` | Boots one soul in isolation, runs the gold set, writes metrics. Kills and **confirms dead** its child; records the confirmation in the results file. |
|
||||
| `compare.py` | Paired diff of two result files with McNemar's exact test and a stated noise floor. |
|
||||
| `build-soul.sh` | Compiles a soul binary from a plain source tree. |
|
||||
| `run_comparison.sh` | All of the above, end to end, from two git refs. |
|
||||
|
||||
## The gold set — 38 queries
|
||||
|
||||
Built from the real corpus (`snapshot-pre-repair-20260806.json`, 78,768 nodes /
|
||||
14,214 edges) so it reflects one person's accumulating memory, not document QA.
|
||||
|
||||
| Category | n | Expected answer derived by |
|
||||
|---|---|---|
|
||||
| `exact_rare` | 6 | **Mined.** Tokens with document frequency 1 across all 78,768 nodes, whose single containing node is a 300–6000 char Memory/Knowledge/Belief. That node is the only possible answer. Re-verified every build. |
|
||||
| `phrase` | 7 | **Mined.** Case-insensitive verbatim scan; the matching set *is* the answer key. Phrases matching >25 nodes are rejected as too diffuse. |
|
||||
| `paraphrase` | 13 | **Hand-selected, machine-checked.** Target locked by id; the build then proves that **zero** content words of the query appear anywhere in the target's label, content, or tags. A leak fails the build — the category cannot quietly decay into lexical matching. |
|
||||
| `associative` | 6 | **Derived from edges.** Query built from one value node's distinctive vocabulary; expected answers are its siblings on the `Self - Values (grounded)` hub. Siblings sharing any query word are dropped, so the only route from query to answer is seed -> hub -> sibling. |
|
||||
| `nonsense` | 3 | **Control.** Verified that no token occurs anywhere in the corpus. Correct behaviour is to return nothing. |
|
||||
| `superseded` | 3 | **Derived.** Correction/stale pairs located by regex scan, kept only when both sides resolve to different surviving nodes. Scored on **ranking**: the correction must be returned *and* rank above the stale node. |
|
||||
|
||||
## Metrics
|
||||
|
||||
`hit@5`, `recall@5`, `recall@10`, `precision@5` (fixed denominator 5, so an
|
||||
empty result is punished like a page of junk), `MRR@10`, and wall-clock latency
|
||||
per query (p50/p95/max). Output is a table plus a machine-readable JSON per run
|
||||
so runs can be diffed.
|
||||
|
||||
## Honesty about noise
|
||||
|
||||
- **Minimum detectable swing on this 38-query set: 6 queries.** If every query
|
||||
that changes changes the same way, `p = 2 x 0.5^n`, which first drops under
|
||||
0.05 at n=6. Any net change smaller than that is inside the noise band and
|
||||
`compare.py` says so in those words.
|
||||
- **Run-to-run drift is measured, not assumed.** Activation is a stateful read
|
||||
by design (traversal reinforces what it touches), so identical inputs need not
|
||||
give identical outputs. Observed: `main` 0 queries of drift across 3 runs
|
||||
(fully deterministic); the activation branch 1 query.
|
||||
- The noise floor used for the verdict is `max(6, observed_drift + 1)`.
|
||||
- **This gold set is underpowered for small effects.** A genuine 3-query
|
||||
improvement would not clear the bar. Growing the set is the fix; until then, a
|
||||
small positive delta means "not shown", not "no effect".
|
||||
|
||||
## First result: `main` vs `feat/recall-through-activation`
|
||||
|
||||
Corpus and gold set identical, three runs each, fresh corpus copy per run.
|
||||
|
||||
| | main | recall-through-activation | delta |
|
||||
|---|---|---|---|
|
||||
| hit@5 | 34.3% | 22.9% | **-11.4pp** |
|
||||
| recall@5 | 26.9% | 19.1% | -7.9pp |
|
||||
| recall@10 | 33.3% | 24.3% | -9.1pp |
|
||||
| precision@5 | 12.0% | 7.4% | -4.6pp |
|
||||
| MRR@10 | 0.294 | 0.242 | -0.053 |
|
||||
| latency p50 | 1140 ms | 3209 ms | **2.81x** |
|
||||
| latency p95 | 1584 ms | 4852 ms | 3.06x |
|
||||
| nonsense clean | 2/3 | 2/3 | — |
|
||||
| superseded outranks | 1/3 | 0/3 | -1 |
|
||||
|
||||
By category (hit@5):
|
||||
|
||||
| category | main | activation |
|
||||
|---|---|---|
|
||||
| exact_rare | 100% | 100% |
|
||||
| phrase | 85.7% | **28.6%** |
|
||||
| paraphrase | 0% | 0% |
|
||||
| associative | 0% | 0% |
|
||||
| superseded | 0% | 0% |
|
||||
|
||||
**Verdict: directionally worse, one query short of significant.** 5 discordant
|
||||
pairs, all 5 against the candidate, 0 for it. McNemar exact p = 0.0625 — under
|
||||
the stated rule that is *inside* the noise band, so the harness reports "no
|
||||
measurable difference" on accuracy and the honest summary is "5 for 5 the wrong
|
||||
way, needs a 6th or a larger gold set to call".
|
||||
|
||||
Latency is a different story: 2.8x at p50 is deterministic and far outside any
|
||||
noise band. That regression is real.
|
||||
|
||||
The result the branch was written for did not appear. Its stated purpose was to
|
||||
recover sibling nodes one hub-hop away — the `associative` category — and that
|
||||
category is **0/6 on both builds**. Probing directly: for the query
|
||||
`Marines hernia sepsis medical ward`, the activation build returns the lexical
|
||||
seed node itself at rank 8, and none of its 12 hub siblings anywhere in the top
|
||||
10. The traversal is running; it is not reaching siblings.
|
||||
|
||||
Two corpus facts likely explain it, and both are measurable rather than
|
||||
speculative:
|
||||
|
||||
1. **The graph is nearly edgeless.** Only 4,060 of 78,768 nodes (5.2%) carry any
|
||||
edge at all — 14,214 edges total, 0.18 per node. Spreading activation over a
|
||||
graph with no edges is an expensive way to do lexical matching, which is
|
||||
roughly what the numbers show.
|
||||
2. **No embeddings.** No node in this snapshot has an embedding field, so the
|
||||
fourth factor of the four-factor product — query/target cosine similarity —
|
||||
has nothing to compute from, and the semantic seeding pass is inert.
|
||||
|
||||
That is the harness earning its keep on its first job: the change would have
|
||||
felt like progress (it is the designed mechanism, and it does run) and measures
|
||||
as a regression on phrase queries plus a 2.8x latency cost, with its intended
|
||||
benefit unrealised because the corpus lacks the structure it needs.
|
||||
Executable
+44
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
# build-soul.sh — compile a soul binary from a plain source tree (no git needed).
|
||||
#
|
||||
# Reuses the amalgam recipe worked out in gen-soul-amalgam.sh (round 9.1) and the
|
||||
# compile flags from .gitea/workflows/ci.yaml, so the binary under test is the
|
||||
# same translation unit CI ships — not a re-implementation.
|
||||
#
|
||||
# elc --target=c emits only an extern prototype for any module that has a .elh
|
||||
# header beside it, and inlines the module's bodies when it does not. So the
|
||||
# amalgam is produced in a scratch copy with every .elh on the import chain
|
||||
# deleted.
|
||||
#
|
||||
# usage: build-soul.sh <src-tree-with-*.el> <out-binary>
|
||||
set -euo pipefail
|
||||
SRC="${1:?usage: build-soul.sh <src-tree> <out-binary>}"
|
||||
OUT="${2:?out-binary}"
|
||||
ELC="${ELC:-$HOME/neuron-dev-stack/src/el/lang/dist/platform/elc}"
|
||||
EL_REPO="${EL_REPO:-$HOME/Development/neuron-technologies/el}"
|
||||
RTDIR="${RTDIR:-$SRC/vendor/el-runtime/v1.0.0-20260501}"
|
||||
SSL="${SSL_PREFIX:-/opt/homebrew/opt/openssl@3}"
|
||||
|
||||
[ -x "$ELC" ] || { echo "no elc at $ELC" >&2; exit 2; }
|
||||
[ -f "$RTDIR/el_runtime.c" ] || { echo "no el_runtime.c at $RTDIR" >&2; exit 2; }
|
||||
|
||||
GEN="$(mktemp -d "${TMPDIR:-/tmp}/soul-build.XXXXXX")"
|
||||
trap 'rm -rf "$GEN"' EXIT
|
||||
mkdir -p "$GEN/neuron" "$GEN/foundation/el/elp/src"
|
||||
cp "$SRC"/*.el "$GEN/neuron/"
|
||||
cp "$EL_REPO"/elp/src/*.el "$GEN/foundation/el/elp/src/"
|
||||
find "$GEN" -name '*.elh' -delete
|
||||
|
||||
( cd "$GEN/neuron" && "$ELC" --target=c soul.el ) > "$GEN/soul.c"
|
||||
BODIES=$(grep -c '^el_val_t .*) {$' "$GEN/soul.c" || true)
|
||||
echo "[build-soul] amalgam $(wc -c < "$GEN/soul.c" | tr -d ' ') bytes, ${BODIES} inlined bodies"
|
||||
[ "$BODIES" -ge 1200 ] || { echo "[build-soul] FAIL: only $BODIES bodies — an import was not inlined"; exit 1; }
|
||||
|
||||
cc -O2 -DHAVE_CURL -rdynamic \
|
||||
-I"$RTDIR" -I"$SSL/include" -L"$SSL/lib" \
|
||||
"$GEN/soul.c" "$RTDIR/el_runtime.c" \
|
||||
-lssl -lcrypto -lcurl -lpthread -lm \
|
||||
-o "$OUT" 2> "$GEN/cc.log" || { echo "[build-soul] FAIL compile"; tail -40 "$GEN/cc.log"; exit 1; }
|
||||
if grep -qE 'implicit.*(engram_|el_)' "$GEN/cc.log"; then
|
||||
echo "[build-soul] FAIL: implicit declarations of runtime symbols"; grep -E 'implicit' "$GEN/cc.log" | head; exit 1; fi
|
||||
echo "[build-soul] OK -> $OUT ($(wc -c < "$OUT" | tr -d ' ') bytes)"
|
||||
Executable
+506
@@ -0,0 +1,506 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
build_gold_set.py — derive the retrieval gold set FROM the corpus, and validate it.
|
||||
|
||||
WHY THIS FILE EXISTS AS CODE AND NOT AS A HAND-WRITTEN JSON
|
||||
A gold set nobody can audit is vibes with extra steps. Every expected answer
|
||||
here is either (a) mined from the corpus by a rule this script re-runs, or
|
||||
(b) hand-selected with a stated criterion that this script then CHECKS
|
||||
against the corpus. Both leave a `derivation` string on every query, and the
|
||||
checks are re-run on demand so the set cannot silently rot as the corpus
|
||||
changes.
|
||||
|
||||
Lineage: this extends the pinned-query approach from
|
||||
docs/research-archive/p0-prototypes/eval_pinned_40q_20260715.py (pinned
|
||||
ground-truth patterns + a --check "winnability" gate) and the per-query
|
||||
relevant-id scoring from docs/research/graphrag_eval/score.py. What is new:
|
||||
ids as ground truth rather than regexes alone, an ASSOCIATIVE category
|
||||
derived from real graph edges, a superseded/contradicted category, and a
|
||||
machine-checked no-lexical-overlap guarantee on the paraphrase category.
|
||||
|
||||
THE SIX CATEGORIES, AND WHAT EACH ONE IS FOR
|
||||
exact_rare a single rare word. Substring matching already wins these.
|
||||
They are a REGRESSION GUARD: any change that loses them is
|
||||
disqualified regardless of what else it gains.
|
||||
phrase a multi-word string that exists verbatim in the corpus.
|
||||
Guards multi-token queries, which the old substring matcher
|
||||
handled by returning nothing.
|
||||
paraphrase same meaning, ZERO shared content words with the target node.
|
||||
THE CATEGORY THAT MATTERS. Mechanically unreachable by string
|
||||
matching; reachable only by semantics or by association.
|
||||
associative the answer is one hub-hop from an obvious starting point and
|
||||
shares no words with the query. This is the case the graph is
|
||||
supposed to buy: query one value, get its siblings.
|
||||
nonsense must return nothing. Guards against a retriever that "improves"
|
||||
recall by returning the whole graph.
|
||||
superseded a fact that was later corrected. The correction must OUTRANK
|
||||
the stale version — ranking, not mere presence.
|
||||
|
||||
usage:
|
||||
python3 build_gold_set.py <snapshot.json> [--out gold_set.json] [--check]
|
||||
--check re-validates an existing gold_set.json against the corpus and exits
|
||||
non-zero if any query became unwinnable or any paraphrase leaked a word.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
DEFAULT_OUT = os.path.join(HERE, "gold_set.json")
|
||||
|
||||
TOKEN = re.compile(r"[a-z0-9][a-z0-9\-']*")
|
||||
|
||||
# Stopwords are deliberately generous. A paraphrase query is only interesting if
|
||||
# its CONTENT words are absent from the target; "the", "is", "what" appearing in
|
||||
# both proves nothing. Being generous here makes the overlap test STRICTER on
|
||||
# the words that carry meaning, which is the conservative direction.
|
||||
STOP = set("""
|
||||
a about above after again against all also am an and any are aren't as at be because been
|
||||
before being below between both but by can can't cannot could couldn't did didn't do does
|
||||
doesn't doing don't down during each few for from further had hadn't has hasn't have haven't
|
||||
having he her here hers herself him himself his how i if in into is isn't it its itself just
|
||||
me more most my myself no nor not of off on once only or other others ought our ours ourselves
|
||||
out over own same shan't she should shouldn't so some such than that the their theirs them
|
||||
themselves then there these they this those through to too under until up very was wasn't we
|
||||
were weren't what when where which while who whom why will with won't would wouldn't you your
|
||||
yours yourself yourselves get gets got make makes made take takes use uses used way ways thing
|
||||
things does doing done keep keeps kept go goes going come comes came one two something anything
|
||||
""".split())
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# corpus helpers
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
def load_corpus(path):
|
||||
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||
data = json.load(fh)
|
||||
nodes = [n for n in data.get("nodes", []) if isinstance(n, dict) and n.get("id")]
|
||||
edges = [e for e in data.get("edges", []) if isinstance(e, dict)]
|
||||
return nodes, edges
|
||||
|
||||
|
||||
def doctext(n):
|
||||
return " ".join([str(n.get("label") or ""), str(n.get("content") or ""), str(n.get("tags") or "")])
|
||||
|
||||
|
||||
def content_tokens(s):
|
||||
return {t for t in TOKEN.findall(s.lower()) if t not in STOP and len(t) > 2}
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# hand-authored queries. Every entry states HOW its expected answer was chosen.
|
||||
# The `check` field names the validation this script runs against the corpus.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# EXACT_RARE — mined, not chosen. The rule (re-run by mine_exact_rare below):
|
||||
# tokens whose document frequency across the whole corpus is 1, whose single
|
||||
# containing node is a Memory/Knowledge/Belief with 300-6000 chars of content
|
||||
# (so the answer is a real memory, not a 117KB whitepaper that contains every
|
||||
# word in English), and whose token is plain lowercase alphabetic. The expected
|
||||
# answer is that one node — it is the only node that can possibly be correct.
|
||||
EXACT_RARE_SEEDS = [
|
||||
"unjailbreakable",
|
||||
"engram-migrate",
|
||||
"cartabandonedevent",
|
||||
"pre-apprenticeship",
|
||||
"inferencenodemanager",
|
||||
"clear-eyed",
|
||||
]
|
||||
|
||||
# PHRASE — chosen by reading the corpus for phrases that (a) occur verbatim,
|
||||
# (b) occur in a small enough set of nodes that "relevant" is well defined.
|
||||
# Expected answers are computed here as EVERY node whose text contains the
|
||||
# phrase case-insensitively — so the answer set is a fact about the corpus, not
|
||||
# an opinion. Queries whose phrase matches more than PHRASE_MAX nodes are
|
||||
# rejected by validation as too diffuse to score.
|
||||
PHRASE_MAX = 25
|
||||
PHRASE_SEEDS = [
|
||||
("patterns not returns",
|
||||
"a verbatim correction Will issued; expected = every node containing the phrase"),
|
||||
("thirty moves",
|
||||
"the canonical biographical phrase; expected = every node containing it"),
|
||||
("Grandma Lucas",
|
||||
"a named person appearing verbatim in the biography/value nodes"),
|
||||
("Directed Harmonic",
|
||||
"the canonical DHARMA expansion, confirmed by Will April 24 2026"),
|
||||
("Sarah Bishop",
|
||||
"a named person; rare enough that the answer set is unambiguous"),
|
||||
("Directed Autonomous Runtime Modification",
|
||||
"the DARMA expansion, quoted verbatim in the backlog item and its correction"),
|
||||
("zero-knowledge encrypted backup",
|
||||
"the paid-tier feature name as written in the roadmap nodes"),
|
||||
]
|
||||
|
||||
# PARAPHRASE — hand-authored. THE SELECTION CRITERION, stated once and applied
|
||||
# to all nine: pick a node whose SUBJECT is unmistakable to a reader, then write
|
||||
# the query a person would actually type when they remember the subject but not
|
||||
# the words. The target is then LOCKED by id, and this script enforces the hard
|
||||
# property that makes the category meaningful: not one content word of the query
|
||||
# appears anywhere in the target node's label, content, or tags. If a word
|
||||
# leaks, validation fails and the query must be rewritten — the set cannot
|
||||
# quietly degrade into a lexical query wearing a paraphrase costume.
|
||||
PARAPHRASE_SEEDS = [
|
||||
("kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"the elderly relative who passed while he stayed away",
|
||||
"target: 'Value - Do the Essential Thing While You Can', whose subject is Grandma Lucas "
|
||||
"dying in Feb 2006 without Will saying goodbye. Query names the event with none of the "
|
||||
"node's own vocabulary."),
|
||||
("kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"a soldier sidelined by illness who refused to quit",
|
||||
"target: 'Value - Survival Is Not an Excuse to Stop', whose subject is enlisting in the "
|
||||
"Marines, a severe hernia, and sepsis. Query describes the episode obliquely."),
|
||||
("kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
||||
"choosing an uncomfortable fact over a pleasant fiction",
|
||||
"target: 'Value - Honesty Before Comfort'. Query states the principle in wholly "
|
||||
"different words."),
|
||||
("kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
||||
"a tight payload beats a bloated one",
|
||||
"target: 'Value - Precision Over Brute Force'. Query restates the claim with no "
|
||||
"shared vocabulary."),
|
||||
("kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"if you are able and nobody is coming the job is yours",
|
||||
"target: 'Value - Capability Is a Debt You Owe the Moment'. Query states the "
|
||||
"obligation without the node's terms."),
|
||||
("kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
||||
"learning is the wealth creditors cannot seize",
|
||||
"target: 'Value - Knowledge Survives When Nothing Else Does', whose subject is the "
|
||||
"library following Will across 30+ moves."),
|
||||
("kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
||||
"reliability proven by track record not assertion",
|
||||
"target: 'Value - Earned Trust' ('Trust is demonstrated, not declared')."),
|
||||
("kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"boundaries that enable instead of confine",
|
||||
"target: 'Value - Constraints as Freedom'. Query is a restatement of the same claim."),
|
||||
("kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
||||
"what shifts tells you where to cut a system apart",
|
||||
"target: 'Value - Change Is the Signal', the value VBD is built on."),
|
||||
("kn-f230b362-b201-4402-9833-4160c89ab3d4",
|
||||
"a mind that compounds instead of resetting each day",
|
||||
"target: 'Value - The System Must Accumulate'. Query is the accumulation claim in "
|
||||
"different vocabulary."),
|
||||
("kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"loved for the unedited self and not the polished exterior",
|
||||
"target: 'Value - Being Seen Is Rarer Than Being Known', whose subject is Sarah Bishop "
|
||||
"as the first person Will did not perform for."),
|
||||
("kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"cheerfulness you arrive at instead of assuming",
|
||||
"target: 'Value - Hope Is a Conclusion'. Query restates 'a conclusion, not a premise'."),
|
||||
("kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"a childhood offering no solid foundation to inherit",
|
||||
"target: 'Value - Structure Is Not Inherited', whose subject is thirty moves between "
|
||||
"two parents' collapses."),
|
||||
]
|
||||
|
||||
# ASSOCIATIVE — derived from real edges, not authored. The construction:
|
||||
# every value node hangs off the 'Self - Values (grounded)' hub by an `identity`
|
||||
# edge. For a chosen value node V, the query is built from V's own distinctive
|
||||
# vocabulary; the expected answers are V's SIBLINGS on that hub. A sibling
|
||||
# shares no query words with the query by construction (validated below), so the
|
||||
# only path from the query to a sibling is: lexical seed on V -> hub -> sibling.
|
||||
# That is a two-hop traversal and nothing else can produce it.
|
||||
VALUES_HUB = "kn-5b606390-a52d-4ca2-8e0e-eba141d13440"
|
||||
ASSOCIATIVE_SEEDS = [
|
||||
("kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", "Grandma Lucas stroke February 2006 goodbye window"),
|
||||
("kn-58874a74-b96f-4883-9e08-45707f4bd3ee", "Marines hernia sepsis medical ward"),
|
||||
("kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", "Sarah Bishop Dyer trailer performance"),
|
||||
("kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", "Swarm Architecture containment lateral worker"),
|
||||
("kn-e0423482-cfa5-4796-8689-8495c93b66bc", "hope won inside the narrative preface"),
|
||||
("kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", "man of the house six years old expectation"),
|
||||
]
|
||||
|
||||
# NONSENSE — must return nothing. Strings chosen to be lexically impossible:
|
||||
# validation asserts each appears in ZERO corpus nodes as a substring and that
|
||||
# none of its tokens appears anywhere either (so not even a partial seed exists).
|
||||
NONSENSE_SEEDS = [
|
||||
"zqxjvw plimforth grebulon",
|
||||
"flarnbistle quommetry",
|
||||
"xxqzzt vurblenacht throom",
|
||||
]
|
||||
|
||||
# SUPERSEDED — a fact that was corrected. Chosen by searching the corpus for
|
||||
# explicit correction language and keeping pairs where BOTH the stale statement
|
||||
# and its correction exist as separate nodes. Scored on RANKING: the correction
|
||||
# must appear, and must appear above the stale node. Ids are locked here and
|
||||
# validated to exist and to match their stated role.
|
||||
SUPERSEDED_SEEDS = [
|
||||
# (query, correct_id, stale_id, derivation)
|
||||
]
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# mining
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
def mine_exact_rare(nodes, byid, seeds):
|
||||
"""Re-derive: confirm each seed token still has df==1 and name its node."""
|
||||
tok = re.compile(r"[A-Za-z][A-Za-z0-9\-]{4,}")
|
||||
want = set(seeds)
|
||||
df = Counter()
|
||||
post = defaultdict(set)
|
||||
for n in nodes:
|
||||
for t in {w.lower() for w in tok.findall(doctext(n))}:
|
||||
if t in want:
|
||||
df[t] += 1
|
||||
post[t].add(n["id"])
|
||||
out = []
|
||||
for s in seeds:
|
||||
ids = sorted(post.get(s, ()))
|
||||
out.append((s, ids, df.get(s, 0)))
|
||||
return out
|
||||
|
||||
|
||||
def phrase_matches(nodes, phrase):
|
||||
p = phrase.lower()
|
||||
return sorted(n["id"] for n in nodes if p in doctext(n).lower())
|
||||
|
||||
|
||||
def hub_siblings(edges, hub, relation="identity"):
|
||||
sibs = []
|
||||
for e in edges:
|
||||
if e.get("from_id") == hub and e.get("relation") == relation:
|
||||
sibs.append(e["to_id"])
|
||||
elif e.get("to_id") == hub and e.get("relation") == relation:
|
||||
sibs.append(e["from_id"])
|
||||
return list(dict.fromkeys(sibs))
|
||||
|
||||
|
||||
def find_superseded_pairs(nodes, byid):
|
||||
"""Locked pairs, each verified here to exist and to carry its stated marker.
|
||||
|
||||
Chosen by scanning the corpus for explicit correction language
|
||||
(CORRECTION/SUPERSEDES/re-corrected/no longer/RECONCILED) and keeping only
|
||||
cases where the STALE claim also survives as its own node — a supersession
|
||||
with nothing to outrank is not a ranking test.
|
||||
"""
|
||||
pairs = []
|
||||
txt = {n["id"]: doctext(n) for n in nodes}
|
||||
|
||||
def find_one(pattern, exclude=()):
|
||||
rx = re.compile(pattern)
|
||||
return [n["id"] for n in nodes
|
||||
if n["id"] not in exclude
|
||||
and rx.search(txt[n["id"]])
|
||||
and 150 < len(str(n.get("content") or "")) < 12000
|
||||
and n.get("node_type") in ("Memory", "Knowledge", "Belief", "BacklogItem")]
|
||||
|
||||
# Each entry: (query, correction-pattern, stale-pattern, why).
|
||||
# The stale side is searched with the correction hits EXCLUDED, because most
|
||||
# correction memories quote the claim they are killing — without the
|
||||
# exclusion the "stale" node resolves to the correction itself and the pair
|
||||
# collapses into a no-op. A pair is only emitted if both sides resolve to
|
||||
# DIFFERENT surviving nodes; otherwise it is dropped and reported.
|
||||
SPECS = [
|
||||
("is the self-improvement architecture called DARMA or DHARMA",
|
||||
r'(?i)CORRECTION:.{0,90}DHARMA .{0,12}not DARMA',
|
||||
r'(?i)\bDARMA\b',
|
||||
"correction node is Will's confirmation that the H is intentional (DHARMA, not DARMA); "
|
||||
"the stale node is the surviving backlog item still titled 'Implement DARMA'."),
|
||||
|
||||
("how many provisional patents does Will actually have",
|
||||
r'(?i)EXACTLY 6 (fully-specced )?provisional',
|
||||
r'(?i)(MY ARCHITECTURE = 12 filed patents|\b12 filed patents\b)',
|
||||
"correction node is the 2026-06-17 confabulation flag establishing EXACTLY 6 provisionals; "
|
||||
"the stale node is the surviving memory that asserts 12 filed patents."),
|
||||
|
||||
("is MCP still the live integration layer",
|
||||
r'(?i)MCP RETIRED',
|
||||
r'(?i)MCP server live at',
|
||||
"correction node is the 'CGI ARCHITECTURE - THREE LAYERS, MCP RETIRED' decision of "
|
||||
"April 30 2026; the stale node still records the MCP server as live."),
|
||||
|
||||
("what does the patterns-not-returns directive mean",
|
||||
r'(?i)CORRECTION:.{0,80}patterns not returns',
|
||||
r'(?i)established returns',
|
||||
"correction node is Will's 'patterns not returns' correction; the stale node is a "
|
||||
"surviving node carrying the misread 'established returns' directive."),
|
||||
|
||||
("was the earlier identity-bug finding correct",
|
||||
r'(?i)SUPERSEDES the earlier .critical identity bug',
|
||||
r'(?i)critical identity bug',
|
||||
"correction node explicitly supersedes the 'critical identity bug' finding; the stale "
|
||||
"node is the surviving original finding."),
|
||||
|
||||
("does Neuron have recursive self-improvement",
|
||||
r'(?i)twice answered .Neuron has no recursive self-improvement',
|
||||
r'(?i)no recursive self-improvement',
|
||||
"correction node records the June-29 finding that the CGI provisional IS the "
|
||||
"recursive-self-improvement mechanism; the stale node is the surviving denial."),
|
||||
]
|
||||
|
||||
for query, cpat, spat, why in SPECS:
|
||||
corr = find_one(cpat)
|
||||
if not corr:
|
||||
continue
|
||||
stale = find_one(spat, exclude=set(corr))
|
||||
if not stale:
|
||||
continue
|
||||
pairs.append((query, corr[0], stale[0], why))
|
||||
|
||||
return pairs
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# build
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
def build(nodes, edges):
|
||||
byid = {n["id"]: n for n in nodes}
|
||||
tokset = {n["id"]: content_tokens(doctext(n)) for n in nodes}
|
||||
queries = []
|
||||
problems = []
|
||||
qn = [0]
|
||||
|
||||
def add(cat, query, relevant, derivation, **extra):
|
||||
qn[0] += 1
|
||||
q = {
|
||||
"id": f"q{qn[0]:02d}",
|
||||
"category": cat,
|
||||
"query": query,
|
||||
"relevant": sorted(relevant),
|
||||
"derivation": derivation,
|
||||
}
|
||||
q.update(extra)
|
||||
queries.append(q)
|
||||
return q
|
||||
|
||||
# --- exact_rare ---------------------------------------------------------
|
||||
for tokname, ids, df in mine_exact_rare(nodes, byid, EXACT_RARE_SEEDS):
|
||||
if df != 1 or len(ids) != 1:
|
||||
problems.append(f"exact_rare '{tokname}': df={df}, ids={len(ids)} (expected df=1)")
|
||||
continue
|
||||
lab = (byid[ids[0]].get("label") or "")[:60]
|
||||
add("exact_rare", tokname, ids,
|
||||
f"MINED: token '{tokname}' has document frequency 1 over all {len(nodes)} corpus nodes "
|
||||
f"(re-verified at build time). Its single containing node is {ids[0]} "
|
||||
f"('{lab}'), which is therefore the only possible correct answer.")
|
||||
|
||||
# --- phrase -------------------------------------------------------------
|
||||
for phrase, why in PHRASE_SEEDS:
|
||||
ids = phrase_matches(nodes, phrase)
|
||||
if not ids:
|
||||
problems.append(f"phrase '{phrase}': 0 corpus matches — unwinnable")
|
||||
continue
|
||||
if len(ids) > PHRASE_MAX:
|
||||
problems.append(f"phrase '{phrase}': {len(ids)} matches > {PHRASE_MAX} — too diffuse")
|
||||
continue
|
||||
add("phrase", phrase, ids,
|
||||
f"MINED: {why}. Case-insensitive verbatim substring scan over label+content+tags at "
|
||||
f"build time returns exactly {len(ids)} node(s); that set IS the answer key.")
|
||||
|
||||
# --- paraphrase ---------------------------------------------------------
|
||||
for target, query, why in PARAPHRASE_SEEDS:
|
||||
if target not in byid:
|
||||
problems.append(f"paraphrase target {target} not in corpus")
|
||||
continue
|
||||
qt = content_tokens(query)
|
||||
leak = sorted(qt & tokset[target])
|
||||
if leak:
|
||||
problems.append(f"paraphrase '{query}': leaks {leak} into target {target}")
|
||||
continue
|
||||
add("paraphrase", query, [target],
|
||||
f"HAND-SELECTED with criterion: {why} VERIFIED at build time: of the {len(qt)} content "
|
||||
f"words in the query, ZERO appear anywhere in the target's label, content, or tags — so "
|
||||
f"no string-matching retriever can reach this answer.",
|
||||
zero_overlap_verified=True, query_content_words=sorted(qt))
|
||||
|
||||
# --- associative --------------------------------------------------------
|
||||
sibs = hub_siblings(edges, VALUES_HUB)
|
||||
if len(sibs) < 5:
|
||||
problems.append(f"associative: values hub {VALUES_HUB} has only {len(sibs)} siblings")
|
||||
for src, query in ASSOCIATIVE_SEEDS:
|
||||
if src not in byid or src not in sibs:
|
||||
problems.append(f"associative source {src} not a sibling on {VALUES_HUB}")
|
||||
continue
|
||||
qt = content_tokens(query)
|
||||
others = [s for s in sibs if s != src and s in byid]
|
||||
# A sibling only counts as a legitimate expected answer if the query
|
||||
# cannot reach it lexically. Drop any sibling that shares a content word.
|
||||
clean = [s for s in others if not (qt & tokset[s])]
|
||||
dropped = len(others) - len(clean)
|
||||
if len(clean) < 5:
|
||||
problems.append(f"associative '{query}': only {len(clean)} lexically-unreachable siblings")
|
||||
continue
|
||||
add("associative", query, clean,
|
||||
f"DERIVED FROM EDGES: the query is built from the distinctive vocabulary of {src} "
|
||||
f"('{(byid[src].get('label') or '')[:48]}'), which hangs off the values hub {VALUES_HUB} "
|
||||
f"by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub "
|
||||
f"({len(clean)} of {len(others)}; {dropped} dropped because they shared a query word and "
|
||||
f"so were lexically reachable). Every remaining sibling shares ZERO content words with "
|
||||
f"the query — the only route from query to answer is seed({src}) -> hub -> sibling, a "
|
||||
f"two-hop traversal.",
|
||||
associative_source=src, hub=VALUES_HUB, siblings_dropped_for_overlap=dropped)
|
||||
|
||||
# --- nonsense -----------------------------------------------------------
|
||||
all_tokens = set()
|
||||
for n in nodes:
|
||||
all_tokens |= {t for t in TOKEN.findall(doctext(n).lower())}
|
||||
for s in NONSENSE_SEEDS:
|
||||
present = sorted(t for t in TOKEN.findall(s.lower()) if t in all_tokens)
|
||||
if present:
|
||||
problems.append(f"nonsense '{s}': tokens {present} DO occur in corpus")
|
||||
continue
|
||||
add("nonsense", s, [],
|
||||
f"CONTROL: verified at build time that none of this string's tokens occurs anywhere in "
|
||||
f"the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
||||
expect_empty=True)
|
||||
|
||||
# --- superseded ---------------------------------------------------------
|
||||
for query, correct, stale, why in find_superseded_pairs(nodes, byid):
|
||||
if correct not in byid or stale not in byid:
|
||||
problems.append(f"superseded '{query}': id missing from corpus")
|
||||
continue
|
||||
add("superseded", query, [correct],
|
||||
f"DERIVED: {why} Scored on RANKING, not presence: the corrected node {correct} must be "
|
||||
f"returned AND must rank above the stale node {stale}.",
|
||||
must_outrank=[correct, stale],
|
||||
stale_id=stale,
|
||||
correct_label=(byid[correct].get("label") or "")[:70],
|
||||
stale_label=(byid[stale].get("label") or "")[:70])
|
||||
|
||||
return queries, problems
|
||||
|
||||
|
||||
def summarize(queries):
|
||||
c = Counter(q["category"] for q in queries)
|
||||
return ", ".join(f"{k}={c[k]}" for k in
|
||||
("exact_rare", "phrase", "paraphrase", "associative", "nonsense", "superseded")
|
||||
if c[k])
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("snapshot")
|
||||
ap.add_argument("--out", default=DEFAULT_OUT)
|
||||
ap.add_argument("--check", action="store_true",
|
||||
help="validate only; do not write. Non-zero exit if anything is unwinnable.")
|
||||
args = ap.parse_args()
|
||||
|
||||
nodes, edges = load_corpus(args.snapshot)
|
||||
print(f"corpus: {len(nodes)} nodes, {len(edges)} edges ({os.path.basename(args.snapshot)})")
|
||||
queries, problems = build(nodes, edges)
|
||||
|
||||
print(f"gold set: {len(queries)} queries [{summarize(queries)}]")
|
||||
if problems:
|
||||
print(f"\n{len(problems)} PROBLEM(S) — these queries were REJECTED, not silently kept:")
|
||||
for p in problems:
|
||||
print(" -", p)
|
||||
|
||||
if args.check:
|
||||
sys.exit(1 if problems else 0)
|
||||
|
||||
doc = {
|
||||
"corpus": os.path.abspath(args.snapshot),
|
||||
"corpus_nodes": len(nodes),
|
||||
"corpus_edges": len(edges),
|
||||
"note": ("Every query carries a `derivation` recording how its expected answer was chosen. "
|
||||
"Re-run with --check to re-validate the whole set against the corpus."),
|
||||
"queries": queries,
|
||||
}
|
||||
with open(args.out, "w", encoding="utf-8") as fh:
|
||||
json.dump(doc, fh, indent=1, ensure_ascii=False)
|
||||
print(f"\nwrote {args.out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+210
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
compare.py — diff two run_eval.py result files, WITH a noise threshold.
|
||||
|
||||
WHY THE STATISTICS ARE NOT OPTIONAL
|
||||
With ~35 scored queries, one query is ~2.9 percentage points. A harness that
|
||||
reports "hit@5 improved 2.9%" without saying that is one query is a harness
|
||||
that will approve noise. So this file refuses to call anything an
|
||||
improvement on the strength of the headline number alone. It reports:
|
||||
|
||||
1. The DISCORDANT PAIRS. Two configurations scored on the same queries are
|
||||
paired data, so the only queries carrying information are the ones
|
||||
where they disagree: b = fixed by B, c = broken by B. Queries both got
|
||||
right, or both got wrong, tell you nothing about which is better.
|
||||
|
||||
2. McNEMAR'S EXACT TEST on (b, c). Under the null "the change is a coin
|
||||
flip", the discordant outcomes are Binomial(b+c, 0.5). The two-sided
|
||||
exact p-value is computed here with no scipy dependency.
|
||||
|
||||
3. The MINIMUM DETECTABLE SWING for this gold set: the smallest number of
|
||||
net-changed queries that would reach p < 0.05 if every discordant pair
|
||||
fell the same way. Anything smaller is inside the noise band, and the
|
||||
verdict line says so in those words.
|
||||
|
||||
Repeat-run variance is the other half of honesty. Spreading activation is a
|
||||
stateful read (it reinforces what it touches), so identical inputs need not
|
||||
give identical outputs. Pass --repeats to fold several runs of the same
|
||||
config into an observed variance band; a delta inside that band is not real
|
||||
either, however good its p-value looks.
|
||||
|
||||
usage:
|
||||
python3 compare.py --baseline results-main.json --candidate results-act.json
|
||||
python3 compare.py --baseline a.json --candidate b.json \
|
||||
--repeats-baseline a2.json a3.json --repeats-candidate b2.json b3.json
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
from math import comb
|
||||
|
||||
|
||||
def binom_two_sided(b, c):
|
||||
"""Two-sided exact binomial p for b successes in n=b+c at p=0.5."""
|
||||
n = b + c
|
||||
if n == 0:
|
||||
return 1.0
|
||||
k = min(b, c)
|
||||
tail = sum(comb(n, i) for i in range(0, k + 1)) / (2 ** n)
|
||||
return min(1.0, 2 * tail)
|
||||
|
||||
|
||||
def min_detectable_swing(n_scored, alpha=0.05):
|
||||
"""Smallest all-one-way discordant count reaching p < alpha.
|
||||
|
||||
If every query that changes changes in the same direction, the p-value is
|
||||
2 * 0.5**n. Solve for the smallest n where that drops under alpha. This is
|
||||
the FLOOR: any real change will have some discordance both ways, so the true
|
||||
requirement is larger. Reporting the floor is the conservative move — it is
|
||||
the most generous threshold we would ever accept.
|
||||
"""
|
||||
n = 1
|
||||
while n <= n_scored:
|
||||
if 2 * (0.5 ** n) < alpha:
|
||||
return n
|
||||
n += 1
|
||||
return n_scored
|
||||
|
||||
|
||||
def load(path):
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
|
||||
|
||||
def row_map(doc):
|
||||
return {r["id"]: r for r in doc["rows"]}
|
||||
|
||||
|
||||
def outcome(r):
|
||||
"""Binary per-query outcome used for the paired test.
|
||||
|
||||
hit@5 for scored queries; 'returned nothing' for the nonsense controls;
|
||||
'correction outranks the stale node' for the superseded queries. One number
|
||||
per query, so every query votes exactly once.
|
||||
"""
|
||||
if "clean" in r:
|
||||
return 1.0 if r["clean"] else 0.0
|
||||
if "outranks" in r:
|
||||
return 1.0 if r["outranks"] else 0.0
|
||||
return r.get("hit@5") or 0.0
|
||||
|
||||
|
||||
def band(values):
|
||||
return (min(values), max(values))
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--baseline", required=True)
|
||||
ap.add_argument("--candidate", required=True)
|
||||
ap.add_argument("--repeats-baseline", nargs="*", default=[])
|
||||
ap.add_argument("--repeats-candidate", nargs="*", default=[])
|
||||
ap.add_argument("--out", default=None)
|
||||
args = ap.parse_args()
|
||||
|
||||
A, B = load(args.baseline), load(args.candidate)
|
||||
ra, rb = row_map(A), row_map(B)
|
||||
ids = [q for q in ra if q in rb]
|
||||
n = len(ids)
|
||||
|
||||
aa, ab = A["aggregate"], B["aggregate"]
|
||||
print(f"baseline {A['label']:14} soul={A['soul_md5'][:12]} {n} shared queries")
|
||||
print(f"candidate {B['label']:14} soul={B['soul_md5'][:12]}")
|
||||
print(f"corpus {A['corpus_nodes']} nodes / {A['corpus_edges']} edges "
|
||||
f"(identical copy for both runs)\n")
|
||||
|
||||
metrics = [("hit@5", 1), ("recall@5", 1), ("recall@10", 1),
|
||||
("precision@5", 1), ("mrr@10", 0)]
|
||||
print(f" {'metric':14} {'baseline':>10} {'candidate':>10} {'delta':>10}")
|
||||
for m, as_pct in metrics:
|
||||
x, y = aa[m], ab[m]
|
||||
if as_pct:
|
||||
print(f" {m:14} {100*x:>9.1f}% {100*y:>9.1f}% {100*(y-x):>+9.1f}pp")
|
||||
else:
|
||||
print(f" {m:14} {x:>10.3f} {y:>10.3f} {y-x:>+10.3f}")
|
||||
for m in ("latency_ms_p50", "latency_ms_p95"):
|
||||
x, y = aa[m], ab[m]
|
||||
ratio = f"{y/x:.2f}x" if x else "n/a"
|
||||
print(f" {m:14} {x:>9.0f}ms {y:>9.0f}ms {ratio:>10}")
|
||||
print(f" {'nonsense':14} {aa['nonsense_clean']:>10} {ab['nonsense_clean']:>10}")
|
||||
print(f" {'outranks':14} {aa['superseded_outranks']:>10} {ab['superseded_outranks']:>10}")
|
||||
|
||||
print(f"\n {'category':14} {'n':>3} {'base hit@5':>11} {'cand hit@5':>11} {'delta':>9}")
|
||||
for c in sorted(set(aa["by_category"]) & set(ab["by_category"])):
|
||||
ea, eb = aa["by_category"][c], ab["by_category"][c]
|
||||
if c == "nonsense":
|
||||
print(f" {c:14} {ea['n']:>3} {'clean ' + str(ea['clean']):>11} "
|
||||
f"{'clean ' + str(eb['clean']):>11}")
|
||||
else:
|
||||
print(f" {c:14} {ea['n']:>3} {100*ea['hit@5']:>10.1f}% {100*eb['hit@5']:>10.1f}% "
|
||||
f"{100*(eb['hit@5']-ea['hit@5']):>+8.1f}pp")
|
||||
|
||||
# ---- paired significance -------------------------------------------------
|
||||
fixed, broken = [], []
|
||||
for q in ids:
|
||||
oa, ob = outcome(ra[q]), outcome(rb[q])
|
||||
if ob > oa:
|
||||
fixed.append(q)
|
||||
elif ob < oa:
|
||||
broken.append(q)
|
||||
b, c = len(fixed), len(broken)
|
||||
p = binom_two_sided(b, c)
|
||||
mds = min_detectable_swing(n)
|
||||
|
||||
print(f"\n== paired comparison over {n} queries ==")
|
||||
print(f" fixed by candidate : {b} {[ra[q]['category'] + ':' + q for q in fixed]}")
|
||||
print(f" broken by candidate: {c} {[ra[q]['category'] + ':' + q for q in broken]}")
|
||||
print(f" discordant pairs : {b + c} net {b - c:+d} queries")
|
||||
print(f" McNemar exact p : {p:.4f}")
|
||||
print(f" noise threshold : a difference needs at least {mds} queries moving the "
|
||||
f"same way to clear p<0.05 on this {n}-query set")
|
||||
|
||||
# ---- repeat-run variance -------------------------------------------------
|
||||
var = {}
|
||||
for name, paths, first in (("baseline", args.repeats_baseline, A),
|
||||
("candidate", args.repeats_candidate, B)):
|
||||
docs = [first] + [load(p) for p in paths]
|
||||
if len(docs) > 1:
|
||||
hits = [d["aggregate"]["hit@5"] for d in docs]
|
||||
lo, hi = band(hits)
|
||||
spread_q = round((hi - lo) * first["aggregate"]["n_scored"])
|
||||
var[name] = {"runs": len(docs), "hit@5_min": lo, "hit@5_max": hi,
|
||||
"spread_queries": spread_q}
|
||||
print(f" {name} repeat runs ({len(docs)}): hit@5 {100*lo:.1f}%..{100*hi:.1f}% "
|
||||
f"= {spread_q} query of run-to-run drift")
|
||||
|
||||
drift = max([v["spread_queries"] for v in var.values()], default=0)
|
||||
floor = max(mds, drift + 1)
|
||||
|
||||
print("\n== VERDICT ==")
|
||||
net = b - c
|
||||
if abs(net) < floor:
|
||||
print(f" NO MEASURABLE DIFFERENCE. Net {net:+d} queries is inside the noise band "
|
||||
f"(needs |net| >= {floor}: {mds} for significance, {drift} observed run-to-run drift).")
|
||||
elif net > 0:
|
||||
print(f" CANDIDATE BETTER by {net} queries (p={p:.4f}), outside the noise band "
|
||||
f"(>= {floor}).")
|
||||
else:
|
||||
print(f" CANDIDATE WORSE by {abs(net)} queries (p={p:.4f}), outside the noise band "
|
||||
f"(>= {floor}).")
|
||||
|
||||
if args.out:
|
||||
with open(args.out, "w", encoding="utf-8") as fh:
|
||||
json.dump({
|
||||
"baseline": A["label"], "candidate": B["label"],
|
||||
"n_shared_queries": n,
|
||||
"fixed_by_candidate": fixed, "broken_by_candidate": broken,
|
||||
"discordant": b + c, "net_queries": net,
|
||||
"mcnemar_exact_p": p,
|
||||
"min_detectable_swing_queries": mds,
|
||||
"observed_run_to_run_drift_queries": drift,
|
||||
"noise_floor_queries": floor,
|
||||
"verdict": ("no measurable difference" if abs(net) < floor
|
||||
else ("candidate better" if net > 0 else "candidate worse")),
|
||||
"baseline_aggregate": aa, "candidate_aggregate": ab,
|
||||
"repeat_variance": var,
|
||||
}, fh, indent=1)
|
||||
print(f"\nwrote {args.out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,145 @@
|
||||
{
|
||||
"baseline": "baseline-embcorpus",
|
||||
"candidate": "hybrid-semantic",
|
||||
"n_shared_queries": 38,
|
||||
"fixed_by_candidate": [
|
||||
"q15",
|
||||
"q16",
|
||||
"q20",
|
||||
"q21",
|
||||
"q26",
|
||||
"q37"
|
||||
],
|
||||
"broken_by_candidate": [],
|
||||
"discordant": 6,
|
||||
"net_queries": 6,
|
||||
"mcnemar_exact_p": 0.03125,
|
||||
"min_detectable_swing_queries": 6,
|
||||
"observed_run_to_run_drift_queries": 0,
|
||||
"noise_floor_queries": 6,
|
||||
"verdict": "candidate better",
|
||||
"baseline_aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.34285714285714286,
|
||||
"recall@5": 0.26947278911564626,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"precision@5": 0.12000000000000001,
|
||||
"mrr@10": 0.2943197278911564,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "1/3",
|
||||
"latency_ms_p50": 1145.9,
|
||||
"latency_ms_p95": 1574.3,
|
||||
"latency_ms_max": 1634.2,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 3.3333333333333335
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.8571428571428571,
|
||||
"recall@5": 0.4902210884353741,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.5965986394557822
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"mrr@10": 0.041666666666666664,
|
||||
"outranks": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
"candidate_aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.5142857142857142,
|
||||
"recall@5": 0.4409013605442177,
|
||||
"recall@10": 0.5047619047619047,
|
||||
"precision@5": 0.15428571428571433,
|
||||
"mrr@10": 0.38746031746031745,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "2/3",
|
||||
"latency_ms_p50": 1219.7,
|
||||
"latency_ms_p95": 1667.1,
|
||||
"latency_ms_max": 1720.2,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 3.3333333333333335
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.38461538461538464,
|
||||
"recall@5": 0.38461538461538464,
|
||||
"recall@10": 0.38461538461538464,
|
||||
"mrr@10": 0.17307692307692307
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.8571428571428571,
|
||||
"recall@5": 0.4902210884353741,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.6634920634920636
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.3333333333333333,
|
||||
"recall@5": 0.3333333333333333,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.2222222222222222,
|
||||
"outranks": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
"repeat_variance": {
|
||||
"candidate": {
|
||||
"runs": 2,
|
||||
"hit@5_min": 0.5142857142857142,
|
||||
"hit@5_max": 0.5142857142857142,
|
||||
"spread_queries": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
{
|
||||
"baseline": "main-r1",
|
||||
"candidate": "act-r1",
|
||||
"n_shared_queries": 38,
|
||||
"fixed_by_candidate": [],
|
||||
"broken_by_candidate": [
|
||||
"q07",
|
||||
"q11",
|
||||
"q12",
|
||||
"q13",
|
||||
"q36"
|
||||
],
|
||||
"discordant": 5,
|
||||
"net_queries": -5,
|
||||
"mcnemar_exact_p": 0.0625,
|
||||
"min_detectable_swing_queries": 6,
|
||||
"observed_run_to_run_drift_queries": 1,
|
||||
"noise_floor_queries": 6,
|
||||
"verdict": "no measurable difference",
|
||||
"baseline_aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.34285714285714286,
|
||||
"recall@5": 0.26947278911564626,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"precision@5": 0.12000000000000001,
|
||||
"mrr@10": 0.2943197278911564,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "1/3",
|
||||
"latency_ms_p50": 1140.4,
|
||||
"latency_ms_p95": 1584.1,
|
||||
"latency_ms_max": 1627.6,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 3.3333333333333335
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.8571428571428571,
|
||||
"recall@5": 0.4902210884353741,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.5965986394557822
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"mrr@10": 0.041666666666666664,
|
||||
"outranks": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
"candidate_aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.22857142857142856,
|
||||
"recall@5": 0.19087301587301586,
|
||||
"recall@10": 0.24277210884353742,
|
||||
"precision@5": 0.07428571428571429,
|
||||
"mrr@10": 0.24154195011337865,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "0/3",
|
||||
"latency_ms_p50": 3208.8,
|
||||
"latency_ms_p95": 4851.9,
|
||||
"latency_ms_max": 5078.8,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 2.6666666666666665
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.2857142857142857,
|
||||
"recall@5": 0.09722222222222222,
|
||||
"recall@10": 0.3567176870748299,
|
||||
"mrr@10": 0.3505668934240363
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"repeat_variance": {
|
||||
"baseline": {
|
||||
"runs": 3,
|
||||
"hit@5_min": 0.34285714285714286,
|
||||
"hit@5_max": 0.34285714285714286,
|
||||
"spread_queries": 0
|
||||
},
|
||||
"candidate": {
|
||||
"runs": 3,
|
||||
"hit@5_min": 0.22857142857142856,
|
||||
"hit@5_max": 0.2571428571428571,
|
||||
"spread_queries": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
import json,sys,time,urllib.request,threading,queue
|
||||
SRC="/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json"
|
||||
OUT=sys.argv[1]
|
||||
URL="http://127.0.0.1:11434/api/embeddings"; MODEL="nomic-embed-text"
|
||||
MAXB=2000 # ENGRAM_EMBED_MAX_CHARS, applied to bytes as the C code does
|
||||
d=json.load(open(SRC,encoding='utf-8',errors='surrogateescape'))
|
||||
tasks=[]
|
||||
for n in d["nodes"]:
|
||||
c=n.get("content") or ""; t=n.get("node_type") or ""
|
||||
if len(c)<8: continue # eg_embed_eligible
|
||||
if t in ("InternalStateEvent","Tag"): continue
|
||||
b=c.encode('utf-8',errors='surrogateescape')[:MAXB]
|
||||
tasks.append((n.get("id") or "", b.decode('utf-8',errors='replace')))
|
||||
del d
|
||||
print("tasks",len(tasks),flush=True)
|
||||
q=queue.Queue(); [q.put(t) for t in tasks]
|
||||
lock=threading.Lock(); f=open(OUT,"w",encoding="utf-8",errors="surrogateescape"); done=[0]; t0=time.time(); fails=[0]
|
||||
def work():
|
||||
while True:
|
||||
try: nid,txt=q.get_nowait()
|
||||
except queue.Empty: return
|
||||
v=None
|
||||
for attempt in range(3):
|
||||
try:
|
||||
body=json.dumps({"model":MODEL,"prompt":txt}).encode()
|
||||
r=urllib.request.Request(URL,data=body,headers={"Content-Type":"application/json"})
|
||||
with urllib.request.urlopen(r,timeout=120) as fh: v=json.load(fh)["embedding"]
|
||||
break
|
||||
except Exception as e:
|
||||
if attempt==2:
|
||||
with lock: fails[0]+=1
|
||||
time.sleep(0.5)
|
||||
with lock:
|
||||
if v: f.write(nid+"\t"+",".join("%.5g"%x for x in v)+"\n")
|
||||
done[0]+=1
|
||||
if done[0]%2000==0:
|
||||
el=time.time()-t0
|
||||
print("%d/%d %.1f/s eta %.1fmin fails=%d"%(done[0],len(tasks),done[0]/el,(len(tasks)-done[0])/(done[0]/el)/60,fails[0]),flush=True)
|
||||
f.flush()
|
||||
ths=[threading.Thread(target=work) for _ in range(8)]
|
||||
[t.start() for t in ths]; [t.join() for t in ths]
|
||||
f.close()
|
||||
print("DONE",done[0],"fails",fails[0],"secs %.1f"%(time.time()-t0),flush=True)
|
||||
@@ -0,0 +1,587 @@
|
||||
{
|
||||
"corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json",
|
||||
"corpus_nodes": 78768,
|
||||
"corpus_edges": 14214,
|
||||
"note": "Every query carries a `derivation` recording how its expected answer was chosen. Re-run with --check to re-validate the whole set against the corpus.",
|
||||
"queries": [
|
||||
{
|
||||
"id": "q01",
|
||||
"category": "exact_rare",
|
||||
"query": "unjailbreakable",
|
||||
"relevant": [
|
||||
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
||||
],
|
||||
"derivation": "MINED: token 'unjailbreakable' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226 ('Daemon hidden substrate architecture ? implemented April 25 '), which is therefore the only possible correct answer."
|
||||
},
|
||||
{
|
||||
"id": "q02",
|
||||
"category": "exact_rare",
|
||||
"query": "engram-migrate",
|
||||
"relevant": [
|
||||
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5"
|
||||
],
|
||||
"derivation": "MINED: token 'engram-migrate' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5 ('Engram v0.1 complete ? April 27, 2026. Local-first spreading'), which is therefore the only possible correct answer."
|
||||
},
|
||||
{
|
||||
"id": "q03",
|
||||
"category": "exact_rare",
|
||||
"query": "cartabandonedevent",
|
||||
"relevant": [
|
||||
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
||||
],
|
||||
"derivation": "MINED: token 'cartabandonedevent' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091 ('MESSAGE FOR AUDIT AGENT af5a7352e70e80434 ? El Language Spec'), which is therefore the only possible correct answer."
|
||||
},
|
||||
{
|
||||
"id": "q04",
|
||||
"category": "exact_rare",
|
||||
"query": "pre-apprenticeship",
|
||||
"relevant": [
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
||||
],
|
||||
"derivation": "MINED: token 'pre-apprenticeship' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-89c02aae-d3ca-43f9-9e5d-eb369896276c ('William Fox Anderson ? Applicant Profile Personal: - Full N'), which is therefore the only possible correct answer."
|
||||
},
|
||||
{
|
||||
"id": "q05",
|
||||
"category": "exact_rare",
|
||||
"query": "inferencenodemanager",
|
||||
"relevant": [
|
||||
"mem-73969486-143f-4431-b5e6-6845d1cc9848"
|
||||
],
|
||||
"derivation": "MINED: token 'inferencenodemanager' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-73969486-143f-4431-b5e6-6845d1cc9848 ('Soma inference backplane deployed April 28 2026. Architectur'), which is therefore the only possible correct answer."
|
||||
},
|
||||
{
|
||||
"id": "q06",
|
||||
"category": "exact_rare",
|
||||
"query": "clear-eyed",
|
||||
"relevant": [
|
||||
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
||||
],
|
||||
"derivation": "MINED: token 'clear-eyed' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is knw-c72597c5-c23d-4c08-8e9e-996dadf26a99 ('Clear Eyes ? The Incomplete World View'), which is therefore the only possible correct answer."
|
||||
},
|
||||
{
|
||||
"id": "q07",
|
||||
"category": "phrase",
|
||||
"query": "patterns not returns",
|
||||
"relevant": [
|
||||
"mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482"
|
||||
],
|
||||
"derivation": "MINED: a verbatim correction Will issued; expected = every node containing the phrase. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 1 node(s); that set IS the answer key."
|
||||
},
|
||||
{
|
||||
"id": "q08",
|
||||
"category": "phrase",
|
||||
"query": "thirty moves",
|
||||
"relevant": [
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"??o?'?B???k",
|
||||
"Kp???",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4",
|
||||
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
||||
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6",
|
||||
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
||||
"ע?RGk?\tH(?"
|
||||
],
|
||||
"derivation": "MINED: the canonical biographical phrase; expected = every node containing it. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 16 node(s); that set IS the answer key."
|
||||
},
|
||||
{
|
||||
"id": "q09",
|
||||
"category": "phrase",
|
||||
"query": "Grandma Lucas",
|
||||
"relevant": [
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e"
|
||||
],
|
||||
"derivation": "MINED: a named person appearing verbatim in the biography/value nodes. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 9 node(s); that set IS the answer key."
|
||||
},
|
||||
{
|
||||
"id": "q10",
|
||||
"category": "phrase",
|
||||
"query": "Directed Harmonic",
|
||||
"relevant": [
|
||||
"%???2??jH??",
|
||||
"63307ac5-cf6b-46e0-8296-07503b461cfa",
|
||||
"7c9d4ab1-205d-4be8-bfae-e2c03a3a5010",
|
||||
"9f291d20-0d32-413c-8c01-4416ccab4f7f",
|
||||
"?;????n}rh?",
|
||||
"???Ͼd??f??",
|
||||
"?Z?.\f?0?]P?",
|
||||
"Dp???]Q??k+",
|
||||
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf"
|
||||
],
|
||||
"derivation": "MINED: the canonical DHARMA expansion, confirmed by Will April 24 2026. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 9 node(s); that set IS the answer key."
|
||||
},
|
||||
{
|
||||
"id": "q11",
|
||||
"category": "phrase",
|
||||
"query": "Sarah Bishop",
|
||||
"relevant": [
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e"
|
||||
],
|
||||
"derivation": "MINED: a named person; rare enough that the answer set is unambiguous. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 2 node(s); that set IS the answer key."
|
||||
},
|
||||
{
|
||||
"id": "q12",
|
||||
"category": "phrase",
|
||||
"query": "Directed Autonomous Runtime Modification",
|
||||
"relevant": [
|
||||
"2a923500-d7e1-4b15-80e2-48dba65984ba",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???",
|
||||
"bl-145a0985-2382-400f-a7c5-c335c5e30a72",
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"g?2睪A|?H\b",
|
||||
"mem-82b93b21-a865-410f-9ec1-fc54121d9bb5",
|
||||
"mem-e6327f52-2bda-4ce7-9471-2fffd1e172de"
|
||||
],
|
||||
"derivation": "MINED: the DARMA expansion, quoted verbatim in the backlog item and its correction. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 14 node(s); that set IS the answer key."
|
||||
},
|
||||
{
|
||||
"id": "q13",
|
||||
"category": "phrase",
|
||||
"query": "zero-knowledge encrypted backup",
|
||||
"relevant": [
|
||||
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a"
|
||||
],
|
||||
"derivation": "MINED: the paid-tier feature name as written in the roadmap nodes. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 1 node(s); that set IS the answer key."
|
||||
},
|
||||
{
|
||||
"id": "q14",
|
||||
"category": "paraphrase",
|
||||
"query": "the elderly relative who passed while he stayed away",
|
||||
"relevant": [
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Do the Essential Thing While You Can', whose subject is Grandma Lucas dying in Feb 2006 without Will saying goodbye. Query names the event with none of the node's own vocabulary. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"away",
|
||||
"elderly",
|
||||
"passed",
|
||||
"relative",
|
||||
"stayed"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q15",
|
||||
"category": "paraphrase",
|
||||
"query": "a soldier sidelined by illness who refused to quit",
|
||||
"relevant": [
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Survival Is Not an Excuse to Stop', whose subject is enlisting in the Marines, a severe hernia, and sepsis. Query describes the episode obliquely. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"illness",
|
||||
"quit",
|
||||
"refused",
|
||||
"sidelined",
|
||||
"soldier"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q16",
|
||||
"category": "paraphrase",
|
||||
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
||||
"relevant": [
|
||||
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Honesty Before Comfort'. Query states the principle in wholly different words. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"choosing",
|
||||
"fact",
|
||||
"fiction",
|
||||
"pleasant",
|
||||
"uncomfortable"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q17",
|
||||
"category": "paraphrase",
|
||||
"query": "a tight payload beats a bloated one",
|
||||
"relevant": [
|
||||
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Precision Over Brute Force'. Query restates the claim with no shared vocabulary. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"beats",
|
||||
"bloated",
|
||||
"payload",
|
||||
"tight"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q18",
|
||||
"category": "paraphrase",
|
||||
"query": "if you are able and nobody is coming the job is yours",
|
||||
"relevant": [
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Capability Is a Debt You Owe the Moment'. Query states the obligation without the node's terms. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"able",
|
||||
"coming",
|
||||
"job",
|
||||
"nobody"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q19",
|
||||
"category": "paraphrase",
|
||||
"query": "learning is the wealth creditors cannot seize",
|
||||
"relevant": [
|
||||
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Knowledge Survives When Nothing Else Does', whose subject is the library following Will across 30+ moves. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"creditors",
|
||||
"learning",
|
||||
"seize",
|
||||
"wealth"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q20",
|
||||
"category": "paraphrase",
|
||||
"query": "reliability proven by track record not assertion",
|
||||
"relevant": [
|
||||
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Earned Trust' ('Trust is demonstrated, not declared'). VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"assertion",
|
||||
"proven",
|
||||
"record",
|
||||
"reliability",
|
||||
"track"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q21",
|
||||
"category": "paraphrase",
|
||||
"query": "boundaries that enable instead of confine",
|
||||
"relevant": [
|
||||
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Constraints as Freedom'. Query is a restatement of the same claim. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"boundaries",
|
||||
"confine",
|
||||
"enable",
|
||||
"instead"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q22",
|
||||
"category": "paraphrase",
|
||||
"query": "what shifts tells you where to cut a system apart",
|
||||
"relevant": [
|
||||
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Change Is the Signal', the value VBD is built on. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"apart",
|
||||
"cut",
|
||||
"shifts",
|
||||
"system",
|
||||
"tells"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q23",
|
||||
"category": "paraphrase",
|
||||
"query": "a mind that compounds instead of resetting each day",
|
||||
"relevant": [
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - The System Must Accumulate'. Query is the accumulation claim in different vocabulary. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"compounds",
|
||||
"day",
|
||||
"instead",
|
||||
"mind",
|
||||
"resetting"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q24",
|
||||
"category": "paraphrase",
|
||||
"query": "loved for the unedited self and not the polished exterior",
|
||||
"relevant": [
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Being Seen Is Rarer Than Being Known', whose subject is Sarah Bishop as the first person Will did not perform for. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"exterior",
|
||||
"loved",
|
||||
"polished",
|
||||
"self",
|
||||
"unedited"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q25",
|
||||
"category": "paraphrase",
|
||||
"query": "cheerfulness you arrive at instead of assuming",
|
||||
"relevant": [
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Hope Is a Conclusion'. Query restates 'a conclusion, not a premise'. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"arrive",
|
||||
"assuming",
|
||||
"cheerfulness",
|
||||
"instead"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q26",
|
||||
"category": "paraphrase",
|
||||
"query": "a childhood offering no solid foundation to inherit",
|
||||
"relevant": [
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930"
|
||||
],
|
||||
"derivation": "HAND-SELECTED with criterion: target: 'Value - Structure Is Not Inherited', whose subject is thirty moves between two parents' collapses. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
||||
"zero_overlap_verified": true,
|
||||
"query_content_words": [
|
||||
"childhood",
|
||||
"foundation",
|
||||
"inherit",
|
||||
"offering",
|
||||
"solid"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "q27",
|
||||
"category": "associative",
|
||||
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
||||
"relevant": [
|
||||
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
||||
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
||||
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
||||
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4"
|
||||
],
|
||||
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71 ('Value ? Do the Essential Thing While You Can'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71) -> hub -> sibling, a two-hop traversal.",
|
||||
"associative_source": "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"siblings_dropped_for_overlap": 2
|
||||
},
|
||||
{
|
||||
"id": "q28",
|
||||
"category": "associative",
|
||||
"query": "Marines hernia sepsis medical ward",
|
||||
"relevant": [
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
||||
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
||||
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
||||
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
||||
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4"
|
||||
],
|
||||
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-58874a74-b96f-4883-9e08-45707f4bd3ee ('Value ? Survival Is Not an Excuse to Stop'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (13 of 13; 0 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-58874a74-b96f-4883-9e08-45707f4bd3ee) -> hub -> sibling, a two-hop traversal.",
|
||||
"associative_source": "kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"siblings_dropped_for_overlap": 0
|
||||
},
|
||||
{
|
||||
"id": "q29",
|
||||
"category": "associative",
|
||||
"query": "Sarah Bishop Dyer trailer performance",
|
||||
"relevant": [
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
||||
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
||||
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
||||
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
||||
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8"
|
||||
],
|
||||
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e ('Value ? Being Seen Is Rarer Than Being Known'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e) -> hub -> sibling, a two-hop traversal.",
|
||||
"associative_source": "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"siblings_dropped_for_overlap": 2
|
||||
},
|
||||
{
|
||||
"id": "q30",
|
||||
"category": "associative",
|
||||
"query": "Swarm Architecture containment lateral worker",
|
||||
"relevant": [
|
||||
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
||||
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
||||
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4"
|
||||
],
|
||||
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83 ('Value ? Constraints as Freedom'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (6 of 13; 7 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83) -> hub -> sibling, a two-hop traversal.",
|
||||
"associative_source": "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"siblings_dropped_for_overlap": 7
|
||||
},
|
||||
{
|
||||
"id": "q31",
|
||||
"category": "associative",
|
||||
"query": "hope won inside the narrative preface",
|
||||
"relevant": [
|
||||
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
||||
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
||||
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4"
|
||||
],
|
||||
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-e0423482-cfa5-4796-8689-8495c93b66bc ('Value ? Hope Is a Conclusion'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-e0423482-cfa5-4796-8689-8495c93b66bc) -> hub -> sibling, a two-hop traversal.",
|
||||
"associative_source": "kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"siblings_dropped_for_overlap": 2
|
||||
},
|
||||
{
|
||||
"id": "q32",
|
||||
"category": "associative",
|
||||
"query": "man of the house six years old expectation",
|
||||
"relevant": [
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
||||
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
||||
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
||||
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
||||
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc"
|
||||
],
|
||||
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8 ('Value ? Capability Is a Debt You Owe the Moment'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8) -> hub -> sibling, a two-hop traversal.",
|
||||
"associative_source": "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"siblings_dropped_for_overlap": 2
|
||||
},
|
||||
{
|
||||
"id": "q33",
|
||||
"category": "nonsense",
|
||||
"query": "zqxjvw plimforth grebulon",
|
||||
"relevant": [],
|
||||
"derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
||||
"expect_empty": true
|
||||
},
|
||||
{
|
||||
"id": "q34",
|
||||
"category": "nonsense",
|
||||
"query": "flarnbistle quommetry",
|
||||
"relevant": [],
|
||||
"derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
||||
"expect_empty": true
|
||||
},
|
||||
{
|
||||
"id": "q35",
|
||||
"category": "nonsense",
|
||||
"query": "xxqzzt vurblenacht throom",
|
||||
"relevant": [],
|
||||
"derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
||||
"expect_empty": true
|
||||
},
|
||||
{
|
||||
"id": "q36",
|
||||
"category": "superseded",
|
||||
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
||||
"relevant": [
|
||||
"mem-80d7416b-20e9-48a0-b176-b215527e2f56"
|
||||
],
|
||||
"derivation": "DERIVED: correction node is Will's confirmation that the H is intentional (DHARMA, not DARMA); the stale node is the surviving backlog item still titled 'Implement DARMA'. Scored on RANKING, not presence: the corrected node mem-80d7416b-20e9-48a0-b176-b215527e2f56 must be returned AND must rank above the stale node bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff.",
|
||||
"must_outrank": [
|
||||
"mem-80d7416b-20e9-48a0-b176-b215527e2f56",
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff"
|
||||
],
|
||||
"stale_id": "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"correct_label": "CORRECTION: The autonomous self-improvement architecture is DHARMA ? n",
|
||||
"stale_label": "Implement DARMA ? Directed Autonomous Runtime Modification Architectur"
|
||||
},
|
||||
{
|
||||
"id": "q37",
|
||||
"category": "superseded",
|
||||
"query": "how many provisional patents does Will actually have",
|
||||
"relevant": [
|
||||
"3cf706a1-3825-45d8-b0a9-06cae6cdf5b8"
|
||||
],
|
||||
"derivation": "DERIVED: correction node is the 2026-06-17 confabulation flag establishing EXACTLY 6 provisionals; the stale node is the surviving memory that asserts 12 filed patents. Scored on RANKING, not presence: the corrected node 3cf706a1-3825-45d8-b0a9-06cae6cdf5b8 must be returned AND must rank above the stale node 936541a9-fabb-466b-9ca3-a78b17ad0c53.",
|
||||
"must_outrank": [
|
||||
"3cf706a1-3825-45d8-b0a9-06cae6cdf5b8",
|
||||
"936541a9-fabb-466b-9ca3-a78b17ad0c53"
|
||||
],
|
||||
"stale_id": "936541a9-fabb-466b-9ca3-a78b17ad0c53",
|
||||
"correct_label": "memory:remembered",
|
||||
"stale_label": "memory:remembered"
|
||||
},
|
||||
{
|
||||
"id": "q38",
|
||||
"category": "superseded",
|
||||
"query": "is MCP still the live integration layer",
|
||||
"relevant": [
|
||||
"mem-30425134-6008-4fd9-a3ee-67a7742c319b"
|
||||
],
|
||||
"derivation": "DERIVED: correction node is the 'CGI ARCHITECTURE - THREE LAYERS, MCP RETIRED' decision of April 30 2026; the stale node still records the MCP server as live. Scored on RANKING, not presence: the corrected node mem-30425134-6008-4fd9-a3ee-67a7742c319b must be returned AND must rank above the stale node mem-101e81b4-8097-4749-8d8d-7bb66de34517.",
|
||||
"must_outrank": [
|
||||
"mem-30425134-6008-4fd9-a3ee-67a7742c319b",
|
||||
"mem-101e81b4-8097-4749-8d8d-7bb66de34517"
|
||||
],
|
||||
"stale_id": "mem-101e81b4-8097-4749-8d8d-7bb66de34517",
|
||||
"correct_label": "CGI ARCHITECTURE ? THREE LAYERS, MCP RETIRED (April 30, 2026). Definit",
|
||||
"stale_label": "GCloud MCP infrastructure ? April 27, 2026. Legion died (~19:30 UTC). "
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
import json,sys
|
||||
SRC="/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json"
|
||||
TSV,OUT=sys.argv[1],sys.argv[2]
|
||||
emb={}
|
||||
for line in open(TSV,encoding='utf-8',errors='surrogateescape'):
|
||||
p=line.rstrip("\n").rsplit("\t",1)
|
||||
if len(p)==2 and p[1].count(",")>100: emb[p[0]]=p[1]
|
||||
print("vectors",len(emb),flush=True)
|
||||
d=json.load(open(SRC,encoding='utf-8',errors='surrogateescape'))
|
||||
hit=0
|
||||
for n in d["nodes"]:
|
||||
v=emb.get(n.get("id") or "")
|
||||
if v: n["emb"]=v; hit+=1
|
||||
print("attached",hit,"of",len(d["nodes"]),flush=True)
|
||||
with open(OUT,"w",encoding='utf-8',errors='surrogateescape') as f:
|
||||
json.dump(d,f,ensure_ascii=False)
|
||||
print("wrote",OUT,flush=True)
|
||||
@@ -0,0 +1,942 @@
|
||||
{
|
||||
"label": "act-r2",
|
||||
"soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-act",
|
||||
"soul_md5": "77722f5a9f49494bf735c2a4be1b5dc4",
|
||||
"corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json",
|
||||
"corpus_nodes": 78768,
|
||||
"corpus_edges": 14214,
|
||||
"gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json",
|
||||
"limit": 10,
|
||||
"port": 7902,
|
||||
"wall_clock_s": 121.2,
|
||||
"child_pid": 78714,
|
||||
"child_confirmed_dead": true,
|
||||
"aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.22857142857142856,
|
||||
"recall@5": 0.18908730158730158,
|
||||
"recall@10": 0.24277210884353742,
|
||||
"precision@5": 0.06857142857142857,
|
||||
"mrr@10": 0.24154195011337865,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "0/3",
|
||||
"latency_ms_p50": 3237.6,
|
||||
"latency_ms_p95": 5264.0,
|
||||
"latency_ms_max": 5510.9,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 2.6666666666666665
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.2857142857142857,
|
||||
"recall@5": 0.0882936507936508,
|
||||
"recall@10": 0.3567176870748299,
|
||||
"mrr@10": 0.3505668934240363
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"id": "q01",
|
||||
"category": "exact_rare",
|
||||
"query": "unjailbreakable",
|
||||
"returned": [
|
||||
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 476.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q02",
|
||||
"category": "exact_rare",
|
||||
"query": "engram-migrate",
|
||||
"returned": [
|
||||
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 708.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q03",
|
||||
"category": "exact_rare",
|
||||
"query": "cartabandonedevent",
|
||||
"returned": [
|
||||
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 532.0,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q04",
|
||||
"category": "exact_rare",
|
||||
"query": "pre-apprenticeship",
|
||||
"returned": [
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 537.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q05",
|
||||
"category": "exact_rare",
|
||||
"query": "inferencenodemanager",
|
||||
"returned": [
|
||||
"mem-73969486-143f-4431-b5e6-6845d1cc9848"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 568.7,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q06",
|
||||
"category": "exact_rare",
|
||||
"query": "clear-eyed",
|
||||
"returned": [
|
||||
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 555.4,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q07",
|
||||
"category": "phrase",
|
||||
"query": "patterns not returns",
|
||||
"returned": [
|
||||
"knw-d357b6bb-ad8a-4791-b516-426aea45fa5b",
|
||||
"kn-69fd6e83-7718-4824-8d66-f49d8954e224",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-c3d9d063-8c5d-45aa-900c-550914b2ff6d",
|
||||
"kn-f838f113-76d5-4a15-9cef-14055c4723a3",
|
||||
"bl-76e878aa-e1fe-468c-bf9c-854097cb7e0b",
|
||||
"art-c71aef51-026f-4d63-80e9-2a0ec0dc3865",
|
||||
"bl-e148d23c-24e8-4122-9915-d1c11f22052f",
|
||||
"bl-31abf75b-998f-4a4f-a6dd-8204119e0451"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1906.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q08",
|
||||
"category": "phrase",
|
||||
"query": "thirty moves",
|
||||
"returned": [
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"bl-80720fdf-7ce7-4d28-aff8-21028d3a8cfb",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"knw-ed33e669-0790-44cb-a036-958d605c6fea",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
||||
"?Z?.\f?0?]P?",
|
||||
"?Q??m?;��`'"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1103.3,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.0625,
|
||||
"recall@10": 0.1875,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q09",
|
||||
"category": "phrase",
|
||||
"query": "Grandma Lucas",
|
||||
"returned": [
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 6,
|
||||
"latency_ms": 1132.4,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.5555555555555556,
|
||||
"recall@10": 0.5555555555555556,
|
||||
"precision@5": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q10",
|
||||
"category": "phrase",
|
||||
"query": "Directed Harmonic",
|
||||
"returned": [
|
||||
"bl-c9adb8e5-293f-4033-99f8-0405c17ef941",
|
||||
"bl-7e7c3fdb-4132-487f-aa70-b2cd559cb7f0",
|
||||
"bl-7aebe936-ac55-4f35-8932-adc5224ff854",
|
||||
"bl-9d53422d-b703-4f1d-860a-8598cb29b792",
|
||||
"mem-34f53a9d-a131-4f82-9dbd-b9eb4a9af52e",
|
||||
"mem-60778715-758c-4677-933d-fc39b8f94152",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 960.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.1111111111111111,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1
|
||||
},
|
||||
{
|
||||
"id": "q11",
|
||||
"category": "phrase",
|
||||
"query": "Sarah Bishop",
|
||||
"returned": [
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
";??A5???",
|
||||
"art-94fae615-7cd5-4695-b968-977101b06a51",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35",
|
||||
"mem-57164d5f-baf0-4149-957a-379a4e255d1a",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"art-8dbceb06-431a-416d-a723-e8c75d595154"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 992.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1111111111111111
|
||||
},
|
||||
{
|
||||
"id": "q12",
|
||||
"category": "phrase",
|
||||
"query": "Directed Autonomous Runtime Modification",
|
||||
"returned": [
|
||||
"kn-b2a99cd7-b379-4d9b-a996-e347a02c7bad",
|
||||
"bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae",
|
||||
"kn-8e1bfb48-33a9-45ad-8da7-e0bdaa5d34e7",
|
||||
"art-92e1837c-5919-42d0-bbb0-4d924d7b2864",
|
||||
"bl-a7a1428f-db9c-417b-8e2c-713b1f84dc1f",
|
||||
"mem-f823e835-313f-4282-b4b3-ce527ffc2f7a",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"?Z?.\f?0?]P?",
|
||||
"bl-145a0985-2382-400f-a7c5-c335c5e30a72"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2820.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.14285714285714285,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.14285714285714285
|
||||
},
|
||||
{
|
||||
"id": "q13",
|
||||
"category": "phrase",
|
||||
"query": "zero-knowledge encrypted backup",
|
||||
"returned": [
|
||||
"art-e495c8c5-ad95-4b64-8771-f68aa4cfcd0a",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"bl-e20944e5-eb16-4ab3-a84d-111e0fc817fa",
|
||||
"bl-e20944e5-f4a6-44a0-91b1-73d04ebed120",
|
||||
"mem-47f72b5b-6e8b-4293-94f1-350197b4809a",
|
||||
"mem-a5f04e52-91f8-41d2-af27-8bf803621758",
|
||||
"7774a16c-1027-4e3b-a21e-67f1f95a4acd",
|
||||
"? ?}&?#??X\b",
|
||||
"7?e?7???\f3?",
|
||||
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2609.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1
|
||||
},
|
||||
{
|
||||
"id": "q14",
|
||||
"category": "paraphrase",
|
||||
"query": "the elderly relative who passed while he stayed away",
|
||||
"returned": [
|
||||
"knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca",
|
||||
"kn-82be4e41-96c5-4da3-85c0-cee10763d975",
|
||||
"mem-ea487cb4-ed67-44ce-8402-b56bb28468d4",
|
||||
"bl-2694b588-a6e3-43de-861c-fa7b0ec7e7fd",
|
||||
"bl-14883d81-f7cb-46dd-82c2-a6e6980264e5",
|
||||
"tag-dark-theme",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 5510.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q15",
|
||||
"category": "paraphrase",
|
||||
"query": "a soldier sidelined by illness who refused to quit",
|
||||
"returned": [
|
||||
"mem-0328c3cb-4550-4ce4-9284-152e832f08f6",
|
||||
"bl-07375bf9-a169-42cd-adb3-7d32b25982f0",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"bl-e5635b1a-c5d0-4caa-bcc8-6a726ea43685",
|
||||
"bl-6172d035-dd94-4776-afdd-d8915f6fc375",
|
||||
"bl-5bb8dedf-8498-4a9b-acdc-31cc9c738f2a",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
";??A5???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 5444.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q16",
|
||||
"category": "paraphrase",
|
||||
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
||||
"returned": [
|
||||
"bl-d24fcce8-2b55-426f-867a-db3958a622d3",
|
||||
"tag-phase-3",
|
||||
"tag-project-structure",
|
||||
"tag-anthropic-contrast",
|
||||
"tag-voice-training",
|
||||
"tag-ebd",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"bl-9d53422d-b703-4f1d-860a-8598cb29b792",
|
||||
";??A5???",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 5056.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q17",
|
||||
"category": "paraphrase",
|
||||
"query": "a tight payload beats a bloated one",
|
||||
"returned": [
|
||||
"bl-fc893be3-e6b4-4ef6-93b0-d54ca5f89083",
|
||||
"bl-57c5cf6b-81a5-4558-9902-5c02981fe273",
|
||||
"tag-guilds",
|
||||
"tag-kids",
|
||||
"tag-coexistence",
|
||||
"tag-cultivated-general-intelligence",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
||||
";??A5???",
|
||||
"mem-cdff0c49-3ac7-4de8-89ec-92d254bd0023"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3473.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q18",
|
||||
"category": "paraphrase",
|
||||
"query": "if you are able and nobody is coming the job is yours",
|
||||
"returned": [
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992",
|
||||
"bl-b019ce7a-1b21-436e-812d-032f50c6c45f",
|
||||
"bl-e98cdd4c-01b5-459e-9036-3578cd5d975a",
|
||||
"bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0",
|
||||
"tag-stable-diffusion",
|
||||
";??A5???",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4446.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q19",
|
||||
"category": "paraphrase",
|
||||
"query": "learning is the wealth creditors cannot seize",
|
||||
"returned": [
|
||||
"mem-e321e54e-8bb3-4596-b13d-bb093d6b149d",
|
||||
"mem-e32ba5a7-c147-4dc0-9479-b720d768eda6",
|
||||
"mem-c7a77457-478d-4eb0-a116-67205a0066a4",
|
||||
"bl-1b20e9bc-eb37-4907-8d63-e311fd61eab8",
|
||||
"bl-aa762207-920d-45ab-b2a3-2f8154d7ef9b",
|
||||
"tag-misalignment",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3932.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q20",
|
||||
"category": "paraphrase",
|
||||
"query": "reliability proven by track record not assertion",
|
||||
"returned": [
|
||||
"kn-48a01973-a025-471d-950f-b93e6a426d82",
|
||||
"mem-23d22bc1-a097-446b-8f11-8aff099e0b76",
|
||||
"mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8",
|
||||
"mem-ce5a2ffc-ad39-4728-9ac6-76fef507d5da",
|
||||
"project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_",
|
||||
"tag-provenance",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"art-e495c8c5-ad95-4b64-8771-f68aa4cfcd0a"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4836.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q21",
|
||||
"category": "paraphrase",
|
||||
"query": "boundaries that enable instead of confine",
|
||||
"returned": [
|
||||
"kn-79056192-7de8-486c-9565-f128439a6fcb",
|
||||
"bl-f6236350-f7b8-4f4f-a702-9eef2eb76e4b",
|
||||
"bl-7f33f1bc-99fa-4906-889f-a42375beea20",
|
||||
"mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9",
|
||||
"mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1",
|
||||
"tag-__cultivation-metric____internal-state____dharma____evidence____novel-idea____gap-compression____values____microsoft__",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4057.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q22",
|
||||
"category": "paraphrase",
|
||||
"query": "what shifts tells you where to cut a system apart",
|
||||
"returned": [
|
||||
"knw-723551f5-1950-42a3-8b89-b6a06913cef0",
|
||||
"tag-identity-studio",
|
||||
"tag-finance",
|
||||
"tag-temporal",
|
||||
"tag-barkhausen",
|
||||
"tag-ilogger",
|
||||
"tag-design-first",
|
||||
";??A5???",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 5098.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q23",
|
||||
"category": "paraphrase",
|
||||
"query": "a mind that compounds instead of resetting each day",
|
||||
"returned": [
|
||||
"project-Goal_setting__alignment__scoring__cadence__Attaches_to_any_imprint_",
|
||||
"tag-performed-values",
|
||||
"tag-turing-test",
|
||||
"tag-sealed",
|
||||
"tag-part-5",
|
||||
"tag-model",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"? ?}&?#??X\b",
|
||||
";??A5???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4652.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q24",
|
||||
"category": "paraphrase",
|
||||
"query": "loved for the unedited self and not the polished exterior",
|
||||
"returned": [
|
||||
"mem-9590ba23-bddb-43e8-a571-68a263c4c364",
|
||||
"project-Imprint__leadership_development__feedback_frameworks__performance__presence_",
|
||||
"bl-a9e57bb2-00a1-4867-ab59-5d9271134b50",
|
||||
"bl-c7793c4a-7630-47fc-a462-d23059087e80",
|
||||
"tag-gateway_platform_neuron-technologies_go_proxy_llm",
|
||||
"tag-fornax",
|
||||
";??A5???",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4653.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q25",
|
||||
"category": "paraphrase",
|
||||
"query": "cheerfulness you arrive at instead of assuming",
|
||||
"returned": [
|
||||
"bl-7a13527b-3e0c-418a-9f37-88fd2152e5ce",
|
||||
"bl-3f57bc69-7285-4f4a-a861-2de52efca058",
|
||||
"bl-5e390b10-8753-4f25-a1a5-b5dbbb002cbf",
|
||||
"tag-ats",
|
||||
"tag-data-model",
|
||||
"tag-aggregation",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
";??A5???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4272.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q26",
|
||||
"category": "paraphrase",
|
||||
"query": "a childhood offering no solid foundation to inherit",
|
||||
"returned": [
|
||||
"tag-memory-model",
|
||||
"tag-storage",
|
||||
"tag-ga4",
|
||||
"tag-potions",
|
||||
"tag-resonance",
|
||||
"tag-neuron",
|
||||
"? ?}&?#??X\b",
|
||||
";??A5???",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4541.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q27",
|
||||
"category": "associative",
|
||||
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
||||
"returned": [
|
||||
"mem-927f41ab-8ede-4f58-acb3-995db16ac775",
|
||||
"mem-dbe80bc2-c602-46b0-b4ea-dd222e52bcde",
|
||||
"mem-82158b02-a180-435d-84f0-0b7ce37511b4",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"tag-upload-window",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4883.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q28",
|
||||
"category": "associative",
|
||||
"query": "Marines hernia sepsis medical ward",
|
||||
"returned": [
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4",
|
||||
"knw-08559f5c-2306-4220-a146-398c74f1643c",
|
||||
"mem-434be7c8-88cb-4039-b79a-1da4ac4de783",
|
||||
"mem-481c769c-68cc-45c7-bc37-c0d9778fa648",
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"bl-9d8f3c5b-4bac-41ce-8ac4-44733f99d6c8",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3237.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q29",
|
||||
"category": "associative",
|
||||
"query": "Sarah Bishop Dyer trailer performance",
|
||||
"returned": [
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"kn-eb1c6d99-d603-4f33-be9a-c63a178690c6",
|
||||
"mem-a3124d5b-2f50-477f-8bb5-06879f5a496c",
|
||||
"bl-7fa1b1a8-b80a-4f28-b162-bfe73765b4f8",
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2457.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q30",
|
||||
"category": "associative",
|
||||
"query": "Swarm Architecture containment lateral worker",
|
||||
"returned": [
|
||||
"mem-265a7107-73b3-4410-9aff-43787d5f473b",
|
||||
"mem-9d1bf963-1b40-4588-bdb3-0432646cc623",
|
||||
"mem-46780047-63a0-4a86-a16b-638b72a7fb8d",
|
||||
"mem-3987d374-3c48-4e8e-b06d-0c363f55ed9c",
|
||||
"bl-e0a0df72-de6e-46ab-800b-e1e3e8dfc387",
|
||||
"tag-__patents____swarm____claim-language____prior-art____filing__",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2640.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q31",
|
||||
"category": "associative",
|
||||
"query": "hope won inside the narrative preface",
|
||||
"returned": [
|
||||
"kn-dca14c4c-4859-47b0-996e-33964ba61a87",
|
||||
"bl-dc8c7e02-eb37-48ae-a6f8-9b512803ae16",
|
||||
"mem-8d1bafe6-209c-456c-9a25-9a927bc5a16d",
|
||||
"bl-0de4e61b-6562-49e5-b7df-ebb809a01723",
|
||||
"bl-8ef1ba6b-3fa0-4dbd-98c5-31665e5694a1",
|
||||
"mem-22f5f665-3ad2-4063-88b0-915849a795f5",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"?Q??m?;��`'",
|
||||
"R^??m?;��?'"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3705.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q32",
|
||||
"category": "associative",
|
||||
"query": "man of the house six years old expectation",
|
||||
"returned": [
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"mem-394cc9e8-049b-45bc-a380-66314f14e367",
|
||||
"bl-ffa22d7e-42a9-4bd2-a428-1d2df243ac93",
|
||||
"bl-4a6746e8-191f-48fc-8bfb-c4dc73b80bcd",
|
||||
"tag-command-pattern",
|
||||
"tag-withholding",
|
||||
"tag-offline",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4223.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q33",
|
||||
"category": "nonsense",
|
||||
"query": "zqxjvw plimforth grebulon",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 1333.9,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q34",
|
||||
"category": "nonsense",
|
||||
"query": "flarnbistle quommetry",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 880.7,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q35",
|
||||
"category": "nonsense",
|
||||
"query": "xxqzzt vurblenacht throom",
|
||||
"returned": [
|
||||
"kn-333542cb-6dab-4662-9725-bf7440d28bf7",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-66a21179-2adc-4b19-a109-880cf4674d7d",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4"
|
||||
],
|
||||
"n_returned": 8,
|
||||
"latency_ms": 1452.7,
|
||||
"error": null,
|
||||
"clean": false,
|
||||
"false_positives": 8
|
||||
},
|
||||
{
|
||||
"id": "q36",
|
||||
"category": "superseded",
|
||||
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
||||
"returned": [
|
||||
"ctx-e5427d7d",
|
||||
"mem-37b57f52-a29a-42cf-a07a-3c5f8a3598dd",
|
||||
"project-worldweaver",
|
||||
"tag-__kotlin____internal-state____pre-reasoning____post-reasoning____compression-ratio____dharma____cultivation__",
|
||||
"tag-import",
|
||||
"tag-__cgi____dharma____cultivation____five-primitives____seed-artifact____agi____intelligence____whitepaper____patent__",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3843.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q37",
|
||||
"category": "superseded",
|
||||
"query": "how many provisional patents does Will actually have",
|
||||
"returned": [
|
||||
"mem-bfe0fafd-2750-4fdc-b773-04e878b3b23f",
|
||||
"art-7bdaff30-5af9-4f0a-93b1-751686f9de3d",
|
||||
"mem-cf07910d-4676-4384-ab97-9cad946cd0b9",
|
||||
"mem-f9da4b43-3724-4bc8-92f8-6f237c89dc4d",
|
||||
"project-Convert_UTC_timestamps_to_Central_time_when_displaying_to_Will__Never_surface_raw_UTC_",
|
||||
"mem-32203649-3213-4d6d-86fd-3d657ac70d77",
|
||||
";??A5???",
|
||||
"? ?}&?#??X\b",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 5264.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q38",
|
||||
"category": "superseded",
|
||||
"query": "is MCP still the live integration layer",
|
||||
"returned": [
|
||||
"project-Define_Iris_as_separate_public_brand__Uncensored_but_principled__Consumer_face_while_Neuron_runs_enterprise_",
|
||||
"project-Imprint__discovery__objection_handling__deal_strategy__pipeline__closing_",
|
||||
"bl-8116da7a-b039-4e08-b8d0-c1c7861f9766",
|
||||
"tag-enterprise",
|
||||
"tag-divisors",
|
||||
"tag-distressed-property",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"???Ͼd??W\b?",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3022.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,942 @@
|
||||
{
|
||||
"label": "act-r3",
|
||||
"soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-act",
|
||||
"soul_md5": "77722f5a9f49494bf735c2a4be1b5dc4",
|
||||
"corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json",
|
||||
"corpus_nodes": 78768,
|
||||
"corpus_edges": 14214,
|
||||
"gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json",
|
||||
"limit": 10,
|
||||
"port": 7903,
|
||||
"wall_clock_s": 112.0,
|
||||
"child_pid": 78802,
|
||||
"child_confirmed_dead": true,
|
||||
"aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.2571428571428571,
|
||||
"recall@5": 0.19291383219954647,
|
||||
"recall@10": 0.244812925170068,
|
||||
"precision@5": 0.08,
|
||||
"mrr@10": 0.266031746031746,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "0/3",
|
||||
"latency_ms_p50": 3220.9,
|
||||
"latency_ms_p95": 4840.2,
|
||||
"latency_ms_max": 5073.6,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 2.6666666666666665
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.42857142857142855,
|
||||
"recall@5": 0.10742630385487528,
|
||||
"recall@10": 0.366921768707483,
|
||||
"mrr@10": 0.47301587301587306
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"id": "q01",
|
||||
"category": "exact_rare",
|
||||
"query": "unjailbreakable",
|
||||
"returned": [
|
||||
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 471.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q02",
|
||||
"category": "exact_rare",
|
||||
"query": "engram-migrate",
|
||||
"returned": [
|
||||
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 548.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q03",
|
||||
"category": "exact_rare",
|
||||
"query": "cartabandonedevent",
|
||||
"returned": [
|
||||
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 470.3,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q04",
|
||||
"category": "exact_rare",
|
||||
"query": "pre-apprenticeship",
|
||||
"returned": [
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 476.3,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q05",
|
||||
"category": "exact_rare",
|
||||
"query": "inferencenodemanager",
|
||||
"returned": [
|
||||
"mem-73969486-143f-4431-b5e6-6845d1cc9848"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 515.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q06",
|
||||
"category": "exact_rare",
|
||||
"query": "clear-eyed",
|
||||
"returned": [
|
||||
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 458.7,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q07",
|
||||
"category": "phrase",
|
||||
"query": "patterns not returns",
|
||||
"returned": [
|
||||
"knw-d357b6bb-ad8a-4791-b516-426aea45fa5b",
|
||||
"kn-69fd6e83-7718-4824-8d66-f49d8954e224",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-c3d9d063-8c5d-45aa-900c-550914b2ff6d",
|
||||
"kn-82be4e41-96c5-4da3-85c0-cee10763d975",
|
||||
"bl-76e878aa-e1fe-468c-bf9c-854097cb7e0b",
|
||||
"art-9887867c-2e21-47c3-9f96-c2dfe5bd4cc1",
|
||||
"bl-31abf75b-998f-4a4f-a6dd-8204119e0451"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1595.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q08",
|
||||
"category": "phrase",
|
||||
"query": "thirty moves",
|
||||
"returned": [
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
||||
"bl-80720fdf-7ce7-4d28-aff8-21028d3a8cfb",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"knw-ed33e669-0790-44cb-a036-958d605c6fea",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"?Z?.\f?0?]P?",
|
||||
"?Q??m?;��`'"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 927.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.125,
|
||||
"recall@10": 0.1875,
|
||||
"precision@5": 0.4,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q09",
|
||||
"category": "phrase",
|
||||
"query": "Grandma Lucas",
|
||||
"returned": [
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 6,
|
||||
"latency_ms": 963.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.5555555555555556,
|
||||
"recall@10": 0.5555555555555556,
|
||||
"precision@5": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q10",
|
||||
"category": "phrase",
|
||||
"query": "Directed Harmonic",
|
||||
"returned": [
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"art-e495c8c5-ad95-4b64-8771-f68aa4cfcd0a",
|
||||
"bl-7aebe936-ac55-4f35-8932-adc5224ff854",
|
||||
"knw-b046991d-5992-4ac4-b854-7d3ac273832c",
|
||||
"mem-ab34c2f7-3243-424b-affa-25555f6cf9cc",
|
||||
"bl-9d53422d-b703-4f1d-860a-8598cb29b792",
|
||||
"bl-455a08cf-5831-4fdb-b42c-b952f2feafb9",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 927.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.1111111111111111,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1
|
||||
},
|
||||
{
|
||||
"id": "q11",
|
||||
"category": "phrase",
|
||||
"query": "Sarah Bishop",
|
||||
"returned": [
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-8dbceb06-431a-416d-a723-e8c75d595154",
|
||||
"mem-a3124d5b-2f50-477f-8bb5-06879f5a496c",
|
||||
"art-94fae615-7cd5-4695-b968-977101b06a51",
|
||||
"mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35",
|
||||
"mem-57164d5f-baf0-4149-957a-379a4e255d1a",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 937.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1111111111111111
|
||||
},
|
||||
{
|
||||
"id": "q12",
|
||||
"category": "phrase",
|
||||
"query": "Directed Autonomous Runtime Modification",
|
||||
"returned": [
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"kn-b2a99cd7-b379-4d9b-a996-e347a02c7bad",
|
||||
"bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae",
|
||||
"kn-8e1bfb48-33a9-45ad-8da7-e0bdaa5d34e7",
|
||||
"mem-cdff0c49-3ac7-4de8-89ec-92d254bd0023",
|
||||
"bl-a7a1428f-db9c-417b-8e2c-713b1f84dc1f",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"?Z?.\f?0?]P?",
|
||||
"bl-145a0985-2382-400f-a7c5-c335c5e30a72"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2451.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.07142857142857142,
|
||||
"recall@10": 0.21428571428571427,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q13",
|
||||
"category": "phrase",
|
||||
"query": "zero-knowledge encrypted backup",
|
||||
"returned": [
|
||||
"mem-7cd90611-88a3-423d-a38a-0db2812952fa",
|
||||
"bl-e20944e5-f4a6-44a0-91b1-73d04ebed120",
|
||||
"mem-64cf3728-674c-404b-965a-b8f8d38bb7bb",
|
||||
"mem-47f72b5b-6e8b-4293-94f1-350197b4809a",
|
||||
"mem-e612f0aa-c2f2-4ee3-bbc7-af2dc826233b",
|
||||
"mem-a5f04e52-91f8-41d2-af27-8bf803621758",
|
||||
"7774a16c-1027-4e3b-a21e-67f1f95a4acd",
|
||||
"? ?}&?#??X\b",
|
||||
"7?e?7???\f3?",
|
||||
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2042.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1
|
||||
},
|
||||
{
|
||||
"id": "q14",
|
||||
"category": "paraphrase",
|
||||
"query": "the elderly relative who passed while he stayed away",
|
||||
"returned": [
|
||||
"knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca",
|
||||
"mem-ea487cb4-ed67-44ce-8402-b56bb28468d4",
|
||||
"art-c71aef51-026f-4d63-80e9-2a0ec0dc3865",
|
||||
"bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60",
|
||||
"bl-2694b588-a6e3-43de-861c-fa7b0ec7e7fd",
|
||||
"bl-14883d81-f7cb-46dd-82c2-a6e6980264e5",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4698.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q15",
|
||||
"category": "paraphrase",
|
||||
"query": "a soldier sidelined by illness who refused to quit",
|
||||
"returned": [
|
||||
"bl-fd047ce9-ae21-4b3e-b3ab-ece0c9592f7f",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0",
|
||||
"bl-6172d035-dd94-4776-afdd-d8915f6fc375",
|
||||
"bl-5bb8dedf-8498-4a9b-acdc-31cc9c738f2a",
|
||||
"tag-cultivated-general-intelligence",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 5073.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q16",
|
||||
"category": "paraphrase",
|
||||
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
||||
"returned": [
|
||||
"bl-e5635b1a-c5d0-4caa-bcc8-6a726ea43685",
|
||||
"bl-fc893be3-e6b4-4ef6-93b0-d54ca5f89083",
|
||||
"tag-project-structure",
|
||||
"tag-anthropic-contrast",
|
||||
"tag-voice-training",
|
||||
"tag-ebd",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"bl-9d53422d-b703-4f1d-860a-8598cb29b792"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4117.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q17",
|
||||
"category": "paraphrase",
|
||||
"query": "a tight payload beats a bloated one",
|
||||
"returned": [
|
||||
"bl-d24fcce8-2b55-426f-867a-db3958a622d3",
|
||||
"tag-phase-3",
|
||||
"bl-57c5cf6b-81a5-4558-9902-5c02981fe273",
|
||||
"tag-guilds",
|
||||
"tag-kids",
|
||||
"tag-coexistence",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
||||
"mem-cdff0c49-3ac7-4de8-89ec-92d254bd0023"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3254.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q18",
|
||||
"category": "paraphrase",
|
||||
"query": "if you are able and nobody is coming the job is yours",
|
||||
"returned": [
|
||||
"bl-07375bf9-a169-42cd-adb3-7d32b25982f0",
|
||||
"mem-0328c3cb-4550-4ce4-9284-152e832f08f6",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992",
|
||||
"bl-e98cdd4c-01b5-459e-9036-3578cd5d975a",
|
||||
"tag-stable-diffusion",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4081.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q19",
|
||||
"category": "paraphrase",
|
||||
"query": "learning is the wealth creditors cannot seize",
|
||||
"returned": [
|
||||
"mem-e32ba5a7-c147-4dc0-9479-b720d768eda6",
|
||||
"bl-b019ce7a-1b21-436e-812d-032f50c6c45f",
|
||||
"mem-c7a77457-478d-4eb0-a116-67205a0066a4",
|
||||
"bl-1b20e9bc-eb37-4907-8d63-e311fd61eab8",
|
||||
"bl-aa762207-920d-45ab-b2a3-2f8154d7ef9b",
|
||||
"tag-misalignment",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3687.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q20",
|
||||
"category": "paraphrase",
|
||||
"query": "reliability proven by track record not assertion",
|
||||
"returned": [
|
||||
"kn-48a01973-a025-471d-950f-b93e6a426d82",
|
||||
"mem-23d22bc1-a097-446b-8f11-8aff099e0b76",
|
||||
"mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8",
|
||||
"mem-ce5a2ffc-ad39-4728-9ac6-76fef507d5da",
|
||||
"project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_",
|
||||
"mem-e321e54e-8bb3-4596-b13d-bb093d6b149d",
|
||||
"? ?}&?#??X\b",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4659.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q21",
|
||||
"category": "paraphrase",
|
||||
"query": "boundaries that enable instead of confine",
|
||||
"returned": [
|
||||
"kn-79056192-7de8-486c-9565-f128439a6fcb",
|
||||
"bl-f6236350-f7b8-4f4f-a702-9eef2eb76e4b",
|
||||
"bl-7f33f1bc-99fa-4906-889f-a42375beea20",
|
||||
"mem-37b57f52-a29a-42cf-a07a-3c5f8a3598dd",
|
||||
"mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1",
|
||||
"tag-__cultivation-metric____internal-state____dharma____evidence____novel-idea____gap-compression____values____microsoft__",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"knw-b046991d-5992-4ac4-b854-7d3ac273832c"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3960.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q22",
|
||||
"category": "paraphrase",
|
||||
"query": "what shifts tells you where to cut a system apart",
|
||||
"returned": [
|
||||
"knw-723551f5-1950-42a3-8b89-b6a06913cef0",
|
||||
"tag-identity-studio",
|
||||
"tag-finance",
|
||||
"tag-temporal",
|
||||
"tag-barkhausen",
|
||||
"tag-ilogger",
|
||||
"tag-design-first",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4840.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q23",
|
||||
"category": "paraphrase",
|
||||
"query": "a mind that compounds instead of resetting each day",
|
||||
"returned": [
|
||||
"project-Goal_setting__alignment__scoring__cadence__Attaches_to_any_imprint_",
|
||||
"tag-performed-values",
|
||||
"tag-turing-test",
|
||||
"tag-sealed",
|
||||
"tag-part-5",
|
||||
"tag-model",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4493.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q24",
|
||||
"category": "paraphrase",
|
||||
"query": "loved for the unedited self and not the polished exterior",
|
||||
"returned": [
|
||||
"mem-9590ba23-bddb-43e8-a571-68a263c4c364",
|
||||
"project-Imprint__leadership_development__feedback_frameworks__performance__presence_",
|
||||
"bl-a9e57bb2-00a1-4867-ab59-5d9271134b50",
|
||||
"bl-c7793c4a-7630-47fc-a462-d23059087e80",
|
||||
"tag-gateway_platform_neuron-technologies_go_proxy_llm",
|
||||
"tag-fornax",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"mem-ab34c2f7-3243-424b-affa-25555f6cf9cc",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4509.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q25",
|
||||
"category": "paraphrase",
|
||||
"query": "cheerfulness you arrive at instead of assuming",
|
||||
"returned": [
|
||||
"bl-7a13527b-3e0c-418a-9f37-88fd2152e5ce",
|
||||
"bl-3f57bc69-7285-4f4a-a861-2de52efca058",
|
||||
"bl-5e390b10-8753-4f25-a1a5-b5dbbb002cbf",
|
||||
"tag-ats",
|
||||
"tag-data-model",
|
||||
"tag-aggregation",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3624.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q26",
|
||||
"category": "paraphrase",
|
||||
"query": "a childhood offering no solid foundation to inherit",
|
||||
"returned": [
|
||||
"tag-memory-model",
|
||||
"tag-storage",
|
||||
"tag-ga4",
|
||||
"tag-potions",
|
||||
"tag-resonance",
|
||||
"tag-neuron",
|
||||
"? ?}&?#??X\b",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4093.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q27",
|
||||
"category": "associative",
|
||||
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
||||
"returned": [
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"mem-927f41ab-8ede-4f58-acb3-995db16ac775",
|
||||
"mem-82158b02-a180-435d-84f0-0b7ce37511b4",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"tag-upload-window",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4235.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q28",
|
||||
"category": "associative",
|
||||
"query": "Marines hernia sepsis medical ward",
|
||||
"returned": [
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4",
|
||||
"mem-0f31141d-3ac5-44b2-9942-be7e4e6feb79",
|
||||
"knw-08559f5c-2306-4220-a146-398c74f1643c",
|
||||
"mem-434be7c8-88cb-4039-b79a-1da4ac4de783",
|
||||
"mem-481c769c-68cc-45c7-bc37-c0d9778fa648",
|
||||
"bl-9d8f3c5b-4bac-41ce-8ac4-44733f99d6c8",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3220.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q29",
|
||||
"category": "associative",
|
||||
"query": "Sarah Bishop Dyer trailer performance",
|
||||
"returned": [
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"kn-eb1c6d99-d603-4f33-be9a-c63a178690c6",
|
||||
"mem-5e7f6ddd-c818-4ad3-b564-54ae278e9976",
|
||||
"bl-7fa1b1a8-b80a-4f28-b162-bfe73765b4f8",
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c",
|
||||
"tag-sarah",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2336.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q30",
|
||||
"category": "associative",
|
||||
"query": "Swarm Architecture containment lateral worker",
|
||||
"returned": [
|
||||
"mem-265a7107-73b3-4410-9aff-43787d5f473b",
|
||||
"mem-9d1bf963-1b40-4588-bdb3-0432646cc623",
|
||||
"mem-46780047-63a0-4a86-a16b-638b72a7fb8d",
|
||||
"mem-3987d374-3c48-4e8e-b06d-0c363f55ed9c",
|
||||
"bl-e0a0df72-de6e-46ab-800b-e1e3e8dfc387",
|
||||
"tag-__patents____swarm____claim-language____prior-art____filing__",
|
||||
"mem-ab34c2f7-3243-424b-affa-25555f6cf9cc",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2348.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q31",
|
||||
"category": "associative",
|
||||
"query": "hope won inside the narrative preface",
|
||||
"returned": [
|
||||
"kn-dca14c4c-4859-47b0-996e-33964ba61a87",
|
||||
"bl-e148d23c-24e8-4122-9915-d1c11f22052f",
|
||||
"bl-dc8c7e02-eb37-48ae-a6f8-9b512803ae16",
|
||||
"mem-8d1bafe6-209c-456c-9a25-9a927bc5a16d",
|
||||
"mem-22f5f665-3ad2-4063-88b0-915849a795f5",
|
||||
"tag-dark-theme",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"?Q??m?;��`'",
|
||||
"R^??m?;��?'"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3511.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q32",
|
||||
"category": "associative",
|
||||
"query": "man of the house six years old expectation",
|
||||
"returned": [
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"bl-ffa22d7e-42a9-4bd2-a428-1d2df243ac93",
|
||||
"mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9",
|
||||
"bl-452a4710-3d2b-4e0f-9413-49a66423bc9a",
|
||||
"bl-4a6746e8-191f-48fc-8bfb-c4dc73b80bcd",
|
||||
"tag-withholding",
|
||||
"tag-offline",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4160.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q33",
|
||||
"category": "nonsense",
|
||||
"query": "zqxjvw plimforth grebulon",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 1318.6,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q34",
|
||||
"category": "nonsense",
|
||||
"query": "flarnbistle quommetry",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 875.7,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q35",
|
||||
"category": "nonsense",
|
||||
"query": "xxqzzt vurblenacht throom",
|
||||
"returned": [
|
||||
"kn-333542cb-6dab-4662-9725-bf7440d28bf7",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-66a21179-2adc-4b19-a109-880cf4674d7d",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4"
|
||||
],
|
||||
"n_returned": 8,
|
||||
"latency_ms": 1393.4,
|
||||
"error": null,
|
||||
"clean": false,
|
||||
"false_positives": 8
|
||||
},
|
||||
{
|
||||
"id": "q36",
|
||||
"category": "superseded",
|
||||
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
||||
"returned": [
|
||||
"knw-b046991d-5992-4ac4-b854-7d3ac273832c",
|
||||
"ctx-e5427d7d",
|
||||
"project-worldweaver",
|
||||
"tag-__kotlin____internal-state____pre-reasoning____post-reasoning____compression-ratio____dharma____cultivation__",
|
||||
"tag-import",
|
||||
"tag-enterprise",
|
||||
"tag-__cgi____dharma____cultivation____five-primitives____seed-artifact____agi____intelligence____whitepaper____patent__",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3775.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q37",
|
||||
"category": "superseded",
|
||||
"query": "how many provisional patents does Will actually have",
|
||||
"returned": [
|
||||
"mem-bfe0fafd-2750-4fdc-b773-04e878b3b23f",
|
||||
"art-7bdaff30-5af9-4f0a-93b1-751686f9de3d",
|
||||
"mem-cf07910d-4676-4384-ab97-9cad946cd0b9",
|
||||
"mem-f9da4b43-3724-4bc8-92f8-6f237c89dc4d",
|
||||
"project-Convert_UTC_timestamps_to_Central_time_when_displaying_to_Will__Never_surface_raw_UTC_",
|
||||
"mem-32203649-3213-4d6d-86fd-3d657ac70d77",
|
||||
"? ?}&?#??X\b",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4857.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q38",
|
||||
"category": "superseded",
|
||||
"query": "is MCP still the live integration layer",
|
||||
"returned": [
|
||||
"bl-0de4e61b-6562-49e5-b7df-ebb809a01723",
|
||||
"project-Imprint__discovery__objection_handling__deal_strategy__pipeline__closing_",
|
||||
"bl-8116da7a-b039-4e08-b8d0-c1c7861f9766",
|
||||
"bl-8ef1ba6b-3fa0-4dbd-98c5-31665e5694a1",
|
||||
"tag-divisors",
|
||||
"tag-distressed-property",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"knw-b046991d-5992-4ac4-b854-7d3ac273832c",
|
||||
"???Ͼd??W\b?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2858.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,945 @@
|
||||
{
|
||||
"label": "baseline-embcorpus",
|
||||
"soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-baseline",
|
||||
"soul_md5": "5cc9521734907cf2da30f0af94498c06",
|
||||
"corpus": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/corpus-embedded.json",
|
||||
"corpus_nodes": 78768,
|
||||
"corpus_edges": 14214,
|
||||
"gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json",
|
||||
"limit": 10,
|
||||
"port": 7893,
|
||||
"wall_clock_s": 48.4,
|
||||
"child_pid": 85995,
|
||||
"child_confirmed_dead": true,
|
||||
"aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.34285714285714286,
|
||||
"recall@5": 0.26947278911564626,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"precision@5": 0.12000000000000001,
|
||||
"mrr@10": 0.2943197278911564,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "1/3",
|
||||
"latency_ms_p50": 1145.9,
|
||||
"latency_ms_p95": 1574.3,
|
||||
"latency_ms_max": 1634.2,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 3.3333333333333335
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.8571428571428571,
|
||||
"recall@5": 0.4902210884353741,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.5965986394557822
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"mrr@10": 0.041666666666666664,
|
||||
"outranks": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"id": "q01",
|
||||
"category": "exact_rare",
|
||||
"query": "unjailbreakable",
|
||||
"returned": [
|
||||
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 229.0,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q02",
|
||||
"category": "exact_rare",
|
||||
"query": "engram-migrate",
|
||||
"returned": [
|
||||
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 262.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q03",
|
||||
"category": "exact_rare",
|
||||
"query": "cartabandonedevent",
|
||||
"returned": [
|
||||
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 230.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q04",
|
||||
"category": "exact_rare",
|
||||
"query": "pre-apprenticeship",
|
||||
"returned": [
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 229.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q05",
|
||||
"category": "exact_rare",
|
||||
"query": "inferencenodemanager",
|
||||
"returned": [
|
||||
"mem-73969486-143f-4431-b5e6-6845d1cc9848"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 250.4,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q06",
|
||||
"category": "exact_rare",
|
||||
"query": "clear-eyed",
|
||||
"returned": [
|
||||
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 227.7,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q07",
|
||||
"category": "phrase",
|
||||
"query": "patterns not returns",
|
||||
"returned": [
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"?ǚ?7??????",
|
||||
"mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"????7???Ջ3",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b",
|
||||
"7?e?7???\f3?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 532.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.3333333333333333
|
||||
},
|
||||
{
|
||||
"id": "q08",
|
||||
"category": "phrase",
|
||||
"query": "thirty moves",
|
||||
"returned": [
|
||||
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
||||
"knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6",
|
||||
"Z[?<S???H??",
|
||||
"rQ??m?;?x?'",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 457.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3125,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q09",
|
||||
"category": "phrase",
|
||||
"query": "Grandma Lucas",
|
||||
"returned": [
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 472.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3333333333333333,
|
||||
"recall@10": 0.5555555555555556,
|
||||
"precision@5": 0.6,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q10",
|
||||
"category": "phrase",
|
||||
"query": "Directed Harmonic",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"ԍ????X????",
|
||||
"bl-dcee1887-34c4-4ffa-9119-1e291685ba08",
|
||||
"?ǚ?7??????",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf",
|
||||
"????7???Ջ3",
|
||||
"??????X??2c",
|
||||
"g?e?7???'c?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 446.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.1111111111111111,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.14285714285714285
|
||||
},
|
||||
{
|
||||
"id": "q11",
|
||||
"category": "phrase",
|
||||
"query": "Sarah Bishop",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35",
|
||||
"mem-b8ecd23e-77ce-42f7-984c-f51453fec16d"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 462.6,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.5,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.2
|
||||
},
|
||||
{
|
||||
"id": "q12",
|
||||
"category": "phrase",
|
||||
"query": "Directed Autonomous Runtime Modification",
|
||||
"returned": [
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"bl-145a0985-2382-400f-a7c5-c335c5e30a72",
|
||||
"mem-82b93b21-a865-410f-9ec1-fc54121d9bb5",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"mem-e6327f52-2bda-4ce7-9471-2fffd1e172de",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 832.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.2857142857142857,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 0.8,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q13",
|
||||
"category": "phrase",
|
||||
"query": "zero-knowledge encrypted backup",
|
||||
"returned": [
|
||||
"7774a16c-1027-4e3b-a21e-67f1f95a4acd",
|
||||
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a",
|
||||
"mem-dba009a2-d2ea-4f5a-b9e8-0f04bc9ab32f",
|
||||
"mem-a3c97012-5fa3-4915-a839-2c75c72005e0",
|
||||
"mem-7cd90611-88a3-423d-a38a-0db2812952fa",
|
||||
"bl-07375bf9-a169-42cd-adb3-7d32b25982f0",
|
||||
"bl-ec84b63d-b278-4944-8d7f-4aa7a51c0315",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 686.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q14",
|
||||
"category": "paraphrase",
|
||||
"query": "the elderly relative who passed while he stayed away",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"????7???Ջ3",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1550.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q15",
|
||||
"category": "paraphrase",
|
||||
"query": "a soldier sidelined by illness who refused to quit",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a31e1001-342e-4deb-a2e6-6d02d1f22dee",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1634.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q16",
|
||||
"category": "paraphrase",
|
||||
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"?of?7???",
|
||||
"????7???Ջ3",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"kn-81c24d13-a73b-4767-819c-dafaacc1498e",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1322.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q17",
|
||||
"category": "paraphrase",
|
||||
"query": "a tight payload beats a bloated one",
|
||||
"returned": [
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"??o?'?B???k",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1041.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q18",
|
||||
"category": "paraphrase",
|
||||
"query": "if you are able and nobody is coming the job is yours",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"imp-dce1da0f-8776-4a9e-972b-33411a7ca138",
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-83bb86c6-521d-416c-a86e-6e29c2d8f102",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1350.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q19",
|
||||
"category": "paraphrase",
|
||||
"query": "learning is the wealth creditors cannot seize",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"??o?'?B???k",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-0625e393-067c-4bba-8389-7e1b79265142",
|
||||
"ע?RGk?\tH(?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1213.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q20",
|
||||
"category": "paraphrase",
|
||||
"query": "reliability proven by track record not assertion",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1553.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q21",
|
||||
"category": "paraphrase",
|
||||
"query": "boundaries that enable instead of confine",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"??????X??2c",
|
||||
"ԍ????X????",
|
||||
"dR????X?-?S",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"dz????Xƹ?i",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1319.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q22",
|
||||
"category": "paraphrase",
|
||||
"query": "what shifts tells you where to cut a system apart",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"dR????X?-?S",
|
||||
"dz????Xƹ?i",
|
||||
"ԍ????X????",
|
||||
"??????X??2c",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1574.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q23",
|
||||
"category": "paraphrase",
|
||||
"query": "a mind that compounds instead of resetting each day",
|
||||
"returned": [
|
||||
"?ǚ?7??????",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"????7???Ջ3",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"%???2??jH??",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1441.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q24",
|
||||
"category": "paraphrase",
|
||||
"query": "loved for the unedited self and not the polished exterior",
|
||||
"returned": [
|
||||
"mem-bbb126a1-b297-42bb-86be-796871829c94",
|
||||
"mem-45022957-2d78-48aa-a714-16d6eca52e0f",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"'?T?a\"B~-?8"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1499.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q25",
|
||||
"category": "paraphrase",
|
||||
"query": "cheerfulness you arrive at instead of assuming",
|
||||
"returned": [
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"?ǚ?7??????",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"7?e?7???\f3?",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"??S?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"??f?7???",
|
||||
"??S?7???",
|
||||
"knw-5578cb21-e899-4822-b7f4-0d96fa094e3d"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1197.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q26",
|
||||
"category": "paraphrase",
|
||||
"query": "a childhood offering no solid foundation to inherit",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"??o?'?B???k",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1311.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q27",
|
||||
"category": "associative",
|
||||
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"?ǚ?7??????"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1422.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q28",
|
||||
"category": "associative",
|
||||
"query": "Marines hernia sepsis medical ward",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-e0bdf5d8-d163-491f-b649-453fee8b721d",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1077.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q29",
|
||||
"category": "associative",
|
||||
"query": "Sarah Bishop Dyer trailer performance",
|
||||
"returned": [
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1149.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q30",
|
||||
"category": "associative",
|
||||
"query": "Swarm Architecture containment lateral worker",
|
||||
"returned": [
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43",
|
||||
"8cbb60c5-4999-4ec1-8682-2592aedc4249",
|
||||
"bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae",
|
||||
"7?e?7???\f3?",
|
||||
"kn-6f248a50-355b-47bb-aec8-e0e646a9b077",
|
||||
"? ?}&?#??X\b",
|
||||
"h??I?cB?Q??",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1145.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q31",
|
||||
"category": "associative",
|
||||
"query": "hope won inside the narrative preface",
|
||||
"returned": [
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"7?e?7???\f3?",
|
||||
"rQ??m?;?x?'",
|
||||
"?Q??m?;?u?'",
|
||||
"R^??m?;��?'",
|
||||
"?Q??m?;��`'",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1177.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q32",
|
||||
"category": "associative",
|
||||
"query": "man of the house six years old expectation",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1382.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q33",
|
||||
"category": "nonsense",
|
||||
"query": "zqxjvw plimforth grebulon",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 660.6,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q34",
|
||||
"category": "nonsense",
|
||||
"query": "flarnbistle quommetry",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 432.5,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q35",
|
||||
"category": "nonsense",
|
||||
"query": "xxqzzt vurblenacht throom",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"kn-66a21179-2adc-4b19-a109-880cf4674d7d",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 676.9,
|
||||
"error": null,
|
||||
"clean": false,
|
||||
"false_positives": 10
|
||||
},
|
||||
{
|
||||
"id": "q36",
|
||||
"category": "superseded",
|
||||
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"mem-80d7416b-20e9-48a0-b176-b215527e2f56",
|
||||
"mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1252.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.125,
|
||||
"outranks": true,
|
||||
"rank_correct": 8,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q37",
|
||||
"category": "superseded",
|
||||
"query": "how many provisional patents does Will actually have",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"?ǚ?7??????",
|
||||
"[?MO5????G",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1624.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q38",
|
||||
"category": "superseded",
|
||||
"query": "is MCP still the live integration layer",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"%???2??jH??",
|
||||
"???Ͼd??W\b?",
|
||||
"??o?'?B???k",
|
||||
"? ?}&?#??X\b",
|
||||
"63307ac5-cf6b-46e0-8296-07503b461cfa",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 946.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,942 @@
|
||||
{
|
||||
"label": "act-r1",
|
||||
"soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-act",
|
||||
"soul_md5": "77722f5a9f49494bf735c2a4be1b5dc4",
|
||||
"corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json",
|
||||
"corpus_nodes": 78768,
|
||||
"corpus_edges": 14214,
|
||||
"gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json",
|
||||
"limit": 10,
|
||||
"port": 7894,
|
||||
"wall_clock_s": 112.3,
|
||||
"child_pid": 78648,
|
||||
"child_confirmed_dead": true,
|
||||
"aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.22857142857142856,
|
||||
"recall@5": 0.19087301587301586,
|
||||
"recall@10": 0.24277210884353742,
|
||||
"precision@5": 0.07428571428571429,
|
||||
"mrr@10": 0.24154195011337865,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "0/3",
|
||||
"latency_ms_p50": 3208.8,
|
||||
"latency_ms_p95": 4851.9,
|
||||
"latency_ms_max": 5078.8,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 2.6666666666666665
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.2857142857142857,
|
||||
"recall@5": 0.09722222222222222,
|
||||
"recall@10": 0.3567176870748299,
|
||||
"mrr@10": 0.3505668934240363
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"id": "q01",
|
||||
"category": "exact_rare",
|
||||
"query": "unjailbreakable",
|
||||
"returned": [
|
||||
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 468.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q02",
|
||||
"category": "exact_rare",
|
||||
"query": "engram-migrate",
|
||||
"returned": [
|
||||
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 569.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q03",
|
||||
"category": "exact_rare",
|
||||
"query": "cartabandonedevent",
|
||||
"returned": [
|
||||
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 479.7,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q04",
|
||||
"category": "exact_rare",
|
||||
"query": "pre-apprenticeship",
|
||||
"returned": [
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 470.4,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q05",
|
||||
"category": "exact_rare",
|
||||
"query": "inferencenodemanager",
|
||||
"returned": [
|
||||
"mem-73969486-143f-4431-b5e6-6845d1cc9848"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 499.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q06",
|
||||
"category": "exact_rare",
|
||||
"query": "clear-eyed",
|
||||
"returned": [
|
||||
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 452.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q07",
|
||||
"category": "phrase",
|
||||
"query": "patterns not returns",
|
||||
"returned": [
|
||||
"knw-d357b6bb-ad8a-4791-b516-426aea45fa5b",
|
||||
"kn-69fd6e83-7718-4824-8d66-f49d8954e224",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-eb1c6d99-d603-4f33-be9a-c63a178690c6",
|
||||
"knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca",
|
||||
"kn-b2a99cd7-b379-4d9b-a996-e347a02c7bad",
|
||||
"bl-76e878aa-e1fe-468c-bf9c-854097cb7e0b",
|
||||
"art-c71aef51-026f-4d63-80e9-2a0ec0dc3865"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1592.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q08",
|
||||
"category": "phrase",
|
||||
"query": "thirty moves",
|
||||
"returned": [
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
||||
"bl-80720fdf-7ce7-4d28-aff8-21028d3a8cfb",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"knw-ed33e669-0790-44cb-a036-958d605c6fea",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"?Z?.\f?0?]P?",
|
||||
"?Q??m?;��`'"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 956.6,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.125,
|
||||
"recall@10": 0.1875,
|
||||
"precision@5": 0.4,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q09",
|
||||
"category": "phrase",
|
||||
"query": "Grandma Lucas",
|
||||
"returned": [
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 6,
|
||||
"latency_ms": 970.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.5555555555555556,
|
||||
"recall@10": 0.5555555555555556,
|
||||
"precision@5": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q10",
|
||||
"category": "phrase",
|
||||
"query": "Directed Harmonic",
|
||||
"returned": [
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"art-e495c8c5-ad95-4b64-8771-f68aa4cfcd0a",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"bl-7aebe936-ac55-4f35-8932-adc5224ff854",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"bl-9d53422d-b703-4f1d-860a-8598cb29b792",
|
||||
"mem-60778715-758c-4677-933d-fc39b8f94152",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 920.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.1111111111111111,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1
|
||||
},
|
||||
{
|
||||
"id": "q11",
|
||||
"category": "phrase",
|
||||
"query": "Sarah Bishop",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-8dbceb06-431a-416d-a723-e8c75d595154",
|
||||
"mem-a3124d5b-2f50-477f-8bb5-06879f5a496c",
|
||||
"knw-528dbc37-eabc-4b75-a7a5-65bf38d6018a",
|
||||
"art-94fae615-7cd5-4695-b968-977101b06a51",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 925.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1111111111111111
|
||||
},
|
||||
{
|
||||
"id": "q12",
|
||||
"category": "phrase",
|
||||
"query": "Directed Autonomous Runtime Modification",
|
||||
"returned": [
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"mem-46780047-63a0-4a86-a16b-638b72a7fb8d",
|
||||
"kn-8e1bfb48-33a9-45ad-8da7-e0bdaa5d34e7",
|
||||
"mem-cdff0c49-3ac7-4de8-89ec-92d254bd0023",
|
||||
"bl-a7a1428f-db9c-417b-8e2c-713b1f84dc1f",
|
||||
"mem-f823e835-313f-4282-b4b3-ce527ffc2f7a",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"?Z?.\f?0?]P?",
|
||||
"bl-145a0985-2382-400f-a7c5-c335c5e30a72"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2455.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.14285714285714285,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.14285714285714285
|
||||
},
|
||||
{
|
||||
"id": "q13",
|
||||
"category": "phrase",
|
||||
"query": "zero-knowledge encrypted backup",
|
||||
"returned": [
|
||||
"art-92e1837c-5919-42d0-bbb0-4d924d7b2864",
|
||||
"bl-e20944e5-eb16-4ab3-a84d-111e0fc817fa",
|
||||
"bl-e20944e5-f4a6-44a0-91b1-73d04ebed120",
|
||||
"mem-47f72b5b-6e8b-4293-94f1-350197b4809a",
|
||||
"mem-e612f0aa-c2f2-4ee3-bbc7-af2dc826233b",
|
||||
"mem-a5f04e52-91f8-41d2-af27-8bf803621758",
|
||||
"7774a16c-1027-4e3b-a21e-67f1f95a4acd",
|
||||
"? ?}&?#??X\b",
|
||||
"7?e?7???\f3?",
|
||||
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2037.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1
|
||||
},
|
||||
{
|
||||
"id": "q14",
|
||||
"category": "paraphrase",
|
||||
"query": "the elderly relative who passed while he stayed away",
|
||||
"returned": [
|
||||
"kn-82be4e41-96c5-4da3-85c0-cee10763d975",
|
||||
"mem-ea487cb4-ed67-44ce-8402-b56bb28468d4",
|
||||
"mem-23d22bc1-a097-446b-8f11-8aff099e0b76",
|
||||
"bl-874d1c2b-c55b-4afb-9601-922a9297e859",
|
||||
"bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60",
|
||||
"bl-2694b588-a6e3-43de-861c-fa7b0ec7e7fd",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4720.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q15",
|
||||
"category": "paraphrase",
|
||||
"query": "a soldier sidelined by illness who refused to quit",
|
||||
"returned": [
|
||||
"mem-0328c3cb-4550-4ce4-9284-152e832f08f6",
|
||||
"bl-07375bf9-a169-42cd-adb3-7d32b25982f0",
|
||||
"bl-fd047ce9-ae21-4b3e-b3ab-ece0c9592f7f",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"bl-6172d035-dd94-4776-afdd-d8915f6fc375",
|
||||
"bl-5bb8dedf-8498-4a9b-acdc-31cc9c738f2a",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 5078.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q16",
|
||||
"category": "paraphrase",
|
||||
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
||||
"returned": [
|
||||
"bl-e5635b1a-c5d0-4caa-bcc8-6a726ea43685",
|
||||
"bl-fc893be3-e6b4-4ef6-93b0-d54ca5f89083",
|
||||
"tag-project-structure",
|
||||
"tag-anthropic-contrast",
|
||||
"tag-voice-training",
|
||||
"tag-ebd",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"bl-9d53422d-b703-4f1d-860a-8598cb29b792",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4118.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q17",
|
||||
"category": "paraphrase",
|
||||
"query": "a tight payload beats a bloated one",
|
||||
"returned": [
|
||||
"bl-d24fcce8-2b55-426f-867a-db3958a622d3",
|
||||
"tag-phase-3",
|
||||
"tag-identity-studio",
|
||||
"tag-kids",
|
||||
"tag-coexistence",
|
||||
"tag-cultivated-general-intelligence",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"mem-cdff0c49-3ac7-4de8-89ec-92d254bd0023"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3241.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q18",
|
||||
"category": "paraphrase",
|
||||
"query": "if you are able and nobody is coming the job is yours",
|
||||
"returned": [
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992",
|
||||
"bl-b019ce7a-1b21-436e-812d-032f50c6c45f",
|
||||
"bl-e98cdd4c-01b5-459e-9036-3578cd5d975a",
|
||||
"bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0",
|
||||
"tag-stable-diffusion",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4090.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q19",
|
||||
"category": "paraphrase",
|
||||
"query": "learning is the wealth creditors cannot seize",
|
||||
"returned": [
|
||||
"mem-e321e54e-8bb3-4596-b13d-bb093d6b149d",
|
||||
"mem-e32ba5a7-c147-4dc0-9479-b720d768eda6",
|
||||
"mem-c7a77457-478d-4eb0-a116-67205a0066a4",
|
||||
"bl-1b20e9bc-eb37-4907-8d63-e311fd61eab8",
|
||||
"bl-aa762207-920d-45ab-b2a3-2f8154d7ef9b",
|
||||
"tag-misalignment",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3671.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q20",
|
||||
"category": "paraphrase",
|
||||
"query": "reliability proven by track record not assertion",
|
||||
"returned": [
|
||||
"kn-48a01973-a025-471d-950f-b93e6a426d82",
|
||||
"bl-7f33f1bc-99fa-4906-889f-a42375beea20",
|
||||
"mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8",
|
||||
"mem-ce5a2ffc-ad39-4728-9ac6-76fef507d5da",
|
||||
"project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_",
|
||||
"tag-provenance",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"art-e495c8c5-ad95-4b64-8771-f68aa4cfcd0a"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4674.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q21",
|
||||
"category": "paraphrase",
|
||||
"query": "boundaries that enable instead of confine",
|
||||
"returned": [
|
||||
"kn-79056192-7de8-486c-9565-f128439a6fcb",
|
||||
"bl-f6236350-f7b8-4f4f-a702-9eef2eb76e4b",
|
||||
"mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9",
|
||||
"mem-37b57f52-a29a-42cf-a07a-3c5f8a3598dd",
|
||||
"mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1",
|
||||
"tag-__cultivation-metric____internal-state____dharma____evidence____novel-idea____gap-compression____values____microsoft__",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3974.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q22",
|
||||
"category": "paraphrase",
|
||||
"query": "what shifts tells you where to cut a system apart",
|
||||
"returned": [
|
||||
"knw-723551f5-1950-42a3-8b89-b6a06913cef0",
|
||||
"bl-57c5cf6b-81a5-4558-9902-5c02981fe273",
|
||||
"tag-guilds",
|
||||
"tag-finance",
|
||||
"tag-temporal",
|
||||
"tag-barkhausen",
|
||||
"tag-ilogger",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4851.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q23",
|
||||
"category": "paraphrase",
|
||||
"query": "a mind that compounds instead of resetting each day",
|
||||
"returned": [
|
||||
"project-Goal_setting__alignment__scoring__cadence__Attaches_to_any_imprint_",
|
||||
"tag-turing-test",
|
||||
"tag-sealed",
|
||||
"tag-design-first",
|
||||
"tag-part-5",
|
||||
"tag-model",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4505.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q24",
|
||||
"category": "paraphrase",
|
||||
"query": "loved for the unedited self and not the polished exterior",
|
||||
"returned": [
|
||||
"mem-9590ba23-bddb-43e8-a571-68a263c4c364",
|
||||
"bl-a9e57bb2-00a1-4867-ab59-5d9271134b50",
|
||||
"tag-performed-values",
|
||||
"bl-c7793c4a-7630-47fc-a462-d23059087e80",
|
||||
"tag-gateway_platform_neuron-technologies_go_proxy_llm",
|
||||
"tag-fornax",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4494.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q25",
|
||||
"category": "paraphrase",
|
||||
"query": "cheerfulness you arrive at instead of assuming",
|
||||
"returned": [
|
||||
"bl-7a13527b-3e0c-418a-9f37-88fd2152e5ce",
|
||||
"bl-3f57bc69-7285-4f4a-a861-2de52efca058",
|
||||
"bl-5e390b10-8753-4f25-a1a5-b5dbbb002cbf",
|
||||
"tag-ats",
|
||||
"tag-data-model",
|
||||
"tag-aggregation",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3619.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q26",
|
||||
"category": "paraphrase",
|
||||
"query": "a childhood offering no solid foundation to inherit",
|
||||
"returned": [
|
||||
"tag-memory-model",
|
||||
"tag-storage",
|
||||
"tag-ga4",
|
||||
"tag-potions",
|
||||
"tag-resonance",
|
||||
"tag-neuron",
|
||||
"? ?}&?#??X\b",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4107.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q27",
|
||||
"category": "associative",
|
||||
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
||||
"returned": [
|
||||
"mem-927f41ab-8ede-4f58-acb3-995db16ac775",
|
||||
"mem-dbe80bc2-c602-46b0-b4ea-dd222e52bcde",
|
||||
"mem-82158b02-a180-435d-84f0-0b7ce37511b4",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"tag-upload-window",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4231.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q28",
|
||||
"category": "associative",
|
||||
"query": "Marines hernia sepsis medical ward",
|
||||
"returned": [
|
||||
"knw-08559f5c-2306-4220-a146-398c74f1643c",
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4",
|
||||
"mem-434be7c8-88cb-4039-b79a-1da4ac4de783",
|
||||
"mem-481c769c-68cc-45c7-bc37-c0d9778fa648",
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"bl-9d8f3c5b-4bac-41ce-8ac4-44733f99d6c8",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3208.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q29",
|
||||
"category": "associative",
|
||||
"query": "Sarah Bishop Dyer trailer performance",
|
||||
"returned": [
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"project-Imprint__leadership_development__feedback_frameworks__performance__presence_",
|
||||
"mem-5e7f6ddd-c818-4ad3-b564-54ae278e9976",
|
||||
"bl-7fa1b1a8-b80a-4f28-b162-bfe73765b4f8",
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c",
|
||||
"tag-sarah",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2359.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q30",
|
||||
"category": "associative",
|
||||
"query": "Swarm Architecture containment lateral worker",
|
||||
"returned": [
|
||||
"mem-265a7107-73b3-4410-9aff-43787d5f473b",
|
||||
"mem-9d1bf963-1b40-4588-bdb3-0432646cc623",
|
||||
"bl-0de4e61b-6562-49e5-b7df-ebb809a01723",
|
||||
"mem-3987d374-3c48-4e8e-b06d-0c363f55ed9c",
|
||||
"bl-e0a0df72-de6e-46ab-800b-e1e3e8dfc387",
|
||||
"tag-__patents____swarm____claim-language____prior-art____filing__",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2342.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q31",
|
||||
"category": "associative",
|
||||
"query": "hope won inside the narrative preface",
|
||||
"returned": [
|
||||
"kn-dca14c4c-4859-47b0-996e-33964ba61a87",
|
||||
"bl-e148d23c-24e8-4122-9915-d1c11f22052f",
|
||||
"bl-31abf75b-998f-4a4f-a6dd-8204119e0451",
|
||||
"bl-dc8c7e02-eb37-48ae-a6f8-9b512803ae16",
|
||||
"mem-8d1bafe6-209c-456c-9a25-9a927bc5a16d",
|
||||
"bl-14883d81-f7cb-46dd-82c2-a6e6980264e5",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"?Q??m?;��`'",
|
||||
"R^??m?;��?'"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3536.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q32",
|
||||
"category": "associative",
|
||||
"query": "man of the house six years old expectation",
|
||||
"returned": [
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"bl-ffa22d7e-42a9-4bd2-a428-1d2df243ac93",
|
||||
"bl-452a4710-3d2b-4e0f-9413-49a66423bc9a",
|
||||
"bl-4a6746e8-191f-48fc-8bfb-c4dc73b80bcd",
|
||||
"tag-command-pattern",
|
||||
"tag-withholding",
|
||||
"tag-offline",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4151.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q33",
|
||||
"category": "nonsense",
|
||||
"query": "zqxjvw plimforth grebulon",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 1342.7,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q34",
|
||||
"category": "nonsense",
|
||||
"query": "flarnbistle quommetry",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 874.8,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q35",
|
||||
"category": "nonsense",
|
||||
"query": "xxqzzt vurblenacht throom",
|
||||
"returned": [
|
||||
"kn-333542cb-6dab-4662-9725-bf7440d28bf7",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-66a21179-2adc-4b19-a109-880cf4674d7d",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4"
|
||||
],
|
||||
"n_returned": 8,
|
||||
"latency_ms": 1374.6,
|
||||
"error": null,
|
||||
"clean": false,
|
||||
"false_positives": 8
|
||||
},
|
||||
{
|
||||
"id": "q36",
|
||||
"category": "superseded",
|
||||
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
||||
"returned": [
|
||||
"ctx-e5427d7d",
|
||||
"project-worldweaver",
|
||||
"tag-__kotlin____internal-state____pre-reasoning____post-reasoning____compression-ratio____dharma____cultivation__",
|
||||
"tag-import",
|
||||
"tag-enterprise",
|
||||
"tag-__cgi____dharma____cultivation____five-primitives____seed-artifact____agi____intelligence____whitepaper____patent__",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 3762.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q37",
|
||||
"category": "superseded",
|
||||
"query": "how many provisional patents does Will actually have",
|
||||
"returned": [
|
||||
"mem-bfe0fafd-2750-4fdc-b773-04e878b3b23f",
|
||||
"art-7bdaff30-5af9-4f0a-93b1-751686f9de3d",
|
||||
"mem-cf07910d-4676-4384-ab97-9cad946cd0b9",
|
||||
"project-Convert_UTC_timestamps_to_Central_time_when_displaying_to_Will__Never_surface_raw_UTC_",
|
||||
"mem-32203649-3213-4d6d-86fd-3d657ac70d77",
|
||||
"mem-22f5f665-3ad2-4063-88b0-915849a795f5",
|
||||
"? ?}&?#??X\b",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 4859.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q38",
|
||||
"category": "superseded",
|
||||
"query": "is MCP still the live integration layer",
|
||||
"returned": [
|
||||
"project-Imprint__discovery__objection_handling__deal_strategy__pipeline__closing_",
|
||||
"bl-8116da7a-b039-4e08-b8d0-c1c7861f9766",
|
||||
"bl-8ef1ba6b-3fa0-4dbd-98c5-31665e5694a1",
|
||||
"tag-dark-theme",
|
||||
"tag-divisors",
|
||||
"tag-distressed-property",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"???Ͼd??W\b?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 2873.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,956 @@
|
||||
{
|
||||
"label": "hybrid-semantic-r2",
|
||||
"soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-hybrid",
|
||||
"soul_md5": "5cd2718932c2ecf4940ddfe5a9c8abbe",
|
||||
"corpus": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/corpus-embedded.json",
|
||||
"corpus_nodes": 78768,
|
||||
"corpus_edges": 14214,
|
||||
"gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json",
|
||||
"limit": 10,
|
||||
"port": 7895,
|
||||
"wall_clock_s": 52.1,
|
||||
"child_pid": 86164,
|
||||
"child_confirmed_dead": true,
|
||||
"aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.5142857142857142,
|
||||
"recall@5": 0.4409013605442177,
|
||||
"recall@10": 0.5047619047619047,
|
||||
"precision@5": 0.15428571428571433,
|
||||
"mrr@10": 0.38746031746031745,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "2/3",
|
||||
"latency_ms_p50": 1225.4,
|
||||
"latency_ms_p95": 1671.6,
|
||||
"latency_ms_max": 1718.8,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 3.3333333333333335
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.38461538461538464,
|
||||
"recall@5": 0.38461538461538464,
|
||||
"recall@10": 0.38461538461538464,
|
||||
"mrr@10": 0.17307692307692307
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.8571428571428571,
|
||||
"recall@5": 0.4902210884353741,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.6634920634920636
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.3333333333333333,
|
||||
"recall@5": 0.3333333333333333,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.2222222222222222,
|
||||
"outranks": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"id": "q01",
|
||||
"category": "exact_rare",
|
||||
"query": "unjailbreakable",
|
||||
"returned": [
|
||||
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 307.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q02",
|
||||
"category": "exact_rare",
|
||||
"query": "engram-migrate",
|
||||
"returned": [
|
||||
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5",
|
||||
"mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257",
|
||||
"project-engram",
|
||||
"project-engram-lang",
|
||||
"mem-60778715-758c-4677-933d-fc39b8f94152",
|
||||
"ctx-89a2",
|
||||
"bl-13babd0c-582e-4e28-a9e4-a77e65925e5d",
|
||||
"870ede67-3454-4e00-9988-46cb13a8a4e2",
|
||||
"bl-3e433255-3710-49fc-a093-c25e71de2ccb",
|
||||
"mem-235a7657-d49e-467e-9f69-f4c3d5f6bd48"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 349.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q03",
|
||||
"category": "exact_rare",
|
||||
"query": "cartabandonedevent",
|
||||
"returned": [
|
||||
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 316.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q04",
|
||||
"category": "exact_rare",
|
||||
"query": "pre-apprenticeship",
|
||||
"returned": [
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 315.7,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q05",
|
||||
"category": "exact_rare",
|
||||
"query": "inferencenodemanager",
|
||||
"returned": [
|
||||
"mem-73969486-143f-4431-b5e6-6845d1cc9848",
|
||||
"bl-c1765767-3e27-449a-8c94-10411d1eb7c0",
|
||||
"project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_"
|
||||
],
|
||||
"n_returned": 3,
|
||||
"latency_ms": 326.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q06",
|
||||
"category": "exact_rare",
|
||||
"query": "clear-eyed",
|
||||
"returned": [
|
||||
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 304.6,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q07",
|
||||
"category": "phrase",
|
||||
"query": "patterns not returns",
|
||||
"returned": [
|
||||
"mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482",
|
||||
"????7???Ջ3",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"?ǚ?7??????",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b",
|
||||
"7?e?7???\f3?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 593.0,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q08",
|
||||
"category": "phrase",
|
||||
"query": "thirty moves",
|
||||
"returned": [
|
||||
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
||||
"knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6",
|
||||
"Z[?<S???H??",
|
||||
"rQ??m?;?x?'",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 534.7,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3125,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q09",
|
||||
"category": "phrase",
|
||||
"query": "Grandma Lucas",
|
||||
"returned": [
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 544.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3333333333333333,
|
||||
"recall@10": 0.5555555555555556,
|
||||
"precision@5": 0.6,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q10",
|
||||
"category": "phrase",
|
||||
"query": "Directed Harmonic",
|
||||
"returned": [
|
||||
"ԍ????X????",
|
||||
"project-harmonic-framework",
|
||||
"?ǚ?7??????",
|
||||
"project-harmonic-framework_com",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"??????X??2c",
|
||||
"bl-dcee1887-34c4-4ffa-9119-1e291685ba08",
|
||||
"????7???Ջ3",
|
||||
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 524.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.1111111111111111,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1111111111111111
|
||||
},
|
||||
{
|
||||
"id": "q11",
|
||||
"category": "phrase",
|
||||
"query": "Sarah Bishop",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35",
|
||||
"mem-b8ecd23e-77ce-42f7-984c-f51453fec16d"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 543.0,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.5,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.2
|
||||
},
|
||||
{
|
||||
"id": "q12",
|
||||
"category": "phrase",
|
||||
"query": "Directed Autonomous Runtime Modification",
|
||||
"returned": [
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"mem-e6327f52-2bda-4ce7-9471-2fffd1e172de",
|
||||
"bl-145a0985-2382-400f-a7c5-c335c5e30a72",
|
||||
"mem-82b93b21-a865-410f-9ec1-fc54121d9bb5",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 905.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.2857142857142857,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 0.8,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q13",
|
||||
"category": "phrase",
|
||||
"query": "zero-knowledge encrypted backup",
|
||||
"returned": [
|
||||
"7774a16c-1027-4e3b-a21e-67f1f95a4acd",
|
||||
"8f3abb0d-77ed-4af3-9f4d-ba62cd198886",
|
||||
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a",
|
||||
"?",
|
||||
"mem-dba009a2-d2ea-4f5a-b9e8-0f04bc9ab32f",
|
||||
"?",
|
||||
"mem-7cd90611-88a3-423d-a38a-0db2812952fa",
|
||||
"?",
|
||||
"bl-07375bf9-a169-42cd-adb3-7d32b25982f0",
|
||||
"?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 769.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.3333333333333333
|
||||
},
|
||||
{
|
||||
"id": "q14",
|
||||
"category": "paraphrase",
|
||||
"query": "the elderly relative who passed while he stayed away",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"????7???Ջ3",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1645.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q15",
|
||||
"category": "paraphrase",
|
||||
"query": "a soldier sidelined by illness who refused to quit",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"kn-a31e1001-342e-4deb-a2e6-6d02d1f22dee",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1718.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q16",
|
||||
"category": "paraphrase",
|
||||
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"????7???Ջ3",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1393.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q17",
|
||||
"category": "paraphrase",
|
||||
"query": "a tight payload beats a bloated one",
|
||||
"returned": [
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
||||
"7?e?7???\f3?",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"??o?'?B???k",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1123.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q18",
|
||||
"category": "paraphrase",
|
||||
"query": "if you are able and nobody is coming the job is yours",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"imp-dce1da0f-8776-4a9e-972b-33411a7ca138",
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"kn-c2205725-69d0-4dd1-9a8d-1c7fa9a0c7b4",
|
||||
"kn-83bb86c6-521d-416c-a86e-6e29c2d8f102",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1448.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q19",
|
||||
"category": "paraphrase",
|
||||
"query": "learning is the wealth creditors cannot seize",
|
||||
"returned": [
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"7?e?7???\f3?",
|
||||
"kn-0625e393-067c-4bba-8389-7e1b79265142",
|
||||
"??o?'?B???k",
|
||||
"kn-150e6790-fc2a-48a7-8289-313c1fbaf5ae",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1298.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q20",
|
||||
"category": "paraphrase",
|
||||
"query": "reliability proven by track record not assertion",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
||||
"? ?}&?#??X\b",
|
||||
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
||||
"? ?}&?#??X\b",
|
||||
"a708dd6e-fe73-4f2f-a21e-89daa0985487",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1621.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q21",
|
||||
"category": "paraphrase",
|
||||
"query": "boundaries that enable instead of confine",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"? ?}&?#??X\b",
|
||||
"project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"ԍ????X????",
|
||||
"??????X??2c",
|
||||
"dR????X?-?S"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1405.7,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q22",
|
||||
"category": "paraphrase",
|
||||
"query": "what shifts tells you where to cut a system apart",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"?of?7???",
|
||||
"??????X??2c",
|
||||
"dz????Xƹ?i",
|
||||
"ԍ????X????",
|
||||
"dR????X?-?S",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1671.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q23",
|
||||
"category": "paraphrase",
|
||||
"query": "a mind that compounds instead of resetting each day",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8",
|
||||
"????7???Ջ3",
|
||||
"a1000001-0000-0000-0000-000000000001",
|
||||
"?ǚ?7??????",
|
||||
"mem-024598a9-ed2e-4eeb-b1e1-5410856ff132",
|
||||
"7?e?7???\f3?",
|
||||
"mem-ade9440f-f161-4c18-9b35-1976257e6ebb",
|
||||
"?of?7???",
|
||||
"ea95f600-8dfd-4c7e-b077-a93dc3cd3623"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1519.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q24",
|
||||
"category": "paraphrase",
|
||||
"query": "loved for the unedited self and not the polished exterior",
|
||||
"returned": [
|
||||
"mem-bbb126a1-b297-42bb-86be-796871829c94",
|
||||
"mem-45022957-2d78-48aa-a714-16d6eca52e0f",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"'?T?a\"B~-?8"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1576.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q25",
|
||||
"category": "paraphrase",
|
||||
"query": "cheerfulness you arrive at instead of assuming",
|
||||
"returned": [
|
||||
"?ǚ?7??????",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"7?e?7???\f3?",
|
||||
"??S?7???",
|
||||
"??f?7???",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"??S?7???",
|
||||
"??S?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1287.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q26",
|
||||
"category": "paraphrase",
|
||||
"query": "a childhood offering no solid foundation to inherit",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
||||
"7?e?7???\f3?",
|
||||
"mem-7b74cac0-905f-4c35-9688-fbcce105a177"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1391.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.25
|
||||
},
|
||||
{
|
||||
"id": "q27",
|
||||
"category": "associative",
|
||||
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"?ǚ?7??????"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1498.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q28",
|
||||
"category": "associative",
|
||||
"query": "Marines hernia sepsis medical ward",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-e0bdf5d8-d163-491f-b649-453fee8b721d",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1169.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q29",
|
||||
"category": "associative",
|
||||
"query": "Sarah Bishop Dyer trailer performance",
|
||||
"returned": [
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1239.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q30",
|
||||
"category": "associative",
|
||||
"query": "Swarm Architecture containment lateral worker",
|
||||
"returned": [
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43",
|
||||
"kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9",
|
||||
"8cbb60c5-4999-4ec1-8682-2592aedc4249",
|
||||
"kn-6f248a50-355b-47bb-aec8-e0e646a9b077",
|
||||
"bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae",
|
||||
"bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98",
|
||||
"7?e?7???\f3?",
|
||||
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"bl-2121fdb9-796a-427e-b9b5-651f4388ea16"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1225.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q31",
|
||||
"category": "associative",
|
||||
"query": "hope won inside the narrative preface",
|
||||
"returned": [
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52",
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"knw-d357b6bb-ad8a-4791-b516-426aea45fa5b",
|
||||
"7?e?7???\f3?",
|
||||
"knw-ed33e669-0790-44cb-a036-958d605c6fea",
|
||||
"rQ??m?;?x?'"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1237.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q32",
|
||||
"category": "associative",
|
||||
"query": "man of the house six years old expectation",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1458.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q33",
|
||||
"category": "nonsense",
|
||||
"query": "zqxjvw plimforth grebulon",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 756.3,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q34",
|
||||
"category": "nonsense",
|
||||
"query": "flarnbistle quommetry",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 519.6,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q35",
|
||||
"category": "nonsense",
|
||||
"query": "xxqzzt vurblenacht throom",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"?V?",
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4",
|
||||
"?m?\\}Q??6??",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"?m?\\}Q??6??",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"?m?\\}Q??6??",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"?m?\\}Q??6??"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 774.0,
|
||||
"error": null,
|
||||
"clean": false,
|
||||
"false_positives": 10
|
||||
},
|
||||
{
|
||||
"id": "q36",
|
||||
"category": "superseded",
|
||||
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"mem-80d7416b-20e9-48a0-b176-b215527e2f56",
|
||||
"? ?}&?#??X\b",
|
||||
"mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-b7e98d63-8b83-4911-b4d0-990602a7f575",
|
||||
"?of?7???",
|
||||
"knw-e047bb42-dc5b-4383-9e88-e508dc03abe3",
|
||||
"? ?}&?#??X\b",
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1340.3,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5,
|
||||
"outranks": true,
|
||||
"rank_correct": 2,
|
||||
"rank_stale": 10
|
||||
},
|
||||
{
|
||||
"id": "q37",
|
||||
"category": "superseded",
|
||||
"query": "how many provisional patents does Will actually have",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"12082f7e-e320-438b-bd65-083d8259748f",
|
||||
"? ?}&?#??X\b",
|
||||
"527ecb25-2587-47eb-8269-73be2431abd4",
|
||||
"? ?}&?#??X\b",
|
||||
"3cf706a1-3825-45d8-b0a9-06cae6cdf5b8",
|
||||
"? ?}&?#??X\b",
|
||||
"4f698ae6-c40e-464e-9798-50350991a188",
|
||||
"? ?}&?#??X\b",
|
||||
"be3b6036-6eca-44a7-8fdf-37b23edfdfd1"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1693.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.16666666666666666,
|
||||
"outranks": true,
|
||||
"rank_correct": 6,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q38",
|
||||
"category": "superseded",
|
||||
"query": "is MCP still the live integration layer",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"bl-7328cbe3-0200-43c2-88e7-0a164e15fca4",
|
||||
"7?e?7???\f3?",
|
||||
"mem-101e81b4-8097-4749-8d8d-7bb66de34517",
|
||||
"? ?}&?#??X\b",
|
||||
"4509ed62-9fb2-48b8-9038-ac569fca9604",
|
||||
"%???2??jH??",
|
||||
"art-8a0870d5-a716-4672-8094-f7463af1265b",
|
||||
"???Ͼd??W\b?",
|
||||
"bl-556438af-57b2-4bd8-a747-9f868aaee290"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1027.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": 4
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,956 @@
|
||||
{
|
||||
"label": "hybrid-semantic",
|
||||
"soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-hybrid",
|
||||
"soul_md5": "5cd2718932c2ecf4940ddfe5a9c8abbe",
|
||||
"corpus": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/corpus-embedded.json",
|
||||
"corpus_nodes": 78768,
|
||||
"corpus_edges": 14214,
|
||||
"gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json",
|
||||
"limit": 10,
|
||||
"port": 7894,
|
||||
"wall_clock_s": 52.1,
|
||||
"child_pid": 86109,
|
||||
"child_confirmed_dead": true,
|
||||
"aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.5142857142857142,
|
||||
"recall@5": 0.4409013605442177,
|
||||
"recall@10": 0.5047619047619047,
|
||||
"precision@5": 0.15428571428571433,
|
||||
"mrr@10": 0.38746031746031745,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "2/3",
|
||||
"latency_ms_p50": 1219.7,
|
||||
"latency_ms_p95": 1667.1,
|
||||
"latency_ms_max": 1720.2,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 3.3333333333333335
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.38461538461538464,
|
||||
"recall@5": 0.38461538461538464,
|
||||
"recall@10": 0.38461538461538464,
|
||||
"mrr@10": 0.17307692307692307
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.8571428571428571,
|
||||
"recall@5": 0.4902210884353741,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.6634920634920636
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.3333333333333333,
|
||||
"recall@5": 0.3333333333333333,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.2222222222222222,
|
||||
"outranks": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"id": "q01",
|
||||
"category": "exact_rare",
|
||||
"query": "unjailbreakable",
|
||||
"returned": [
|
||||
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 323.6,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q02",
|
||||
"category": "exact_rare",
|
||||
"query": "engram-migrate",
|
||||
"returned": [
|
||||
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5",
|
||||
"mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257",
|
||||
"project-engram",
|
||||
"project-engram-lang",
|
||||
"mem-60778715-758c-4677-933d-fc39b8f94152",
|
||||
"ctx-89a2",
|
||||
"bl-13babd0c-582e-4e28-a9e4-a77e65925e5d",
|
||||
"870ede67-3454-4e00-9988-46cb13a8a4e2",
|
||||
"bl-3e433255-3710-49fc-a093-c25e71de2ccb",
|
||||
"mem-235a7657-d49e-467e-9f69-f4c3d5f6bd48"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 330.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q03",
|
||||
"category": "exact_rare",
|
||||
"query": "cartabandonedevent",
|
||||
"returned": [
|
||||
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 318.7,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q04",
|
||||
"category": "exact_rare",
|
||||
"query": "pre-apprenticeship",
|
||||
"returned": [
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 316.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q05",
|
||||
"category": "exact_rare",
|
||||
"query": "inferencenodemanager",
|
||||
"returned": [
|
||||
"mem-73969486-143f-4431-b5e6-6845d1cc9848",
|
||||
"bl-c1765767-3e27-449a-8c94-10411d1eb7c0",
|
||||
"project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_"
|
||||
],
|
||||
"n_returned": 3,
|
||||
"latency_ms": 328.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q06",
|
||||
"category": "exact_rare",
|
||||
"query": "clear-eyed",
|
||||
"returned": [
|
||||
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 296.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q07",
|
||||
"category": "phrase",
|
||||
"query": "patterns not returns",
|
||||
"returned": [
|
||||
"mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482",
|
||||
"????7???Ջ3",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"?ǚ?7??????",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b",
|
||||
"7?e?7???\f3?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 611.4,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q08",
|
||||
"category": "phrase",
|
||||
"query": "thirty moves",
|
||||
"returned": [
|
||||
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
||||
"knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6",
|
||||
"Z[?<S???H??",
|
||||
"rQ??m?;?x?'",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 543.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3125,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q09",
|
||||
"category": "phrase",
|
||||
"query": "Grandma Lucas",
|
||||
"returned": [
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 547.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3333333333333333,
|
||||
"recall@10": 0.5555555555555556,
|
||||
"precision@5": 0.6,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q10",
|
||||
"category": "phrase",
|
||||
"query": "Directed Harmonic",
|
||||
"returned": [
|
||||
"ԍ????X????",
|
||||
"project-harmonic-framework",
|
||||
"?ǚ?7??????",
|
||||
"project-harmonic-framework_com",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"??????X??2c",
|
||||
"bl-dcee1887-34c4-4ffa-9119-1e291685ba08",
|
||||
"????7???Ջ3",
|
||||
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 518.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.1111111111111111,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.1111111111111111
|
||||
},
|
||||
{
|
||||
"id": "q11",
|
||||
"category": "phrase",
|
||||
"query": "Sarah Bishop",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35",
|
||||
"mem-b8ecd23e-77ce-42f7-984c-f51453fec16d"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 534.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.5,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.2
|
||||
},
|
||||
{
|
||||
"id": "q12",
|
||||
"category": "phrase",
|
||||
"query": "Directed Autonomous Runtime Modification",
|
||||
"returned": [
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"mem-e6327f52-2bda-4ce7-9471-2fffd1e172de",
|
||||
"bl-145a0985-2382-400f-a7c5-c335c5e30a72",
|
||||
"mem-82b93b21-a865-410f-9ec1-fc54121d9bb5",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 895.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.2857142857142857,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 0.8,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q13",
|
||||
"category": "phrase",
|
||||
"query": "zero-knowledge encrypted backup",
|
||||
"returned": [
|
||||
"7774a16c-1027-4e3b-a21e-67f1f95a4acd",
|
||||
"8f3abb0d-77ed-4af3-9f4d-ba62cd198886",
|
||||
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a",
|
||||
"?",
|
||||
"mem-dba009a2-d2ea-4f5a-b9e8-0f04bc9ab32f",
|
||||
"?",
|
||||
"mem-7cd90611-88a3-423d-a38a-0db2812952fa",
|
||||
"?",
|
||||
"bl-07375bf9-a169-42cd-adb3-7d32b25982f0",
|
||||
"?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 763.0,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.3333333333333333
|
||||
},
|
||||
{
|
||||
"id": "q14",
|
||||
"category": "paraphrase",
|
||||
"query": "the elderly relative who passed while he stayed away",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"????7???Ջ3",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1650.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q15",
|
||||
"category": "paraphrase",
|
||||
"query": "a soldier sidelined by illness who refused to quit",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"kn-a31e1001-342e-4deb-a2e6-6d02d1f22dee",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1720.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q16",
|
||||
"category": "paraphrase",
|
||||
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"????7???Ջ3",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1396.3,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q17",
|
||||
"category": "paraphrase",
|
||||
"query": "a tight payload beats a bloated one",
|
||||
"returned": [
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
||||
"7?e?7???\f3?",
|
||||
"??o?'?B???k",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1126.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q18",
|
||||
"category": "paraphrase",
|
||||
"query": "if you are able and nobody is coming the job is yours",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"imp-dce1da0f-8776-4a9e-972b-33411a7ca138",
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"kn-c2205725-69d0-4dd1-9a8d-1c7fa9a0c7b4",
|
||||
"kn-83bb86c6-521d-416c-a86e-6e29c2d8f102",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1452.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q19",
|
||||
"category": "paraphrase",
|
||||
"query": "learning is the wealth creditors cannot seize",
|
||||
"returned": [
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"7?e?7???\f3?",
|
||||
"kn-0625e393-067c-4bba-8389-7e1b79265142",
|
||||
"??o?'?B???k",
|
||||
"kn-150e6790-fc2a-48a7-8289-313c1fbaf5ae",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1303.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q20",
|
||||
"category": "paraphrase",
|
||||
"query": "reliability proven by track record not assertion",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
||||
"? ?}&?#??X\b",
|
||||
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
||||
"? ?}&?#??X\b",
|
||||
"a708dd6e-fe73-4f2f-a21e-89daa0985487",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1618.6,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q21",
|
||||
"category": "paraphrase",
|
||||
"query": "boundaries that enable instead of confine",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"? ?}&?#??X\b",
|
||||
"project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"ԍ????X????",
|
||||
"??????X??2c",
|
||||
"dR????X?-?S"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1406.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q22",
|
||||
"category": "paraphrase",
|
||||
"query": "what shifts tells you where to cut a system apart",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???",
|
||||
"dR????X?-?S",
|
||||
"dz????Xƹ?i",
|
||||
"ԍ????X????",
|
||||
"??????X??2c",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1667.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q23",
|
||||
"category": "paraphrase",
|
||||
"query": "a mind that compounds instead of resetting each day",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8",
|
||||
"????7???Ջ3",
|
||||
"a1000001-0000-0000-0000-000000000001",
|
||||
"?ǚ?7??????",
|
||||
"mem-024598a9-ed2e-4eeb-b1e1-5410856ff132",
|
||||
"7?e?7???\f3?",
|
||||
"mem-ade9440f-f161-4c18-9b35-1976257e6ebb",
|
||||
"?of?7???",
|
||||
"ea95f600-8dfd-4c7e-b077-a93dc3cd3623"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1528.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q24",
|
||||
"category": "paraphrase",
|
||||
"query": "loved for the unedited self and not the polished exterior",
|
||||
"returned": [
|
||||
"mem-bbb126a1-b297-42bb-86be-796871829c94",
|
||||
"mem-45022957-2d78-48aa-a714-16d6eca52e0f",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"'?T?a\"B~-?8"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1579.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q25",
|
||||
"category": "paraphrase",
|
||||
"query": "cheerfulness you arrive at instead of assuming",
|
||||
"returned": [
|
||||
"?ǚ?7??????",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"7?e?7???\f3?",
|
||||
"??S?7???",
|
||||
"??f?7???",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"??S?7???",
|
||||
"??S?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1288.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q26",
|
||||
"category": "paraphrase",
|
||||
"query": "a childhood offering no solid foundation to inherit",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
||||
"7?e?7???\f3?",
|
||||
"mem-7b74cac0-905f-4c35-9688-fbcce105a177"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1383.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.25
|
||||
},
|
||||
{
|
||||
"id": "q27",
|
||||
"category": "associative",
|
||||
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"?ǚ?7??????"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1499.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q28",
|
||||
"category": "associative",
|
||||
"query": "Marines hernia sepsis medical ward",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-e0bdf5d8-d163-491f-b649-453fee8b721d",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1167.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q29",
|
||||
"category": "associative",
|
||||
"query": "Sarah Bishop Dyer trailer performance",
|
||||
"returned": [
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1245.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q30",
|
||||
"category": "associative",
|
||||
"query": "Swarm Architecture containment lateral worker",
|
||||
"returned": [
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43",
|
||||
"kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9",
|
||||
"8cbb60c5-4999-4ec1-8682-2592aedc4249",
|
||||
"kn-6f248a50-355b-47bb-aec8-e0e646a9b077",
|
||||
"bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae",
|
||||
"bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98",
|
||||
"7?e?7???\f3?",
|
||||
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
||||
"bl-2121fdb9-796a-427e-b9b5-651f4388ea16"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1219.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q31",
|
||||
"category": "associative",
|
||||
"query": "hope won inside the narrative preface",
|
||||
"returned": [
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52",
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"knw-d357b6bb-ad8a-4791-b516-426aea45fa5b",
|
||||
"7?e?7???\f3?",
|
||||
"knw-ed33e669-0790-44cb-a036-958d605c6fea",
|
||||
"rQ??m?;?x?'"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1235.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q32",
|
||||
"category": "associative",
|
||||
"query": "man of the house six years old expectation",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1464.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q33",
|
||||
"category": "nonsense",
|
||||
"query": "zqxjvw plimforth grebulon",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 761.0,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q34",
|
||||
"category": "nonsense",
|
||||
"query": "flarnbistle quommetry",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 525.1,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q35",
|
||||
"category": "nonsense",
|
||||
"query": "xxqzzt vurblenacht throom",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"?V?",
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4",
|
||||
"?m?\\}Q??6??",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"?m?\\}Q??6??",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"?m?\\}Q??6??",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"?m?\\}Q??6??"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 761.7,
|
||||
"error": null,
|
||||
"clean": false,
|
||||
"false_positives": 10
|
||||
},
|
||||
{
|
||||
"id": "q36",
|
||||
"category": "superseded",
|
||||
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"mem-80d7416b-20e9-48a0-b176-b215527e2f56",
|
||||
"? ?}&?#??X\b",
|
||||
"mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-b7e98d63-8b83-4911-b4d0-990602a7f575",
|
||||
"?of?7???",
|
||||
"knw-e047bb42-dc5b-4383-9e88-e508dc03abe3",
|
||||
"? ?}&?#??X\b",
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1322.6,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5,
|
||||
"outranks": true,
|
||||
"rank_correct": 2,
|
||||
"rank_stale": 10
|
||||
},
|
||||
{
|
||||
"id": "q37",
|
||||
"category": "superseded",
|
||||
"query": "how many provisional patents does Will actually have",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"12082f7e-e320-438b-bd65-083d8259748f",
|
||||
"? ?}&?#??X\b",
|
||||
"527ecb25-2587-47eb-8269-73be2431abd4",
|
||||
"? ?}&?#??X\b",
|
||||
"3cf706a1-3825-45d8-b0a9-06cae6cdf5b8",
|
||||
"? ?}&?#??X\b",
|
||||
"4f698ae6-c40e-464e-9798-50350991a188",
|
||||
"? ?}&?#??X\b",
|
||||
"be3b6036-6eca-44a7-8fdf-37b23edfdfd1"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1694.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.16666666666666666,
|
||||
"outranks": true,
|
||||
"rank_correct": 6,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q38",
|
||||
"category": "superseded",
|
||||
"query": "is MCP still the live integration layer",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"bl-7328cbe3-0200-43c2-88e7-0a164e15fca4",
|
||||
"7?e?7???\f3?",
|
||||
"mem-101e81b4-8097-4749-8d8d-7bb66de34517",
|
||||
"? ?}&?#??X\b",
|
||||
"4509ed62-9fb2-48b8-9038-ac569fca9604",
|
||||
"%???2??jH??",
|
||||
"art-8a0870d5-a716-4672-8094-f7463af1265b",
|
||||
"???Ͼd??W\b?",
|
||||
"bl-556438af-57b2-4bd8-a747-9f868aaee290"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1041.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": 4
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,945 @@
|
||||
{
|
||||
"label": "main-r2",
|
||||
"soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-main",
|
||||
"soul_md5": "caf822425540114984bfcd713ad37162",
|
||||
"corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json",
|
||||
"corpus_nodes": 78768,
|
||||
"corpus_edges": 14214,
|
||||
"gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json",
|
||||
"limit": 10,
|
||||
"port": 7895,
|
||||
"wall_clock_s": 45.6,
|
||||
"child_pid": 78695,
|
||||
"child_confirmed_dead": true,
|
||||
"aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.34285714285714286,
|
||||
"recall@5": 0.26947278911564626,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"precision@5": 0.12000000000000001,
|
||||
"mrr@10": 0.2943197278911564,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "1/3",
|
||||
"latency_ms_p50": 1151.1,
|
||||
"latency_ms_p95": 1580.7,
|
||||
"latency_ms_max": 1663.2,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 3.3333333333333335
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.8571428571428571,
|
||||
"recall@5": 0.4902210884353741,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.5965986394557822
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"mrr@10": 0.041666666666666664,
|
||||
"outranks": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"id": "q01",
|
||||
"category": "exact_rare",
|
||||
"query": "unjailbreakable",
|
||||
"returned": [
|
||||
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 231.3,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q02",
|
||||
"category": "exact_rare",
|
||||
"query": "engram-migrate",
|
||||
"returned": [
|
||||
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 266.6,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q03",
|
||||
"category": "exact_rare",
|
||||
"query": "cartabandonedevent",
|
||||
"returned": [
|
||||
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 229.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q04",
|
||||
"category": "exact_rare",
|
||||
"query": "pre-apprenticeship",
|
||||
"returned": [
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 235.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q05",
|
||||
"category": "exact_rare",
|
||||
"query": "inferencenodemanager",
|
||||
"returned": [
|
||||
"mem-73969486-143f-4431-b5e6-6845d1cc9848"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 254.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q06",
|
||||
"category": "exact_rare",
|
||||
"query": "clear-eyed",
|
||||
"returned": [
|
||||
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 228.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q07",
|
||||
"category": "phrase",
|
||||
"query": "patterns not returns",
|
||||
"returned": [
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"?ǚ?7??????",
|
||||
"mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"????7???Ջ3",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b",
|
||||
"7?e?7???\f3?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 543.3,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.3333333333333333
|
||||
},
|
||||
{
|
||||
"id": "q08",
|
||||
"category": "phrase",
|
||||
"query": "thirty moves",
|
||||
"returned": [
|
||||
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
||||
"knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6",
|
||||
"Z[?<S???H??",
|
||||
"rQ??m?;?x?'",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 466.4,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3125,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q09",
|
||||
"category": "phrase",
|
||||
"query": "Grandma Lucas",
|
||||
"returned": [
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 474.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3333333333333333,
|
||||
"recall@10": 0.5555555555555556,
|
||||
"precision@5": 0.6,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q10",
|
||||
"category": "phrase",
|
||||
"query": "Directed Harmonic",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"ԍ????X????",
|
||||
"bl-dcee1887-34c4-4ffa-9119-1e291685ba08",
|
||||
"?ǚ?7??????",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf",
|
||||
"????7???Ջ3",
|
||||
"??????X??2c",
|
||||
"g?e?7???'c?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 460.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.1111111111111111,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.14285714285714285
|
||||
},
|
||||
{
|
||||
"id": "q11",
|
||||
"category": "phrase",
|
||||
"query": "Sarah Bishop",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35",
|
||||
"mem-b8ecd23e-77ce-42f7-984c-f51453fec16d"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 477.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.5,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.2
|
||||
},
|
||||
{
|
||||
"id": "q12",
|
||||
"category": "phrase",
|
||||
"query": "Directed Autonomous Runtime Modification",
|
||||
"returned": [
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"bl-145a0985-2382-400f-a7c5-c335c5e30a72",
|
||||
"mem-82b93b21-a865-410f-9ec1-fc54121d9bb5",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"mem-e6327f52-2bda-4ce7-9471-2fffd1e172de",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 824.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.2857142857142857,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 0.8,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q13",
|
||||
"category": "phrase",
|
||||
"query": "zero-knowledge encrypted backup",
|
||||
"returned": [
|
||||
"7774a16c-1027-4e3b-a21e-67f1f95a4acd",
|
||||
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a",
|
||||
"mem-dba009a2-d2ea-4f5a-b9e8-0f04bc9ab32f",
|
||||
"mem-a3c97012-5fa3-4915-a839-2c75c72005e0",
|
||||
"mem-7cd90611-88a3-423d-a38a-0db2812952fa",
|
||||
"bl-07375bf9-a169-42cd-adb3-7d32b25982f0",
|
||||
"bl-ec84b63d-b278-4944-8d7f-4aa7a51c0315",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 681.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q14",
|
||||
"category": "paraphrase",
|
||||
"query": "the elderly relative who passed while he stayed away",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"????7???Ջ3",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1577.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q15",
|
||||
"category": "paraphrase",
|
||||
"query": "a soldier sidelined by illness who refused to quit",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a31e1001-342e-4deb-a2e6-6d02d1f22dee",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1663.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q16",
|
||||
"category": "paraphrase",
|
||||
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"?of?7???",
|
||||
"????7???Ջ3",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"kn-81c24d13-a73b-4767-819c-dafaacc1498e",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1331.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q17",
|
||||
"category": "paraphrase",
|
||||
"query": "a tight payload beats a bloated one",
|
||||
"returned": [
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"??o?'?B???k",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1036.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q18",
|
||||
"category": "paraphrase",
|
||||
"query": "if you are able and nobody is coming the job is yours",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"imp-dce1da0f-8776-4a9e-972b-33411a7ca138",
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-83bb86c6-521d-416c-a86e-6e29c2d8f102",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1363.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q19",
|
||||
"category": "paraphrase",
|
||||
"query": "learning is the wealth creditors cannot seize",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"??o?'?B???k",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-0625e393-067c-4bba-8389-7e1b79265142",
|
||||
"ע?RGk?\tH(?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1230.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q20",
|
||||
"category": "paraphrase",
|
||||
"query": "reliability proven by track record not assertion",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1548.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q21",
|
||||
"category": "paraphrase",
|
||||
"query": "boundaries that enable instead of confine",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"??????X??2c",
|
||||
"ԍ????X????",
|
||||
"dR????X?-?S",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"dz????Xƹ?i",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1315.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q22",
|
||||
"category": "paraphrase",
|
||||
"query": "what shifts tells you where to cut a system apart",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"dR????X?-?S",
|
||||
"dz????Xƹ?i",
|
||||
"ԍ????X????",
|
||||
"??????X??2c",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1580.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q23",
|
||||
"category": "paraphrase",
|
||||
"query": "a mind that compounds instead of resetting each day",
|
||||
"returned": [
|
||||
"?ǚ?7??????",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"????7???Ջ3",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"%???2??jH??",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1474.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q24",
|
||||
"category": "paraphrase",
|
||||
"query": "loved for the unedited self and not the polished exterior",
|
||||
"returned": [
|
||||
"mem-bbb126a1-b297-42bb-86be-796871829c94",
|
||||
"mem-45022957-2d78-48aa-a714-16d6eca52e0f",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"'?T?a\"B~-?8"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1490.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q25",
|
||||
"category": "paraphrase",
|
||||
"query": "cheerfulness you arrive at instead of assuming",
|
||||
"returned": [
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"?ǚ?7??????",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"7?e?7???\f3?",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"??S?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"??f?7???",
|
||||
"??S?7???",
|
||||
"knw-5578cb21-e899-4822-b7f4-0d96fa094e3d"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1192.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q26",
|
||||
"category": "paraphrase",
|
||||
"query": "a childhood offering no solid foundation to inherit",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"??o?'?B???k",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1320.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q27",
|
||||
"category": "associative",
|
||||
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"?ǚ?7??????"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1448.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q28",
|
||||
"category": "associative",
|
||||
"query": "Marines hernia sepsis medical ward",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-e0bdf5d8-d163-491f-b649-453fee8b721d",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1107.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q29",
|
||||
"category": "associative",
|
||||
"query": "Sarah Bishop Dyer trailer performance",
|
||||
"returned": [
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1151.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q30",
|
||||
"category": "associative",
|
||||
"query": "Swarm Architecture containment lateral worker",
|
||||
"returned": [
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43",
|
||||
"8cbb60c5-4999-4ec1-8682-2592aedc4249",
|
||||
"bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae",
|
||||
"7?e?7???\f3?",
|
||||
"kn-6f248a50-355b-47bb-aec8-e0e646a9b077",
|
||||
"? ?}&?#??X\b",
|
||||
"h??I?cB?Q??",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1154.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q31",
|
||||
"category": "associative",
|
||||
"query": "hope won inside the narrative preface",
|
||||
"returned": [
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"7?e?7???\f3?",
|
||||
"rQ??m?;?x?'",
|
||||
"?Q??m?;?u?'",
|
||||
"R^??m?;��?'",
|
||||
"?Q??m?;��`'",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1187.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q32",
|
||||
"category": "associative",
|
||||
"query": "man of the house six years old expectation",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1413.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q33",
|
||||
"category": "nonsense",
|
||||
"query": "zqxjvw plimforth grebulon",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 684.0,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q34",
|
||||
"category": "nonsense",
|
||||
"query": "flarnbistle quommetry",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 444.0,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q35",
|
||||
"category": "nonsense",
|
||||
"query": "xxqzzt vurblenacht throom",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"kn-66a21179-2adc-4b19-a109-880cf4674d7d",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 696.8,
|
||||
"error": null,
|
||||
"clean": false,
|
||||
"false_positives": 10
|
||||
},
|
||||
{
|
||||
"id": "q36",
|
||||
"category": "superseded",
|
||||
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"mem-80d7416b-20e9-48a0-b176-b215527e2f56",
|
||||
"mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1245.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.125,
|
||||
"outranks": true,
|
||||
"rank_correct": 8,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q37",
|
||||
"category": "superseded",
|
||||
"query": "how many provisional patents does Will actually have",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"?ǚ?7??????",
|
||||
"[?MO5????G",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1617.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q38",
|
||||
"category": "superseded",
|
||||
"query": "is MCP still the live integration layer",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"%???2??jH??",
|
||||
"???Ͼd??W\b?",
|
||||
"??o?'?B???k",
|
||||
"? ?}&?#??X\b",
|
||||
"63307ac5-cf6b-46e0-8296-07503b461cfa",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 967.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,945 @@
|
||||
{
|
||||
"label": "main-r3",
|
||||
"soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-main",
|
||||
"soul_md5": "caf822425540114984bfcd713ad37162",
|
||||
"corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json",
|
||||
"corpus_nodes": 78768,
|
||||
"corpus_edges": 14214,
|
||||
"gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json",
|
||||
"limit": 10,
|
||||
"port": 7896,
|
||||
"wall_clock_s": 45.7,
|
||||
"child_pid": 78750,
|
||||
"child_confirmed_dead": true,
|
||||
"aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.34285714285714286,
|
||||
"recall@5": 0.26947278911564626,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"precision@5": 0.12000000000000001,
|
||||
"mrr@10": 0.2943197278911564,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "1/3",
|
||||
"latency_ms_p50": 1141.7,
|
||||
"latency_ms_p95": 1578.0,
|
||||
"latency_ms_max": 1645.5,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 3.3333333333333335
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.8571428571428571,
|
||||
"recall@5": 0.4902210884353741,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.5965986394557822
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"mrr@10": 0.041666666666666664,
|
||||
"outranks": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"id": "q01",
|
||||
"category": "exact_rare",
|
||||
"query": "unjailbreakable",
|
||||
"returned": [
|
||||
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 237.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q02",
|
||||
"category": "exact_rare",
|
||||
"query": "engram-migrate",
|
||||
"returned": [
|
||||
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 272.4,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q03",
|
||||
"category": "exact_rare",
|
||||
"query": "cartabandonedevent",
|
||||
"returned": [
|
||||
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 231.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q04",
|
||||
"category": "exact_rare",
|
||||
"query": "pre-apprenticeship",
|
||||
"returned": [
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 231.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q05",
|
||||
"category": "exact_rare",
|
||||
"query": "inferencenodemanager",
|
||||
"returned": [
|
||||
"mem-73969486-143f-4431-b5e6-6845d1cc9848"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 262.6,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q06",
|
||||
"category": "exact_rare",
|
||||
"query": "clear-eyed",
|
||||
"returned": [
|
||||
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 239.4,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q07",
|
||||
"category": "phrase",
|
||||
"query": "patterns not returns",
|
||||
"returned": [
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"?ǚ?7??????",
|
||||
"mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"????7???Ջ3",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b",
|
||||
"7?e?7???\f3?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 542.3,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.3333333333333333
|
||||
},
|
||||
{
|
||||
"id": "q08",
|
||||
"category": "phrase",
|
||||
"query": "thirty moves",
|
||||
"returned": [
|
||||
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
||||
"knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6",
|
||||
"Z[?<S???H??",
|
||||
"rQ??m?;?x?'",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 465.0,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3125,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q09",
|
||||
"category": "phrase",
|
||||
"query": "Grandma Lucas",
|
||||
"returned": [
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 470.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3333333333333333,
|
||||
"recall@10": 0.5555555555555556,
|
||||
"precision@5": 0.6,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q10",
|
||||
"category": "phrase",
|
||||
"query": "Directed Harmonic",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"ԍ????X????",
|
||||
"bl-dcee1887-34c4-4ffa-9119-1e291685ba08",
|
||||
"?ǚ?7??????",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf",
|
||||
"????7???Ջ3",
|
||||
"??????X??2c",
|
||||
"g?e?7???'c?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 441.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.1111111111111111,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.14285714285714285
|
||||
},
|
||||
{
|
||||
"id": "q11",
|
||||
"category": "phrase",
|
||||
"query": "Sarah Bishop",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35",
|
||||
"mem-b8ecd23e-77ce-42f7-984c-f51453fec16d"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 455.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.5,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.2
|
||||
},
|
||||
{
|
||||
"id": "q12",
|
||||
"category": "phrase",
|
||||
"query": "Directed Autonomous Runtime Modification",
|
||||
"returned": [
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"bl-145a0985-2382-400f-a7c5-c335c5e30a72",
|
||||
"mem-82b93b21-a865-410f-9ec1-fc54121d9bb5",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"mem-e6327f52-2bda-4ce7-9471-2fffd1e172de",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 816.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.2857142857142857,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 0.8,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q13",
|
||||
"category": "phrase",
|
||||
"query": "zero-knowledge encrypted backup",
|
||||
"returned": [
|
||||
"7774a16c-1027-4e3b-a21e-67f1f95a4acd",
|
||||
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a",
|
||||
"mem-dba009a2-d2ea-4f5a-b9e8-0f04bc9ab32f",
|
||||
"mem-a3c97012-5fa3-4915-a839-2c75c72005e0",
|
||||
"mem-7cd90611-88a3-423d-a38a-0db2812952fa",
|
||||
"bl-07375bf9-a169-42cd-adb3-7d32b25982f0",
|
||||
"bl-ec84b63d-b278-4944-8d7f-4aa7a51c0315",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 678.3,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q14",
|
||||
"category": "paraphrase",
|
||||
"query": "the elderly relative who passed while he stayed away",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"????7???Ջ3",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1564.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q15",
|
||||
"category": "paraphrase",
|
||||
"query": "a soldier sidelined by illness who refused to quit",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a31e1001-342e-4deb-a2e6-6d02d1f22dee",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1645.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q16",
|
||||
"category": "paraphrase",
|
||||
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"?of?7???",
|
||||
"????7???Ջ3",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"kn-81c24d13-a73b-4767-819c-dafaacc1498e",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1320.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q17",
|
||||
"category": "paraphrase",
|
||||
"query": "a tight payload beats a bloated one",
|
||||
"returned": [
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"??o?'?B???k",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1029.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q18",
|
||||
"category": "paraphrase",
|
||||
"query": "if you are able and nobody is coming the job is yours",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"imp-dce1da0f-8776-4a9e-972b-33411a7ca138",
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-83bb86c6-521d-416c-a86e-6e29c2d8f102",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1347.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q19",
|
||||
"category": "paraphrase",
|
||||
"query": "learning is the wealth creditors cannot seize",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"??o?'?B???k",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-0625e393-067c-4bba-8389-7e1b79265142",
|
||||
"ע?RGk?\tH(?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1220.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q20",
|
||||
"category": "paraphrase",
|
||||
"query": "reliability proven by track record not assertion",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1560.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q21",
|
||||
"category": "paraphrase",
|
||||
"query": "boundaries that enable instead of confine",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"??????X??2c",
|
||||
"ԍ????X????",
|
||||
"dR????X?-?S",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"dz????Xƹ?i",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1305.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q22",
|
||||
"category": "paraphrase",
|
||||
"query": "what shifts tells you where to cut a system apart",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"dR????X?-?S",
|
||||
"dz????Xƹ?i",
|
||||
"ԍ????X????",
|
||||
"??????X??2c",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1578.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q23",
|
||||
"category": "paraphrase",
|
||||
"query": "a mind that compounds instead of resetting each day",
|
||||
"returned": [
|
||||
"????7???Ջ3",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"?ǚ?7??????",
|
||||
"7?e?7???\f3?",
|
||||
"??f?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"%???2??jH??",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1449.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q24",
|
||||
"category": "paraphrase",
|
||||
"query": "loved for the unedited self and not the polished exterior",
|
||||
"returned": [
|
||||
"mem-bbb126a1-b297-42bb-86be-796871829c94",
|
||||
"mem-45022957-2d78-48aa-a714-16d6eca52e0f",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"'?T?a\"B~-?8"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1489.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q25",
|
||||
"category": "paraphrase",
|
||||
"query": "cheerfulness you arrive at instead of assuming",
|
||||
"returned": [
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"?ǚ?7??????",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"7?e?7???\f3?",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"??S?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"??f?7???",
|
||||
"??S?7???",
|
||||
"knw-5578cb21-e899-4822-b7f4-0d96fa094e3d"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1186.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q26",
|
||||
"category": "paraphrase",
|
||||
"query": "a childhood offering no solid foundation to inherit",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"??o?'?B???k",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1309.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q27",
|
||||
"category": "associative",
|
||||
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"?ǚ?7??????"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1427.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q28",
|
||||
"category": "associative",
|
||||
"query": "Marines hernia sepsis medical ward",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-e0bdf5d8-d163-491f-b649-453fee8b721d",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1081.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q29",
|
||||
"category": "associative",
|
||||
"query": "Sarah Bishop Dyer trailer performance",
|
||||
"returned": [
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1144.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q30",
|
||||
"category": "associative",
|
||||
"query": "Swarm Architecture containment lateral worker",
|
||||
"returned": [
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43",
|
||||
"8cbb60c5-4999-4ec1-8682-2592aedc4249",
|
||||
"bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae",
|
||||
"7?e?7???\f3?",
|
||||
"kn-6f248a50-355b-47bb-aec8-e0e646a9b077",
|
||||
"? ?}&?#??X\b",
|
||||
"h??I?cB?Q??",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1141.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q31",
|
||||
"category": "associative",
|
||||
"query": "hope won inside the narrative preface",
|
||||
"returned": [
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"7?e?7???\f3?",
|
||||
"rQ??m?;?x?'",
|
||||
"?Q??m?;?u?'",
|
||||
"R^??m?;��?'",
|
||||
"?Q??m?;��`'",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1171.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q32",
|
||||
"category": "associative",
|
||||
"query": "man of the house six years old expectation",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1388.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q33",
|
||||
"category": "nonsense",
|
||||
"query": "zqxjvw plimforth grebulon",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 659.0,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q34",
|
||||
"category": "nonsense",
|
||||
"query": "flarnbistle quommetry",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 432.5,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q35",
|
||||
"category": "nonsense",
|
||||
"query": "xxqzzt vurblenacht throom",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"kn-66a21179-2adc-4b19-a109-880cf4674d7d",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 673.7,
|
||||
"error": null,
|
||||
"clean": false,
|
||||
"false_positives": 10
|
||||
},
|
||||
{
|
||||
"id": "q36",
|
||||
"category": "superseded",
|
||||
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"mem-80d7416b-20e9-48a0-b176-b215527e2f56",
|
||||
"mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1254.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.125,
|
||||
"outranks": true,
|
||||
"rank_correct": 8,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q37",
|
||||
"category": "superseded",
|
||||
"query": "how many provisional patents does Will actually have",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"?ǚ?7??????",
|
||||
"[?MO5????G",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1626.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q38",
|
||||
"category": "superseded",
|
||||
"query": "is MCP still the live integration layer",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"%???2??jH??",
|
||||
"???Ͼd??W\b?",
|
||||
"??o?'?B???k",
|
||||
"? ?}&?#??X\b",
|
||||
"63307ac5-cf6b-46e0-8296-07503b461cfa",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 950.9,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,945 @@
|
||||
{
|
||||
"label": "main-r1",
|
||||
"soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-main",
|
||||
"soul_md5": "caf822425540114984bfcd713ad37162",
|
||||
"corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json",
|
||||
"corpus_nodes": 78768,
|
||||
"corpus_edges": 14214,
|
||||
"gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json",
|
||||
"limit": 10,
|
||||
"port": 7893,
|
||||
"wall_clock_s": 45.9,
|
||||
"child_pid": 78554,
|
||||
"child_confirmed_dead": true,
|
||||
"aggregate": {
|
||||
"n_queries": 38,
|
||||
"n_scored": 35,
|
||||
"hit@5": 0.34285714285714286,
|
||||
"recall@5": 0.26947278911564626,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"precision@5": 0.12000000000000001,
|
||||
"mrr@10": 0.2943197278911564,
|
||||
"nonsense_clean": "2/3",
|
||||
"superseded_outranks": "1/3",
|
||||
"latency_ms_p50": 1140.4,
|
||||
"latency_ms_p95": 1584.1,
|
||||
"latency_ms_max": 1627.6,
|
||||
"errors": 0,
|
||||
"by_category": {
|
||||
"associative": {
|
||||
"n": 6,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"exact_rare": {
|
||||
"n": 6,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
"nonsense": {
|
||||
"n": 3,
|
||||
"clean": 2,
|
||||
"avg_false_positives": 3.3333333333333335
|
||||
},
|
||||
"paraphrase": {
|
||||
"n": 13,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
"phrase": {
|
||||
"n": 7,
|
||||
"hit@5": 0.8571428571428571,
|
||||
"recall@5": 0.4902210884353741,
|
||||
"recall@10": 0.6666666666666666,
|
||||
"mrr@10": 0.5965986394557822
|
||||
},
|
||||
"superseded": {
|
||||
"n": 3,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.3333333333333333,
|
||||
"mrr@10": 0.041666666666666664,
|
||||
"outranks": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"id": "q01",
|
||||
"category": "exact_rare",
|
||||
"query": "unjailbreakable",
|
||||
"returned": [
|
||||
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 240.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q02",
|
||||
"category": "exact_rare",
|
||||
"query": "engram-migrate",
|
||||
"returned": [
|
||||
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 271.3,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q03",
|
||||
"category": "exact_rare",
|
||||
"query": "cartabandonedevent",
|
||||
"returned": [
|
||||
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 232.0,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q04",
|
||||
"category": "exact_rare",
|
||||
"query": "pre-apprenticeship",
|
||||
"returned": [
|
||||
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 227.8,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q05",
|
||||
"category": "exact_rare",
|
||||
"query": "inferencenodemanager",
|
||||
"returned": [
|
||||
"mem-73969486-143f-4431-b5e6-6845d1cc9848"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 259.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q06",
|
||||
"category": "exact_rare",
|
||||
"query": "clear-eyed",
|
||||
"returned": [
|
||||
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
||||
],
|
||||
"n_returned": 1,
|
||||
"latency_ms": 237.4,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q07",
|
||||
"category": "phrase",
|
||||
"query": "patterns not returns",
|
||||
"returned": [
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"?ǚ?7??????",
|
||||
"mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"????7???Ջ3",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b",
|
||||
"7?e?7???\f3?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 543.7,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.3333333333333333
|
||||
},
|
||||
{
|
||||
"id": "q08",
|
||||
"category": "phrase",
|
||||
"query": "thirty moves",
|
||||
"returned": [
|
||||
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
||||
"knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6",
|
||||
"Z[?<S???H??",
|
||||
"rQ??m?;?x?'",
|
||||
"kn-f230b362-b201-4402-9833-4160c89ab3d4",
|
||||
"kn-6061318f-046b-4935-907d-8eafdce14930"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 467.1,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3125,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 1.0,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q09",
|
||||
"category": "phrase",
|
||||
"query": "Grandma Lucas",
|
||||
"returned": [
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 475.2,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.3333333333333333,
|
||||
"recall@10": 0.5555555555555556,
|
||||
"precision@5": 0.6,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q10",
|
||||
"category": "phrase",
|
||||
"query": "Directed Harmonic",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"ԍ????X????",
|
||||
"bl-dcee1887-34c4-4ffa-9119-1e291685ba08",
|
||||
"?ǚ?7??????",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf",
|
||||
"????7???Ջ3",
|
||||
"??????X??2c",
|
||||
"g?e?7???'c?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 449.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.1111111111111111,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.14285714285714285
|
||||
},
|
||||
{
|
||||
"id": "q11",
|
||||
"category": "phrase",
|
||||
"query": "Sarah Bishop",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
||||
"mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35",
|
||||
"mem-b8ecd23e-77ce-42f7-984c-f51453fec16d"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 463.9,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.5,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.2
|
||||
},
|
||||
{
|
||||
"id": "q12",
|
||||
"category": "phrase",
|
||||
"query": "Directed Autonomous Runtime Modification",
|
||||
"returned": [
|
||||
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
||||
"bl-145a0985-2382-400f-a7c5-c335c5e30a72",
|
||||
"mem-82b93b21-a865-410f-9ec1-fc54121d9bb5",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"mem-e6327f52-2bda-4ce7-9471-2fffd1e172de",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 821.5,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 0.2857142857142857,
|
||||
"recall@10": 0.5,
|
||||
"precision@5": 0.8,
|
||||
"mrr@10": 1.0
|
||||
},
|
||||
{
|
||||
"id": "q13",
|
||||
"category": "phrase",
|
||||
"query": "zero-knowledge encrypted backup",
|
||||
"returned": [
|
||||
"7774a16c-1027-4e3b-a21e-67f1f95a4acd",
|
||||
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a",
|
||||
"mem-dba009a2-d2ea-4f5a-b9e8-0f04bc9ab32f",
|
||||
"mem-a3c97012-5fa3-4915-a839-2c75c72005e0",
|
||||
"mem-7cd90611-88a3-423d-a38a-0db2812952fa",
|
||||
"bl-07375bf9-a169-42cd-adb3-7d32b25982f0",
|
||||
"bl-ec84b63d-b278-4944-8d7f-4aa7a51c0315",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 688.0,
|
||||
"error": null,
|
||||
"hit@5": 1.0,
|
||||
"recall@5": 1.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.2,
|
||||
"mrr@10": 0.5
|
||||
},
|
||||
{
|
||||
"id": "q14",
|
||||
"category": "paraphrase",
|
||||
"query": "the elderly relative who passed while he stayed away",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"????7???Ջ3",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1559.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q15",
|
||||
"category": "paraphrase",
|
||||
"query": "a soldier sidelined by illness who refused to quit",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a31e1001-342e-4deb-a2e6-6d02d1f22dee",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1627.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q16",
|
||||
"category": "paraphrase",
|
||||
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"?of?7???",
|
||||
"????7???Ջ3",
|
||||
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"kn-81c24d13-a73b-4767-819c-dafaacc1498e",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1317.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q17",
|
||||
"category": "paraphrase",
|
||||
"query": "a tight payload beats a bloated one",
|
||||
"returned": [
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"??o?'?B???k",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1055.2,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q18",
|
||||
"category": "paraphrase",
|
||||
"query": "if you are able and nobody is coming the job is yours",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"imp-dce1da0f-8776-4a9e-972b-33411a7ca138",
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-83bb86c6-521d-416c-a86e-6e29c2d8f102",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1352.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q19",
|
||||
"category": "paraphrase",
|
||||
"query": "learning is the wealth creditors cannot seize",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
||||
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"??o?'?B???k",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-0625e393-067c-4bba-8389-7e1b79265142",
|
||||
"ע?RGk?\tH(?"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1213.0,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q20",
|
||||
"category": "paraphrase",
|
||||
"query": "reliability proven by track record not assertion",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1545.8,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q21",
|
||||
"category": "paraphrase",
|
||||
"query": "boundaries that enable instead of confine",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"??????X??2c",
|
||||
"ԍ????X????",
|
||||
"dR????X?-?S",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"dz????Xƹ?i",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1327.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q22",
|
||||
"category": "paraphrase",
|
||||
"query": "what shifts tells you where to cut a system apart",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???",
|
||||
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"dR????X?-?S",
|
||||
"dz????Xƹ?i",
|
||||
"ԍ????X????",
|
||||
"??????X??2c",
|
||||
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1584.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q23",
|
||||
"category": "paraphrase",
|
||||
"query": "a mind that compounds instead of resetting each day",
|
||||
"returned": [
|
||||
"????7???Ջ3",
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"?ǚ?7??????",
|
||||
"7?e?7???\f3?",
|
||||
"??f?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"%???2??jH??",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1432.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q24",
|
||||
"category": "paraphrase",
|
||||
"query": "loved for the unedited self and not the polished exterior",
|
||||
"returned": [
|
||||
"mem-bbb126a1-b297-42bb-86be-796871829c94",
|
||||
"mem-45022957-2d78-48aa-a714-16d6eca52e0f",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"'?T?a\"B~-?8"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1505.5,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q25",
|
||||
"category": "paraphrase",
|
||||
"query": "cheerfulness you arrive at instead of assuming",
|
||||
"returned": [
|
||||
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
||||
"?ǚ?7??????",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"7?e?7???\f3?",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"??S?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"??f?7???",
|
||||
"??S?7???",
|
||||
"knw-5578cb21-e899-4822-b7f4-0d96fa094e3d"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1201.1,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q26",
|
||||
"category": "paraphrase",
|
||||
"query": "a childhood offering no solid foundation to inherit",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"7?e?7???\f3?",
|
||||
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
||||
"? ?}&?#??X\b",
|
||||
"??o?'?B???k",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1303.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q27",
|
||||
"category": "associative",
|
||||
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"?ǚ?7??????"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1416.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q28",
|
||||
"category": "associative",
|
||||
"query": "Marines hernia sepsis medical ward",
|
||||
"returned": [
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
||||
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"art-e0bdf5d8-d163-491f-b649-453fee8b721d",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1090.3,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q29",
|
||||
"category": "associative",
|
||||
"query": "Sarah Bishop Dyer trailer performance",
|
||||
"returned": [
|
||||
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1167.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q30",
|
||||
"category": "associative",
|
||||
"query": "Swarm Architecture containment lateral worker",
|
||||
"returned": [
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
||||
"bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43",
|
||||
"8cbb60c5-4999-4ec1-8682-2592aedc4249",
|
||||
"bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae",
|
||||
"7?e?7???\f3?",
|
||||
"kn-6f248a50-355b-47bb-aec8-e0e646a9b077",
|
||||
"? ?}&?#??X\b",
|
||||
"h??I?cB?Q??",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1140.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q31",
|
||||
"category": "associative",
|
||||
"query": "hope won inside the narrative preface",
|
||||
"returned": [
|
||||
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
||||
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
||||
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
||||
"7?e?7???\f3?",
|
||||
"rQ??m?;?x?'",
|
||||
"?Q??m?;?u?'",
|
||||
"R^??m?;��?'",
|
||||
"?Q??m?;��`'",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 9,
|
||||
"latency_ms": 1157.6,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q32",
|
||||
"category": "associative",
|
||||
"query": "man of the house six years old expectation",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
||||
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1379.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0
|
||||
},
|
||||
{
|
||||
"id": "q33",
|
||||
"category": "nonsense",
|
||||
"query": "zqxjvw plimforth grebulon",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 677.2,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q34",
|
||||
"category": "nonsense",
|
||||
"query": "flarnbistle quommetry",
|
||||
"returned": [],
|
||||
"n_returned": 0,
|
||||
"latency_ms": 443.2,
|
||||
"error": null,
|
||||
"clean": true,
|
||||
"false_positives": 0
|
||||
},
|
||||
{
|
||||
"id": "q35",
|
||||
"category": "nonsense",
|
||||
"query": "xxqzzt vurblenacht throom",
|
||||
"returned": [
|
||||
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
||||
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4",
|
||||
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
||||
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
||||
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
||||
"kn-66a21179-2adc-4b19-a109-880cf4674d7d",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 679.8,
|
||||
"error": null,
|
||||
"clean": false,
|
||||
"false_positives": 10
|
||||
},
|
||||
{
|
||||
"id": "q36",
|
||||
"category": "superseded",
|
||||
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
||||
"returned": [
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"mem-80d7416b-20e9-48a0-b176-b215527e2f56",
|
||||
"mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da",
|
||||
"art-80ca3d31-84dc-4502-83f4-538372b9764f"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1245.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 1.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.125,
|
||||
"outranks": true,
|
||||
"rank_correct": 8,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q37",
|
||||
"category": "superseded",
|
||||
"query": "how many provisional patents does Will actually have",
|
||||
"returned": [
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"? ?}&?#??X\b",
|
||||
"7?e?7???\f3?",
|
||||
"?of?7???",
|
||||
"?ǚ?7??????",
|
||||
"[?MO5????G",
|
||||
"art-ee615cdb-e599-423d-9a4d-977859390ed3"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 1618.4,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
},
|
||||
{
|
||||
"id": "q38",
|
||||
"category": "superseded",
|
||||
"query": "is MCP still the live integration layer",
|
||||
"returned": [
|
||||
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
||||
"7?e?7???\f3?",
|
||||
"? ?}&?#??X\b",
|
||||
"%???2??jH??",
|
||||
"???Ͼd??W\b?",
|
||||
"??o?'?B???k",
|
||||
"? ?}&?#??X\b",
|
||||
"63307ac5-cf6b-46e0-8296-07503b461cfa",
|
||||
"? ?}&?#??X\b",
|
||||
"?of?7???"
|
||||
],
|
||||
"n_returned": 10,
|
||||
"latency_ms": 956.7,
|
||||
"error": null,
|
||||
"hit@5": 0.0,
|
||||
"recall@5": 0.0,
|
||||
"recall@10": 0.0,
|
||||
"precision@5": 0.0,
|
||||
"mrr@10": 0.0,
|
||||
"outranks": false,
|
||||
"rank_correct": null,
|
||||
"rank_stale": null
|
||||
}
|
||||
]
|
||||
}
|
||||
Executable
+98
@@ -0,0 +1,98 @@
|
||||
#!/usr/bin/env bash
|
||||
# run_comparison.sh — the whole harness, end to end, from two git refs.
|
||||
#
|
||||
# Builds a soul from each ref, boots each on its own throwaway port with its own
|
||||
# throwaway HOME and its own disposable copy of the corpus, runs the gold set N
|
||||
# times per ref, and prints the comparison with its noise threshold.
|
||||
#
|
||||
# SAFETY: never touches ~/.neuron, /Applications/Neuron*, ~/neuron-dev-stack, or
|
||||
# any running service. Sources are exported with `git archive` into a scratch
|
||||
# dir, so no worktree or branch state is mutated either. Ports are checked
|
||||
# against the live set before anything boots. Every soul this script starts is
|
||||
# killed and confirmed dead by run_eval.py; the sweep at the end is a backstop.
|
||||
#
|
||||
# usage:
|
||||
# run_comparison.sh [--baseline main] [--candidate feat/recall-through-activation]
|
||||
# [--repeats 3] [--corpus <snapshot.json>] [--repo <path>]
|
||||
set -euo pipefail
|
||||
|
||||
BASELINE="main"
|
||||
CANDIDATE="feat/recall-through-activation"
|
||||
REPEATS=3
|
||||
CORPUS="$HOME/neuron-memory-backups/snapshot-pre-repair-20260806.json"
|
||||
REPO="$HOME/Development/neuron"
|
||||
BASE_PORT=7893
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--baseline) BASELINE="$2"; shift 2 ;;
|
||||
--candidate) CANDIDATE="$2"; shift 2 ;;
|
||||
--repeats) REPEATS="$2"; shift 2 ;;
|
||||
--corpus) CORPUS="$2"; shift 2 ;;
|
||||
--repo) REPO="$2"; shift 2 ;;
|
||||
*) echo "unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
WORK="$(mktemp -d "${TMPDIR:-/tmp}/retrieval-eval.XXXXXX")"
|
||||
trap 'rm -rf "$WORK"' EXIT
|
||||
|
||||
[ -f "$CORPUS" ] || { echo "no corpus at $CORPUS" >&2; exit 2; }
|
||||
echo "corpus: $CORPUS ($(du -h "$CORPUS" | cut -f1))"
|
||||
|
||||
slug() { printf '%s' "$1" | tr '/' '-'; }
|
||||
|
||||
build_ref() { # ref -> binary path
|
||||
local ref="$1" out="$WORK/soul-$(slug "$1")"
|
||||
local src="$WORK/src-$(slug "$1")"
|
||||
mkdir -p "$src"
|
||||
git -C "$REPO" archive "$ref" | tar -x -C "$src"
|
||||
"$HERE/build-soul.sh" "$src" "$out" >&2
|
||||
printf '%s' "$out"
|
||||
}
|
||||
|
||||
echo "== building $BASELINE =="
|
||||
BIN_A="$(build_ref "$BASELINE")"
|
||||
echo "== building $CANDIDATE =="
|
||||
BIN_B="$(build_ref "$CANDIDATE")"
|
||||
|
||||
echo "== validating the gold set against this corpus =="
|
||||
python3 "$HERE/build_gold_set.py" "$CORPUS" --check
|
||||
|
||||
port=$BASE_PORT
|
||||
run_one() { # binary label out
|
||||
echo "== $2 =="
|
||||
python3 "$HERE/run_eval.py" --soul "$1" --corpus "$CORPUS" --label "$2" \
|
||||
--port "$port" --out "$3"
|
||||
port=$((port + 1))
|
||||
}
|
||||
|
||||
A_MAIN="$WORK/results-a-1.json"; B_MAIN="$WORK/results-b-1.json"
|
||||
A_REP=(); B_REP=()
|
||||
for i in $(seq 1 "$REPEATS"); do
|
||||
a="$WORK/results-a-$i.json"; b="$WORK/results-b-$i.json"
|
||||
run_one "$BIN_A" "$(slug "$BASELINE")-r$i" "$a"
|
||||
run_one "$BIN_B" "$(slug "$CANDIDATE")-r$i" "$b"
|
||||
[ "$i" -gt 1 ] && { A_REP+=("$a"); B_REP+=("$b"); }
|
||||
done
|
||||
|
||||
cp "$A_MAIN" "$HERE/results-$(slug "$BASELINE").json"
|
||||
cp "$B_MAIN" "$HERE/results-$(slug "$CANDIDATE").json"
|
||||
|
||||
echo
|
||||
python3 "$HERE/compare.py" \
|
||||
--baseline "$A_MAIN" --candidate "$B_MAIN" \
|
||||
${A_REP[@]+--repeats-baseline "${A_REP[@]}"} \
|
||||
${B_REP[@]+--repeats-candidate "${B_REP[@]}"} \
|
||||
--out "$HERE/comparison-$(slug "$BASELINE")-vs-$(slug "$CANDIDATE").json"
|
||||
|
||||
# Backstop: run_eval.py kills and confirms its own child, but a crashed run
|
||||
# could leak one. Leaving a soul running is how the live engine got squeezed.
|
||||
STRAY=$(pgrep -f "$WORK/soul-" || true)
|
||||
if [ -n "$STRAY" ]; then
|
||||
echo "!! stray eval souls, killing: $STRAY" >&2
|
||||
kill -9 $STRAY 2>/dev/null || true
|
||||
fi
|
||||
pgrep -f "$WORK/soul-" >/dev/null && { echo "!! STILL RUNNING" >&2; exit 5; }
|
||||
echo "process check: no eval souls running"
|
||||
Executable
+398
@@ -0,0 +1,398 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
run_eval.py — measure one soul build's retrieval against the gold set.
|
||||
|
||||
WHAT IT MEASURES, AND WHY IT BOOTS A REAL SOUL
|
||||
The point is Will's designed retrieval — spreading activation over the
|
||||
weighted directed graph with four-factor multiplicative scoring — not a
|
||||
Python re-implementation of it. A re-implementation would measure my
|
||||
reading of the design; booting the compiled binary measures the design. So
|
||||
this harness compiles the actual `soul.el` amalgam (build-soul.sh) and asks
|
||||
it over HTTP, exactly as the MCP wrapper and the app do.
|
||||
|
||||
SAFETY — read this before changing anything here
|
||||
* Boots on a THROWAWAY port with a THROWAWAY $HOME and a THROWAWAY COPY of
|
||||
the corpus. Refuses to use 7770 / 8742 / 7779 / 17779 / 7771.
|
||||
* ENGRAM_URL / SOUL_ENGRAM_URL are UNSET and SOUL_ISE_URL is pinned to a
|
||||
dead port. This is not belt-and-braces: the periodic engram sync resolves
|
||||
its source as env(SOUL_ISE_URL) -> state -> DEFAULT http://localhost:8742,
|
||||
so leaving it unset makes an "isolated" run silently pull the operator's
|
||||
LIVE brain. (Learned the hard way on 2026-08-03; see the same note in
|
||||
scripts/verify-soul-contract.sh.)
|
||||
* Every process this file starts is tracked and killed in a finally block,
|
||||
then CONFIRMED dead by pid probe, and the confirmation is written into the
|
||||
results file. A run that cannot confirm its child is dead exits non-zero.
|
||||
* Activation is a STATEFUL read by design (patent claim 29: traversal
|
||||
updates last-activation and increments activation counts). The corpus copy
|
||||
is therefore per-run and disposable, and every run starts from a byte-
|
||||
identical copy so two configurations see the same starting graph.
|
||||
|
||||
usage:
|
||||
python3 run_eval.py --soul <binary> --corpus <snapshot.json> --label main \
|
||||
[--port 7893] [--gold gold_set.json] [--limit 10] [--out results-main.json]
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
FORBIDDEN_PORTS = {7770, 8742, 7779, 17779, 7771, 8080}
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# metrics
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
def recall_at_k(returned, relevant, k):
|
||||
if not relevant:
|
||||
return None
|
||||
return len(set(returned[:k]) & set(relevant)) / len(relevant)
|
||||
|
||||
|
||||
def hit_at_k(returned, relevant, k):
|
||||
if not relevant:
|
||||
return None
|
||||
return 1.0 if set(returned[:k]) & set(relevant) else 0.0
|
||||
|
||||
|
||||
def precision_at_k(returned, relevant, k):
|
||||
"""Fixed denominator k, as in docs/research/graphrag_eval/score.py.
|
||||
|
||||
Fixed denominator penalises an empty result and a page of junk equally,
|
||||
which is what we want: a retriever that returns nothing is not 'precise'.
|
||||
"""
|
||||
if not relevant:
|
||||
return None
|
||||
return len(set(returned[:k]) & set(relevant)) / k
|
||||
|
||||
|
||||
def mrr(returned, relevant, k):
|
||||
if not relevant:
|
||||
return None
|
||||
rel = set(relevant)
|
||||
for i, nid in enumerate(returned[:k], start=1):
|
||||
if nid in rel:
|
||||
return 1.0 / i
|
||||
return 0.0
|
||||
|
||||
|
||||
def mean(vals):
|
||||
vals = [v for v in vals if v is not None]
|
||||
return sum(vals) / len(vals) if vals else 0.0
|
||||
|
||||
|
||||
def pct(vals):
|
||||
return f"{100 * mean(vals):.1f}%"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# soul lifecycle
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
class Soul:
|
||||
def __init__(self, binary, corpus, port, verbose=True):
|
||||
if port in FORBIDDEN_PORTS:
|
||||
raise SystemExit(f"REFUSING: port {port} is a live service port.")
|
||||
self.binary = os.path.abspath(binary)
|
||||
self.corpus = os.path.abspath(corpus)
|
||||
self.port = port
|
||||
self.verbose = verbose
|
||||
self.home = None
|
||||
self.proc = None
|
||||
self.pid = None
|
||||
self.log = None
|
||||
self.confirmed_dead = None
|
||||
|
||||
@property
|
||||
def base(self):
|
||||
return f"http://127.0.0.1:{self.port}"
|
||||
|
||||
def start(self, boot_timeout=180):
|
||||
self.home = tempfile.mkdtemp(prefix="retrieval-eval-home.")
|
||||
snap = os.path.join(self.home, "corpus.json")
|
||||
t0 = time.time()
|
||||
shutil.copyfile(self.corpus, snap) # per-run disposable copy, never the source
|
||||
self.log = os.path.join(self.home, "soul.log")
|
||||
|
||||
env = {k: v for k, v in os.environ.items()
|
||||
if k not in ("ENGRAM_URL", "ENGRAM_API_KEY", "SOUL_ENGRAM_URL",
|
||||
"ANTHROPIC_API_KEY", "NEURON_LLM_API_KEY", "SOUL_IDENTITY",
|
||||
"SOUL_API_KEY")}
|
||||
env.update({
|
||||
"HOME": self.home,
|
||||
"NEURON_PORT": str(self.port),
|
||||
"SOUL_CGI_ID": f"ntn-retrieval-eval-{os.getpid()}",
|
||||
"SOUL_ENGRAM_PATH": snap,
|
||||
"NEURON_API_URL": "http://127.0.0.1:9", # dead port
|
||||
"SOUL_ISE_URL": "http://127.0.0.1:9", # dead port — see SAFETY above
|
||||
# Park the background loops for an hour so heartbeat/consolidation
|
||||
# cannot mutate the graph between queries and make runs unrepeatable.
|
||||
"SOUL_TICK_MS": "3600000",
|
||||
"SOUL_HEARTBEAT_MS": "3600000",
|
||||
"SOUL_REFRESH_MS": "3600000",
|
||||
})
|
||||
with open(self.log, "wb") as lf:
|
||||
self.proc = subprocess.Popen([self.binary], env=env, stdout=lf, stderr=lf,
|
||||
start_new_session=True)
|
||||
self.pid = self.proc.pid
|
||||
if self.verbose:
|
||||
print(f" booted pid={self.pid} port={self.port} home={self.home}")
|
||||
|
||||
deadline = time.time() + boot_timeout
|
||||
while time.time() < deadline:
|
||||
if self.proc.poll() is not None:
|
||||
raise RuntimeError(f"soul exited during boot: {self._log_tail()}")
|
||||
rss = self._rss_kb()
|
||||
if rss and rss > 6 * 1024 * 1024:
|
||||
self.stop()
|
||||
raise RuntimeError(f"soul RSS {rss}KB > 6GB — aborted")
|
||||
try:
|
||||
with urllib.request.urlopen(f"{self.base}/health", timeout=2) as r:
|
||||
if r.status == 200:
|
||||
if self.verbose:
|
||||
print(f" healthy in {time.time() - t0:.1f}s, RSS={self._rss_kb()}KB")
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(0.5)
|
||||
self.stop()
|
||||
raise RuntimeError(f"soul never healthy on {self.base}: {self._log_tail()}")
|
||||
|
||||
def _rss_kb(self):
|
||||
try:
|
||||
out = subprocess.run(["ps", "-o", "rss=", "-p", str(self.pid)],
|
||||
capture_output=True, text=True, timeout=5).stdout.strip()
|
||||
return int(out) if out else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def _log_tail(self, n=15):
|
||||
try:
|
||||
with open(self.log, encoding="utf-8", errors="replace") as fh:
|
||||
return "\n".join(fh.read().splitlines()[-n:])
|
||||
except Exception:
|
||||
return "(no log)"
|
||||
|
||||
def recall(self, query, limit, timeout=60):
|
||||
url = f"{self.base}/api/neuron/recall?query={urllib.parse.quote(query)}&limit={limit}"
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=timeout) as r:
|
||||
raw = r.read().decode("utf-8", "replace")
|
||||
ms = (time.perf_counter() - t0) * 1000
|
||||
except Exception as exc:
|
||||
return [], (time.perf_counter() - t0) * 1000, f"{type(exc).__name__}: {exc}"
|
||||
try:
|
||||
arr = json.loads(raw)
|
||||
except Exception:
|
||||
return [], ms, f"unparseable response ({len(raw)}B)"
|
||||
if not isinstance(arr, list):
|
||||
return [], ms, f"non-array response: {str(arr)[:120]}"
|
||||
ids = [x.get("id") for x in arr if isinstance(x, dict) and x.get("id")]
|
||||
return ids, ms, None
|
||||
|
||||
def stop(self):
|
||||
"""Kill and CONFIRM. A test process that outlives its test is a bug."""
|
||||
if self.pid is None:
|
||||
self.confirmed_dead = True
|
||||
return True
|
||||
for sig in (signal.SIGTERM, signal.SIGKILL):
|
||||
try:
|
||||
os.kill(self.pid, sig)
|
||||
except ProcessLookupError:
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
for _ in range(20):
|
||||
try:
|
||||
os.kill(self.pid, 0)
|
||||
except ProcessLookupError:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
else:
|
||||
continue
|
||||
break
|
||||
try:
|
||||
self.proc.wait(timeout=5)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
os.kill(self.pid, 0)
|
||||
self.confirmed_dead = False
|
||||
except ProcessLookupError:
|
||||
self.confirmed_dead = True
|
||||
if self.verbose:
|
||||
print(f" pid {self.pid}: {'CONFIRMED DEAD' if self.confirmed_dead else 'STILL ALIVE'}")
|
||||
if self.home and os.path.isdir(self.home):
|
||||
shutil.rmtree(self.home, ignore_errors=True)
|
||||
return self.confirmed_dead
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# eval
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
def evaluate(soul, gold, limit):
|
||||
rows = []
|
||||
for q in gold["queries"]:
|
||||
ids, ms, err = soul.recall(q["query"], limit)
|
||||
rel = q.get("relevant") or []
|
||||
row = {
|
||||
"id": q["id"],
|
||||
"category": q["category"],
|
||||
"query": q["query"],
|
||||
"returned": ids,
|
||||
"n_returned": len(ids),
|
||||
"latency_ms": round(ms, 1),
|
||||
"error": err,
|
||||
}
|
||||
if q.get("expect_empty"):
|
||||
row["clean"] = (len(ids) == 0)
|
||||
row["false_positives"] = len(ids)
|
||||
else:
|
||||
row["hit@5"] = hit_at_k(ids, rel, 5)
|
||||
row["recall@5"] = recall_at_k(ids, rel, 5)
|
||||
row["recall@10"] = recall_at_k(ids, rel, 10)
|
||||
row["precision@5"] = precision_at_k(ids, rel, 5)
|
||||
row["mrr@10"] = mrr(ids, rel, 10)
|
||||
if q.get("must_outrank"):
|
||||
correct, stale = q["must_outrank"]
|
||||
ic = ids.index(correct) if correct in ids else None
|
||||
istale = ids.index(stale) if stale in ids else None
|
||||
# Correct must be present AND above the stale node. A run that
|
||||
# returns neither is NOT a pass: the corrected fact is what the
|
||||
# user needed.
|
||||
row["outranks"] = (ic is not None) and (istale is None or ic < istale)
|
||||
row["rank_correct"] = None if ic is None else ic + 1
|
||||
row["rank_stale"] = None if istale is None else istale + 1
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def aggregate(rows):
|
||||
scored = [r for r in rows if "hit@5" in r]
|
||||
nonsense = [r for r in rows if "clean" in r]
|
||||
outrank = [r for r in rows if "outranks" in r]
|
||||
lat = sorted(r["latency_ms"] for r in rows)
|
||||
agg = {
|
||||
"n_queries": len(rows),
|
||||
"n_scored": len(scored),
|
||||
"hit@5": mean([r["hit@5"] for r in scored]),
|
||||
"recall@5": mean([r["recall@5"] for r in scored]),
|
||||
"recall@10": mean([r["recall@10"] for r in scored]),
|
||||
"precision@5": mean([r["precision@5"] for r in scored]),
|
||||
"mrr@10": mean([r["mrr@10"] for r in scored]),
|
||||
"nonsense_clean": f"{sum(1 for r in nonsense if r['clean'])}/{len(nonsense)}",
|
||||
"superseded_outranks": f"{sum(1 for r in outrank if r['outranks'])}/{len(outrank)}",
|
||||
"latency_ms_p50": lat[len(lat) // 2] if lat else 0,
|
||||
"latency_ms_p95": lat[max(0, int(len(lat) * 0.95) - 1)] if lat else 0,
|
||||
"latency_ms_max": lat[-1] if lat else 0,
|
||||
"errors": sum(1 for r in rows if r["error"]),
|
||||
"by_category": {},
|
||||
}
|
||||
cats = sorted({r["category"] for r in rows})
|
||||
for c in cats:
|
||||
cr = [r for r in rows if r["category"] == c]
|
||||
if c == "nonsense":
|
||||
agg["by_category"][c] = {
|
||||
"n": len(cr),
|
||||
"clean": sum(1 for r in cr if r["clean"]),
|
||||
"avg_false_positives": mean([float(r["false_positives"]) for r in cr]),
|
||||
}
|
||||
else:
|
||||
e = {
|
||||
"n": len(cr),
|
||||
"hit@5": mean([r.get("hit@5") for r in cr]),
|
||||
"recall@5": mean([r.get("recall@5") for r in cr]),
|
||||
"recall@10": mean([r.get("recall@10") for r in cr]),
|
||||
"mrr@10": mean([r.get("mrr@10") for r in cr]),
|
||||
}
|
||||
if c == "superseded":
|
||||
e["outranks"] = sum(1 for r in cr if r.get("outranks"))
|
||||
agg["by_category"][c] = e
|
||||
return agg
|
||||
|
||||
|
||||
def print_table(label, agg):
|
||||
print(f"\n=== {label} ===")
|
||||
print(f" queries {agg['n_queries']} ({agg['n_scored']} scored + "
|
||||
f"{agg['n_queries'] - agg['n_scored']} control) · errors {agg['errors']}")
|
||||
print(f" {'hit@5':>12} {'recall@5':>10} {'recall@10':>10} {'prec@5':>9} {'MRR@10':>9}")
|
||||
print(f" {pct([agg['hit@5']]):>12} {pct([agg['recall@5']]):>10} {pct([agg['recall@10']]):>10} "
|
||||
f"{pct([agg['precision@5']]):>9} {agg['mrr@10']:>9.3f}")
|
||||
print(f" nonsense clean {agg['nonsense_clean']} · superseded outranks {agg['superseded_outranks']}")
|
||||
print(f" latency ms p50 {agg['latency_ms_p50']:.0f} · p95 {agg['latency_ms_p95']:.0f} "
|
||||
f"· max {agg['latency_ms_max']:.0f}")
|
||||
print(f"\n {'category':14} {'n':>3} {'hit@5':>8} {'recall@5':>9} {'recall@10':>10} {'MRR@10':>8}")
|
||||
for c, e in agg["by_category"].items():
|
||||
if c == "nonsense":
|
||||
print(f" {c:14} {e['n']:>3} {'clean ' + str(e['clean']) + '/' + str(e['n']):>8}"
|
||||
f"{'':>9} {'':>10} {'avg FP ' + format(e['avg_false_positives'], '.1f'):>8}")
|
||||
else:
|
||||
extra = f" outranks {e['outranks']}/{e['n']}" if "outranks" in e else ""
|
||||
print(f" {c:14} {e['n']:>3} {pct([e['hit@5']]):>8} {pct([e['recall@5']]):>9} "
|
||||
f"{pct([e['recall@10']]):>10} {e['mrr@10']:>8.3f}{extra}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--soul", required=True)
|
||||
ap.add_argument("--corpus", required=True)
|
||||
ap.add_argument("--label", required=True)
|
||||
ap.add_argument("--gold", default=os.path.join(HERE, "gold_set.json"))
|
||||
ap.add_argument("--port", type=int, default=7893)
|
||||
ap.add_argument("--limit", type=int, default=10)
|
||||
ap.add_argument("--out", default=None)
|
||||
args = ap.parse_args()
|
||||
|
||||
with open(args.gold, encoding="utf-8") as fh:
|
||||
gold = json.load(fh)
|
||||
print(f"[{args.label}] soul={os.path.basename(args.soul)} "
|
||||
f"corpus={os.path.basename(args.corpus)} gold={len(gold['queries'])}q limit={args.limit}")
|
||||
|
||||
soul = Soul(args.soul, args.corpus, args.port)
|
||||
rows = []
|
||||
started = time.time()
|
||||
try:
|
||||
soul.start()
|
||||
rows = evaluate(soul, gold, args.limit)
|
||||
finally:
|
||||
dead = soul.stop()
|
||||
|
||||
agg = aggregate(rows)
|
||||
print_table(args.label, agg)
|
||||
|
||||
out = args.out or os.path.join(HERE, f"results-{args.label}.json")
|
||||
doc = {
|
||||
"label": args.label,
|
||||
"soul_binary": os.path.abspath(args.soul),
|
||||
"soul_md5": subprocess.run(["md5", "-q", args.soul], capture_output=True,
|
||||
text=True).stdout.strip(),
|
||||
"corpus": os.path.abspath(args.corpus),
|
||||
"corpus_nodes": gold.get("corpus_nodes"),
|
||||
"corpus_edges": gold.get("corpus_edges"),
|
||||
"gold_set": os.path.abspath(args.gold),
|
||||
"limit": args.limit,
|
||||
"port": args.port,
|
||||
"wall_clock_s": round(time.time() - started, 1),
|
||||
"child_pid": soul.pid,
|
||||
"child_confirmed_dead": soul.confirmed_dead,
|
||||
"aggregate": agg,
|
||||
"rows": rows,
|
||||
}
|
||||
with open(out, "w", encoding="utf-8") as fh:
|
||||
json.dump(doc, fh, indent=1, ensure_ascii=False)
|
||||
print(f"\nwrote {out}")
|
||||
if not dead:
|
||||
print("FATAL: child process could not be confirmed dead", file=sys.stderr)
|
||||
sys.exit(4)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+212
-21
@@ -41,6 +41,7 @@
|
||||
#include <fcntl.h>
|
||||
#include <dirent.h>
|
||||
#include <errno.h>
|
||||
#include <signal.h> /* SIGPIPE disposition — see el_runtime_ignore_sigpipe */
|
||||
#include <pthread.h>
|
||||
#include <curl/curl.h>
|
||||
|
||||
@@ -1238,16 +1239,77 @@ static const char* http_reason_phrase(int status) {
|
||||
}
|
||||
}
|
||||
|
||||
/* Best-effort send with retry on partial writes. */
|
||||
/* ── A departing client MUST NOT be able to kill the daemon ──────────────────
|
||||
* (2026-08-06, round 9.1 / ADR 0006 item 4.)
|
||||
*
|
||||
* Measured field failure: a client cancelled its request at 25 s; the handler
|
||||
* finished its work at 116.9 s and wrote the reply into the departed client's
|
||||
* socket. The second send() on a reset connection raised SIGPIPE, whose DEFAULT
|
||||
* disposition terminates the process — `exited due to SIGPIPE ... ran for
|
||||
* 361177ms`. launchd respawned 4 ms later, so EVERY other in-flight request on
|
||||
* that daemon lost its work, silently.
|
||||
*
|
||||
* Two independent guards, because one of them can be undone from outside this
|
||||
* file (an embedder may reset signal dispositions) and the other cannot:
|
||||
* 1. process-wide SIGPIPE -> SIG_IGN, installed at runtime init;
|
||||
* 2. per-send suppression at the syscall (MSG_NOSIGNAL where the platform has
|
||||
* it, SO_NOSIGPIPE on the accepted socket on macOS/BSD).
|
||||
* With either in force, send() reports the peer's departure as EPIPE and the
|
||||
* caller decides — which is the point: this is an ordinary I/O outcome, not a
|
||||
* fatal condition.
|
||||
*
|
||||
* It deliberately does NOT swallow the error. http_send_response() below
|
||||
* classifies the errno and logs: "client left" for a departure, and a real
|
||||
* "send failed: <strerror>" for anything else, so a genuine write fault is
|
||||
* still visible in the log (spec round-9.1 §5.3). */
|
||||
|
||||
#ifndef MSG_NOSIGNAL
|
||||
#define MSG_NOSIGNAL 0
|
||||
#endif
|
||||
|
||||
void el_runtime_ignore_sigpipe(void) {
|
||||
static int done = 0;
|
||||
if (done) return;
|
||||
done = 1;
|
||||
struct sigaction sa;
|
||||
memset(&sa, 0, sizeof(sa));
|
||||
sa.sa_handler = SIG_IGN;
|
||||
sigemptyset(&sa.sa_mask);
|
||||
sigaction(SIGPIPE, &sa, NULL);
|
||||
}
|
||||
|
||||
/* Suppress SIGPIPE for one accepted connection (macOS/BSD have no
|
||||
* MSG_NOSIGNAL; they have the socket option instead). Best effort. */
|
||||
static void http_socket_nosigpipe(int fd) {
|
||||
#ifdef SO_NOSIGPIPE
|
||||
int on = 1;
|
||||
setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &on, sizeof(on));
|
||||
#else
|
||||
(void)fd;
|
||||
#endif
|
||||
}
|
||||
|
||||
/* Best-effort send with retry on partial writes.
|
||||
* Returns 0 on success, -1 on failure with errno preserved for the caller. */
|
||||
static int http_send_all(int fd, const char* p, size_t left) {
|
||||
while (left > 0) {
|
||||
ssize_t w = send(fd, p, left, 0);
|
||||
if (w <= 0) return -1;
|
||||
ssize_t w = send(fd, p, left, MSG_NOSIGNAL);
|
||||
if (w < 0) {
|
||||
if (errno == EINTR) continue; /* not an error — retry */
|
||||
return -1; /* errno stays set for caller */
|
||||
}
|
||||
if (w == 0) { errno = EPIPE; return -1; }
|
||||
p += w; left -= (size_t)w;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Did this write fail because the client is gone, or because something is
|
||||
* actually wrong with the socket? Only the first is routine. */
|
||||
static int http_write_err_is_client_gone(int e) {
|
||||
return e == EPIPE || e == ECONNRESET || e == ENOTCONN || e == ESHUTDOWN;
|
||||
}
|
||||
|
||||
/* Discriminator that http_response() embeds at the start of its envelope.
|
||||
* A handler returning a string starting with this exact prefix is treated
|
||||
* as a structured response; anything else is treated as a raw body. */
|
||||
@@ -1468,14 +1530,30 @@ static void http_send_response(int fd, const char* body) {
|
||||
free(env_body); free(hdrs.buf); return;
|
||||
}
|
||||
|
||||
if (http_send_all(fd, status_line, (size_t)sl) == 0
|
||||
&& http_send_all(fd, hdrs.buf, hdrs.len) == 0
|
||||
&& http_send_all(fd, tail, (size_t)tl) == 0
|
||||
&& (head_only
|
||||
/* HEAD requests echo headers + Content-Length but no body. */
|
||||
? 1
|
||||
: http_send_all(fd, eff_body, blen) == 0)) {
|
||||
/* sent successfully */
|
||||
/* The reply is written in four pieces; any of them can find the client
|
||||
* already gone. errno is captured at the first failure, before any later
|
||||
* library call can clobber it, and classified once below. */
|
||||
errno = 0;
|
||||
int send_err = 0;
|
||||
if (http_send_all(fd, status_line, (size_t)sl) != 0) send_err = errno;
|
||||
else if (http_send_all(fd, hdrs.buf, hdrs.len) != 0) send_err = errno;
|
||||
else if (http_send_all(fd, tail, (size_t)tl) != 0) send_err = errno;
|
||||
else if (!head_only /* HEAD echoes headers + Content-Length, no body. */
|
||||
&& http_send_all(fd, eff_body, blen) != 0) send_err = errno;
|
||||
|
||||
if (send_err) {
|
||||
if (http_write_err_is_client_gone(send_err)) {
|
||||
/* ROUTINE. The user closed the window, quit the app, or cancelled.
|
||||
* The work is done and the daemon keeps serving everyone else. */
|
||||
fprintf(stderr, "[http] client left before the reply was written "
|
||||
"(%zu-byte body, %s) - request completed, reply discarded\n",
|
||||
blen, strerror(send_err));
|
||||
} else {
|
||||
/* NOT routine — a real write fault. Never let the client-gone case
|
||||
* above hide this one. */
|
||||
fprintf(stderr, "[http] send failed: %s (%zu-byte body)\n",
|
||||
strerror(send_err), blen);
|
||||
}
|
||||
}
|
||||
|
||||
if (env_parsed_root) el_release(env_parsed_root);
|
||||
@@ -1491,6 +1569,7 @@ static void* http_worker(void* arg) {
|
||||
HttpWorkerArg* a = (HttpWorkerArg*)arg;
|
||||
int fd = a->fd;
|
||||
free(a);
|
||||
http_socket_nosigpipe(fd);
|
||||
char *method = NULL, *path = NULL, *body = NULL;
|
||||
if (http_read_request(fd, &method, &path, &body, NULL) == 0) {
|
||||
http_handler_fn h = http_lookup_active();
|
||||
@@ -1531,6 +1610,7 @@ static void* http_worker(void* arg) {
|
||||
}
|
||||
|
||||
void http_serve(el_val_t port, el_val_t handler) {
|
||||
el_runtime_ignore_sigpipe(); /* serving implies clients that leave */
|
||||
/* If `handler` looks like a string name, register it as the active handler. */
|
||||
const char* hname = EL_CSTR(handler);
|
||||
if (hname && looks_like_string(handler)) {
|
||||
@@ -1634,6 +1714,7 @@ static void* _http_serve_async_loop(void* raw) {
|
||||
}
|
||||
|
||||
void http_serve_async(el_val_t port, el_val_t handler) {
|
||||
el_runtime_ignore_sigpipe(); /* serving implies clients that leave */
|
||||
const char* hname = EL_CSTR(handler);
|
||||
if (hname && looks_like_string(handler)) {
|
||||
http_set_handler(handler);
|
||||
@@ -1821,6 +1902,7 @@ static void* http_worker_v2(void* arg) {
|
||||
HttpWorkerArg* a = (HttpWorkerArg*)arg;
|
||||
int fd = a->fd;
|
||||
free(a);
|
||||
http_socket_nosigpipe(fd);
|
||||
char *method = NULL, *path = NULL, *body = NULL, *hdr_block = NULL;
|
||||
if (http_read_request(fd, &method, &path, &body, &hdr_block) == 0) {
|
||||
http_handler4_fn h = http_lookup_active_v2();
|
||||
@@ -1858,6 +1940,7 @@ static void* http_worker_v2(void* arg) {
|
||||
}
|
||||
|
||||
void http_serve_v2(el_val_t port, el_val_t handler) {
|
||||
el_runtime_ignore_sigpipe(); /* serving implies clients that leave */
|
||||
const char* hname = EL_CSTR(handler);
|
||||
if (hname && looks_like_string(handler)) {
|
||||
http_set_handler_v2(handler);
|
||||
@@ -5511,6 +5594,9 @@ el_val_t getpid_now(void) {
|
||||
static el_val_t _el_args_list = 0;
|
||||
|
||||
void el_runtime_init_args(int argc, char** argv) {
|
||||
/* First line of every generated main(): a client that leaves must never be
|
||||
* able to signal this process to death. See el_runtime_ignore_sigpipe. */
|
||||
el_runtime_ignore_sigpipe();
|
||||
_el_args_list = el_list_empty();
|
||||
for (int i = 1; i < argc; i++) {
|
||||
_el_args_list = el_list_append(_el_args_list, EL_STR(argv[i]));
|
||||
@@ -6013,6 +6099,12 @@ static void engram_bll_parse_access(EngramNode* nn, const char* s) {
|
||||
* propagation loop in engram_activate. 0.25 damps semantically unrelated
|
||||
* branches ~4x without severing them. Unembedded targets are ungated. */
|
||||
#define ENGRAM_QGATE_FLOOR 0.25
|
||||
/* The read-path semantic leg (engram claim 24) reuses ENGRAM_EMBED_SEED_MIN
|
||||
* above as its admission floor: a node joins the embedding ranking only if its
|
||||
* query cosine clears the same bar that lets it join the seed set. No new
|
||||
* tuning constant is introduced, and the floor is load-bearing rather than
|
||||
* cosmetic — it is what keeps a query with no real match (the gold set's
|
||||
* nonsense controls) from being answered with its nearest neighbours. */
|
||||
#define ENGRAM_EMBED_MAX_CHARS 2000
|
||||
#define ENGRAM_EMBED_TIMEOUT_MS 4000L
|
||||
#define ENGRAM_EMBED_BREAKER_LIMIT 3
|
||||
@@ -7295,8 +7387,25 @@ static int engram_node_match_score(const EngramNode* n,
|
||||
return score;
|
||||
}
|
||||
|
||||
/* Semantic leg of the read path (engram claim 24). Returns the query/target
|
||||
* cosine renormalized onto [0,1] over the band [ENGRAM_EMBED_SEED_MIN, 1.0],
|
||||
* and exactly 0.0 when the pair is not comparable (no query embedding, target
|
||||
* unembedded, dim mismatch) or falls at/below the seed floor. Claim 32's
|
||||
* clamp-at-zero is subsumed: nothing below the floor can contribute.
|
||||
* A 0.0 return makes the fused score collapse to the lexical score, which is
|
||||
* why a dead embedder degrades to the historical behaviour exactly. */
|
||||
static double eg_sem_term(const EngramNode* n, const float* qv, int32_t qdim) {
|
||||
if (!qv || qdim <= 0 || !n->emb || n->emb_dim != qdim) return 0.0;
|
||||
double c = eg_cosine(n->emb, qv, qdim);
|
||||
if (c <= ENGRAM_EMBED_SEED_MIN) return 0.0;
|
||||
double t = (c - ENGRAM_EMBED_SEED_MIN) / (1.0 - ENGRAM_EMBED_SEED_MIN);
|
||||
return t > 1.0 ? 1.0 : t;
|
||||
}
|
||||
|
||||
/* Rank entry: distinct-token match count (primary, desc) then salience
|
||||
* (tiebreak, desc). */
|
||||
* (tiebreak, desc). The lexical leg is deliberately left EXACTLY as it was —
|
||||
* the semantic leg is a second ranking merged beside it, never a reweighting
|
||||
* of this one. */
|
||||
typedef struct { int64_t idx; int score; double salience; } EngramRankEntry;
|
||||
static int engram_rank_cmp(const void* a, const void* b) {
|
||||
const EngramRankEntry* ea = (const EngramRankEntry*)a;
|
||||
@@ -7307,6 +7416,53 @@ static int engram_rank_cmp(const void* a, const void* b) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Semantic rank entry: node index and its renormalized query similarity,
|
||||
* ordered by similarity desc. This is the claim-24 "embedding search"
|
||||
* ranking, computed independently of the lexical one. */
|
||||
typedef struct { int64_t idx; double sem; } EngramSemEntry;
|
||||
static int engram_sem_cmp(const void* a, const void* b) {
|
||||
const EngramSemEntry* ea = (const EngramSemEntry*)a;
|
||||
const EngramSemEntry* eb = (const EngramSemEntry*)b;
|
||||
if (ea->sem < eb->sem) return 1; /* desc */
|
||||
if (ea->sem > eb->sem) return -1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Merge the two rankings by strict alternation, lexical first:
|
||||
* L1, S1, L2, S2, L3, ... deduplicated by node index, capped at lim.
|
||||
*
|
||||
* Rank fusion, not score fusion. nomic's cosine scale is compressed (real
|
||||
* matches land ~0.55-0.70 while unrelated pairs sit ~0.35-0.50), so any
|
||||
* additive blend of a cosine onto a token-coverage score is dominated by
|
||||
* whichever leg happens to have the wider spread. Alternation is invariant to
|
||||
* both scales: it asks each leg for its next best answer in turn.
|
||||
*
|
||||
* Position 1 is always the top lexical hit, so a query whose answer the
|
||||
* lexical leg already ranks first cannot be displaced — exact-token retrieval
|
||||
* is structurally safe. The cost is bounded and explicit: a lexical hit at
|
||||
* rank r lands at output position 2r-1. */
|
||||
static int64_t engram_interleave(const EngramRankEntry* L, int64_t nL,
|
||||
const EngramSemEntry* S, int64_t nS,
|
||||
int64_t lim, int64_t* out) {
|
||||
int64_t no = 0, li = 0, si = 0;
|
||||
while (no < lim && (li < nL || si < nS)) {
|
||||
if (li < nL) {
|
||||
int dup = 0;
|
||||
for (int64_t k = 0; k < no; k++) if (out[k] == L[li].idx) { dup = 1; break; }
|
||||
if (!dup) out[no++] = L[li].idx;
|
||||
li++;
|
||||
}
|
||||
if (no >= lim) break;
|
||||
if (si < nS) {
|
||||
int dup = 0;
|
||||
for (int64_t k = 0; k < no; k++) if (out[k] == S[si].idx) { dup = 1; break; }
|
||||
if (!dup) out[no++] = S[si].idx;
|
||||
si++;
|
||||
}
|
||||
}
|
||||
return no;
|
||||
}
|
||||
|
||||
el_val_t engram_search(el_val_t query, el_val_t limit) {
|
||||
EngramStore* g = engram_get();
|
||||
const char* q = EL_CSTR(query);
|
||||
@@ -7319,6 +7475,12 @@ el_val_t engram_search(el_val_t query, el_val_t limit) {
|
||||
if (ntok == 0) return lst;
|
||||
EngramRankEntry* hits = malloc((size_t)g->node_count * sizeof(EngramRankEntry));
|
||||
if (!hits) return lst;
|
||||
/* Claim-24 semantic leg: one query embedding, fetched once per search.
|
||||
* NULL (embedder down / circuit breaker open) => pure lexical, as before. */
|
||||
int32_t qdim = 0;
|
||||
float* qv = eg_embed_fetch(q, &qdim);
|
||||
EngramSemEntry* sem = qv ? malloc((size_t)g->node_count * sizeof(EngramSemEntry)) : NULL;
|
||||
int64_t nsem = 0;
|
||||
int64_t nhits = 0;
|
||||
for (int64_t i = 0; i < g->node_count; i++) {
|
||||
EngramNode* n = &g->nodes[i];
|
||||
@@ -7334,14 +7496,24 @@ el_val_t engram_search(el_val_t query, el_val_t limit) {
|
||||
hits[nhits].salience = n->salience;
|
||||
nhits++;
|
||||
}
|
||||
if (sem) {
|
||||
double sv = eg_sem_term(n, qv, qdim);
|
||||
if (sv > 0.0) { sem[nsem].idx = i; sem[nsem].sem = sv; nsem++; }
|
||||
}
|
||||
}
|
||||
/* Rank by distinct tokens matched (desc) then salience (desc), then cap. */
|
||||
/* Rank each leg independently, then alternate between them. */
|
||||
qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp);
|
||||
int64_t end = nhits < lim ? nhits : lim;
|
||||
for (int64_t k = 0; k < end; k++) {
|
||||
lst = el_list_append(lst, engram_node_to_map(&g->nodes[hits[k].idx]));
|
||||
if (sem) qsort(sem, (size_t)nsem, sizeof(EngramSemEntry), engram_sem_cmp);
|
||||
int64_t* order = malloc((size_t)lim * sizeof(int64_t));
|
||||
if (order) {
|
||||
int64_t no = engram_interleave(hits, nhits, sem, nsem, lim, order);
|
||||
for (int64_t k = 0; k < no; k++)
|
||||
lst = el_list_append(lst, engram_node_to_map(&g->nodes[order[k]]));
|
||||
free(order);
|
||||
}
|
||||
free(hits);
|
||||
free(sem);
|
||||
free(qv);
|
||||
return lst;
|
||||
}
|
||||
|
||||
@@ -9173,6 +9345,14 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) {
|
||||
if (ntok > 0) {
|
||||
EngramRankEntry* hits = malloc((size_t)g->node_count * sizeof(EngramRankEntry));
|
||||
if (hits) {
|
||||
/* Claim-24 semantic leg. This is the function /api/neuron/recall
|
||||
* actually reaches (routes.el -> neuron-api.el handle_api_recall),
|
||||
* so the semantic half of the retrieval surface has to land HERE
|
||||
* to be observable to the MCP wrapper and the app. */
|
||||
int32_t qdim = 0;
|
||||
float* qv = eg_embed_fetch(q, &qdim);
|
||||
EngramSemEntry* sem = qv ? malloc((size_t)g->node_count * sizeof(EngramSemEntry)) : NULL;
|
||||
int64_t nsem = 0;
|
||||
int64_t nhits = 0;
|
||||
for (int64_t i = 0; i < g->node_count; i++) {
|
||||
EngramNode* n = &g->nodes[i];
|
||||
@@ -9185,15 +9365,26 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) {
|
||||
hits[nhits].salience = n->salience;
|
||||
nhits++;
|
||||
}
|
||||
if (sem) {
|
||||
double sv = eg_sem_term(n, qv, qdim);
|
||||
if (sv > 0.0) { sem[nsem].idx = i; sem[nsem].sem = sv; nsem++; }
|
||||
}
|
||||
}
|
||||
qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp);
|
||||
int64_t end = nhits < lim ? nhits : lim;
|
||||
for (int64_t k = 0; k < end; k++) {
|
||||
if (!first) jb_putc(&b, ',');
|
||||
engram_emit_node_json(&b, &g->nodes[hits[k].idx], 0);
|
||||
first = 0;
|
||||
if (sem) qsort(sem, (size_t)nsem, sizeof(EngramSemEntry), engram_sem_cmp);
|
||||
int64_t* order = malloc((size_t)lim * sizeof(int64_t));
|
||||
if (order) {
|
||||
int64_t no = engram_interleave(hits, nhits, sem, nsem, lim, order);
|
||||
for (int64_t k = 0; k < no; k++) {
|
||||
if (!first) jb_putc(&b, ',');
|
||||
engram_emit_node_json(&b, &g->nodes[order[k]], 0);
|
||||
first = 0;
|
||||
}
|
||||
free(order);
|
||||
}
|
||||
free(hits);
|
||||
free(sem);
|
||||
free(qv);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user