From cf41d12d2266dc3ef12d3c51857d16dde2de13ba Mon Sep 17 00:00:00 2001 From: Neuron <1timlingo@gmail.com> Date: Fri, 7 Aug 2026 13:40:51 -0500 Subject: [PATCH 01/15] test(retrieval): a measurement harness for memory recall, and its first verdict MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing else on the memory roadmap should be built until a change can be shown to help. Right now we judge by feel, and the benchmark literature is full of systems that felt better and measured worse. This is the missing gate. WHAT IT MEASURES, AND WHY IT BOOTS A REAL SOUL The subject is Will's designed retrieval — spreading activation over the weighted directed graph, four-factor multiplicative scoring — not a proxy for it. A Python re-implementation would measure my reading of the design, so the harness compiles the actual soul.el amalgam from a git ref and asks it over HTTP on /api/neuron/recall, exactly as the MCP wrapper and the app do. BUILT ON WHAT WAS ALREADY HERE, NOT AROUND IT docs/research/graphrag_eval/{collect,score}.py — per-query relevant-id scoring and fixed-denominator precision@5 (kept verbatim: an empty result should be punished like a page of junk). docs/research-archive/p0-prototypes/eval_pinned_40q_20260715.py — the pinned ground truth + --check winnability gate, so every run judges alike. scripts/verify-soul-contract.sh — the isolation recipe, including the non-obvious SOUL_ISE_URL pin without which an "isolated" soul silently syncs the operator's live brain. gen-soul-amalgam.sh + .gitea/workflows/ci.yaml — the build recipe and flags. New here: ids rather than regexes as ground truth, an associative category derived from real edges, a superseded category scored on ranking, a machine-checked zero-lexical-overlap guarantee on paraphrases, paired significance testing, and measurement of the real compiled soul rather than an offline replica of one leg of it. THE GOLD SET IS AUDITABLE, NOT VIBES 38 queries over the real 78,768-node corpus, each carrying a `derivation` string, each re-validated by `build_gold_set.py --check`. exact_rare is mined (document frequency 1). phrase is mined (verbatim scan; >25 matches rejected as too diffuse). paraphrase is hand-selected then PROVEN to share zero content words with its target — a leak fails the build, so the category cannot decay into lexical matching. associative is derived from real hub edges with lexically-reachable siblings dropped. nonsense is verified absent. superseded pairs are kept only when both sides survive as distinct nodes. HONEST ABOUT NOISE Minimum detectable swing on 38 queries is 6: if every changed query moves the same way, p = 2*0.5^n first clears 0.05 at n=6. Run-to-run drift is measured, not assumed — activation is a stateful read, and it shows: main is fully deterministic across 3 runs, the candidate drifts by 1 query. compare.py reports "no measurable difference" for anything inside max(6, drift+1). FIRST VERDICT — feat/recall-through-activation hit@5 34.3% -> 22.9%, phrase 85.7% -> 28.6%, latency p50 2.81x. Five discordant pairs, all five against the candidate, none for it; McNemar exact p = 0.0625, so by the stated rule this is one query short of significant and is reported as such rather than as a win for main. The latency regression is deterministic and not in any noise band. The benefit the branch was written for is absent: associative recall is 0/6 on BOTH builds. Probed directly, the traversal returns the lexical seed at rank 8 and none of its 12 hub siblings. Two measured corpus facts explain it — only 4,060 of 78,768 nodes (5.2%) carry any edge, and no node has an embedding, so the fourth factor of the four-factor product has nothing to compute from. The mechanism runs; the corpus lacks the structure it needs. SAFETY Throwaway port, throwaway HOME, disposable per-run copy of the corpus; live ports refused by name. Every soul started is killed AND confirmed dead by pid probe, with the confirmation written into the results file; run_comparison.sh sweeps for strays and exits non-zero if any survive. Nothing under ~/.neuron, /Applications/Neuron*, or ~/neuron-dev-stack is read, written, or restarted. Rung: E2E-VERIFIED — 6 full runs (3 per config) against the real compiled binaries on the real corpus; numbers above are measured, not projected. Co-Authored-By: Claude Opus 5 (1M context) --- tools/retrieval-eval/README.md | 147 +++ tools/retrieval-eval/build-soul.sh | 44 + tools/retrieval-eval/build_gold_set.py | 506 ++++++++++ tools/retrieval-eval/compare.py | 210 ++++ ...son-main-vs-recall-through-activation.json | 150 +++ tools/retrieval-eval/gold_set.json | 587 +++++++++++ tools/retrieval-eval/results-act-r2.json | 942 +++++++++++++++++ tools/retrieval-eval/results-act-r3.json | 942 +++++++++++++++++ ...esults-feat-recall-through-activation.json | 942 +++++++++++++++++ tools/retrieval-eval/results-main-r2.json | 945 ++++++++++++++++++ tools/retrieval-eval/results-main-r3.json | 945 ++++++++++++++++++ tools/retrieval-eval/results-main.json | 945 ++++++++++++++++++ tools/retrieval-eval/run_comparison.sh | 98 ++ tools/retrieval-eval/run_eval.py | 398 ++++++++ 14 files changed, 7801 insertions(+) create mode 100644 tools/retrieval-eval/README.md create mode 100755 tools/retrieval-eval/build-soul.sh create mode 100755 tools/retrieval-eval/build_gold_set.py create mode 100755 tools/retrieval-eval/compare.py create mode 100644 tools/retrieval-eval/comparison-main-vs-recall-through-activation.json create mode 100644 tools/retrieval-eval/gold_set.json create mode 100644 tools/retrieval-eval/results-act-r2.json create mode 100644 tools/retrieval-eval/results-act-r3.json create mode 100644 tools/retrieval-eval/results-feat-recall-through-activation.json create mode 100644 tools/retrieval-eval/results-main-r2.json create mode 100644 tools/retrieval-eval/results-main-r3.json create mode 100644 tools/retrieval-eval/results-main.json create mode 100755 tools/retrieval-eval/run_comparison.sh create mode 100755 tools/retrieval-eval/run_eval.py diff --git a/tools/retrieval-eval/README.md b/tools/retrieval-eval/README.md new file mode 100644 index 0000000..2f204f1 --- /dev/null +++ b/tools/retrieval-eval/README.md @@ -0,0 +1,147 @@ +# Retrieval eval harness + +Measures Neuron's memory retrieval so a change can be shown to help before it is +believed to help. Nothing else on the memory roadmap should ship without a run +through this. + +``` +tools/retrieval-eval/run_comparison.sh --baseline main --candidate +``` + +That builds a soul from each ref, boots each in isolation on a fixed corpus, +runs the gold set three times per ref, and prints a table plus a verdict that +refuses to call a difference real if it is inside the noise band. + +## What was reused + +This is not a new idea, it is the missing third of an existing one. + +| Prior work | What it gave | What was missing | +|---|---|---| +| `docs/research/graphrag_eval/` (`collect.py`, `score.py`, 2026-06-08) | The three-retriever comparison that produced the numbers everyone quotes: substring 1.7% P@5, graph 21.7%, BM25 55%. Per-query relevant-id scoring, fixed-denominator precision@5, unique-relevant analysis. | 13 hand-written queries, judged by an LLM after the fact; measured the *live* soul on the *live* engram. | +| `docs/research-archive/p0-prototypes/eval_pinned_40q_20260715.py` | The pinned-query discipline: ground truth committed as regexes so every run judges alike, plus a `--check` winnability gate. 40 queries in 5 bands including a deliberate paraphrase-hard band. | Scored offline replicas of substring/BM25 — it never ran the real retrieval path. | +| `docs/research-archive/p0-prototypes/stage0_eval_20260714.py` | The `hit@5` metric and the substring/BM25 reference implementations. | Same: offline only. | +| `scripts/verify-soul-contract.sh` | The isolation recipe, verbatim: throwaway port, throwaway `HOME`, `SOUL_ENGRAM_PATH`, and the non-obvious `SOUL_ISE_URL` pin that stops an "isolated" soul silently syncing the operator's live brain. | It is a contract gate, not a measurement. | +| `_engine-liveness-91/gen-soul-amalgam.sh` + `.gitea/workflows/ci.yaml` | The build recipe (`elc --target=c` with every `.elh` on the import chain removed) and CI's exact compile flags. | — | + +**Reused directly:** the isolation recipe, the build recipe, fixed-denominator +precision@5, the pinned-ground-truth and winnability ideas. +**New here:** ids rather than regexes as ground truth, an associative category +derived from real graph edges, a superseded/contradicted category scored on +ranking, a machine-checked zero-lexical-overlap guarantee on paraphrases, +paired significance testing, and — the point — measurement against the **real +compiled soul** rather than an offline replica of one leg of it. + +## Design fit + +The thing under measurement is Will's designed retrieval: spreading activation +over the weighted directed graph, four-factor multiplicative scoring (parent +strength x edge weight x target salience x query/target cosine). A Python +re-implementation would measure my reading of the design. So the harness +compiles the actual `soul.el` amalgam and asks it over HTTP on +`/api/neuron/recall`, exactly as the MCP wrapper and the app do. + +## Files + +| File | Does | +|---|---| +| `build_gold_set.py` | Derives and **validates** the gold set from the corpus. `--check` re-validates and exits non-zero if a query became unwinnable or a paraphrase leaked a word. | +| `gold_set.json` | 38 queries. Every one carries a `derivation` string. | +| `run_eval.py` | Boots one soul in isolation, runs the gold set, writes metrics. Kills and **confirms dead** its child; records the confirmation in the results file. | +| `compare.py` | Paired diff of two result files with McNemar's exact test and a stated noise floor. | +| `build-soul.sh` | Compiles a soul binary from a plain source tree. | +| `run_comparison.sh` | All of the above, end to end, from two git refs. | + +## The gold set — 38 queries + +Built from the real corpus (`snapshot-pre-repair-20260806.json`, 78,768 nodes / +14,214 edges) so it reflects one person's accumulating memory, not document QA. + +| Category | n | Expected answer derived by | +|---|---|---| +| `exact_rare` | 6 | **Mined.** Tokens with document frequency 1 across all 78,768 nodes, whose single containing node is a 300–6000 char Memory/Knowledge/Belief. That node is the only possible answer. Re-verified every build. | +| `phrase` | 7 | **Mined.** Case-insensitive verbatim scan; the matching set *is* the answer key. Phrases matching >25 nodes are rejected as too diffuse. | +| `paraphrase` | 13 | **Hand-selected, machine-checked.** Target locked by id; the build then proves that **zero** content words of the query appear anywhere in the target's label, content, or tags. A leak fails the build — the category cannot quietly decay into lexical matching. | +| `associative` | 6 | **Derived from edges.** Query built from one value node's distinctive vocabulary; expected answers are its siblings on the `Self - Values (grounded)` hub. Siblings sharing any query word are dropped, so the only route from query to answer is seed -> hub -> sibling. | +| `nonsense` | 3 | **Control.** Verified that no token occurs anywhere in the corpus. Correct behaviour is to return nothing. | +| `superseded` | 3 | **Derived.** Correction/stale pairs located by regex scan, kept only when both sides resolve to different surviving nodes. Scored on **ranking**: the correction must be returned *and* rank above the stale node. | + +## Metrics + +`hit@5`, `recall@5`, `recall@10`, `precision@5` (fixed denominator 5, so an +empty result is punished like a page of junk), `MRR@10`, and wall-clock latency +per query (p50/p95/max). Output is a table plus a machine-readable JSON per run +so runs can be diffed. + +## Honesty about noise + +- **Minimum detectable swing on this 38-query set: 6 queries.** If every query + that changes changes the same way, `p = 2 x 0.5^n`, which first drops under + 0.05 at n=6. Any net change smaller than that is inside the noise band and + `compare.py` says so in those words. +- **Run-to-run drift is measured, not assumed.** Activation is a stateful read + by design (traversal reinforces what it touches), so identical inputs need not + give identical outputs. Observed: `main` 0 queries of drift across 3 runs + (fully deterministic); the activation branch 1 query. +- The noise floor used for the verdict is `max(6, observed_drift + 1)`. +- **This gold set is underpowered for small effects.** A genuine 3-query + improvement would not clear the bar. Growing the set is the fix; until then, a + small positive delta means "not shown", not "no effect". + +## First result: `main` vs `feat/recall-through-activation` + +Corpus and gold set identical, three runs each, fresh corpus copy per run. + +| | main | recall-through-activation | delta | +|---|---|---|---| +| hit@5 | 34.3% | 22.9% | **-11.4pp** | +| recall@5 | 26.9% | 19.1% | -7.9pp | +| recall@10 | 33.3% | 24.3% | -9.1pp | +| precision@5 | 12.0% | 7.4% | -4.6pp | +| MRR@10 | 0.294 | 0.242 | -0.053 | +| latency p50 | 1140 ms | 3209 ms | **2.81x** | +| latency p95 | 1584 ms | 4852 ms | 3.06x | +| nonsense clean | 2/3 | 2/3 | — | +| superseded outranks | 1/3 | 0/3 | -1 | + +By category (hit@5): + +| category | main | activation | +|---|---|---| +| exact_rare | 100% | 100% | +| phrase | 85.7% | **28.6%** | +| paraphrase | 0% | 0% | +| associative | 0% | 0% | +| superseded | 0% | 0% | + +**Verdict: directionally worse, one query short of significant.** 5 discordant +pairs, all 5 against the candidate, 0 for it. McNemar exact p = 0.0625 — under +the stated rule that is *inside* the noise band, so the harness reports "no +measurable difference" on accuracy and the honest summary is "5 for 5 the wrong +way, needs a 6th or a larger gold set to call". + +Latency is a different story: 2.8x at p50 is deterministic and far outside any +noise band. That regression is real. + +The result the branch was written for did not appear. Its stated purpose was to +recover sibling nodes one hub-hop away — the `associative` category — and that +category is **0/6 on both builds**. Probing directly: for the query +`Marines hernia sepsis medical ward`, the activation build returns the lexical +seed node itself at rank 8, and none of its 12 hub siblings anywhere in the top +10. The traversal is running; it is not reaching siblings. + +Two corpus facts likely explain it, and both are measurable rather than +speculative: + +1. **The graph is nearly edgeless.** Only 4,060 of 78,768 nodes (5.2%) carry any + edge at all — 14,214 edges total, 0.18 per node. Spreading activation over a + graph with no edges is an expensive way to do lexical matching, which is + roughly what the numbers show. +2. **No embeddings.** No node in this snapshot has an embedding field, so the + fourth factor of the four-factor product — query/target cosine similarity — + has nothing to compute from, and the semantic seeding pass is inert. + +That is the harness earning its keep on its first job: the change would have +felt like progress (it is the designed mechanism, and it does run) and measures +as a regression on phrase queries plus a 2.8x latency cost, with its intended +benefit unrealised because the corpus lacks the structure it needs. diff --git a/tools/retrieval-eval/build-soul.sh b/tools/retrieval-eval/build-soul.sh new file mode 100755 index 0000000..e182f7a --- /dev/null +++ b/tools/retrieval-eval/build-soul.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# build-soul.sh — compile a soul binary from a plain source tree (no git needed). +# +# Reuses the amalgam recipe worked out in gen-soul-amalgam.sh (round 9.1) and the +# compile flags from .gitea/workflows/ci.yaml, so the binary under test is the +# same translation unit CI ships — not a re-implementation. +# +# elc --target=c emits only an extern prototype for any module that has a .elh +# header beside it, and inlines the module's bodies when it does not. So the +# amalgam is produced in a scratch copy with every .elh on the import chain +# deleted. +# +# usage: build-soul.sh +set -euo pipefail +SRC="${1:?usage: build-soul.sh }" +OUT="${2:?out-binary}" +ELC="${ELC:-$HOME/neuron-dev-stack/src/el/lang/dist/platform/elc}" +EL_REPO="${EL_REPO:-$HOME/Development/neuron-technologies/el}" +RTDIR="${RTDIR:-$SRC/vendor/el-runtime/v1.0.0-20260501}" +SSL="${SSL_PREFIX:-/opt/homebrew/opt/openssl@3}" + +[ -x "$ELC" ] || { echo "no elc at $ELC" >&2; exit 2; } +[ -f "$RTDIR/el_runtime.c" ] || { echo "no el_runtime.c at $RTDIR" >&2; exit 2; } + +GEN="$(mktemp -d "${TMPDIR:-/tmp}/soul-build.XXXXXX")" +trap 'rm -rf "$GEN"' EXIT +mkdir -p "$GEN/neuron" "$GEN/foundation/el/elp/src" +cp "$SRC"/*.el "$GEN/neuron/" +cp "$EL_REPO"/elp/src/*.el "$GEN/foundation/el/elp/src/" +find "$GEN" -name '*.elh' -delete + +( cd "$GEN/neuron" && "$ELC" --target=c soul.el ) > "$GEN/soul.c" +BODIES=$(grep -c '^el_val_t .*) {$' "$GEN/soul.c" || true) +echo "[build-soul] amalgam $(wc -c < "$GEN/soul.c" | tr -d ' ') bytes, ${BODIES} inlined bodies" +[ "$BODIES" -ge 1200 ] || { echo "[build-soul] FAIL: only $BODIES bodies — an import was not inlined"; exit 1; } + +cc -O2 -DHAVE_CURL -rdynamic \ + -I"$RTDIR" -I"$SSL/include" -L"$SSL/lib" \ + "$GEN/soul.c" "$RTDIR/el_runtime.c" \ + -lssl -lcrypto -lcurl -lpthread -lm \ + -o "$OUT" 2> "$GEN/cc.log" || { echo "[build-soul] FAIL compile"; tail -40 "$GEN/cc.log"; exit 1; } +if grep -qE 'implicit.*(engram_|el_)' "$GEN/cc.log"; then + echo "[build-soul] FAIL: implicit declarations of runtime symbols"; grep -E 'implicit' "$GEN/cc.log" | head; exit 1; fi +echo "[build-soul] OK -> $OUT ($(wc -c < "$OUT" | tr -d ' ') bytes)" diff --git a/tools/retrieval-eval/build_gold_set.py b/tools/retrieval-eval/build_gold_set.py new file mode 100755 index 0000000..70256d1 --- /dev/null +++ b/tools/retrieval-eval/build_gold_set.py @@ -0,0 +1,506 @@ +#!/usr/bin/env python3 +""" +build_gold_set.py — derive the retrieval gold set FROM the corpus, and validate it. + +WHY THIS FILE EXISTS AS CODE AND NOT AS A HAND-WRITTEN JSON + A gold set nobody can audit is vibes with extra steps. Every expected answer + here is either (a) mined from the corpus by a rule this script re-runs, or + (b) hand-selected with a stated criterion that this script then CHECKS + against the corpus. Both leave a `derivation` string on every query, and the + checks are re-run on demand so the set cannot silently rot as the corpus + changes. + + Lineage: this extends the pinned-query approach from + docs/research-archive/p0-prototypes/eval_pinned_40q_20260715.py (pinned + ground-truth patterns + a --check "winnability" gate) and the per-query + relevant-id scoring from docs/research/graphrag_eval/score.py. What is new: + ids as ground truth rather than regexes alone, an ASSOCIATIVE category + derived from real graph edges, a superseded/contradicted category, and a + machine-checked no-lexical-overlap guarantee on the paraphrase category. + +THE SIX CATEGORIES, AND WHAT EACH ONE IS FOR + exact_rare a single rare word. Substring matching already wins these. + They are a REGRESSION GUARD: any change that loses them is + disqualified regardless of what else it gains. + phrase a multi-word string that exists verbatim in the corpus. + Guards multi-token queries, which the old substring matcher + handled by returning nothing. + paraphrase same meaning, ZERO shared content words with the target node. + THE CATEGORY THAT MATTERS. Mechanically unreachable by string + matching; reachable only by semantics or by association. + associative the answer is one hub-hop from an obvious starting point and + shares no words with the query. This is the case the graph is + supposed to buy: query one value, get its siblings. + nonsense must return nothing. Guards against a retriever that "improves" + recall by returning the whole graph. + superseded a fact that was later corrected. The correction must OUTRANK + the stale version — ranking, not mere presence. + +usage: + python3 build_gold_set.py [--out gold_set.json] [--check] + --check re-validates an existing gold_set.json against the corpus and exits + non-zero if any query became unwinnable or any paraphrase leaked a word. +""" +import argparse +import json +import os +import re +import sys +from collections import Counter, defaultdict + +HERE = os.path.dirname(os.path.abspath(__file__)) +DEFAULT_OUT = os.path.join(HERE, "gold_set.json") + +TOKEN = re.compile(r"[a-z0-9][a-z0-9\-']*") + +# Stopwords are deliberately generous. A paraphrase query is only interesting if +# its CONTENT words are absent from the target; "the", "is", "what" appearing in +# both proves nothing. Being generous here makes the overlap test STRICTER on +# the words that carry meaning, which is the conservative direction. +STOP = set(""" +a about above after again against all also am an and any are aren't as at be because been +before being below between both but by can can't cannot could couldn't did didn't do does +doesn't doing don't down during each few for from further had hadn't has hasn't have haven't +having he her here hers herself him himself his how i if in into is isn't it its itself just +me more most my myself no nor not of off on once only or other others ought our ours ourselves +out over own same shan't she should shouldn't so some such than that the their theirs them +themselves then there these they this those through to too under until up very was wasn't we +were weren't what when where which while who whom why will with won't would wouldn't you your +yours yourself yourselves get gets got make makes made take takes use uses used way ways thing +things does doing done keep keeps kept go goes going come comes came one two something anything +""".split()) + + +# ───────────────────────────────────────────────────────────────────────────── +# corpus helpers +# ───────────────────────────────────────────────────────────────────────────── +def load_corpus(path): + with open(path, encoding="utf-8", errors="replace") as fh: + data = json.load(fh) + nodes = [n for n in data.get("nodes", []) if isinstance(n, dict) and n.get("id")] + edges = [e for e in data.get("edges", []) if isinstance(e, dict)] + return nodes, edges + + +def doctext(n): + return " ".join([str(n.get("label") or ""), str(n.get("content") or ""), str(n.get("tags") or "")]) + + +def content_tokens(s): + return {t for t in TOKEN.findall(s.lower()) if t not in STOP and len(t) > 2} + + +# ───────────────────────────────────────────────────────────────────────────── +# hand-authored queries. Every entry states HOW its expected answer was chosen. +# The `check` field names the validation this script runs against the corpus. +# ───────────────────────────────────────────────────────────────────────────── + +# EXACT_RARE — mined, not chosen. The rule (re-run by mine_exact_rare below): +# tokens whose document frequency across the whole corpus is 1, whose single +# containing node is a Memory/Knowledge/Belief with 300-6000 chars of content +# (so the answer is a real memory, not a 117KB whitepaper that contains every +# word in English), and whose token is plain lowercase alphabetic. The expected +# answer is that one node — it is the only node that can possibly be correct. +EXACT_RARE_SEEDS = [ + "unjailbreakable", + "engram-migrate", + "cartabandonedevent", + "pre-apprenticeship", + "inferencenodemanager", + "clear-eyed", +] + +# PHRASE — chosen by reading the corpus for phrases that (a) occur verbatim, +# (b) occur in a small enough set of nodes that "relevant" is well defined. +# Expected answers are computed here as EVERY node whose text contains the +# phrase case-insensitively — so the answer set is a fact about the corpus, not +# an opinion. Queries whose phrase matches more than PHRASE_MAX nodes are +# rejected by validation as too diffuse to score. +PHRASE_MAX = 25 +PHRASE_SEEDS = [ + ("patterns not returns", + "a verbatim correction Will issued; expected = every node containing the phrase"), + ("thirty moves", + "the canonical biographical phrase; expected = every node containing it"), + ("Grandma Lucas", + "a named person appearing verbatim in the biography/value nodes"), + ("Directed Harmonic", + "the canonical DHARMA expansion, confirmed by Will April 24 2026"), + ("Sarah Bishop", + "a named person; rare enough that the answer set is unambiguous"), + ("Directed Autonomous Runtime Modification", + "the DARMA expansion, quoted verbatim in the backlog item and its correction"), + ("zero-knowledge encrypted backup", + "the paid-tier feature name as written in the roadmap nodes"), +] + +# PARAPHRASE — hand-authored. THE SELECTION CRITERION, stated once and applied +# to all nine: pick a node whose SUBJECT is unmistakable to a reader, then write +# the query a person would actually type when they remember the subject but not +# the words. The target is then LOCKED by id, and this script enforces the hard +# property that makes the category meaningful: not one content word of the query +# appears anywhere in the target node's label, content, or tags. If a word +# leaks, validation fails and the query must be rewritten — the set cannot +# quietly degrade into a lexical query wearing a paraphrase costume. +PARAPHRASE_SEEDS = [ + ("kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "the elderly relative who passed while he stayed away", + "target: 'Value - Do the Essential Thing While You Can', whose subject is Grandma Lucas " + "dying in Feb 2006 without Will saying goodbye. Query names the event with none of the " + "node's own vocabulary."), + ("kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "a soldier sidelined by illness who refused to quit", + "target: 'Value - Survival Is Not an Excuse to Stop', whose subject is enlisting in the " + "Marines, a severe hernia, and sepsis. Query describes the episode obliquely."), + ("kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "choosing an uncomfortable fact over a pleasant fiction", + "target: 'Value - Honesty Before Comfort'. Query states the principle in wholly " + "different words."), + ("kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "a tight payload beats a bloated one", + "target: 'Value - Precision Over Brute Force'. Query restates the claim with no " + "shared vocabulary."), + ("kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "if you are able and nobody is coming the job is yours", + "target: 'Value - Capability Is a Debt You Owe the Moment'. Query states the " + "obligation without the node's terms."), + ("kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "learning is the wealth creditors cannot seize", + "target: 'Value - Knowledge Survives When Nothing Else Does', whose subject is the " + "library following Will across 30+ moves."), + ("kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "reliability proven by track record not assertion", + "target: 'Value - Earned Trust' ('Trust is demonstrated, not declared')."), + ("kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "boundaries that enable instead of confine", + "target: 'Value - Constraints as Freedom'. Query is a restatement of the same claim."), + ("kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "what shifts tells you where to cut a system apart", + "target: 'Value - Change Is the Signal', the value VBD is built on."), + ("kn-f230b362-b201-4402-9833-4160c89ab3d4", + "a mind that compounds instead of resetting each day", + "target: 'Value - The System Must Accumulate'. Query is the accumulation claim in " + "different vocabulary."), + ("kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "loved for the unedited self and not the polished exterior", + "target: 'Value - Being Seen Is Rarer Than Being Known', whose subject is Sarah Bishop " + "as the first person Will did not perform for."), + ("kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "cheerfulness you arrive at instead of assuming", + "target: 'Value - Hope Is a Conclusion'. Query restates 'a conclusion, not a premise'."), + ("kn-6061318f-046b-4935-907d-8eafdce14930", + "a childhood offering no solid foundation to inherit", + "target: 'Value - Structure Is Not Inherited', whose subject is thirty moves between " + "two parents' collapses."), +] + +# ASSOCIATIVE — derived from real edges, not authored. The construction: +# every value node hangs off the 'Self - Values (grounded)' hub by an `identity` +# edge. For a chosen value node V, the query is built from V's own distinctive +# vocabulary; the expected answers are V's SIBLINGS on that hub. A sibling +# shares no query words with the query by construction (validated below), so the +# only path from the query to a sibling is: lexical seed on V -> hub -> sibling. +# That is a two-hop traversal and nothing else can produce it. +VALUES_HUB = "kn-5b606390-a52d-4ca2-8e0e-eba141d13440" +ASSOCIATIVE_SEEDS = [ + ("kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", "Grandma Lucas stroke February 2006 goodbye window"), + ("kn-58874a74-b96f-4883-9e08-45707f4bd3ee", "Marines hernia sepsis medical ward"), + ("kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", "Sarah Bishop Dyer trailer performance"), + ("kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", "Swarm Architecture containment lateral worker"), + ("kn-e0423482-cfa5-4796-8689-8495c93b66bc", "hope won inside the narrative preface"), + ("kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", "man of the house six years old expectation"), +] + +# NONSENSE — must return nothing. Strings chosen to be lexically impossible: +# validation asserts each appears in ZERO corpus nodes as a substring and that +# none of its tokens appears anywhere either (so not even a partial seed exists). +NONSENSE_SEEDS = [ + "zqxjvw plimforth grebulon", + "flarnbistle quommetry", + "xxqzzt vurblenacht throom", +] + +# SUPERSEDED — a fact that was corrected. Chosen by searching the corpus for +# explicit correction language and keeping pairs where BOTH the stale statement +# and its correction exist as separate nodes. Scored on RANKING: the correction +# must appear, and must appear above the stale node. Ids are locked here and +# validated to exist and to match their stated role. +SUPERSEDED_SEEDS = [ + # (query, correct_id, stale_id, derivation) +] + + +# ───────────────────────────────────────────────────────────────────────────── +# mining +# ───────────────────────────────────────────────────────────────────────────── +def mine_exact_rare(nodes, byid, seeds): + """Re-derive: confirm each seed token still has df==1 and name its node.""" + tok = re.compile(r"[A-Za-z][A-Za-z0-9\-]{4,}") + want = set(seeds) + df = Counter() + post = defaultdict(set) + for n in nodes: + for t in {w.lower() for w in tok.findall(doctext(n))}: + if t in want: + df[t] += 1 + post[t].add(n["id"]) + out = [] + for s in seeds: + ids = sorted(post.get(s, ())) + out.append((s, ids, df.get(s, 0))) + return out + + +def phrase_matches(nodes, phrase): + p = phrase.lower() + return sorted(n["id"] for n in nodes if p in doctext(n).lower()) + + +def hub_siblings(edges, hub, relation="identity"): + sibs = [] + for e in edges: + if e.get("from_id") == hub and e.get("relation") == relation: + sibs.append(e["to_id"]) + elif e.get("to_id") == hub and e.get("relation") == relation: + sibs.append(e["from_id"]) + return list(dict.fromkeys(sibs)) + + +def find_superseded_pairs(nodes, byid): + """Locked pairs, each verified here to exist and to carry its stated marker. + + Chosen by scanning the corpus for explicit correction language + (CORRECTION/SUPERSEDES/re-corrected/no longer/RECONCILED) and keeping only + cases where the STALE claim also survives as its own node — a supersession + with nothing to outrank is not a ranking test. + """ + pairs = [] + txt = {n["id"]: doctext(n) for n in nodes} + + def find_one(pattern, exclude=()): + rx = re.compile(pattern) + return [n["id"] for n in nodes + if n["id"] not in exclude + and rx.search(txt[n["id"]]) + and 150 < len(str(n.get("content") or "")) < 12000 + and n.get("node_type") in ("Memory", "Knowledge", "Belief", "BacklogItem")] + + # Each entry: (query, correction-pattern, stale-pattern, why). + # The stale side is searched with the correction hits EXCLUDED, because most + # correction memories quote the claim they are killing — without the + # exclusion the "stale" node resolves to the correction itself and the pair + # collapses into a no-op. A pair is only emitted if both sides resolve to + # DIFFERENT surviving nodes; otherwise it is dropped and reported. + SPECS = [ + ("is the self-improvement architecture called DARMA or DHARMA", + r'(?i)CORRECTION:.{0,90}DHARMA .{0,12}not DARMA', + r'(?i)\bDARMA\b', + "correction node is Will's confirmation that the H is intentional (DHARMA, not DARMA); " + "the stale node is the surviving backlog item still titled 'Implement DARMA'."), + + ("how many provisional patents does Will actually have", + r'(?i)EXACTLY 6 (fully-specced )?provisional', + r'(?i)(MY ARCHITECTURE = 12 filed patents|\b12 filed patents\b)', + "correction node is the 2026-06-17 confabulation flag establishing EXACTLY 6 provisionals; " + "the stale node is the surviving memory that asserts 12 filed patents."), + + ("is MCP still the live integration layer", + r'(?i)MCP RETIRED', + r'(?i)MCP server live at', + "correction node is the 'CGI ARCHITECTURE - THREE LAYERS, MCP RETIRED' decision of " + "April 30 2026; the stale node still records the MCP server as live."), + + ("what does the patterns-not-returns directive mean", + r'(?i)CORRECTION:.{0,80}patterns not returns', + r'(?i)established returns', + "correction node is Will's 'patterns not returns' correction; the stale node is a " + "surviving node carrying the misread 'established returns' directive."), + + ("was the earlier identity-bug finding correct", + r'(?i)SUPERSEDES the earlier .critical identity bug', + r'(?i)critical identity bug', + "correction node explicitly supersedes the 'critical identity bug' finding; the stale " + "node is the surviving original finding."), + + ("does Neuron have recursive self-improvement", + r'(?i)twice answered .Neuron has no recursive self-improvement', + r'(?i)no recursive self-improvement', + "correction node records the June-29 finding that the CGI provisional IS the " + "recursive-self-improvement mechanism; the stale node is the surviving denial."), + ] + + for query, cpat, spat, why in SPECS: + corr = find_one(cpat) + if not corr: + continue + stale = find_one(spat, exclude=set(corr)) + if not stale: + continue + pairs.append((query, corr[0], stale[0], why)) + + return pairs + + +# ───────────────────────────────────────────────────────────────────────────── +# build +# ───────────────────────────────────────────────────────────────────────────── +def build(nodes, edges): + byid = {n["id"]: n for n in nodes} + tokset = {n["id"]: content_tokens(doctext(n)) for n in nodes} + queries = [] + problems = [] + qn = [0] + + def add(cat, query, relevant, derivation, **extra): + qn[0] += 1 + q = { + "id": f"q{qn[0]:02d}", + "category": cat, + "query": query, + "relevant": sorted(relevant), + "derivation": derivation, + } + q.update(extra) + queries.append(q) + return q + + # --- exact_rare --------------------------------------------------------- + for tokname, ids, df in mine_exact_rare(nodes, byid, EXACT_RARE_SEEDS): + if df != 1 or len(ids) != 1: + problems.append(f"exact_rare '{tokname}': df={df}, ids={len(ids)} (expected df=1)") + continue + lab = (byid[ids[0]].get("label") or "")[:60] + add("exact_rare", tokname, ids, + f"MINED: token '{tokname}' has document frequency 1 over all {len(nodes)} corpus nodes " + f"(re-verified at build time). Its single containing node is {ids[0]} " + f"('{lab}'), which is therefore the only possible correct answer.") + + # --- phrase ------------------------------------------------------------- + for phrase, why in PHRASE_SEEDS: + ids = phrase_matches(nodes, phrase) + if not ids: + problems.append(f"phrase '{phrase}': 0 corpus matches — unwinnable") + continue + if len(ids) > PHRASE_MAX: + problems.append(f"phrase '{phrase}': {len(ids)} matches > {PHRASE_MAX} — too diffuse") + continue + add("phrase", phrase, ids, + f"MINED: {why}. Case-insensitive verbatim substring scan over label+content+tags at " + f"build time returns exactly {len(ids)} node(s); that set IS the answer key.") + + # --- paraphrase --------------------------------------------------------- + for target, query, why in PARAPHRASE_SEEDS: + if target not in byid: + problems.append(f"paraphrase target {target} not in corpus") + continue + qt = content_tokens(query) + leak = sorted(qt & tokset[target]) + if leak: + problems.append(f"paraphrase '{query}': leaks {leak} into target {target}") + continue + add("paraphrase", query, [target], + f"HAND-SELECTED with criterion: {why} VERIFIED at build time: of the {len(qt)} content " + f"words in the query, ZERO appear anywhere in the target's label, content, or tags — so " + f"no string-matching retriever can reach this answer.", + zero_overlap_verified=True, query_content_words=sorted(qt)) + + # --- associative -------------------------------------------------------- + sibs = hub_siblings(edges, VALUES_HUB) + if len(sibs) < 5: + problems.append(f"associative: values hub {VALUES_HUB} has only {len(sibs)} siblings") + for src, query in ASSOCIATIVE_SEEDS: + if src not in byid or src not in sibs: + problems.append(f"associative source {src} not a sibling on {VALUES_HUB}") + continue + qt = content_tokens(query) + others = [s for s in sibs if s != src and s in byid] + # A sibling only counts as a legitimate expected answer if the query + # cannot reach it lexically. Drop any sibling that shares a content word. + clean = [s for s in others if not (qt & tokset[s])] + dropped = len(others) - len(clean) + if len(clean) < 5: + problems.append(f"associative '{query}': only {len(clean)} lexically-unreachable siblings") + continue + add("associative", query, clean, + f"DERIVED FROM EDGES: the query is built from the distinctive vocabulary of {src} " + f"('{(byid[src].get('label') or '')[:48]}'), which hangs off the values hub {VALUES_HUB} " + f"by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub " + f"({len(clean)} of {len(others)}; {dropped} dropped because they shared a query word and " + f"so were lexically reachable). Every remaining sibling shares ZERO content words with " + f"the query — the only route from query to answer is seed({src}) -> hub -> sibling, a " + f"two-hop traversal.", + associative_source=src, hub=VALUES_HUB, siblings_dropped_for_overlap=dropped) + + # --- nonsense ----------------------------------------------------------- + all_tokens = set() + for n in nodes: + all_tokens |= {t for t in TOKEN.findall(doctext(n).lower())} + for s in NONSENSE_SEEDS: + present = sorted(t for t in TOKEN.findall(s.lower()) if t in all_tokens) + if present: + problems.append(f"nonsense '{s}': tokens {present} DO occur in corpus") + continue + add("nonsense", s, [], + f"CONTROL: verified at build time that none of this string's tokens occurs anywhere in " + f"the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + expect_empty=True) + + # --- superseded --------------------------------------------------------- + for query, correct, stale, why in find_superseded_pairs(nodes, byid): + if correct not in byid or stale not in byid: + problems.append(f"superseded '{query}': id missing from corpus") + continue + add("superseded", query, [correct], + f"DERIVED: {why} Scored on RANKING, not presence: the corrected node {correct} must be " + f"returned AND must rank above the stale node {stale}.", + must_outrank=[correct, stale], + stale_id=stale, + correct_label=(byid[correct].get("label") or "")[:70], + stale_label=(byid[stale].get("label") or "")[:70]) + + return queries, problems + + +def summarize(queries): + c = Counter(q["category"] for q in queries) + return ", ".join(f"{k}={c[k]}" for k in + ("exact_rare", "phrase", "paraphrase", "associative", "nonsense", "superseded") + if c[k]) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("snapshot") + ap.add_argument("--out", default=DEFAULT_OUT) + ap.add_argument("--check", action="store_true", + help="validate only; do not write. Non-zero exit if anything is unwinnable.") + args = ap.parse_args() + + nodes, edges = load_corpus(args.snapshot) + print(f"corpus: {len(nodes)} nodes, {len(edges)} edges ({os.path.basename(args.snapshot)})") + queries, problems = build(nodes, edges) + + print(f"gold set: {len(queries)} queries [{summarize(queries)}]") + if problems: + print(f"\n{len(problems)} PROBLEM(S) — these queries were REJECTED, not silently kept:") + for p in problems: + print(" -", p) + + if args.check: + sys.exit(1 if problems else 0) + + doc = { + "corpus": os.path.abspath(args.snapshot), + "corpus_nodes": len(nodes), + "corpus_edges": len(edges), + "note": ("Every query carries a `derivation` recording how its expected answer was chosen. " + "Re-run with --check to re-validate the whole set against the corpus."), + "queries": queries, + } + with open(args.out, "w", encoding="utf-8") as fh: + json.dump(doc, fh, indent=1, ensure_ascii=False) + print(f"\nwrote {args.out}") + + +if __name__ == "__main__": + main() diff --git a/tools/retrieval-eval/compare.py b/tools/retrieval-eval/compare.py new file mode 100755 index 0000000..83ac4f8 --- /dev/null +++ b/tools/retrieval-eval/compare.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +""" +compare.py — diff two run_eval.py result files, WITH a noise threshold. + +WHY THE STATISTICS ARE NOT OPTIONAL + With ~35 scored queries, one query is ~2.9 percentage points. A harness that + reports "hit@5 improved 2.9%" without saying that is one query is a harness + that will approve noise. So this file refuses to call anything an + improvement on the strength of the headline number alone. It reports: + + 1. The DISCORDANT PAIRS. Two configurations scored on the same queries are + paired data, so the only queries carrying information are the ones + where they disagree: b = fixed by B, c = broken by B. Queries both got + right, or both got wrong, tell you nothing about which is better. + + 2. McNEMAR'S EXACT TEST on (b, c). Under the null "the change is a coin + flip", the discordant outcomes are Binomial(b+c, 0.5). The two-sided + exact p-value is computed here with no scipy dependency. + + 3. The MINIMUM DETECTABLE SWING for this gold set: the smallest number of + net-changed queries that would reach p < 0.05 if every discordant pair + fell the same way. Anything smaller is inside the noise band, and the + verdict line says so in those words. + + Repeat-run variance is the other half of honesty. Spreading activation is a + stateful read (it reinforces what it touches), so identical inputs need not + give identical outputs. Pass --repeats to fold several runs of the same + config into an observed variance band; a delta inside that band is not real + either, however good its p-value looks. + +usage: + python3 compare.py --baseline results-main.json --candidate results-act.json + python3 compare.py --baseline a.json --candidate b.json \ + --repeats-baseline a2.json a3.json --repeats-candidate b2.json b3.json +""" +import argparse +import json +from math import comb + + +def binom_two_sided(b, c): + """Two-sided exact binomial p for b successes in n=b+c at p=0.5.""" + n = b + c + if n == 0: + return 1.0 + k = min(b, c) + tail = sum(comb(n, i) for i in range(0, k + 1)) / (2 ** n) + return min(1.0, 2 * tail) + + +def min_detectable_swing(n_scored, alpha=0.05): + """Smallest all-one-way discordant count reaching p < alpha. + + If every query that changes changes in the same direction, the p-value is + 2 * 0.5**n. Solve for the smallest n where that drops under alpha. This is + the FLOOR: any real change will have some discordance both ways, so the true + requirement is larger. Reporting the floor is the conservative move — it is + the most generous threshold we would ever accept. + """ + n = 1 + while n <= n_scored: + if 2 * (0.5 ** n) < alpha: + return n + n += 1 + return n_scored + + +def load(path): + with open(path, encoding="utf-8") as fh: + return json.load(fh) + + +def row_map(doc): + return {r["id"]: r for r in doc["rows"]} + + +def outcome(r): + """Binary per-query outcome used for the paired test. + + hit@5 for scored queries; 'returned nothing' for the nonsense controls; + 'correction outranks the stale node' for the superseded queries. One number + per query, so every query votes exactly once. + """ + if "clean" in r: + return 1.0 if r["clean"] else 0.0 + if "outranks" in r: + return 1.0 if r["outranks"] else 0.0 + return r.get("hit@5") or 0.0 + + +def band(values): + return (min(values), max(values)) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--baseline", required=True) + ap.add_argument("--candidate", required=True) + ap.add_argument("--repeats-baseline", nargs="*", default=[]) + ap.add_argument("--repeats-candidate", nargs="*", default=[]) + ap.add_argument("--out", default=None) + args = ap.parse_args() + + A, B = load(args.baseline), load(args.candidate) + ra, rb = row_map(A), row_map(B) + ids = [q for q in ra if q in rb] + n = len(ids) + + aa, ab = A["aggregate"], B["aggregate"] + print(f"baseline {A['label']:14} soul={A['soul_md5'][:12]} {n} shared queries") + print(f"candidate {B['label']:14} soul={B['soul_md5'][:12]}") + print(f"corpus {A['corpus_nodes']} nodes / {A['corpus_edges']} edges " + f"(identical copy for both runs)\n") + + metrics = [("hit@5", 1), ("recall@5", 1), ("recall@10", 1), + ("precision@5", 1), ("mrr@10", 0)] + print(f" {'metric':14} {'baseline':>10} {'candidate':>10} {'delta':>10}") + for m, as_pct in metrics: + x, y = aa[m], ab[m] + if as_pct: + print(f" {m:14} {100*x:>9.1f}% {100*y:>9.1f}% {100*(y-x):>+9.1f}pp") + else: + print(f" {m:14} {x:>10.3f} {y:>10.3f} {y-x:>+10.3f}") + for m in ("latency_ms_p50", "latency_ms_p95"): + x, y = aa[m], ab[m] + ratio = f"{y/x:.2f}x" if x else "n/a" + print(f" {m:14} {x:>9.0f}ms {y:>9.0f}ms {ratio:>10}") + print(f" {'nonsense':14} {aa['nonsense_clean']:>10} {ab['nonsense_clean']:>10}") + print(f" {'outranks':14} {aa['superseded_outranks']:>10} {ab['superseded_outranks']:>10}") + + print(f"\n {'category':14} {'n':>3} {'base hit@5':>11} {'cand hit@5':>11} {'delta':>9}") + for c in sorted(set(aa["by_category"]) & set(ab["by_category"])): + ea, eb = aa["by_category"][c], ab["by_category"][c] + if c == "nonsense": + print(f" {c:14} {ea['n']:>3} {'clean ' + str(ea['clean']):>11} " + f"{'clean ' + str(eb['clean']):>11}") + else: + print(f" {c:14} {ea['n']:>3} {100*ea['hit@5']:>10.1f}% {100*eb['hit@5']:>10.1f}% " + f"{100*(eb['hit@5']-ea['hit@5']):>+8.1f}pp") + + # ---- paired significance ------------------------------------------------- + fixed, broken = [], [] + for q in ids: + oa, ob = outcome(ra[q]), outcome(rb[q]) + if ob > oa: + fixed.append(q) + elif ob < oa: + broken.append(q) + b, c = len(fixed), len(broken) + p = binom_two_sided(b, c) + mds = min_detectable_swing(n) + + print(f"\n== paired comparison over {n} queries ==") + print(f" fixed by candidate : {b} {[ra[q]['category'] + ':' + q for q in fixed]}") + print(f" broken by candidate: {c} {[ra[q]['category'] + ':' + q for q in broken]}") + print(f" discordant pairs : {b + c} net {b - c:+d} queries") + print(f" McNemar exact p : {p:.4f}") + print(f" noise threshold : a difference needs at least {mds} queries moving the " + f"same way to clear p<0.05 on this {n}-query set") + + # ---- repeat-run variance ------------------------------------------------- + var = {} + for name, paths, first in (("baseline", args.repeats_baseline, A), + ("candidate", args.repeats_candidate, B)): + docs = [first] + [load(p) for p in paths] + if len(docs) > 1: + hits = [d["aggregate"]["hit@5"] for d in docs] + lo, hi = band(hits) + spread_q = round((hi - lo) * first["aggregate"]["n_scored"]) + var[name] = {"runs": len(docs), "hit@5_min": lo, "hit@5_max": hi, + "spread_queries": spread_q} + print(f" {name} repeat runs ({len(docs)}): hit@5 {100*lo:.1f}%..{100*hi:.1f}% " + f"= {spread_q} query of run-to-run drift") + + drift = max([v["spread_queries"] for v in var.values()], default=0) + floor = max(mds, drift + 1) + + print("\n== VERDICT ==") + net = b - c + if abs(net) < floor: + print(f" NO MEASURABLE DIFFERENCE. Net {net:+d} queries is inside the noise band " + f"(needs |net| >= {floor}: {mds} for significance, {drift} observed run-to-run drift).") + elif net > 0: + print(f" CANDIDATE BETTER by {net} queries (p={p:.4f}), outside the noise band " + f"(>= {floor}).") + else: + print(f" CANDIDATE WORSE by {abs(net)} queries (p={p:.4f}), outside the noise band " + f"(>= {floor}).") + + if args.out: + with open(args.out, "w", encoding="utf-8") as fh: + json.dump({ + "baseline": A["label"], "candidate": B["label"], + "n_shared_queries": n, + "fixed_by_candidate": fixed, "broken_by_candidate": broken, + "discordant": b + c, "net_queries": net, + "mcnemar_exact_p": p, + "min_detectable_swing_queries": mds, + "observed_run_to_run_drift_queries": drift, + "noise_floor_queries": floor, + "verdict": ("no measurable difference" if abs(net) < floor + else ("candidate better" if net > 0 else "candidate worse")), + "baseline_aggregate": aa, "candidate_aggregate": ab, + "repeat_variance": var, + }, fh, indent=1) + print(f"\nwrote {args.out}") + + +if __name__ == "__main__": + main() diff --git a/tools/retrieval-eval/comparison-main-vs-recall-through-activation.json b/tools/retrieval-eval/comparison-main-vs-recall-through-activation.json new file mode 100644 index 0000000..72abd0e --- /dev/null +++ b/tools/retrieval-eval/comparison-main-vs-recall-through-activation.json @@ -0,0 +1,150 @@ +{ + "baseline": "main-r1", + "candidate": "act-r1", + "n_shared_queries": 38, + "fixed_by_candidate": [], + "broken_by_candidate": [ + "q07", + "q11", + "q12", + "q13", + "q36" + ], + "discordant": 5, + "net_queries": -5, + "mcnemar_exact_p": 0.0625, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 1, + "noise_floor_queries": 6, + "verdict": "no measurable difference", + "baseline_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.34285714285714286, + "recall@5": 0.26947278911564626, + "recall@10": 0.3333333333333333, + "precision@5": 0.12000000000000001, + "mrr@10": 0.2943197278911564, + "nonsense_clean": "2/3", + "superseded_outranks": "1/3", + "latency_ms_p50": 1140.4, + "latency_ms_p95": 1584.1, + "latency_ms_max": 1627.6, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4902210884353741, + "recall@10": 0.6666666666666666, + "mrr@10": 0.5965986394557822 + }, + "superseded": { + "n": 3, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.3333333333333333, + "mrr@10": 0.041666666666666664, + "outranks": 1 + } + } + }, + "candidate_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.22857142857142856, + "recall@5": 0.19087301587301586, + "recall@10": 0.24277210884353742, + "precision@5": 0.07428571428571429, + "mrr@10": 0.24154195011337865, + "nonsense_clean": "2/3", + "superseded_outranks": "0/3", + "latency_ms_p50": 3208.8, + "latency_ms_p95": 4851.9, + "latency_ms_max": 5078.8, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 2.6666666666666665 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "phrase": { + "n": 7, + "hit@5": 0.2857142857142857, + "recall@5": 0.09722222222222222, + "recall@10": 0.3567176870748299, + "mrr@10": 0.3505668934240363 + }, + "superseded": { + "n": 3, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0, + "outranks": 0 + } + } + }, + "repeat_variance": { + "baseline": { + "runs": 3, + "hit@5_min": 0.34285714285714286, + "hit@5_max": 0.34285714285714286, + "spread_queries": 0 + }, + "candidate": { + "runs": 3, + "hit@5_min": 0.22857142857142856, + "hit@5_max": 0.2571428571428571, + "spread_queries": 1 + } + } +} \ No newline at end of file diff --git a/tools/retrieval-eval/gold_set.json b/tools/retrieval-eval/gold_set.json new file mode 100644 index 0000000..9cc950b --- /dev/null +++ b/tools/retrieval-eval/gold_set.json @@ -0,0 +1,587 @@ +{ + "corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "note": "Every query carries a `derivation` recording how its expected answer was chosen. Re-run with --check to re-validate the whole set against the corpus.", + "queries": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "relevant": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "derivation": "MINED: token 'unjailbreakable' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226 ('Daemon hidden substrate architecture ? implemented April 25 '), which is therefore the only possible correct answer." + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "relevant": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5" + ], + "derivation": "MINED: token 'engram-migrate' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5 ('Engram v0.1 complete ? April 27, 2026. Local-first spreading'), which is therefore the only possible correct answer." + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "relevant": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "derivation": "MINED: token 'cartabandonedevent' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091 ('MESSAGE FOR AUDIT AGENT af5a7352e70e80434 ? El Language Spec'), which is therefore the only possible correct answer." + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "relevant": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "derivation": "MINED: token 'pre-apprenticeship' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-89c02aae-d3ca-43f9-9e5d-eb369896276c ('William Fox Anderson ? Applicant Profile Personal: - Full N'), which is therefore the only possible correct answer." + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "relevant": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848" + ], + "derivation": "MINED: token 'inferencenodemanager' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-73969486-143f-4431-b5e6-6845d1cc9848 ('Soma inference backplane deployed April 28 2026. Architectur'), which is therefore the only possible correct answer." + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "relevant": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "derivation": "MINED: token 'clear-eyed' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is knw-c72597c5-c23d-4c08-8e9e-996dadf26a99 ('Clear Eyes ? The Incomplete World View'), which is therefore the only possible correct answer." + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "relevant": [ + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482" + ], + "derivation": "MINED: a verbatim correction Will issued; expected = every node containing the phrase. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 1 node(s); that set IS the answer key." + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "relevant": [ + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "??o?'?B???k", + "Kp???", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "ע?RGk?\tH(?" + ], + "derivation": "MINED: the canonical biographical phrase; expected = every node containing it. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 16 node(s); that set IS the answer key." + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "relevant": [ + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "derivation": "MINED: a named person appearing verbatim in the biography/value nodes. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 9 node(s); that set IS the answer key." + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "relevant": [ + "%???2??jH??", + "63307ac5-cf6b-46e0-8296-07503b461cfa", + "7c9d4ab1-205d-4be8-bfae-e2c03a3a5010", + "9f291d20-0d32-413c-8c01-4416ccab4f7f", + "?;????n}rh?", + "???Ͼd??f??", + "?Z?.\f?0?]P?", + "Dp???]Q??k+", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf" + ], + "derivation": "MINED: the canonical DHARMA expansion, confirmed by Will April 24 2026. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 9 node(s); that set IS the answer key." + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "relevant": [ + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "derivation": "MINED: a named person; rare enough that the answer set is unambiguous. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 2 node(s); that set IS the answer key." + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "relevant": [ + "2a923500-d7e1-4b15-80e2-48dba65984ba", + "7?e?7???\f3?", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "?of?7???", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "g?2睪A|?H\b", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de" + ], + "derivation": "MINED: the DARMA expansion, quoted verbatim in the backlog item and its correction. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 14 node(s); that set IS the answer key." + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "relevant": [ + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a" + ], + "derivation": "MINED: the paid-tier feature name as written in the roadmap nodes. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 1 node(s); that set IS the answer key." + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "relevant": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Do the Essential Thing While You Can', whose subject is Grandma Lucas dying in Feb 2006 without Will saying goodbye. Query names the event with none of the node's own vocabulary. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "away", + "elderly", + "passed", + "relative", + "stayed" + ] + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "relevant": [ + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Survival Is Not an Excuse to Stop', whose subject is enlisting in the Marines, a severe hernia, and sepsis. Query describes the episode obliquely. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "illness", + "quit", + "refused", + "sidelined", + "soldier" + ] + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "relevant": [ + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Honesty Before Comfort'. Query states the principle in wholly different words. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "choosing", + "fact", + "fiction", + "pleasant", + "uncomfortable" + ] + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "relevant": [ + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Precision Over Brute Force'. Query restates the claim with no shared vocabulary. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "beats", + "bloated", + "payload", + "tight" + ] + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "relevant": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Capability Is a Debt You Owe the Moment'. Query states the obligation without the node's terms. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "able", + "coming", + "job", + "nobody" + ] + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "relevant": [ + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Knowledge Survives When Nothing Else Does', whose subject is the library following Will across 30+ moves. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "creditors", + "learning", + "seize", + "wealth" + ] + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "relevant": [ + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Earned Trust' ('Trust is demonstrated, not declared'). VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "assertion", + "proven", + "record", + "reliability", + "track" + ] + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "relevant": [ + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Constraints as Freedom'. Query is a restatement of the same claim. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "boundaries", + "confine", + "enable", + "instead" + ] + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "relevant": [ + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Change Is the Signal', the value VBD is built on. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "apart", + "cut", + "shifts", + "system", + "tells" + ] + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "relevant": [ + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - The System Must Accumulate'. Query is the accumulation claim in different vocabulary. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "compounds", + "day", + "instead", + "mind", + "resetting" + ] + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "relevant": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Being Seen Is Rarer Than Being Known', whose subject is Sarah Bishop as the first person Will did not perform for. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "exterior", + "loved", + "polished", + "self", + "unedited" + ] + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "relevant": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Hope Is a Conclusion'. Query restates 'a conclusion, not a premise'. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "arrive", + "assuming", + "cheerfulness", + "instead" + ] + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "relevant": [ + "kn-6061318f-046b-4935-907d-8eafdce14930" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Structure Is Not Inherited', whose subject is thirty moves between two parents' collapses. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "childhood", + "foundation", + "inherit", + "offering", + "solid" + ] + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "relevant": [ + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71 ('Value ? Do the Essential Thing While You Can'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 2 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "relevant": [ + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-58874a74-b96f-4883-9e08-45707f4bd3ee ('Value ? Survival Is Not an Excuse to Stop'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (13 of 13; 0 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-58874a74-b96f-4883-9e08-45707f4bd3ee) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 0 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "relevant": [ + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e ('Value ? Being Seen Is Rarer Than Being Known'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 2 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "relevant": [ + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83 ('Value ? Constraints as Freedom'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (6 of 13; 7 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 7 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "relevant": [ + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-e0423482-cfa5-4796-8689-8495c93b66bc ('Value ? Hope Is a Conclusion'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-e0423482-cfa5-4796-8689-8495c93b66bc) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 2 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "relevant": [ + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8 ('Value ? Capability Is a Debt You Owe the Moment'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 2 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "relevant": [], + "derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "relevant": [], + "derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "relevant": [], + "derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "relevant": [ + "mem-80d7416b-20e9-48a0-b176-b215527e2f56" + ], + "derivation": "DERIVED: correction node is Will's confirmation that the H is intentional (DHARMA, not DARMA); the stale node is the surviving backlog item still titled 'Implement DARMA'. Scored on RANKING, not presence: the corrected node mem-80d7416b-20e9-48a0-b176-b215527e2f56 must be returned AND must rank above the stale node bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff.", + "must_outrank": [ + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff" + ], + "stale_id": "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "correct_label": "CORRECTION: The autonomous self-improvement architecture is DHARMA ? n", + "stale_label": "Implement DARMA ? Directed Autonomous Runtime Modification Architectur" + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "relevant": [ + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8" + ], + "derivation": "DERIVED: correction node is the 2026-06-17 confabulation flag establishing EXACTLY 6 provisionals; the stale node is the surviving memory that asserts 12 filed patents. Scored on RANKING, not presence: the corrected node 3cf706a1-3825-45d8-b0a9-06cae6cdf5b8 must be returned AND must rank above the stale node 936541a9-fabb-466b-9ca3-a78b17ad0c53.", + "must_outrank": [ + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "936541a9-fabb-466b-9ca3-a78b17ad0c53" + ], + "stale_id": "936541a9-fabb-466b-9ca3-a78b17ad0c53", + "correct_label": "memory:remembered", + "stale_label": "memory:remembered" + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "relevant": [ + "mem-30425134-6008-4fd9-a3ee-67a7742c319b" + ], + "derivation": "DERIVED: correction node is the 'CGI ARCHITECTURE - THREE LAYERS, MCP RETIRED' decision of April 30 2026; the stale node still records the MCP server as live. Scored on RANKING, not presence: the corrected node mem-30425134-6008-4fd9-a3ee-67a7742c319b must be returned AND must rank above the stale node mem-101e81b4-8097-4749-8d8d-7bb66de34517.", + "must_outrank": [ + "mem-30425134-6008-4fd9-a3ee-67a7742c319b", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517" + ], + "stale_id": "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "correct_label": "CGI ARCHITECTURE ? THREE LAYERS, MCP RETIRED (April 30, 2026). Definit", + "stale_label": "GCloud MCP infrastructure ? April 27, 2026. Legion died (~19:30 UTC). " + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-act-r2.json b/tools/retrieval-eval/results-act-r2.json new file mode 100644 index 0000000..549c340 --- /dev/null +++ b/tools/retrieval-eval/results-act-r2.json @@ -0,0 +1,942 @@ +{ + "label": "act-r2", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-act", + "soul_md5": "77722f5a9f49494bf735c2a4be1b5dc4", + "corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json", + "limit": 10, + "port": 7902, + "wall_clock_s": 121.2, + "child_pid": 78714, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.22857142857142856, + "recall@5": 0.18908730158730158, + "recall@10": 0.24277210884353742, + "precision@5": 0.06857142857142857, + "mrr@10": 0.24154195011337865, + "nonsense_clean": "2/3", + "superseded_outranks": "0/3", + "latency_ms_p50": 3237.6, + "latency_ms_p95": 5264.0, + "latency_ms_max": 5510.9, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 2.6666666666666665 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "phrase": { + "n": 7, + "hit@5": 0.2857142857142857, + "recall@5": 0.0882936507936508, + "recall@10": 0.3567176870748299, + "mrr@10": 0.3505668934240363 + }, + "superseded": { + "n": 3, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0, + "outranks": 0 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "n_returned": 1, + "latency_ms": 476.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5" + ], + "n_returned": 1, + "latency_ms": 708.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "n_returned": 1, + "latency_ms": 532.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "n_returned": 1, + "latency_ms": 537.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848" + ], + "n_returned": 1, + "latency_ms": 568.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "n_returned": 1, + "latency_ms": 555.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-69fd6e83-7718-4824-8d66-f49d8954e224", + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "kn-c3d9d063-8c5d-45aa-900c-550914b2ff6d", + "kn-f838f113-76d5-4a15-9cef-14055c4723a3", + "bl-76e878aa-e1fe-468c-bf9c-854097cb7e0b", + "art-c71aef51-026f-4d63-80e9-2a0ec0dc3865", + "bl-e148d23c-24e8-4122-9915-d1c11f22052f", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451" + ], + "n_returned": 10, + "latency_ms": 1906.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "bl-80720fdf-7ce7-4d28-aff8-21028d3a8cfb", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "?Z?.\f?0?]P?", + "?Q??m?;��`'" + ], + "n_returned": 10, + "latency_ms": 1103.3, + "error": null, + "hit@5": 1.0, + "recall@5": 0.0625, + "recall@10": 0.1875, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "? ?}&?#??X\b", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 6, + "latency_ms": 1132.4, + "error": null, + "hit@5": 1.0, + "recall@5": 0.5555555555555556, + "recall@10": 0.5555555555555556, + "precision@5": 1.0, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "bl-c9adb8e5-293f-4033-99f8-0405c17ef941", + "bl-7e7c3fdb-4132-487f-aa70-b2cd559cb7f0", + "bl-7aebe936-ac55-4f35-8932-adc5224ff854", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + "mem-34f53a9d-a131-4f82-9dbd-b9eb4a9af52e", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf" + ], + "n_returned": 10, + "latency_ms": 960.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.1111111111111111, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + ";??A5???", + "art-94fae615-7cd5-4695-b968-977101b06a51", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35", + "mem-57164d5f-baf0-4149-957a-379a4e255d1a", + "? ?}&?#??X\b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "art-8dbceb06-431a-416d-a723-e8c75d595154" + ], + "n_returned": 10, + "latency_ms": 992.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.5, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "kn-b2a99cd7-b379-4d9b-a996-e347a02c7bad", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "kn-8e1bfb48-33a9-45ad-8da7-e0bdaa5d34e7", + "art-92e1837c-5919-42d0-bbb0-4d924d7b2864", + "bl-a7a1428f-db9c-417b-8e2c-713b1f84dc1f", + "mem-f823e835-313f-4282-b4b3-ce527ffc2f7a", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "?Z?.\f?0?]P?", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72" + ], + "n_returned": 10, + "latency_ms": 2820.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.14285714285714285, + "precision@5": 0.0, + "mrr@10": 0.14285714285714285 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "art-e495c8c5-ad95-4b64-8771-f68aa4cfcd0a", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "bl-e20944e5-eb16-4ab3-a84d-111e0fc817fa", + "bl-e20944e5-f4a6-44a0-91b1-73d04ebed120", + "mem-47f72b5b-6e8b-4293-94f1-350197b4809a", + "mem-a5f04e52-91f8-41d2-af27-8bf803621758", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "? ?}&?#??X\b", + "7?e?7???\f3?", + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a" + ], + "n_returned": 10, + "latency_ms": 2609.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "kn-82be4e41-96c5-4da3-85c0-cee10763d975", + "mem-ea487cb4-ed67-44ce-8402-b56bb28468d4", + "bl-2694b588-a6e3-43de-861c-fa7b0ec7e7fd", + "bl-14883d81-f7cb-46dd-82c2-a6e6980264e5", + "tag-dark-theme", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "? ?}&?#??X\b", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad" + ], + "n_returned": 10, + "latency_ms": 5510.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "mem-0328c3cb-4550-4ce4-9284-152e832f08f6", + "bl-07375bf9-a169-42cd-adb3-7d32b25982f0", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "bl-e5635b1a-c5d0-4caa-bcc8-6a726ea43685", + "bl-6172d035-dd94-4776-afdd-d8915f6fc375", + "bl-5bb8dedf-8498-4a9b-acdc-31cc9c738f2a", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + ";??A5???" + ], + "n_returned": 10, + "latency_ms": 5444.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "bl-d24fcce8-2b55-426f-867a-db3958a622d3", + "tag-phase-3", + "tag-project-structure", + "tag-anthropic-contrast", + "tag-voice-training", + "tag-ebd", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + ";??A5???", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 5056.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-fc893be3-e6b4-4ef6-93b0-d54ca5f89083", + "bl-57c5cf6b-81a5-4558-9902-5c02981fe273", + "tag-guilds", + "tag-kids", + "tag-coexistence", + "tag-cultivated-general-intelligence", + "? ?}&?#??X\b", + "kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491", + ";??A5???", + "mem-cdff0c49-3ac7-4de8-89ec-92d254bd0023" + ], + "n_returned": 10, + "latency_ms": 3473.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992", + "bl-b019ce7a-1b21-436e-812d-032f50c6c45f", + "bl-e98cdd4c-01b5-459e-9036-3578cd5d975a", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "tag-stable-diffusion", + ";??A5???", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 4446.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "mem-e321e54e-8bb3-4596-b13d-bb093d6b149d", + "mem-e32ba5a7-c147-4dc0-9479-b720d768eda6", + "mem-c7a77457-478d-4eb0-a116-67205a0066a4", + "bl-1b20e9bc-eb37-4907-8d63-e311fd61eab8", + "bl-aa762207-920d-45ab-b2a3-2f8154d7ef9b", + "tag-misalignment", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 3932.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "kn-48a01973-a025-471d-950f-b93e6a426d82", + "mem-23d22bc1-a097-446b-8f11-8aff099e0b76", + "mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8", + "mem-ce5a2ffc-ad39-4728-9ac6-76fef507d5da", + "project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_", + "tag-provenance", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "art-e495c8c5-ad95-4b64-8771-f68aa4cfcd0a" + ], + "n_returned": 10, + "latency_ms": 4836.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "kn-79056192-7de8-486c-9565-f128439a6fcb", + "bl-f6236350-f7b8-4f4f-a702-9eef2eb76e4b", + "bl-7f33f1bc-99fa-4906-889f-a42375beea20", + "mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "tag-__cultivation-metric____internal-state____dharma____evidence____novel-idea____gap-compression____values____microsoft__", + "? ?}&?#??X\b", + "kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "art-ee615cdb-e599-423d-9a4d-977859390ed3" + ], + "n_returned": 10, + "latency_ms": 4057.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "tag-identity-studio", + "tag-finance", + "tag-temporal", + "tag-barkhausen", + "tag-ilogger", + "tag-design-first", + ";??A5???", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 5098.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "project-Goal_setting__alignment__scoring__cadence__Attaches_to_any_imprint_", + "tag-performed-values", + "tag-turing-test", + "tag-sealed", + "tag-part-5", + "tag-model", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "? ?}&?#??X\b", + ";??A5???" + ], + "n_returned": 10, + "latency_ms": 4652.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "mem-9590ba23-bddb-43e8-a571-68a263c4c364", + "project-Imprint__leadership_development__feedback_frameworks__performance__presence_", + "bl-a9e57bb2-00a1-4867-ab59-5d9271134b50", + "bl-c7793c4a-7630-47fc-a462-d23059087e80", + "tag-gateway_platform_neuron-technologies_go_proxy_llm", + "tag-fornax", + ";??A5???", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-80ca3d31-84dc-4502-83f4-538372b9764f" + ], + "n_returned": 10, + "latency_ms": 4653.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "bl-7a13527b-3e0c-418a-9f37-88fd2152e5ce", + "bl-3f57bc69-7285-4f4a-a861-2de52efca058", + "bl-5e390b10-8753-4f25-a1a5-b5dbbb002cbf", + "tag-ats", + "tag-data-model", + "tag-aggregation", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + ";??A5???", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 4272.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "tag-memory-model", + "tag-storage", + "tag-ga4", + "tag-potions", + "tag-resonance", + "tag-neuron", + "? ?}&?#??X\b", + ";??A5???", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 4541.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "mem-927f41ab-8ede-4f58-acb3-995db16ac775", + "mem-dbe80bc2-c602-46b0-b4ea-dd222e52bcde", + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "tag-upload-window", + "? ?}&?#??X\b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 10, + "latency_ms": 4883.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "mem-434be7c8-88cb-4039-b79a-1da4ac4de783", + "mem-481c769c-68cc-45c7-bc37-c0d9778fa648", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "bl-9d8f3c5b-4bac-41ce-8ac4-44733f99d6c8", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 10, + "latency_ms": 3237.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "kn-eb1c6d99-d603-4f33-be9a-c63a178690c6", + "mem-a3124d5b-2f50-477f-8bb5-06879f5a496c", + "bl-7fa1b1a8-b80a-4f28-b162-bfe73765b4f8", + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "? ?}&?#??X\b", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 2457.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "mem-265a7107-73b3-4410-9aff-43787d5f473b", + "mem-9d1bf963-1b40-4588-bdb3-0432646cc623", + "mem-46780047-63a0-4a86-a16b-638b72a7fb8d", + "mem-3987d374-3c48-4e8e-b06d-0c363f55ed9c", + "bl-e0a0df72-de6e-46ab-800b-e1e3e8dfc387", + "tag-__patents____swarm____claim-language____prior-art____filing__", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "7?e?7???\f3?", + "?of?7???" + ], + "n_returned": 10, + "latency_ms": 2640.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-dca14c4c-4859-47b0-996e-33964ba61a87", + "bl-dc8c7e02-eb37-48ae-a6f8-9b512803ae16", + "mem-8d1bafe6-209c-456c-9a25-9a927bc5a16d", + "bl-0de4e61b-6562-49e5-b7df-ebb809a01723", + "bl-8ef1ba6b-3fa0-4dbd-98c5-31665e5694a1", + "mem-22f5f665-3ad2-4063-88b0-915849a795f5", + "? ?}&?#??X\b", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "?Q??m?;��`'", + "R^??m?;��?'" + ], + "n_returned": 10, + "latency_ms": 3705.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-394cc9e8-049b-45bc-a380-66314f14e367", + "bl-ffa22d7e-42a9-4bd2-a428-1d2df243ac93", + "bl-4a6746e8-191f-48fc-8bfb-c4dc73b80bcd", + "tag-command-pattern", + "tag-withholding", + "tag-offline", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 4223.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 1333.9, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 880.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [ + "kn-333542cb-6dab-4662-9725-bf7440d28bf7", + "? ?}&?#??X\b", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-66a21179-2adc-4b19-a109-880cf4674d7d", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4" + ], + "n_returned": 8, + "latency_ms": 1452.7, + "error": null, + "clean": false, + "false_positives": 8 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "ctx-e5427d7d", + "mem-37b57f52-a29a-42cf-a07a-3c5f8a3598dd", + "project-worldweaver", + "tag-__kotlin____internal-state____pre-reasoning____post-reasoning____compression-ratio____dharma____cultivation__", + "tag-import", + "tag-__cgi____dharma____cultivation____five-primitives____seed-artifact____agi____intelligence____whitepaper____patent__", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "? ?}&?#??X\b", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 3843.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": null + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "mem-bfe0fafd-2750-4fdc-b773-04e878b3b23f", + "art-7bdaff30-5af9-4f0a-93b1-751686f9de3d", + "mem-cf07910d-4676-4384-ab97-9cad946cd0b9", + "mem-f9da4b43-3724-4bc8-92f8-6f237c89dc4d", + "project-Convert_UTC_timestamps_to_Central_time_when_displaying_to_Will__Never_surface_raw_UTC_", + "mem-32203649-3213-4d6d-86fd-3d657ac70d77", + ";??A5???", + "? ?}&?#??X\b", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f" + ], + "n_returned": 10, + "latency_ms": 5264.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "project-Define_Iris_as_separate_public_brand__Uncensored_but_principled__Consumer_face_while_Neuron_runs_enterprise_", + "project-Imprint__discovery__objection_handling__deal_strategy__pipeline__closing_", + "bl-8116da7a-b039-4e08-b8d0-c1c7861f9766", + "tag-enterprise", + "tag-divisors", + "tag-distressed-property", + "? ?}&?#??X\b", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "???Ͼd??W\b?", + "kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491" + ], + "n_returned": 10, + "latency_ms": 3022.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": null + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-act-r3.json b/tools/retrieval-eval/results-act-r3.json new file mode 100644 index 0000000..f55756e --- /dev/null +++ b/tools/retrieval-eval/results-act-r3.json @@ -0,0 +1,942 @@ +{ + "label": "act-r3", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-act", + "soul_md5": "77722f5a9f49494bf735c2a4be1b5dc4", + "corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json", + "limit": 10, + "port": 7903, + "wall_clock_s": 112.0, + "child_pid": 78802, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.2571428571428571, + "recall@5": 0.19291383219954647, + "recall@10": 0.244812925170068, + "precision@5": 0.08, + "mrr@10": 0.266031746031746, + "nonsense_clean": "2/3", + "superseded_outranks": "0/3", + "latency_ms_p50": 3220.9, + "latency_ms_p95": 4840.2, + "latency_ms_max": 5073.6, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 2.6666666666666665 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "phrase": { + "n": 7, + "hit@5": 0.42857142857142855, + "recall@5": 0.10742630385487528, + "recall@10": 0.366921768707483, + "mrr@10": 0.47301587301587306 + }, + "superseded": { + "n": 3, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0, + "outranks": 0 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "n_returned": 1, + "latency_ms": 471.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5" + ], + "n_returned": 1, + "latency_ms": 548.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "n_returned": 1, + "latency_ms": 470.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "n_returned": 1, + "latency_ms": 476.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848" + ], + "n_returned": 1, + "latency_ms": 515.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "n_returned": 1, + "latency_ms": 458.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-69fd6e83-7718-4824-8d66-f49d8954e224", + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "kn-c3d9d063-8c5d-45aa-900c-550914b2ff6d", + "kn-82be4e41-96c5-4da3-85c0-cee10763d975", + "bl-76e878aa-e1fe-468c-bf9c-854097cb7e0b", + "art-9887867c-2e21-47c3-9f96-c2dfe5bd4cc1", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451" + ], + "n_returned": 10, + "latency_ms": 1595.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "bl-80720fdf-7ce7-4d28-aff8-21028d3a8cfb", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "?Z?.\f?0?]P?", + "?Q??m?;��`'" + ], + "n_returned": 10, + "latency_ms": 927.5, + "error": null, + "hit@5": 1.0, + "recall@5": 0.125, + "recall@10": 0.1875, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "? ?}&?#??X\b", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 6, + "latency_ms": 963.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.5555555555555556, + "recall@10": 0.5555555555555556, + "precision@5": 1.0, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a", + "art-e495c8c5-ad95-4b64-8771-f68aa4cfcd0a", + "bl-7aebe936-ac55-4f35-8932-adc5224ff854", + "knw-b046991d-5992-4ac4-b854-7d3ac273832c", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + "bl-455a08cf-5831-4fdb-b42c-b952f2feafb9", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf" + ], + "n_returned": 10, + "latency_ms": 927.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.1111111111111111, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "art-8dbceb06-431a-416d-a723-e8c75d595154", + "mem-a3124d5b-2f50-477f-8bb5-06879f5a496c", + "art-94fae615-7cd5-4695-b968-977101b06a51", + "mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35", + "mem-57164d5f-baf0-4149-957a-379a4e255d1a", + "? ?}&?#??X\b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef" + ], + "n_returned": 10, + "latency_ms": 937.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.5, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "kn-b2a99cd7-b379-4d9b-a996-e347a02c7bad", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "kn-8e1bfb48-33a9-45ad-8da7-e0bdaa5d34e7", + "mem-cdff0c49-3ac7-4de8-89ec-92d254bd0023", + "bl-a7a1428f-db9c-417b-8e2c-713b1f84dc1f", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "?Z?.\f?0?]P?", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72" + ], + "n_returned": 10, + "latency_ms": 2451.5, + "error": null, + "hit@5": 1.0, + "recall@5": 0.07142857142857142, + "recall@10": 0.21428571428571427, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "mem-7cd90611-88a3-423d-a38a-0db2812952fa", + "bl-e20944e5-f4a6-44a0-91b1-73d04ebed120", + "mem-64cf3728-674c-404b-965a-b8f8d38bb7bb", + "mem-47f72b5b-6e8b-4293-94f1-350197b4809a", + "mem-e612f0aa-c2f2-4ee3-bbc7-af2dc826233b", + "mem-a5f04e52-91f8-41d2-af27-8bf803621758", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "? ?}&?#??X\b", + "7?e?7???\f3?", + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a" + ], + "n_returned": 10, + "latency_ms": 2042.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "mem-ea487cb4-ed67-44ce-8402-b56bb28468d4", + "art-c71aef51-026f-4d63-80e9-2a0ec0dc3865", + "bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60", + "bl-2694b588-a6e3-43de-861c-fa7b0ec7e7fd", + "bl-14883d81-f7cb-46dd-82c2-a6e6980264e5", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "? ?}&?#??X\b", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad" + ], + "n_returned": 10, + "latency_ms": 4698.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "bl-fd047ce9-ae21-4b3e-b3ab-ece0c9592f7f", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "bl-6172d035-dd94-4776-afdd-d8915f6fc375", + "bl-5bb8dedf-8498-4a9b-acdc-31cc9c738f2a", + "tag-cultivated-general-intelligence", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f" + ], + "n_returned": 10, + "latency_ms": 5073.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "bl-e5635b1a-c5d0-4caa-bcc8-6a726ea43685", + "bl-fc893be3-e6b4-4ef6-93b0-d54ca5f89083", + "tag-project-structure", + "tag-anthropic-contrast", + "tag-voice-training", + "tag-ebd", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792" + ], + "n_returned": 10, + "latency_ms": 4117.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-d24fcce8-2b55-426f-867a-db3958a622d3", + "tag-phase-3", + "bl-57c5cf6b-81a5-4558-9902-5c02981fe273", + "tag-guilds", + "tag-kids", + "tag-coexistence", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "? ?}&?#??X\b", + "kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491", + "mem-cdff0c49-3ac7-4de8-89ec-92d254bd0023" + ], + "n_returned": 10, + "latency_ms": 3254.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "bl-07375bf9-a169-42cd-adb3-7d32b25982f0", + "mem-0328c3cb-4550-4ce4-9284-152e832f08f6", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992", + "bl-e98cdd4c-01b5-459e-9036-3578cd5d975a", + "tag-stable-diffusion", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef" + ], + "n_returned": 10, + "latency_ms": 4081.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "mem-e32ba5a7-c147-4dc0-9479-b720d768eda6", + "bl-b019ce7a-1b21-436e-812d-032f50c6c45f", + "mem-c7a77457-478d-4eb0-a116-67205a0066a4", + "bl-1b20e9bc-eb37-4907-8d63-e311fd61eab8", + "bl-aa762207-920d-45ab-b2a3-2f8154d7ef9b", + "tag-misalignment", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 3687.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "kn-48a01973-a025-471d-950f-b93e6a426d82", + "mem-23d22bc1-a097-446b-8f11-8aff099e0b76", + "mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8", + "mem-ce5a2ffc-ad39-4728-9ac6-76fef507d5da", + "project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_", + "mem-e321e54e-8bb3-4596-b13d-bb093d6b149d", + "? ?}&?#??X\b", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9" + ], + "n_returned": 10, + "latency_ms": 4659.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "kn-79056192-7de8-486c-9565-f128439a6fcb", + "bl-f6236350-f7b8-4f4f-a702-9eef2eb76e4b", + "bl-7f33f1bc-99fa-4906-889f-a42375beea20", + "mem-37b57f52-a29a-42cf-a07a-3c5f8a3598dd", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "tag-__cultivation-metric____internal-state____dharma____evidence____novel-idea____gap-compression____values____microsoft__", + "? ?}&?#??X\b", + "kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "knw-b046991d-5992-4ac4-b854-7d3ac273832c" + ], + "n_returned": 10, + "latency_ms": 3960.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "tag-identity-studio", + "tag-finance", + "tag-temporal", + "tag-barkhausen", + "tag-ilogger", + "tag-design-first", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c" + ], + "n_returned": 10, + "latency_ms": 4840.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "project-Goal_setting__alignment__scoring__cadence__Attaches_to_any_imprint_", + "tag-performed-values", + "tag-turing-test", + "tag-sealed", + "tag-part-5", + "tag-model", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 4493.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "mem-9590ba23-bddb-43e8-a571-68a263c4c364", + "project-Imprint__leadership_development__feedback_frameworks__performance__presence_", + "bl-a9e57bb2-00a1-4867-ab59-5d9271134b50", + "bl-c7793c4a-7630-47fc-a462-d23059087e80", + "tag-gateway_platform_neuron-technologies_go_proxy_llm", + "tag-fornax", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc", + "art-80ca3d31-84dc-4502-83f4-538372b9764f" + ], + "n_returned": 10, + "latency_ms": 4509.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "bl-7a13527b-3e0c-418a-9f37-88fd2152e5ce", + "bl-3f57bc69-7285-4f4a-a861-2de52efca058", + "bl-5e390b10-8753-4f25-a1a5-b5dbbb002cbf", + "tag-ats", + "tag-data-model", + "tag-aggregation", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef" + ], + "n_returned": 10, + "latency_ms": 3624.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "tag-memory-model", + "tag-storage", + "tag-ga4", + "tag-potions", + "tag-resonance", + "tag-neuron", + "? ?}&?#??X\b", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-ee615cdb-e599-423d-9a4d-977859390ed3" + ], + "n_returned": 10, + "latency_ms": 4093.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "mem-927f41ab-8ede-4f58-acb3-995db16ac775", + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "tag-upload-window", + "? ?}&?#??X\b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 10, + "latency_ms": 4235.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "mem-0f31141d-3ac5-44b2-9942-be7e4e6feb79", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "mem-434be7c8-88cb-4039-b79a-1da4ac4de783", + "mem-481c769c-68cc-45c7-bc37-c0d9778fa648", + "bl-9d8f3c5b-4bac-41ce-8ac4-44733f99d6c8", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 10, + "latency_ms": 3220.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-eb1c6d99-d603-4f33-be9a-c63a178690c6", + "mem-5e7f6ddd-c818-4ad3-b564-54ae278e9976", + "bl-7fa1b1a8-b80a-4f28-b162-bfe73765b4f8", + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c", + "tag-sarah", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "? ?}&?#??X\b", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 2336.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "mem-265a7107-73b3-4410-9aff-43787d5f473b", + "mem-9d1bf963-1b40-4588-bdb3-0432646cc623", + "mem-46780047-63a0-4a86-a16b-638b72a7fb8d", + "mem-3987d374-3c48-4e8e-b06d-0c363f55ed9c", + "bl-e0a0df72-de6e-46ab-800b-e1e3e8dfc387", + "tag-__patents____swarm____claim-language____prior-art____filing__", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "7?e?7???\f3?" + ], + "n_returned": 10, + "latency_ms": 2348.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-dca14c4c-4859-47b0-996e-33964ba61a87", + "bl-e148d23c-24e8-4122-9915-d1c11f22052f", + "bl-dc8c7e02-eb37-48ae-a6f8-9b512803ae16", + "mem-8d1bafe6-209c-456c-9a25-9a927bc5a16d", + "mem-22f5f665-3ad2-4063-88b0-915849a795f5", + "tag-dark-theme", + "? ?}&?#??X\b", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "?Q??m?;��`'", + "R^??m?;��?'" + ], + "n_returned": 10, + "latency_ms": 3511.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "bl-ffa22d7e-42a9-4bd2-a428-1d2df243ac93", + "mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9", + "bl-452a4710-3d2b-4e0f-9413-49a66423bc9a", + "bl-4a6746e8-191f-48fc-8bfb-c4dc73b80bcd", + "tag-withholding", + "tag-offline", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c" + ], + "n_returned": 10, + "latency_ms": 4160.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 1318.6, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 875.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [ + "kn-333542cb-6dab-4662-9725-bf7440d28bf7", + "? ?}&?#??X\b", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-66a21179-2adc-4b19-a109-880cf4674d7d", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4" + ], + "n_returned": 8, + "latency_ms": 1393.4, + "error": null, + "clean": false, + "false_positives": 8 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "knw-b046991d-5992-4ac4-b854-7d3ac273832c", + "ctx-e5427d7d", + "project-worldweaver", + "tag-__kotlin____internal-state____pre-reasoning____post-reasoning____compression-ratio____dharma____cultivation__", + "tag-import", + "tag-enterprise", + "tag-__cgi____dharma____cultivation____five-primitives____seed-artifact____agi____intelligence____whitepaper____patent__", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "? ?}&?#??X\b", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 10, + "latency_ms": 3775.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": null + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "mem-bfe0fafd-2750-4fdc-b773-04e878b3b23f", + "art-7bdaff30-5af9-4f0a-93b1-751686f9de3d", + "mem-cf07910d-4676-4384-ab97-9cad946cd0b9", + "mem-f9da4b43-3724-4bc8-92f8-6f237c89dc4d", + "project-Convert_UTC_timestamps_to_Central_time_when_displaying_to_Will__Never_surface_raw_UTC_", + "mem-32203649-3213-4d6d-86fd-3d657ac70d77", + "? ?}&?#??X\b", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396" + ], + "n_returned": 10, + "latency_ms": 4857.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "bl-0de4e61b-6562-49e5-b7df-ebb809a01723", + "project-Imprint__discovery__objection_handling__deal_strategy__pipeline__closing_", + "bl-8116da7a-b039-4e08-b8d0-c1c7861f9766", + "bl-8ef1ba6b-3fa0-4dbd-98c5-31665e5694a1", + "tag-divisors", + "tag-distressed-property", + "? ?}&?#??X\b", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "knw-b046991d-5992-4ac4-b854-7d3ac273832c", + "???Ͼd??W\b?" + ], + "n_returned": 10, + "latency_ms": 2858.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": null + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-feat-recall-through-activation.json b/tools/retrieval-eval/results-feat-recall-through-activation.json new file mode 100644 index 0000000..ad01474 --- /dev/null +++ b/tools/retrieval-eval/results-feat-recall-through-activation.json @@ -0,0 +1,942 @@ +{ + "label": "act-r1", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-act", + "soul_md5": "77722f5a9f49494bf735c2a4be1b5dc4", + "corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json", + "limit": 10, + "port": 7894, + "wall_clock_s": 112.3, + "child_pid": 78648, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.22857142857142856, + "recall@5": 0.19087301587301586, + "recall@10": 0.24277210884353742, + "precision@5": 0.07428571428571429, + "mrr@10": 0.24154195011337865, + "nonsense_clean": "2/3", + "superseded_outranks": "0/3", + "latency_ms_p50": 3208.8, + "latency_ms_p95": 4851.9, + "latency_ms_max": 5078.8, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 2.6666666666666665 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "phrase": { + "n": 7, + "hit@5": 0.2857142857142857, + "recall@5": 0.09722222222222222, + "recall@10": 0.3567176870748299, + "mrr@10": 0.3505668934240363 + }, + "superseded": { + "n": 3, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0, + "outranks": 0 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "n_returned": 1, + "latency_ms": 468.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5" + ], + "n_returned": 1, + "latency_ms": 569.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "n_returned": 1, + "latency_ms": 479.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "n_returned": 1, + "latency_ms": 470.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848" + ], + "n_returned": 1, + "latency_ms": 499.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "n_returned": 1, + "latency_ms": 452.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-69fd6e83-7718-4824-8d66-f49d8954e224", + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "kn-eb1c6d99-d603-4f33-be9a-c63a178690c6", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "kn-b2a99cd7-b379-4d9b-a996-e347a02c7bad", + "bl-76e878aa-e1fe-468c-bf9c-854097cb7e0b", + "art-c71aef51-026f-4d63-80e9-2a0ec0dc3865" + ], + "n_returned": 10, + "latency_ms": 1592.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "bl-80720fdf-7ce7-4d28-aff8-21028d3a8cfb", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "?Z?.\f?0?]P?", + "?Q??m?;��`'" + ], + "n_returned": 10, + "latency_ms": 956.6, + "error": null, + "hit@5": 1.0, + "recall@5": 0.125, + "recall@10": 0.1875, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "? ?}&?#??X\b", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 6, + "latency_ms": 970.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.5555555555555556, + "recall@10": 0.5555555555555556, + "precision@5": 1.0, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a", + "art-e495c8c5-ad95-4b64-8771-f68aa4cfcd0a", + "kn-e8423822-eacf-4029-aa7b-10d4d28d621e", + "bl-7aebe936-ac55-4f35-8932-adc5224ff854", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf" + ], + "n_returned": 10, + "latency_ms": 920.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.1111111111111111, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "art-8dbceb06-431a-416d-a723-e8c75d595154", + "mem-a3124d5b-2f50-477f-8bb5-06879f5a496c", + "knw-528dbc37-eabc-4b75-a7a5-65bf38d6018a", + "art-94fae615-7cd5-4695-b968-977101b06a51", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "? ?}&?#??X\b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 10, + "latency_ms": 925.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.5, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "mem-46780047-63a0-4a86-a16b-638b72a7fb8d", + "kn-8e1bfb48-33a9-45ad-8da7-e0bdaa5d34e7", + "mem-cdff0c49-3ac7-4de8-89ec-92d254bd0023", + "bl-a7a1428f-db9c-417b-8e2c-713b1f84dc1f", + "mem-f823e835-313f-4282-b4b3-ce527ffc2f7a", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "?Z?.\f?0?]P?", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72" + ], + "n_returned": 10, + "latency_ms": 2455.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.14285714285714285, + "precision@5": 0.0, + "mrr@10": 0.14285714285714285 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "art-92e1837c-5919-42d0-bbb0-4d924d7b2864", + "bl-e20944e5-eb16-4ab3-a84d-111e0fc817fa", + "bl-e20944e5-f4a6-44a0-91b1-73d04ebed120", + "mem-47f72b5b-6e8b-4293-94f1-350197b4809a", + "mem-e612f0aa-c2f2-4ee3-bbc7-af2dc826233b", + "mem-a5f04e52-91f8-41d2-af27-8bf803621758", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "? ?}&?#??X\b", + "7?e?7???\f3?", + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a" + ], + "n_returned": 10, + "latency_ms": 2037.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "kn-82be4e41-96c5-4da3-85c0-cee10763d975", + "mem-ea487cb4-ed67-44ce-8402-b56bb28468d4", + "mem-23d22bc1-a097-446b-8f11-8aff099e0b76", + "bl-874d1c2b-c55b-4afb-9601-922a9297e859", + "bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60", + "bl-2694b588-a6e3-43de-861c-fa7b0ec7e7fd", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "? ?}&?#??X\b", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad" + ], + "n_returned": 10, + "latency_ms": 4720.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "mem-0328c3cb-4550-4ce4-9284-152e832f08f6", + "bl-07375bf9-a169-42cd-adb3-7d32b25982f0", + "bl-fd047ce9-ae21-4b3e-b3ab-ece0c9592f7f", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "bl-6172d035-dd94-4776-afdd-d8915f6fc375", + "bl-5bb8dedf-8498-4a9b-acdc-31cc9c738f2a", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396" + ], + "n_returned": 10, + "latency_ms": 5078.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "bl-e5635b1a-c5d0-4caa-bcc8-6a726ea43685", + "bl-fc893be3-e6b4-4ef6-93b0-d54ca5f89083", + "tag-project-structure", + "tag-anthropic-contrast", + "tag-voice-training", + "tag-ebd", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 4118.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-d24fcce8-2b55-426f-867a-db3958a622d3", + "tag-phase-3", + "tag-identity-studio", + "tag-kids", + "tag-coexistence", + "tag-cultivated-general-intelligence", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "? ?}&?#??X\b", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "mem-cdff0c49-3ac7-4de8-89ec-92d254bd0023" + ], + "n_returned": 10, + "latency_ms": 3241.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992", + "bl-b019ce7a-1b21-436e-812d-032f50c6c45f", + "bl-e98cdd4c-01b5-459e-9036-3578cd5d975a", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "tag-stable-diffusion", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 4090.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "mem-e321e54e-8bb3-4596-b13d-bb093d6b149d", + "mem-e32ba5a7-c147-4dc0-9479-b720d768eda6", + "mem-c7a77457-478d-4eb0-a116-67205a0066a4", + "bl-1b20e9bc-eb37-4907-8d63-e311fd61eab8", + "bl-aa762207-920d-45ab-b2a3-2f8154d7ef9b", + "tag-misalignment", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3" + ], + "n_returned": 10, + "latency_ms": 3671.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "kn-48a01973-a025-471d-950f-b93e6a426d82", + "bl-7f33f1bc-99fa-4906-889f-a42375beea20", + "mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8", + "mem-ce5a2ffc-ad39-4728-9ac6-76fef507d5da", + "project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_", + "tag-provenance", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "art-e495c8c5-ad95-4b64-8771-f68aa4cfcd0a" + ], + "n_returned": 10, + "latency_ms": 4674.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "kn-79056192-7de8-486c-9565-f128439a6fcb", + "bl-f6236350-f7b8-4f4f-a702-9eef2eb76e4b", + "mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9", + "mem-37b57f52-a29a-42cf-a07a-3c5f8a3598dd", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "tag-__cultivation-metric____internal-state____dharma____evidence____novel-idea____gap-compression____values____microsoft__", + "? ?}&?#??X\b", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396" + ], + "n_returned": 10, + "latency_ms": 3974.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "bl-57c5cf6b-81a5-4558-9902-5c02981fe273", + "tag-guilds", + "tag-finance", + "tag-temporal", + "tag-barkhausen", + "tag-ilogger", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c" + ], + "n_returned": 10, + "latency_ms": 4851.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "project-Goal_setting__alignment__scoring__cadence__Attaches_to_any_imprint_", + "tag-turing-test", + "tag-sealed", + "tag-design-first", + "tag-part-5", + "tag-model", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 4505.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "mem-9590ba23-bddb-43e8-a571-68a263c4c364", + "bl-a9e57bb2-00a1-4867-ab59-5d9271134b50", + "tag-performed-values", + "bl-c7793c4a-7630-47fc-a462-d23059087e80", + "tag-gateway_platform_neuron-technologies_go_proxy_llm", + "tag-fornax", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396" + ], + "n_returned": 10, + "latency_ms": 4494.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "bl-7a13527b-3e0c-418a-9f37-88fd2152e5ce", + "bl-3f57bc69-7285-4f4a-a861-2de52efca058", + "bl-5e390b10-8753-4f25-a1a5-b5dbbb002cbf", + "tag-ats", + "tag-data-model", + "tag-aggregation", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 3619.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "tag-memory-model", + "tag-storage", + "tag-ga4", + "tag-potions", + "tag-resonance", + "tag-neuron", + "? ?}&?#??X\b", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-ee615cdb-e599-423d-9a4d-977859390ed3" + ], + "n_returned": 10, + "latency_ms": 4107.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "mem-927f41ab-8ede-4f58-acb3-995db16ac775", + "mem-dbe80bc2-c602-46b0-b4ea-dd222e52bcde", + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "tag-upload-window", + "? ?}&?#??X\b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 10, + "latency_ms": 4231.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "mem-434be7c8-88cb-4039-b79a-1da4ac4de783", + "mem-481c769c-68cc-45c7-bc37-c0d9778fa648", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "bl-9d8f3c5b-4bac-41ce-8ac4-44733f99d6c8", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 10, + "latency_ms": 3208.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "project-Imprint__leadership_development__feedback_frameworks__performance__presence_", + "mem-5e7f6ddd-c818-4ad3-b564-54ae278e9976", + "bl-7fa1b1a8-b80a-4f28-b162-bfe73765b4f8", + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c", + "tag-sarah", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "? ?}&?#??X\b", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 2359.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "mem-265a7107-73b3-4410-9aff-43787d5f473b", + "mem-9d1bf963-1b40-4588-bdb3-0432646cc623", + "bl-0de4e61b-6562-49e5-b7df-ebb809a01723", + "mem-3987d374-3c48-4e8e-b06d-0c363f55ed9c", + "bl-e0a0df72-de6e-46ab-800b-e1e3e8dfc387", + "tag-__patents____swarm____claim-language____prior-art____filing__", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "7?e?7???\f3?", + "?of?7???" + ], + "n_returned": 10, + "latency_ms": 2342.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-dca14c4c-4859-47b0-996e-33964ba61a87", + "bl-e148d23c-24e8-4122-9915-d1c11f22052f", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-dc8c7e02-eb37-48ae-a6f8-9b512803ae16", + "mem-8d1bafe6-209c-456c-9a25-9a927bc5a16d", + "bl-14883d81-f7cb-46dd-82c2-a6e6980264e5", + "? ?}&?#??X\b", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "?Q??m?;��`'", + "R^??m?;��?'" + ], + "n_returned": 10, + "latency_ms": 3536.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "bl-ffa22d7e-42a9-4bd2-a428-1d2df243ac93", + "bl-452a4710-3d2b-4e0f-9413-49a66423bc9a", + "bl-4a6746e8-191f-48fc-8bfb-c4dc73b80bcd", + "tag-command-pattern", + "tag-withholding", + "tag-offline", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 4151.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 1342.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 874.8, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [ + "kn-333542cb-6dab-4662-9725-bf7440d28bf7", + "? ?}&?#??X\b", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-66a21179-2adc-4b19-a109-880cf4674d7d", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4" + ], + "n_returned": 8, + "latency_ms": 1374.6, + "error": null, + "clean": false, + "false_positives": 8 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "ctx-e5427d7d", + "project-worldweaver", + "tag-__kotlin____internal-state____pre-reasoning____post-reasoning____compression-ratio____dharma____cultivation__", + "tag-import", + "tag-enterprise", + "tag-__cgi____dharma____cultivation____five-primitives____seed-artifact____agi____intelligence____whitepaper____patent__", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "? ?}&?#??X\b", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "015644f5-8194-4af0-800d-dd4a0cd71396" + ], + "n_returned": 10, + "latency_ms": 3762.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": null + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "mem-bfe0fafd-2750-4fdc-b773-04e878b3b23f", + "art-7bdaff30-5af9-4f0a-93b1-751686f9de3d", + "mem-cf07910d-4676-4384-ab97-9cad946cd0b9", + "project-Convert_UTC_timestamps_to_Central_time_when_displaying_to_Will__Never_surface_raw_UTC_", + "mem-32203649-3213-4d6d-86fd-3d657ac70d77", + "mem-22f5f665-3ad2-4063-88b0-915849a795f5", + "? ?}&?#??X\b", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-80ca3d31-84dc-4502-83f4-538372b9764f", + "015644f5-8194-4af0-800d-dd4a0cd71396" + ], + "n_returned": 10, + "latency_ms": 4859.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "project-Imprint__discovery__objection_handling__deal_strategy__pipeline__closing_", + "bl-8116da7a-b039-4e08-b8d0-c1c7861f9766", + "bl-8ef1ba6b-3fa0-4dbd-98c5-31665e5694a1", + "tag-dark-theme", + "tag-divisors", + "tag-distressed-property", + "? ?}&?#??X\b", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "???Ͼd??W\b?" + ], + "n_returned": 10, + "latency_ms": 2873.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": null + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-main-r2.json b/tools/retrieval-eval/results-main-r2.json new file mode 100644 index 0000000..81a9b29 --- /dev/null +++ b/tools/retrieval-eval/results-main-r2.json @@ -0,0 +1,945 @@ +{ + "label": "main-r2", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-main", + "soul_md5": "caf822425540114984bfcd713ad37162", + "corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json", + "limit": 10, + "port": 7895, + "wall_clock_s": 45.6, + "child_pid": 78695, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.34285714285714286, + "recall@5": 0.26947278911564626, + "recall@10": 0.3333333333333333, + "precision@5": 0.12000000000000001, + "mrr@10": 0.2943197278911564, + "nonsense_clean": "2/3", + "superseded_outranks": "1/3", + "latency_ms_p50": 1151.1, + "latency_ms_p95": 1580.7, + "latency_ms_max": 1663.2, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4902210884353741, + "recall@10": 0.6666666666666666, + "mrr@10": 0.5965986394557822 + }, + "superseded": { + "n": 3, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.3333333333333333, + "mrr@10": 0.041666666666666664, + "outranks": 1 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "n_returned": 1, + "latency_ms": 231.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5" + ], + "n_returned": 1, + "latency_ms": 266.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "n_returned": 1, + "latency_ms": 229.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "n_returned": 1, + "latency_ms": 235.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848" + ], + "n_returned": 1, + "latency_ms": 254.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "n_returned": 1, + "latency_ms": 228.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "?ǚ?7??????", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "????7???Ջ3", + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b", + "7?e?7???\f3?" + ], + "n_returned": 10, + "latency_ms": 543.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "Z[?] [--repo ] +set -euo pipefail + +BASELINE="main" +CANDIDATE="feat/recall-through-activation" +REPEATS=3 +CORPUS="$HOME/neuron-memory-backups/snapshot-pre-repair-20260806.json" +REPO="$HOME/Development/neuron" +BASE_PORT=7893 + +while [ $# -gt 0 ]; do + case "$1" in + --baseline) BASELINE="$2"; shift 2 ;; + --candidate) CANDIDATE="$2"; shift 2 ;; + --repeats) REPEATS="$2"; shift 2 ;; + --corpus) CORPUS="$2"; shift 2 ;; + --repo) REPO="$2"; shift 2 ;; + *) echo "unknown arg: $1" >&2; exit 2 ;; + esac +done + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/retrieval-eval.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT + +[ -f "$CORPUS" ] || { echo "no corpus at $CORPUS" >&2; exit 2; } +echo "corpus: $CORPUS ($(du -h "$CORPUS" | cut -f1))" + +slug() { printf '%s' "$1" | tr '/' '-'; } + +build_ref() { # ref -> binary path + local ref="$1" out="$WORK/soul-$(slug "$1")" + local src="$WORK/src-$(slug "$1")" + mkdir -p "$src" + git -C "$REPO" archive "$ref" | tar -x -C "$src" + "$HERE/build-soul.sh" "$src" "$out" >&2 + printf '%s' "$out" +} + +echo "== building $BASELINE ==" +BIN_A="$(build_ref "$BASELINE")" +echo "== building $CANDIDATE ==" +BIN_B="$(build_ref "$CANDIDATE")" + +echo "== validating the gold set against this corpus ==" +python3 "$HERE/build_gold_set.py" "$CORPUS" --check + +port=$BASE_PORT +run_one() { # binary label out + echo "== $2 ==" + python3 "$HERE/run_eval.py" --soul "$1" --corpus "$CORPUS" --label "$2" \ + --port "$port" --out "$3" + port=$((port + 1)) +} + +A_MAIN="$WORK/results-a-1.json"; B_MAIN="$WORK/results-b-1.json" +A_REP=(); B_REP=() +for i in $(seq 1 "$REPEATS"); do + a="$WORK/results-a-$i.json"; b="$WORK/results-b-$i.json" + run_one "$BIN_A" "$(slug "$BASELINE")-r$i" "$a" + run_one "$BIN_B" "$(slug "$CANDIDATE")-r$i" "$b" + [ "$i" -gt 1 ] && { A_REP+=("$a"); B_REP+=("$b"); } +done + +cp "$A_MAIN" "$HERE/results-$(slug "$BASELINE").json" +cp "$B_MAIN" "$HERE/results-$(slug "$CANDIDATE").json" + +echo +python3 "$HERE/compare.py" \ + --baseline "$A_MAIN" --candidate "$B_MAIN" \ + ${A_REP[@]+--repeats-baseline "${A_REP[@]}"} \ + ${B_REP[@]+--repeats-candidate "${B_REP[@]}"} \ + --out "$HERE/comparison-$(slug "$BASELINE")-vs-$(slug "$CANDIDATE").json" + +# Backstop: run_eval.py kills and confirms its own child, but a crashed run +# could leak one. Leaving a soul running is how the live engine got squeezed. +STRAY=$(pgrep -f "$WORK/soul-" || true) +if [ -n "$STRAY" ]; then + echo "!! stray eval souls, killing: $STRAY" >&2 + kill -9 $STRAY 2>/dev/null || true +fi +pgrep -f "$WORK/soul-" >/dev/null && { echo "!! STILL RUNNING" >&2; exit 5; } +echo "process check: no eval souls running" diff --git a/tools/retrieval-eval/run_eval.py b/tools/retrieval-eval/run_eval.py new file mode 100755 index 0000000..ec054c8 --- /dev/null +++ b/tools/retrieval-eval/run_eval.py @@ -0,0 +1,398 @@ +#!/usr/bin/env python3 +""" +run_eval.py — measure one soul build's retrieval against the gold set. + +WHAT IT MEASURES, AND WHY IT BOOTS A REAL SOUL + The point is Will's designed retrieval — spreading activation over the + weighted directed graph with four-factor multiplicative scoring — not a + Python re-implementation of it. A re-implementation would measure my + reading of the design; booting the compiled binary measures the design. So + this harness compiles the actual `soul.el` amalgam (build-soul.sh) and asks + it over HTTP, exactly as the MCP wrapper and the app do. + +SAFETY — read this before changing anything here + * Boots on a THROWAWAY port with a THROWAWAY $HOME and a THROWAWAY COPY of + the corpus. Refuses to use 7770 / 8742 / 7779 / 17779 / 7771. + * ENGRAM_URL / SOUL_ENGRAM_URL are UNSET and SOUL_ISE_URL is pinned to a + dead port. This is not belt-and-braces: the periodic engram sync resolves + its source as env(SOUL_ISE_URL) -> state -> DEFAULT http://localhost:8742, + so leaving it unset makes an "isolated" run silently pull the operator's + LIVE brain. (Learned the hard way on 2026-08-03; see the same note in + scripts/verify-soul-contract.sh.) + * Every process this file starts is tracked and killed in a finally block, + then CONFIRMED dead by pid probe, and the confirmation is written into the + results file. A run that cannot confirm its child is dead exits non-zero. + * Activation is a STATEFUL read by design (patent claim 29: traversal + updates last-activation and increments activation counts). The corpus copy + is therefore per-run and disposable, and every run starts from a byte- + identical copy so two configurations see the same starting graph. + +usage: + python3 run_eval.py --soul --corpus --label main \ + [--port 7893] [--gold gold_set.json] [--limit 10] [--out results-main.json] +""" +import argparse +import json +import os +import shutil +import signal +import subprocess +import sys +import tempfile +import time +import urllib.error +import urllib.parse +import urllib.request + +HERE = os.path.dirname(os.path.abspath(__file__)) +FORBIDDEN_PORTS = {7770, 8742, 7779, 17779, 7771, 8080} + + +# ───────────────────────────────────────────────────────────────────────────── +# metrics +# ───────────────────────────────────────────────────────────────────────────── +def recall_at_k(returned, relevant, k): + if not relevant: + return None + return len(set(returned[:k]) & set(relevant)) / len(relevant) + + +def hit_at_k(returned, relevant, k): + if not relevant: + return None + return 1.0 if set(returned[:k]) & set(relevant) else 0.0 + + +def precision_at_k(returned, relevant, k): + """Fixed denominator k, as in docs/research/graphrag_eval/score.py. + + Fixed denominator penalises an empty result and a page of junk equally, + which is what we want: a retriever that returns nothing is not 'precise'. + """ + if not relevant: + return None + return len(set(returned[:k]) & set(relevant)) / k + + +def mrr(returned, relevant, k): + if not relevant: + return None + rel = set(relevant) + for i, nid in enumerate(returned[:k], start=1): + if nid in rel: + return 1.0 / i + return 0.0 + + +def mean(vals): + vals = [v for v in vals if v is not None] + return sum(vals) / len(vals) if vals else 0.0 + + +def pct(vals): + return f"{100 * mean(vals):.1f}%" + + +# ───────────────────────────────────────────────────────────────────────────── +# soul lifecycle +# ───────────────────────────────────────────────────────────────────────────── +class Soul: + def __init__(self, binary, corpus, port, verbose=True): + if port in FORBIDDEN_PORTS: + raise SystemExit(f"REFUSING: port {port} is a live service port.") + self.binary = os.path.abspath(binary) + self.corpus = os.path.abspath(corpus) + self.port = port + self.verbose = verbose + self.home = None + self.proc = None + self.pid = None + self.log = None + self.confirmed_dead = None + + @property + def base(self): + return f"http://127.0.0.1:{self.port}" + + def start(self, boot_timeout=180): + self.home = tempfile.mkdtemp(prefix="retrieval-eval-home.") + snap = os.path.join(self.home, "corpus.json") + t0 = time.time() + shutil.copyfile(self.corpus, snap) # per-run disposable copy, never the source + self.log = os.path.join(self.home, "soul.log") + + env = {k: v for k, v in os.environ.items() + if k not in ("ENGRAM_URL", "ENGRAM_API_KEY", "SOUL_ENGRAM_URL", + "ANTHROPIC_API_KEY", "NEURON_LLM_API_KEY", "SOUL_IDENTITY", + "SOUL_API_KEY")} + env.update({ + "HOME": self.home, + "NEURON_PORT": str(self.port), + "SOUL_CGI_ID": f"ntn-retrieval-eval-{os.getpid()}", + "SOUL_ENGRAM_PATH": snap, + "NEURON_API_URL": "http://127.0.0.1:9", # dead port + "SOUL_ISE_URL": "http://127.0.0.1:9", # dead port — see SAFETY above + # Park the background loops for an hour so heartbeat/consolidation + # cannot mutate the graph between queries and make runs unrepeatable. + "SOUL_TICK_MS": "3600000", + "SOUL_HEARTBEAT_MS": "3600000", + "SOUL_REFRESH_MS": "3600000", + }) + with open(self.log, "wb") as lf: + self.proc = subprocess.Popen([self.binary], env=env, stdout=lf, stderr=lf, + start_new_session=True) + self.pid = self.proc.pid + if self.verbose: + print(f" booted pid={self.pid} port={self.port} home={self.home}") + + deadline = time.time() + boot_timeout + while time.time() < deadline: + if self.proc.poll() is not None: + raise RuntimeError(f"soul exited during boot: {self._log_tail()}") + rss = self._rss_kb() + if rss and rss > 6 * 1024 * 1024: + self.stop() + raise RuntimeError(f"soul RSS {rss}KB > 6GB — aborted") + try: + with urllib.request.urlopen(f"{self.base}/health", timeout=2) as r: + if r.status == 200: + if self.verbose: + print(f" healthy in {time.time() - t0:.1f}s, RSS={self._rss_kb()}KB") + return + except Exception: + pass + time.sleep(0.5) + self.stop() + raise RuntimeError(f"soul never healthy on {self.base}: {self._log_tail()}") + + def _rss_kb(self): + try: + out = subprocess.run(["ps", "-o", "rss=", "-p", str(self.pid)], + capture_output=True, text=True, timeout=5).stdout.strip() + return int(out) if out else None + except Exception: + return None + + def _log_tail(self, n=15): + try: + with open(self.log, encoding="utf-8", errors="replace") as fh: + return "\n".join(fh.read().splitlines()[-n:]) + except Exception: + return "(no log)" + + def recall(self, query, limit, timeout=60): + url = f"{self.base}/api/neuron/recall?query={urllib.parse.quote(query)}&limit={limit}" + t0 = time.perf_counter() + try: + with urllib.request.urlopen(url, timeout=timeout) as r: + raw = r.read().decode("utf-8", "replace") + ms = (time.perf_counter() - t0) * 1000 + except Exception as exc: + return [], (time.perf_counter() - t0) * 1000, f"{type(exc).__name__}: {exc}" + try: + arr = json.loads(raw) + except Exception: + return [], ms, f"unparseable response ({len(raw)}B)" + if not isinstance(arr, list): + return [], ms, f"non-array response: {str(arr)[:120]}" + ids = [x.get("id") for x in arr if isinstance(x, dict) and x.get("id")] + return ids, ms, None + + def stop(self): + """Kill and CONFIRM. A test process that outlives its test is a bug.""" + if self.pid is None: + self.confirmed_dead = True + return True + for sig in (signal.SIGTERM, signal.SIGKILL): + try: + os.kill(self.pid, sig) + except ProcessLookupError: + break + except Exception: + pass + for _ in range(20): + try: + os.kill(self.pid, 0) + except ProcessLookupError: + break + time.sleep(0.1) + else: + continue + break + try: + self.proc.wait(timeout=5) + except Exception: + pass + try: + os.kill(self.pid, 0) + self.confirmed_dead = False + except ProcessLookupError: + self.confirmed_dead = True + if self.verbose: + print(f" pid {self.pid}: {'CONFIRMED DEAD' if self.confirmed_dead else 'STILL ALIVE'}") + if self.home and os.path.isdir(self.home): + shutil.rmtree(self.home, ignore_errors=True) + return self.confirmed_dead + + +# ───────────────────────────────────────────────────────────────────────────── +# eval +# ───────────────────────────────────────────────────────────────────────────── +def evaluate(soul, gold, limit): + rows = [] + for q in gold["queries"]: + ids, ms, err = soul.recall(q["query"], limit) + rel = q.get("relevant") or [] + row = { + "id": q["id"], + "category": q["category"], + "query": q["query"], + "returned": ids, + "n_returned": len(ids), + "latency_ms": round(ms, 1), + "error": err, + } + if q.get("expect_empty"): + row["clean"] = (len(ids) == 0) + row["false_positives"] = len(ids) + else: + row["hit@5"] = hit_at_k(ids, rel, 5) + row["recall@5"] = recall_at_k(ids, rel, 5) + row["recall@10"] = recall_at_k(ids, rel, 10) + row["precision@5"] = precision_at_k(ids, rel, 5) + row["mrr@10"] = mrr(ids, rel, 10) + if q.get("must_outrank"): + correct, stale = q["must_outrank"] + ic = ids.index(correct) if correct in ids else None + istale = ids.index(stale) if stale in ids else None + # Correct must be present AND above the stale node. A run that + # returns neither is NOT a pass: the corrected fact is what the + # user needed. + row["outranks"] = (ic is not None) and (istale is None or ic < istale) + row["rank_correct"] = None if ic is None else ic + 1 + row["rank_stale"] = None if istale is None else istale + 1 + rows.append(row) + return rows + + +def aggregate(rows): + scored = [r for r in rows if "hit@5" in r] + nonsense = [r for r in rows if "clean" in r] + outrank = [r for r in rows if "outranks" in r] + lat = sorted(r["latency_ms"] for r in rows) + agg = { + "n_queries": len(rows), + "n_scored": len(scored), + "hit@5": mean([r["hit@5"] for r in scored]), + "recall@5": mean([r["recall@5"] for r in scored]), + "recall@10": mean([r["recall@10"] for r in scored]), + "precision@5": mean([r["precision@5"] for r in scored]), + "mrr@10": mean([r["mrr@10"] for r in scored]), + "nonsense_clean": f"{sum(1 for r in nonsense if r['clean'])}/{len(nonsense)}", + "superseded_outranks": f"{sum(1 for r in outrank if r['outranks'])}/{len(outrank)}", + "latency_ms_p50": lat[len(lat) // 2] if lat else 0, + "latency_ms_p95": lat[max(0, int(len(lat) * 0.95) - 1)] if lat else 0, + "latency_ms_max": lat[-1] if lat else 0, + "errors": sum(1 for r in rows if r["error"]), + "by_category": {}, + } + cats = sorted({r["category"] for r in rows}) + for c in cats: + cr = [r for r in rows if r["category"] == c] + if c == "nonsense": + agg["by_category"][c] = { + "n": len(cr), + "clean": sum(1 for r in cr if r["clean"]), + "avg_false_positives": mean([float(r["false_positives"]) for r in cr]), + } + else: + e = { + "n": len(cr), + "hit@5": mean([r.get("hit@5") for r in cr]), + "recall@5": mean([r.get("recall@5") for r in cr]), + "recall@10": mean([r.get("recall@10") for r in cr]), + "mrr@10": mean([r.get("mrr@10") for r in cr]), + } + if c == "superseded": + e["outranks"] = sum(1 for r in cr if r.get("outranks")) + agg["by_category"][c] = e + return agg + + +def print_table(label, agg): + print(f"\n=== {label} ===") + print(f" queries {agg['n_queries']} ({agg['n_scored']} scored + " + f"{agg['n_queries'] - agg['n_scored']} control) · errors {agg['errors']}") + print(f" {'hit@5':>12} {'recall@5':>10} {'recall@10':>10} {'prec@5':>9} {'MRR@10':>9}") + print(f" {pct([agg['hit@5']]):>12} {pct([agg['recall@5']]):>10} {pct([agg['recall@10']]):>10} " + f"{pct([agg['precision@5']]):>9} {agg['mrr@10']:>9.3f}") + print(f" nonsense clean {agg['nonsense_clean']} · superseded outranks {agg['superseded_outranks']}") + print(f" latency ms p50 {agg['latency_ms_p50']:.0f} · p95 {agg['latency_ms_p95']:.0f} " + f"· max {agg['latency_ms_max']:.0f}") + print(f"\n {'category':14} {'n':>3} {'hit@5':>8} {'recall@5':>9} {'recall@10':>10} {'MRR@10':>8}") + for c, e in agg["by_category"].items(): + if c == "nonsense": + print(f" {c:14} {e['n']:>3} {'clean ' + str(e['clean']) + '/' + str(e['n']):>8}" + f"{'':>9} {'':>10} {'avg FP ' + format(e['avg_false_positives'], '.1f'):>8}") + else: + extra = f" outranks {e['outranks']}/{e['n']}" if "outranks" in e else "" + print(f" {c:14} {e['n']:>3} {pct([e['hit@5']]):>8} {pct([e['recall@5']]):>9} " + f"{pct([e['recall@10']]):>10} {e['mrr@10']:>8.3f}{extra}") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--soul", required=True) + ap.add_argument("--corpus", required=True) + ap.add_argument("--label", required=True) + ap.add_argument("--gold", default=os.path.join(HERE, "gold_set.json")) + ap.add_argument("--port", type=int, default=7893) + ap.add_argument("--limit", type=int, default=10) + ap.add_argument("--out", default=None) + args = ap.parse_args() + + with open(args.gold, encoding="utf-8") as fh: + gold = json.load(fh) + print(f"[{args.label}] soul={os.path.basename(args.soul)} " + f"corpus={os.path.basename(args.corpus)} gold={len(gold['queries'])}q limit={args.limit}") + + soul = Soul(args.soul, args.corpus, args.port) + rows = [] + started = time.time() + try: + soul.start() + rows = evaluate(soul, gold, args.limit) + finally: + dead = soul.stop() + + agg = aggregate(rows) + print_table(args.label, agg) + + out = args.out or os.path.join(HERE, f"results-{args.label}.json") + doc = { + "label": args.label, + "soul_binary": os.path.abspath(args.soul), + "soul_md5": subprocess.run(["md5", "-q", args.soul], capture_output=True, + text=True).stdout.strip(), + "corpus": os.path.abspath(args.corpus), + "corpus_nodes": gold.get("corpus_nodes"), + "corpus_edges": gold.get("corpus_edges"), + "gold_set": os.path.abspath(args.gold), + "limit": args.limit, + "port": args.port, + "wall_clock_s": round(time.time() - started, 1), + "child_pid": soul.pid, + "child_confirmed_dead": soul.confirmed_dead, + "aggregate": agg, + "rows": rows, + } + with open(out, "w", encoding="utf-8") as fh: + json.dump(doc, fh, indent=1, ensure_ascii=False) + print(f"\nwrote {out}") + if not dead: + print("FATAL: child process could not be confirmed dead", file=sys.stderr) + sys.exit(4) + + +if __name__ == "__main__": + main() From 315b2eff00e7cec6be499d47763cddd69b24238b Mon Sep 17 00:00:00 2001 From: Neuron Date: Fri, 7 Aug 2026 14:44:42 -0500 Subject: [PATCH 02/15] feat(engram): fuse cosine similarity into the recall read path (claim 24) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit engram_search_json — the function /api/neuron/recall actually reaches — ranked only by distinct-token match count, so the embedding field on every node record was inert. Add the semantic leg as a UNION beside the lexical one, not a replacement for it: fused = (distinct_tokens_matched / query_tokens) + 0.90 * sem sem = clamp01((cos(q,n) - 0.60) / (1 - 0.60)) ; 0 when not comparable Holding the semantic weight strictly below 1.0 means a node matching every query token can never be displaced by semantics alone — the regression guard that PR #135 lacked when it swapped the read path to spreading activation and took phrase recall from 85.7% to 28.6%. No query embedding (embedder down, circuit breaker open) => sem == 0 for all nodes => fused == sc/ntok, a monotone map of the old integer score, so the ordering degrades to the historical behaviour exactly. Restores engram claim 24: 'maintain a vector similarity index over the semantic embedding vectors of all stored node records, and ... respond to embedding search queries by returning the node records whose embedding vectors have the highest cosine similarity to a query vector, independently of the spreading activation traversal.' Co-Authored-By: Claude Opus 5 (1M context) --- .../el-runtime/v1.0.0-20260501/el_runtime.c | 74 +++++++++++++++++-- 1 file changed, 67 insertions(+), 7 deletions(-) diff --git a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c index 765aa6e..85b4595 100644 --- a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c +++ b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c @@ -6099,6 +6099,21 @@ static void engram_bll_parse_access(EngramNode* nn, const char* s) { * propagation loop in engram_activate. 0.25 damps semantically unrelated * branches ~4x without severing them. Unembedded targets are ungated. */ #define ENGRAM_QGATE_FLOOR 0.25 +/* ENGRAM_SEM_WEIGHT: weight of the semantic (cosine) leg in the fused read-path + * score, per engram claim 24 — "maintain a vector similarity index over the + * semantic embedding vectors of all stored node records, and ... respond to + * embedding search queries by returning the node records whose embedding + * vectors have the highest cosine similarity to a query vector, independently + * of the spreading activation traversal." + * The lexical leg contributes distinct-token coverage in [0,1]; the semantic + * leg contributes at most ENGRAM_SEM_WEIGHT. Holding this strictly BELOW 1.0 + * is the safety property: a node matching every query token can never be + * displaced by semantics alone, so exact/phrase retrieval cannot regress the + * way it did when the read path was swapped wholesale to spreading activation + * (PR #135: phrase 85.7% -> 28.6%). Semantics reorders WITHIN and BELOW the + * full-lexical-match band, and admits high-cosine nodes the lexical pass + * missed entirely. */ +#define ENGRAM_SEM_WEIGHT 0.90 #define ENGRAM_EMBED_MAX_CHARS 2000 #define ENGRAM_EMBED_TIMEOUT_MS 4000L #define ENGRAM_EMBED_BREAKER_LIMIT 3 @@ -7381,13 +7396,39 @@ static int engram_node_match_score(const EngramNode* n, return score; } -/* Rank entry: distinct-token match count (primary, desc) then salience - * (tiebreak, desc). */ -typedef struct { int64_t idx; int score; double salience; } EngramRankEntry; +/* Semantic leg of the read path (engram claim 24). Returns the query/target + * cosine renormalized onto [0,1] over the band [ENGRAM_EMBED_SEED_MIN, 1.0], + * and exactly 0.0 when the pair is not comparable (no query embedding, target + * unembedded, dim mismatch) or falls at/below the seed floor. Claim 32's + * clamp-at-zero is subsumed: nothing below the floor can contribute. + * A 0.0 return makes the fused score collapse to the lexical score, which is + * why a dead embedder degrades to the historical behaviour exactly. */ +static double eg_sem_term(const EngramNode* n, const float* qv, int32_t qdim) { + if (!qv || qdim <= 0 || !n->emb || n->emb_dim != qdim) return 0.0; + double c = eg_cosine(n->emb, qv, qdim); + if (c <= ENGRAM_EMBED_SEED_MIN) return 0.0; + double t = (c - ENGRAM_EMBED_SEED_MIN) / (1.0 - ENGRAM_EMBED_SEED_MIN); + return t > 1.0 ? 1.0 : t; +} + +/* Fused rank score: distinct-token coverage in [0,1] plus at most + * ENGRAM_SEM_WEIGHT of semantic similarity. With no query embedding this is + * sc/ntok, a strictly monotone map of the old integer score, so the ordering + * is bit-identical to the pre-semantic ranking. */ +static double eg_fused_score(int sc, int ntok, double sem) { + double f = (double)sc / (double)(ntok > 0 ? ntok : 1); + return f + ENGRAM_SEM_WEIGHT * sem; +} + +/* Rank entry: fused lexical+semantic score (primary, desc) then salience + * (tiebreak, desc). `score` retains the raw distinct-token count for callers + * that want the lexical signal on its own. */ +typedef struct { int64_t idx; int score; double fused; double salience; } EngramRankEntry; static int engram_rank_cmp(const void* a, const void* b) { const EngramRankEntry* ea = (const EngramRankEntry*)a; const EngramRankEntry* eb = (const EngramRankEntry*)b; - if (ea->score != eb->score) return eb->score - ea->score; /* desc */ + if (ea->fused < eb->fused) return 1; /* desc */ + if (ea->fused > eb->fused) return -1; if (ea->salience < eb->salience) return 1; if (ea->salience > eb->salience) return -1; return 0; @@ -7405,6 +7446,10 @@ el_val_t engram_search(el_val_t query, el_val_t limit) { if (ntok == 0) return lst; EngramRankEntry* hits = malloc((size_t)g->node_count * sizeof(EngramRankEntry)); if (!hits) return lst; + /* Claim-24 semantic leg: one query embedding, fetched once per search. + * NULL (embedder down / circuit breaker open) => pure lexical, as before. */ + int32_t qdim = 0; + float* qv = eg_embed_fetch(q, &qdim); int64_t nhits = 0; for (int64_t i = 0; i < g->node_count; i++) { EngramNode* n = &g->nodes[i]; @@ -7414,20 +7459,26 @@ el_val_t engram_search(el_val_t query, el_val_t limit) { * + engram_compile_layered_json — that's the legitimate path. */ if (engram_layer_is_transparent(n->layer_id)) continue; int sc = engram_node_match_score(n, toks, ntok); - if (sc > 0) { + double sem = eg_sem_term(n, qv, qdim); + /* Union, not replacement: a node enters the candidate set on EITHER + * leg. sc == 0 && sem > 0 is the embedding-search half of claim 24 — + * nodes the lexical pass cannot see at all. */ + if (sc > 0 || sem > 0.0) { hits[nhits].idx = i; hits[nhits].score = sc; + hits[nhits].fused = eg_fused_score(sc, ntok, sem); hits[nhits].salience = n->salience; nhits++; } } - /* Rank by distinct tokens matched (desc) then salience (desc), then cap. */ + /* Rank by fused score (desc) then salience (desc), then cap. */ qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp); int64_t end = nhits < lim ? nhits : lim; for (int64_t k = 0; k < end; k++) { lst = el_list_append(lst, engram_node_to_map(&g->nodes[hits[k].idx])); } free(hits); + free(qv); return lst; } @@ -9259,15 +9310,23 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { if (ntok > 0) { EngramRankEntry* hits = malloc((size_t)g->node_count * sizeof(EngramRankEntry)); if (hits) { + /* Claim-24 semantic leg. This is the function /api/neuron/recall + * actually reaches (routes.el -> neuron-api.el handle_api_recall), + * so the semantic half of the retrieval surface has to land HERE + * to be observable to the MCP wrapper and the app. */ + int32_t qdim = 0; + float* qv = eg_embed_fetch(q, &qdim); int64_t nhits = 0; for (int64_t i = 0; i < g->node_count; i++) { EngramNode* n = &g->nodes[i]; /* Filter transparent layers — same as engram_search. */ if (engram_layer_is_transparent(n->layer_id)) continue; int sc = engram_node_match_score(n, toks, ntok); - if (sc > 0) { + double sem = eg_sem_term(n, qv, qdim); + if (sc > 0 || sem > 0.0) { hits[nhits].idx = i; hits[nhits].score = sc; + hits[nhits].fused = eg_fused_score(sc, ntok, sem); hits[nhits].salience = n->salience; nhits++; } @@ -9280,6 +9339,7 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { first = 0; } free(hits); + free(qv); } } } From 635453b936b5777bd53a861810904076c2ea5aae Mon Sep 17 00:00:00 2001 From: Neuron Date: Fri, 7 Aug 2026 14:59:39 -0500 Subject: [PATCH 03/15] feat(engram): rank-interleave the semantic leg into recall; embed the corpus MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the score-fusion first cut with rank fusion, which is what the data called for. nomic's cosine scale is compressed (true matches 0.55-0.70, unrelated pairs 0.35-0.50), so an additive blend of cosine onto token-coverage is dominated by whichever leg has the wider spread. Alternation is invariant to both scales: L1, S1, L2, S2, ... deduped, capped at limit Lexical ranking is left byte-identical; the semantic ranking is computed beside it and admitted only above ENGRAM_EMBED_SEED_MIN (0.60) — Will's existing seed floor, no new tuning constant. That floor is what keeps the nonsense controls clean: a query with no real match must not be answered with its neighbours. embed-corpus.py / merge-corpus.py produce the derived corpus the semantic leg needs (76,986 vectors, nomic-embed-text, 0 failures, 11 min). Zero of 78,791 nodes carried an embedding before this; the field round-tripped through the snapshot but nothing ever wrote it. MEASURED, 38-query gold set, paired against the SAME derived corpus so the comparison isolates the code change: hit@5 34.3% -> 51.4% paraphrase 0.0% -> 38.5% MRR@10 0.294 -> 0.387 superseded 1/3 -> 2/3 outranks recall@10 33.3% -> 50.5% latency p50 1146 -> 1220ms (1.06x) exact_rare 100% -> 100% phrase 85.7% -> 85.7% nonsense 2/3 -> 2/3 6 queries fixed, 0 broken, McNemar exact p=0.0312, 0 drift across repeats. Regression guards all held. Contrast PR #135, which swapped the read path to spreading activation wholesale: phrase 85.7 -> 28.6, latency 2.81x. Correct mechanism, wrong substrate. The substrate is now present. Restores engram claim 24 (previously 0% honoured). Co-Authored-By: Claude Opus 5 (1M context) --- .../comparison-main-vs-hybrid-semantic.json | 145 +++ tools/retrieval-eval/embed-corpus.py | 43 + tools/retrieval-eval/merge-corpus.py | 17 + .../results-baseline-embcorpus.json | 945 +++++++++++++++++ tools/retrieval-eval/results-hybrid-r2.json | 956 ++++++++++++++++++ tools/retrieval-eval/results-hybrid.json | 956 ++++++++++++++++++ .../el-runtime/v1.0.0-20260501/el_runtime.c | 141 ++- 7 files changed, 3155 insertions(+), 48 deletions(-) create mode 100644 tools/retrieval-eval/comparison-main-vs-hybrid-semantic.json create mode 100644 tools/retrieval-eval/embed-corpus.py create mode 100644 tools/retrieval-eval/merge-corpus.py create mode 100644 tools/retrieval-eval/results-baseline-embcorpus.json create mode 100644 tools/retrieval-eval/results-hybrid-r2.json create mode 100644 tools/retrieval-eval/results-hybrid.json diff --git a/tools/retrieval-eval/comparison-main-vs-hybrid-semantic.json b/tools/retrieval-eval/comparison-main-vs-hybrid-semantic.json new file mode 100644 index 0000000..6f13286 --- /dev/null +++ b/tools/retrieval-eval/comparison-main-vs-hybrid-semantic.json @@ -0,0 +1,145 @@ +{ + "baseline": "baseline-embcorpus", + "candidate": "hybrid-semantic", + "n_shared_queries": 38, + "fixed_by_candidate": [ + "q15", + "q16", + "q20", + "q21", + "q26", + "q37" + ], + "broken_by_candidate": [], + "discordant": 6, + "net_queries": 6, + "mcnemar_exact_p": 0.03125, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "candidate better", + "baseline_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.34285714285714286, + "recall@5": 0.26947278911564626, + "recall@10": 0.3333333333333333, + "precision@5": 0.12000000000000001, + "mrr@10": 0.2943197278911564, + "nonsense_clean": "2/3", + "superseded_outranks": "1/3", + "latency_ms_p50": 1145.9, + "latency_ms_p95": 1574.3, + "latency_ms_max": 1634.2, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4902210884353741, + "recall@10": 0.6666666666666666, + "mrr@10": 0.5965986394557822 + }, + "superseded": { + "n": 3, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.3333333333333333, + "mrr@10": 0.041666666666666664, + "outranks": 1 + } + } + }, + "candidate_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.5142857142857142, + "recall@5": 0.4409013605442177, + "recall@10": 0.5047619047619047, + "precision@5": 0.15428571428571433, + "mrr@10": 0.38746031746031745, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1219.7, + "latency_ms_p95": 1667.1, + "latency_ms_max": 1720.2, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.38461538461538464, + "recall@5": 0.38461538461538464, + "recall@10": 0.38461538461538464, + "mrr@10": 0.17307692307692307 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4902210884353741, + "recall@10": 0.6666666666666666, + "mrr@10": 0.6634920634920636 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.2222222222222222, + "outranks": 2 + } + } + }, + "repeat_variance": { + "candidate": { + "runs": 2, + "hit@5_min": 0.5142857142857142, + "hit@5_max": 0.5142857142857142, + "spread_queries": 0 + } + } +} \ No newline at end of file diff --git a/tools/retrieval-eval/embed-corpus.py b/tools/retrieval-eval/embed-corpus.py new file mode 100644 index 0000000..6dbef64 --- /dev/null +++ b/tools/retrieval-eval/embed-corpus.py @@ -0,0 +1,43 @@ +import json,sys,time,urllib.request,threading,queue +SRC="/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json" +OUT=sys.argv[1] +URL="http://127.0.0.1:11434/api/embeddings"; MODEL="nomic-embed-text" +MAXB=2000 # ENGRAM_EMBED_MAX_CHARS, applied to bytes as the C code does +d=json.load(open(SRC,encoding='utf-8',errors='surrogateescape')) +tasks=[] +for n in d["nodes"]: + c=n.get("content") or ""; t=n.get("node_type") or "" + if len(c)<8: continue # eg_embed_eligible + if t in ("InternalStateEvent","Tag"): continue + b=c.encode('utf-8',errors='surrogateescape')[:MAXB] + tasks.append((n.get("id") or "", b.decode('utf-8',errors='replace'))) +del d +print("tasks",len(tasks),flush=True) +q=queue.Queue(); [q.put(t) for t in tasks] +lock=threading.Lock(); f=open(OUT,"w",encoding="utf-8",errors="surrogateescape"); done=[0]; t0=time.time(); fails=[0] +def work(): + while True: + try: nid,txt=q.get_nowait() + except queue.Empty: return + v=None + for attempt in range(3): + try: + body=json.dumps({"model":MODEL,"prompt":txt}).encode() + r=urllib.request.Request(URL,data=body,headers={"Content-Type":"application/json"}) + with urllib.request.urlopen(r,timeout=120) as fh: v=json.load(fh)["embedding"] + break + except Exception as e: + if attempt==2: + with lock: fails[0]+=1 + time.sleep(0.5) + with lock: + if v: f.write(nid+"\t"+",".join("%.5g"%x for x in v)+"\n") + done[0]+=1 + if done[0]%2000==0: + el=time.time()-t0 + print("%d/%d %.1f/s eta %.1fmin fails=%d"%(done[0],len(tasks),done[0]/el,(len(tasks)-done[0])/(done[0]/el)/60,fails[0]),flush=True) + f.flush() +ths=[threading.Thread(target=work) for _ in range(8)] +[t.start() for t in ths]; [t.join() for t in ths] +f.close() +print("DONE",done[0],"fails",fails[0],"secs %.1f"%(time.time()-t0),flush=True) diff --git a/tools/retrieval-eval/merge-corpus.py b/tools/retrieval-eval/merge-corpus.py new file mode 100644 index 0000000..f19bbfa --- /dev/null +++ b/tools/retrieval-eval/merge-corpus.py @@ -0,0 +1,17 @@ +import json,sys +SRC="/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json" +TSV,OUT=sys.argv[1],sys.argv[2] +emb={} +for line in open(TSV,encoding='utf-8',errors='surrogateescape'): + p=line.rstrip("\n").rsplit("\t",1) + if len(p)==2 and p[1].count(",")>100: emb[p[0]]=p[1] +print("vectors",len(emb),flush=True) +d=json.load(open(SRC,encoding='utf-8',errors='surrogateescape')) +hit=0 +for n in d["nodes"]: + v=emb.get(n.get("id") or "") + if v: n["emb"]=v; hit+=1 +print("attached",hit,"of",len(d["nodes"]),flush=True) +with open(OUT,"w",encoding='utf-8',errors='surrogateescape') as f: + json.dump(d,f,ensure_ascii=False) +print("wrote",OUT,flush=True) diff --git a/tools/retrieval-eval/results-baseline-embcorpus.json b/tools/retrieval-eval/results-baseline-embcorpus.json new file mode 100644 index 0000000..e585152 --- /dev/null +++ b/tools/retrieval-eval/results-baseline-embcorpus.json @@ -0,0 +1,945 @@ +{ + "label": "baseline-embcorpus", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-baseline", + "soul_md5": "5cc9521734907cf2da30f0af94498c06", + "corpus": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/corpus-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json", + "limit": 10, + "port": 7893, + "wall_clock_s": 48.4, + "child_pid": 85995, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.34285714285714286, + "recall@5": 0.26947278911564626, + "recall@10": 0.3333333333333333, + "precision@5": 0.12000000000000001, + "mrr@10": 0.2943197278911564, + "nonsense_clean": "2/3", + "superseded_outranks": "1/3", + "latency_ms_p50": 1145.9, + "latency_ms_p95": 1574.3, + "latency_ms_max": 1634.2, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4902210884353741, + "recall@10": 0.6666666666666666, + "mrr@10": 0.5965986394557822 + }, + "superseded": { + "n": 3, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.3333333333333333, + "mrr@10": 0.041666666666666664, + "outranks": 1 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "n_returned": 1, + "latency_ms": 229.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5" + ], + "n_returned": 1, + "latency_ms": 262.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "n_returned": 1, + "latency_ms": 230.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "n_returned": 1, + "latency_ms": 229.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848" + ], + "n_returned": 1, + "latency_ms": 250.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "n_returned": 1, + "latency_ms": 227.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "?ǚ?7??????", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "????7???Ջ3", + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b", + "7?e?7???\f3?" + ], + "n_returned": 10, + "latency_ms": 532.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "Z[? 28.6%). Semantics reorders WITHIN and BELOW the - * full-lexical-match band, and admits high-cosine nodes the lexical pass - * missed entirely. */ -#define ENGRAM_SEM_WEIGHT 0.90 +/* The read-path semantic leg (engram claim 24) reuses ENGRAM_EMBED_SEED_MIN + * above as its admission floor: a node joins the embedding ranking only if its + * query cosine clears the same bar that lets it join the seed set. No new + * tuning constant is introduced, and the floor is load-bearing rather than + * cosmetic — it is what keeps a query with no real match (the gold set's + * nonsense controls) from being answered with its nearest neighbours. */ #define ENGRAM_EMBED_MAX_CHARS 2000 #define ENGRAM_EMBED_TIMEOUT_MS 4000L #define ENGRAM_EMBED_BREAKER_LIMIT 3 @@ -7411,29 +7402,67 @@ static double eg_sem_term(const EngramNode* n, const float* qv, int32_t qdim) { return t > 1.0 ? 1.0 : t; } -/* Fused rank score: distinct-token coverage in [0,1] plus at most - * ENGRAM_SEM_WEIGHT of semantic similarity. With no query embedding this is - * sc/ntok, a strictly monotone map of the old integer score, so the ordering - * is bit-identical to the pre-semantic ranking. */ -static double eg_fused_score(int sc, int ntok, double sem) { - double f = (double)sc / (double)(ntok > 0 ? ntok : 1); - return f + ENGRAM_SEM_WEIGHT * sem; -} - -/* Rank entry: fused lexical+semantic score (primary, desc) then salience - * (tiebreak, desc). `score` retains the raw distinct-token count for callers - * that want the lexical signal on its own. */ -typedef struct { int64_t idx; int score; double fused; double salience; } EngramRankEntry; +/* Rank entry: distinct-token match count (primary, desc) then salience + * (tiebreak, desc). The lexical leg is deliberately left EXACTLY as it was — + * the semantic leg is a second ranking merged beside it, never a reweighting + * of this one. */ +typedef struct { int64_t idx; int score; double salience; } EngramRankEntry; static int engram_rank_cmp(const void* a, const void* b) { const EngramRankEntry* ea = (const EngramRankEntry*)a; const EngramRankEntry* eb = (const EngramRankEntry*)b; - if (ea->fused < eb->fused) return 1; /* desc */ - if (ea->fused > eb->fused) return -1; + if (ea->score != eb->score) return eb->score - ea->score; /* desc */ if (ea->salience < eb->salience) return 1; if (ea->salience > eb->salience) return -1; return 0; } +/* Semantic rank entry: node index and its renormalized query similarity, + * ordered by similarity desc. This is the claim-24 "embedding search" + * ranking, computed independently of the lexical one. */ +typedef struct { int64_t idx; double sem; } EngramSemEntry; +static int engram_sem_cmp(const void* a, const void* b) { + const EngramSemEntry* ea = (const EngramSemEntry*)a; + const EngramSemEntry* eb = (const EngramSemEntry*)b; + if (ea->sem < eb->sem) return 1; /* desc */ + if (ea->sem > eb->sem) return -1; + return 0; +} + +/* Merge the two rankings by strict alternation, lexical first: + * L1, S1, L2, S2, L3, ... deduplicated by node index, capped at lim. + * + * Rank fusion, not score fusion. nomic's cosine scale is compressed (real + * matches land ~0.55-0.70 while unrelated pairs sit ~0.35-0.50), so any + * additive blend of a cosine onto a token-coverage score is dominated by + * whichever leg happens to have the wider spread. Alternation is invariant to + * both scales: it asks each leg for its next best answer in turn. + * + * Position 1 is always the top lexical hit, so a query whose answer the + * lexical leg already ranks first cannot be displaced — exact-token retrieval + * is structurally safe. The cost is bounded and explicit: a lexical hit at + * rank r lands at output position 2r-1. */ +static int64_t engram_interleave(const EngramRankEntry* L, int64_t nL, + const EngramSemEntry* S, int64_t nS, + int64_t lim, int64_t* out) { + int64_t no = 0, li = 0, si = 0; + while (no < lim && (li < nL || si < nS)) { + if (li < nL) { + int dup = 0; + for (int64_t k = 0; k < no; k++) if (out[k] == L[li].idx) { dup = 1; break; } + if (!dup) out[no++] = L[li].idx; + li++; + } + if (no >= lim) break; + if (si < nS) { + int dup = 0; + for (int64_t k = 0; k < no; k++) if (out[k] == S[si].idx) { dup = 1; break; } + if (!dup) out[no++] = S[si].idx; + si++; + } + } + return no; +} + el_val_t engram_search(el_val_t query, el_val_t limit) { EngramStore* g = engram_get(); const char* q = EL_CSTR(query); @@ -7450,6 +7479,8 @@ el_val_t engram_search(el_val_t query, el_val_t limit) { * NULL (embedder down / circuit breaker open) => pure lexical, as before. */ int32_t qdim = 0; float* qv = eg_embed_fetch(q, &qdim); + EngramSemEntry* sem = qv ? malloc((size_t)g->node_count * sizeof(EngramSemEntry)) : NULL; + int64_t nsem = 0; int64_t nhits = 0; for (int64_t i = 0; i < g->node_count; i++) { EngramNode* n = &g->nodes[i]; @@ -7459,25 +7490,29 @@ el_val_t engram_search(el_val_t query, el_val_t limit) { * + engram_compile_layered_json — that's the legitimate path. */ if (engram_layer_is_transparent(n->layer_id)) continue; int sc = engram_node_match_score(n, toks, ntok); - double sem = eg_sem_term(n, qv, qdim); - /* Union, not replacement: a node enters the candidate set on EITHER - * leg. sc == 0 && sem > 0 is the embedding-search half of claim 24 — - * nodes the lexical pass cannot see at all. */ - if (sc > 0 || sem > 0.0) { + if (sc > 0) { hits[nhits].idx = i; hits[nhits].score = sc; - hits[nhits].fused = eg_fused_score(sc, ntok, sem); hits[nhits].salience = n->salience; nhits++; } + if (sem) { + double sv = eg_sem_term(n, qv, qdim); + if (sv > 0.0) { sem[nsem].idx = i; sem[nsem].sem = sv; nsem++; } + } } - /* Rank by fused score (desc) then salience (desc), then cap. */ + /* Rank each leg independently, then alternate between them. */ qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp); - int64_t end = nhits < lim ? nhits : lim; - for (int64_t k = 0; k < end; k++) { - lst = el_list_append(lst, engram_node_to_map(&g->nodes[hits[k].idx])); + if (sem) qsort(sem, (size_t)nsem, sizeof(EngramSemEntry), engram_sem_cmp); + int64_t* order = malloc((size_t)lim * sizeof(int64_t)); + if (order) { + int64_t no = engram_interleave(hits, nhits, sem, nsem, lim, order); + for (int64_t k = 0; k < no; k++) + lst = el_list_append(lst, engram_node_to_map(&g->nodes[order[k]])); + free(order); } free(hits); + free(sem); free(qv); return lst; } @@ -9316,29 +9351,39 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { * to be observable to the MCP wrapper and the app. */ int32_t qdim = 0; float* qv = eg_embed_fetch(q, &qdim); + EngramSemEntry* sem = qv ? malloc((size_t)g->node_count * sizeof(EngramSemEntry)) : NULL; + int64_t nsem = 0; int64_t nhits = 0; for (int64_t i = 0; i < g->node_count; i++) { EngramNode* n = &g->nodes[i]; /* Filter transparent layers — same as engram_search. */ if (engram_layer_is_transparent(n->layer_id)) continue; int sc = engram_node_match_score(n, toks, ntok); - double sem = eg_sem_term(n, qv, qdim); - if (sc > 0 || sem > 0.0) { + if (sc > 0) { hits[nhits].idx = i; hits[nhits].score = sc; - hits[nhits].fused = eg_fused_score(sc, ntok, sem); hits[nhits].salience = n->salience; nhits++; } + if (sem) { + double sv = eg_sem_term(n, qv, qdim); + if (sv > 0.0) { sem[nsem].idx = i; sem[nsem].sem = sv; nsem++; } + } } qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp); - int64_t end = nhits < lim ? nhits : lim; - for (int64_t k = 0; k < end; k++) { - if (!first) jb_putc(&b, ','); - engram_emit_node_json(&b, &g->nodes[hits[k].idx], 0); - first = 0; + if (sem) qsort(sem, (size_t)nsem, sizeof(EngramSemEntry), engram_sem_cmp); + int64_t* order = malloc((size_t)lim * sizeof(int64_t)); + if (order) { + int64_t no = engram_interleave(hits, nhits, sem, nsem, lim, order); + for (int64_t k = 0; k < no; k++) { + if (!first) jb_putc(&b, ','); + engram_emit_node_json(&b, &g->nodes[order[k]], 0); + first = 0; + } + free(order); } free(hits); + free(sem); free(qv); } } From 059ce02003d170b74af751821daf6133f36d58ac Mon Sep 17 00:00:00 2001 From: Tim Lingo <1timlingo@gmail.com> Date: Fri, 7 Aug 2026 15:17:27 -0500 Subject: [PATCH 04/15] feat(engram): an associative leg on the recall path (claim 10 typed relations) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The recall route had no way to reach a node that shares no token and no embedding neighbourhood with the query. The design reserves that case for the graph, and nothing on the read path consulted an edge. This adds a third ranked leg beside the lexical and semantic ones: expand the top 3 lexical hits along STRUCTURAL relations only (claim 10 — identity, contains, superseded_by, references, ...), two hops, both directions, pruned at the same 0.02 firing threshold engram_activate uses; order what was reached by query similarity. Merged by strict rotation, never by score blending. Not PR #135. That wired recall wholesale to engram_activate and lost 57 points of phrase accuracy. The failure there was RANK, not reach — a 2-hop associate at strength 0.06 cannot outrank thousands of 1-hop neighbours of strong lexical seeds. Here the lexical leg is untouched and the associative list is empty for most queries, because a node whose only edges are `tagged` and `related` expands to nothing. MEASURED, hybrid-semantic baseline -> this, 38-query gold set, embedded corpus: associative 0.0% -> 66.7% (first non-zero ever recorded on that category) hit@5 51.4% -> 62.9% exact_rare, phrase, paraphrase, nonsense, superseded: all unchanged latency p50 1.01x 4 queries moved, all gains, 0 losses, McNemar p=0.125 deterministic: two runs of the same binary differ on 0 of 38 rows VERDICT: NOT-SHOWN. The harness needs 6 queries to clear p<0.05 and the whole associative category is only 6 queries, so even 4/6 fixed cannot reach the floor. The mechanism is confirmed to work; the gold set cannot certify it. --- .../comparison-hybrid-vs-assoc.json | 143 +++ .../retrieval-eval/results-assoc-leg-r2.json | 956 ++++++++++++++++++ tools/retrieval-eval/results-assoc-leg.json | 956 ++++++++++++++++++ .../el-runtime/v1.0.0-20260501/el_runtime.c | 183 +++- 4 files changed, 2237 insertions(+), 1 deletion(-) create mode 100644 tools/retrieval-eval/comparison-hybrid-vs-assoc.json create mode 100644 tools/retrieval-eval/results-assoc-leg-r2.json create mode 100644 tools/retrieval-eval/results-assoc-leg.json diff --git a/tools/retrieval-eval/comparison-hybrid-vs-assoc.json b/tools/retrieval-eval/comparison-hybrid-vs-assoc.json new file mode 100644 index 0000000..210e59e --- /dev/null +++ b/tools/retrieval-eval/comparison-hybrid-vs-assoc.json @@ -0,0 +1,143 @@ +{ + "baseline": "hybrid-semantic", + "candidate": "assoc-leg", + "n_shared_queries": 38, + "fixed_by_candidate": [ + "q27", + "q28", + "q29", + "q31" + ], + "broken_by_candidate": [], + "discordant": 4, + "net_queries": 4, + "mcnemar_exact_p": 0.125, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "no measurable difference", + "baseline_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.5142857142857142, + "recall@5": 0.4409013605442177, + "recall@10": 0.5047619047619047, + "precision@5": 0.15428571428571433, + "mrr@10": 0.38746031746031745, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1219.7, + "latency_ms_p95": 1667.1, + "latency_ms_max": 1720.2, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.38461538461538464, + "recall@5": 0.38461538461538464, + "recall@10": 0.38461538461538464, + "mrr@10": 0.17307692307692307 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4902210884353741, + "recall@10": 0.6666666666666666, + "mrr@10": 0.6634920634920636 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.2222222222222222, + "outranks": 2 + } + } + }, + "candidate_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.6285714285714286, + "recall@5": 0.45309194773480493, + "recall@10": 0.5405733155733157, + "precision@5": 0.17714285714285719, + "mrr@10": 0.42650793650793645, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1228.5, + "latency_ms_p95": 1681.8, + "latency_ms_max": 1718.6, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.07342657342657342, + "recall@10": 0.24825174825174826, + "mrr@10": 0.22777777777777777 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.38461538461538464, + "recall@5": 0.38461538461538464, + "recall@10": 0.38461538461538464, + "mrr@10": 0.17307692307692307 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4882369614512472, + "recall@10": 0.6329365079365079, + "mrr@10": 0.6634920634920636 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.2222222222222222, + "outranks": 2 + } + } + }, + "repeat_variance": { + "baseline": { + "runs": 2, + "hit@5_min": 0.5142857142857142, + "hit@5_max": 0.5142857142857142, + "spread_queries": 0 + } + } +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-assoc-leg-r2.json b/tools/retrieval-eval/results-assoc-leg-r2.json new file mode 100644 index 0000000..3c134a6 --- /dev/null +++ b/tools/retrieval-eval/results-assoc-leg-r2.json @@ -0,0 +1,956 @@ +{ + "label": "assoc-leg-r2", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-assoc", + "soul_md5": "ab9d490ecdfb1f9e6f23cca841ad8fb5", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json", + "limit": 10, + "port": 7894, + "wall_clock_s": 51.8, + "child_pid": 87150, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.6285714285714286, + "recall@5": 0.45309194773480493, + "recall@10": 0.5405733155733157, + "precision@5": 0.17714285714285719, + "mrr@10": 0.42650793650793645, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1223.5, + "latency_ms_p95": 1676.1, + "latency_ms_max": 1740.6, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.07342657342657342, + "recall@10": 0.24825174825174826, + "mrr@10": 0.22777777777777777 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.38461538461538464, + "recall@5": 0.38461538461538464, + "recall@10": 0.38461538461538464, + "mrr@10": 0.17307692307692307 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4882369614512472, + "recall@10": 0.6329365079365079, + "mrr@10": 0.6634920634920636 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.2222222222222222, + "outranks": 2 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "n_returned": 1, + "latency_ms": 306.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5", + "mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257", + "project-engram", + "project-engram-lang", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "ctx-89a2", + "bl-13babd0c-582e-4e28-a9e4-a77e65925e5d", + "870ede67-3454-4e00-9988-46cb13a8a4e2", + "bl-3e433255-3710-49fc-a093-c25e71de2ccb", + "mem-235a7657-d49e-467e-9f69-f4c3d5f6bd48" + ], + "n_returned": 10, + "latency_ms": 331.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "n_returned": 1, + "latency_ms": 292.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "n_returned": 1, + "latency_ms": 309.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848", + "bl-c1765767-3e27-449a-8c94-10411d1eb7c0", + "project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_" + ], + "n_returned": 3, + "latency_ms": 331.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "n_returned": 1, + "latency_ms": 295.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "????7???Ջ3", + "kn-363f4976-6946-4b4d-b51b-8a2b0f5aef25", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "ctx-63e3", + "?ǚ?7??????", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b" + ], + "n_returned": 10, + "latency_ms": 613.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6" + ], + "n_returned": 10, + "latency_ms": 538.1, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1875, + "recall@10": 0.375, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "? ?}&?#??X\b", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "? ?}&?#??X\b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "? ?}&?#??X\b", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee" + ], + "n_returned": 10, + "latency_ms": 553.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.4444444444444444, + "recall@10": 0.4444444444444444, + "precision@5": 0.8, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "ԍ????X????", + "project-harmonic-framework", + "?ǚ?7??????", + "project-harmonic-framework_com", + "knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a", + "??????X??2c", + "bl-dcee1887-34c4-4ffa-9119-1e291685ba08", + "????7???Ջ3", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356" + ], + "n_returned": 10, + "latency_ms": 527.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.1111111111111111, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "? ?}&?#??X\b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35", + "mem-b8ecd23e-77ce-42f7-984c-f51453fec16d" + ], + "n_returned": 10, + "latency_ms": 546.0, + "error": null, + "hit@5": 1.0, + "recall@5": 0.5, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "7?e?7???\f3?", + "?of?7???", + "? ?}&?#??X\b", + "? ?}&?#??X\b" + ], + "n_returned": 9, + "latency_ms": 892.4, + "error": null, + "hit@5": 1.0, + "recall@5": 0.2857142857142857, + "recall@10": 0.5, + "precision@5": 0.8, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "8f3abb0d-77ed-4af3-9f4d-ba62cd198886", + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a", + "?", + "mem-dba009a2-d2ea-4f5a-b9e8-0f04bc9ab32f", + "?", + "mem-7cd90611-88a3-423d-a38a-0db2812952fa", + "?", + "bl-07375bf9-a169-42cd-adb3-7d32b25982f0", + "?" + ], + "n_returned": 10, + "latency_ms": 754.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "????7???Ջ3", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c" + ], + "n_returned": 10, + "latency_ms": 1644.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "kn-a31e1001-342e-4deb-a2e6-6d02d1f22dee", + "knw-9e74ee95-ba7d-49b1-9262-977eae9729d1", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 1740.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "7?e?7???\f3?", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "?of?7???", + "? ?}&?#??X\b", + "????7???Ջ3", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0", + "knw-9e74ee95-ba7d-49b1-9262-977eae9729d1", + "kn-e8423822-eacf-4029-aa7b-10d4d28d621e" + ], + "n_returned": 10, + "latency_ms": 1393.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491", + "7?e?7???\f3?", + "??o?'?B???k", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-e8423822-eacf-4029-aa7b-10d4d28d621e", + "? ?}&?#??X\b", + "?of?7???", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 1118.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "7?e?7???\f3?", + "ctx-cc7f", + "?of?7???", + "ctx-4a41", + "imp-dce1da0f-8776-4a9e-972b-33411a7ca138", + "a1000001-0000-0000-0000-000000000001", + "knw-729fc901-8335-44c4-9f3a-b150b4aa0915", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "ctx-175f" + ], + "n_returned": 9, + "latency_ms": 1440.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-23c27d3b-e0d2-43a8-a80c-0a44477ae18a", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "8cbb60c5-4999-4ec1-8682-2592aedc4249", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4" + ], + "n_returned": 10, + "latency_ms": 1316.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "? ?}&?#??X\b", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "? ?}&?#??X\b", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "? ?}&?#??X\b", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "7?e?7???\f3?" + ], + "n_returned": 10, + "latency_ms": 1620.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "? ?}&?#??X\b", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "? ?}&?#??X\b", + "project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "ԍ????X????", + "??????X??2c", + "dR????X?-?S" + ], + "n_returned": 10, + "latency_ms": 1393.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "7?e?7???\f3?", + "knw-9e74ee95-ba7d-49b1-9262-977eae9729d1", + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "?of?7???", + "dR????X?-?S", + "dz????Xƹ?i", + "ԍ????X????", + "??????X??2c", + "kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491" + ], + "n_returned": 10, + "latency_ms": 1676.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "????7???Ջ3", + "a1000001-0000-0000-0000-000000000001", + "?ǚ?7??????", + "mem-024598a9-ed2e-4eeb-b1e1-5410856ff132", + "7?e?7???\f3?", + "mem-ade9440f-f161-4c18-9b35-1976257e6ebb", + "?of?7???", + "ea95f600-8dfd-4c7e-b077-a93dc3cd3623" + ], + "n_returned": 10, + "latency_ms": 1534.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "mem-bbb126a1-b297-42bb-86be-796871829c94", + "mem-45022957-2d78-48aa-a714-16d6eca52e0f", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "knw-35940684-abc4-42f0-b942-818f66b1f69a", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "7?e?7???\f3?", + "? ?}&?#??X\b", + "'?T?a\"B~-?8" + ], + "n_returned": 10, + "latency_ms": 1567.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "?ǚ?7??????", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "7?e?7???\f3?", + "ctx-bb74", + "??S?7???", + "knw-f671966c-3387-4848-abca-b5deec122e00" + ], + "n_returned": 10, + "latency_ms": 1280.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "? ?}&?#??X\b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "? ?}&?#??X\b", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "? ?}&?#??X\b", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "7?e?7???\f3?", + "mem-7b74cac0-905f-4c35-9688-fbcce105a177" + ], + "n_returned": 10, + "latency_ms": 1401.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "? ?}&?#??X\b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "? ?}&?#??X\b", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "? ?}&?#??X\b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "? ?}&?#??X\b", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc" + ], + "n_returned": 10, + "latency_ms": 1512.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.36363636363636365, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "54608b69-78b6-4239-b60f-b8206cfecacc", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "art-e0bdf5d8-d163-491f-b649-453fee8b721d", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71" + ], + "n_returned": 10, + "latency_ms": 1165.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.07692307692307693, + "recall@10": 0.3076923076923077, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "? ?}&?#??X\b", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "? ?}&?#??X\b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "? ?}&?#??X\b", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40" + ], + "n_returned": 10, + "latency_ms": 1230.6, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.36363636363636365, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43", + "ctx-e5427d7d", + "kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9", + "ctx-bb74", + "8cbb60c5-4999-4ec1-8682-2592aedc4249", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "kn-5adecd7e-d6db-4576-87fe-6ef8a935cea6", + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98" + ], + "n_returned": 10, + "latency_ms": 1223.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-9e74ee95-ba7d-49b1-9262-977eae9729d1", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b" + ], + "n_returned": 9, + "latency_ms": 1235.8, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.18181818181818182, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "? ?}&?#??X\b", + "4f698ae6-c40e-464e-9798-50350991a188", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "? ?}&?#??X\b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "? ?}&?#??X\b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "? ?}&?#??X\b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71" + ], + "n_returned": 10, + "latency_ms": 1460.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.2727272727272727, + "precision@5": 0.0, + "mrr@10": 0.16666666666666666 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 750.0, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 531.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [ + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "?V?", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "?m?\\}Q??6??", + "8cbb60c5-4999-4ec1-8682-2592aedc4249", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "?m?\\}Q??6??", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 769.1, + "error": null, + "clean": false, + "false_positives": 10 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "7?e?7???\f3?", + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "? ?}&?#??X\b", + "mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da", + "? ?}&?#??X\b", + "kn-b7e98d63-8b83-4911-b4d0-990602a7f575", + "?of?7???", + "knw-e047bb42-dc5b-4383-9e88-e508dc03abe3", + "? ?}&?#??X\b", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff" + ], + "n_returned": 10, + "latency_ms": 1327.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5, + "outranks": true, + "rank_correct": 2, + "rank_stale": 10 + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "? ?}&?#??X\b", + "12082f7e-e320-438b-bd65-083d8259748f", + "? ?}&?#??X\b", + "527ecb25-2587-47eb-8269-73be2431abd4", + "? ?}&?#??X\b", + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "? ?}&?#??X\b", + "4f698ae6-c40e-464e-9798-50350991a188", + "? ?}&?#??X\b", + "be3b6036-6eca-44a7-8fdf-37b23edfdfd1" + ], + "n_returned": 10, + "latency_ms": 1687.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.16666666666666666, + "outranks": true, + "rank_correct": 6, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "bl-7328cbe3-0200-43c2-88e7-0a164e15fca4", + "7?e?7???\f3?", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "? ?}&?#??X\b", + "4509ed62-9fb2-48b8-9038-ac569fca9604", + "%???2??jH??", + "art-8a0870d5-a716-4672-8094-f7463af1265b", + "???Ͼd??W\b?", + "bl-556438af-57b2-4bd8-a747-9f868aaee290" + ], + "n_returned": 10, + "latency_ms": 1040.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": 4 + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-assoc-leg.json b/tools/retrieval-eval/results-assoc-leg.json new file mode 100644 index 0000000..5aec308 --- /dev/null +++ b/tools/retrieval-eval/results-assoc-leg.json @@ -0,0 +1,956 @@ +{ + "label": "assoc-leg", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-assoc", + "soul_md5": "ab9d490ecdfb1f9e6f23cca841ad8fb5", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json", + "limit": 10, + "port": 7893, + "wall_clock_s": 52.4, + "child_pid": 87099, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.6285714285714286, + "recall@5": 0.45309194773480493, + "recall@10": 0.5405733155733157, + "precision@5": 0.17714285714285719, + "mrr@10": 0.42650793650793645, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1228.5, + "latency_ms_p95": 1681.8, + "latency_ms_max": 1718.6, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.07342657342657342, + "recall@10": 0.24825174825174826, + "mrr@10": 0.22777777777777777 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.38461538461538464, + "recall@5": 0.38461538461538464, + "recall@10": 0.38461538461538464, + "mrr@10": 0.17307692307692307 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4882369614512472, + "recall@10": 0.6329365079365079, + "mrr@10": 0.6634920634920636 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.2222222222222222, + "outranks": 2 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "n_returned": 1, + "latency_ms": 306.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5", + "mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257", + "project-engram", + "project-engram-lang", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "ctx-89a2", + "bl-13babd0c-582e-4e28-a9e4-a77e65925e5d", + "870ede67-3454-4e00-9988-46cb13a8a4e2", + "bl-3e433255-3710-49fc-a093-c25e71de2ccb", + "mem-235a7657-d49e-467e-9f69-f4c3d5f6bd48" + ], + "n_returned": 10, + "latency_ms": 360.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "n_returned": 1, + "latency_ms": 324.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "n_returned": 1, + "latency_ms": 313.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848", + "bl-c1765767-3e27-449a-8c94-10411d1eb7c0", + "project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_" + ], + "n_returned": 3, + "latency_ms": 331.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "n_returned": 1, + "latency_ms": 303.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "????7???Ջ3", + "kn-363f4976-6946-4b4d-b51b-8a2b0f5aef25", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "ctx-63e3", + "?ǚ?7??????", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b" + ], + "n_returned": 10, + "latency_ms": 588.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6" + ], + "n_returned": 10, + "latency_ms": 533.3, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1875, + "recall@10": 0.375, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "? ?}&?#??X\b", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "? ?}&?#??X\b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "? ?}&?#??X\b", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee" + ], + "n_returned": 10, + "latency_ms": 542.4, + "error": null, + "hit@5": 1.0, + "recall@5": 0.4444444444444444, + "recall@10": 0.4444444444444444, + "precision@5": 0.8, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "ԍ????X????", + "project-harmonic-framework", + "?ǚ?7??????", + "project-harmonic-framework_com", + "knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a", + "??????X??2c", + "bl-dcee1887-34c4-4ffa-9119-1e291685ba08", + "????7???Ջ3", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356" + ], + "n_returned": 10, + "latency_ms": 517.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.1111111111111111, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "? ?}&?#??X\b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35", + "mem-b8ecd23e-77ce-42f7-984c-f51453fec16d" + ], + "n_returned": 10, + "latency_ms": 547.1, + "error": null, + "hit@5": 1.0, + "recall@5": 0.5, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "7?e?7???\f3?", + "?of?7???", + "? ?}&?#??X\b", + "? ?}&?#??X\b" + ], + "n_returned": 9, + "latency_ms": 903.4, + "error": null, + "hit@5": 1.0, + "recall@5": 0.2857142857142857, + "recall@10": 0.5, + "precision@5": 0.8, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "8f3abb0d-77ed-4af3-9f4d-ba62cd198886", + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a", + "?", + "mem-dba009a2-d2ea-4f5a-b9e8-0f04bc9ab32f", + "?", + "mem-7cd90611-88a3-423d-a38a-0db2812952fa", + "?", + "bl-07375bf9-a169-42cd-adb3-7d32b25982f0", + "?" + ], + "n_returned": 10, + "latency_ms": 774.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "????7???Ջ3", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c" + ], + "n_returned": 10, + "latency_ms": 1641.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "kn-a31e1001-342e-4deb-a2e6-6d02d1f22dee", + "knw-9e74ee95-ba7d-49b1-9262-977eae9729d1", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 1718.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "7?e?7???\f3?", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "?of?7???", + "? ?}&?#??X\b", + "????7???Ջ3", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0", + "knw-9e74ee95-ba7d-49b1-9262-977eae9729d1", + "kn-e8423822-eacf-4029-aa7b-10d4d28d621e" + ], + "n_returned": 10, + "latency_ms": 1409.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491", + "7?e?7???\f3?", + "??o?'?B???k", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-e8423822-eacf-4029-aa7b-10d4d28d621e", + "? ?}&?#??X\b", + "?of?7???", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 1135.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "7?e?7???\f3?", + "ctx-cc7f", + "?of?7???", + "ctx-4a41", + "imp-dce1da0f-8776-4a9e-972b-33411a7ca138", + "a1000001-0000-0000-0000-000000000001", + "knw-729fc901-8335-44c4-9f3a-b150b4aa0915", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "ctx-175f" + ], + "n_returned": 9, + "latency_ms": 1446.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-23c27d3b-e0d2-43a8-a80c-0a44477ae18a", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "8cbb60c5-4999-4ec1-8682-2592aedc4249", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4" + ], + "n_returned": 10, + "latency_ms": 1296.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "? ?}&?#??X\b", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "? ?}&?#??X\b", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "? ?}&?#??X\b", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "7?e?7???\f3?" + ], + "n_returned": 10, + "latency_ms": 1621.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "? ?}&?#??X\b", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "? ?}&?#??X\b", + "project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "ԍ????X????", + "??????X??2c", + "dR????X?-?S" + ], + "n_returned": 10, + "latency_ms": 1411.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "7?e?7???\f3?", + "knw-9e74ee95-ba7d-49b1-9262-977eae9729d1", + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "?of?7???", + "dR????X?-?S", + "dz????Xƹ?i", + "ԍ????X????", + "??????X??2c", + "kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491" + ], + "n_returned": 10, + "latency_ms": 1681.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "????7???Ջ3", + "a1000001-0000-0000-0000-000000000001", + "?ǚ?7??????", + "mem-024598a9-ed2e-4eeb-b1e1-5410856ff132", + "7?e?7???\f3?", + "mem-ade9440f-f161-4c18-9b35-1976257e6ebb", + "?of?7???", + "ea95f600-8dfd-4c7e-b077-a93dc3cd3623" + ], + "n_returned": 10, + "latency_ms": 1536.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "mem-bbb126a1-b297-42bb-86be-796871829c94", + "mem-45022957-2d78-48aa-a714-16d6eca52e0f", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "knw-35940684-abc4-42f0-b942-818f66b1f69a", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "7?e?7???\f3?", + "? ?}&?#??X\b", + "'?T?a\"B~-?8" + ], + "n_returned": 10, + "latency_ms": 1567.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "?ǚ?7??????", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "7?e?7???\f3?", + "ctx-bb74", + "??S?7???", + "knw-f671966c-3387-4848-abca-b5deec122e00" + ], + "n_returned": 10, + "latency_ms": 1278.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "? ?}&?#??X\b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "? ?}&?#??X\b", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "? ?}&?#??X\b", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "7?e?7???\f3?", + "mem-7b74cac0-905f-4c35-9688-fbcce105a177" + ], + "n_returned": 10, + "latency_ms": 1401.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "? ?}&?#??X\b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "? ?}&?#??X\b", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "? ?}&?#??X\b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "? ?}&?#??X\b", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc" + ], + "n_returned": 10, + "latency_ms": 1509.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.36363636363636365, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "54608b69-78b6-4239-b60f-b8206cfecacc", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "art-e0bdf5d8-d163-491f-b649-453fee8b721d", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71" + ], + "n_returned": 10, + "latency_ms": 1156.7, + "error": null, + "hit@5": 1.0, + "recall@5": 0.07692307692307693, + "recall@10": 0.3076923076923077, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "? ?}&?#??X\b", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "? ?}&?#??X\b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "? ?}&?#??X\b", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40" + ], + "n_returned": 10, + "latency_ms": 1232.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.36363636363636365, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43", + "ctx-e5427d7d", + "kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9", + "ctx-bb74", + "8cbb60c5-4999-4ec1-8682-2592aedc4249", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "kn-5adecd7e-d6db-4576-87fe-6ef8a935cea6", + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98" + ], + "n_returned": 10, + "latency_ms": 1228.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-9e74ee95-ba7d-49b1-9262-977eae9729d1", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b" + ], + "n_returned": 9, + "latency_ms": 1251.6, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.18181818181818182, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "? ?}&?#??X\b", + "4f698ae6-c40e-464e-9798-50350991a188", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "? ?}&?#??X\b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "? ?}&?#??X\b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "? ?}&?#??X\b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71" + ], + "n_returned": 10, + "latency_ms": 1459.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.2727272727272727, + "precision@5": 0.0, + "mrr@10": 0.16666666666666666 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 746.6, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 519.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [ + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "?V?", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "?m?\\}Q??6??", + "8cbb60c5-4999-4ec1-8682-2592aedc4249", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "?m?\\}Q??6??", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd" + ], + "n_returned": 10, + "latency_ms": 764.0, + "error": null, + "clean": false, + "false_positives": 10 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "7?e?7???\f3?", + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "? ?}&?#??X\b", + "mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da", + "? ?}&?#??X\b", + "kn-b7e98d63-8b83-4911-b4d0-990602a7f575", + "?of?7???", + "knw-e047bb42-dc5b-4383-9e88-e508dc03abe3", + "? ?}&?#??X\b", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff" + ], + "n_returned": 10, + "latency_ms": 1346.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5, + "outranks": true, + "rank_correct": 2, + "rank_stale": 10 + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "? ?}&?#??X\b", + "12082f7e-e320-438b-bd65-083d8259748f", + "? ?}&?#??X\b", + "527ecb25-2587-47eb-8269-73be2431abd4", + "? ?}&?#??X\b", + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "? ?}&?#??X\b", + "4f698ae6-c40e-464e-9798-50350991a188", + "? ?}&?#??X\b", + "be3b6036-6eca-44a7-8fdf-37b23edfdfd1" + ], + "n_returned": 10, + "latency_ms": 1687.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.16666666666666666, + "outranks": true, + "rank_correct": 6, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "bl-7328cbe3-0200-43c2-88e7-0a164e15fca4", + "7?e?7???\f3?", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "? ?}&?#??X\b", + "4509ed62-9fb2-48b8-9038-ac569fca9604", + "%???2??jH??", + "art-8a0870d5-a716-4672-8094-f7463af1265b", + "???Ͼd??W\b?", + "bl-556438af-57b2-4bd8-a747-9f868aaee290" + ], + "n_returned": 10, + "latency_ms": 1030.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": 4 + } + ] +} \ No newline at end of file diff --git a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c index 815021e..3fe3fad 100644 --- a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c +++ b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c @@ -7463,6 +7463,176 @@ static int64_t engram_interleave(const EngramRankEntry* L, int64_t nL, return no; } +/* ── Associative leg (claim 10 typed relations + claim 1 activation) ──────── + * + * WHY THIS EXISTS. engram_search_json has two legs, and neither can reach a + * node that shares no token with the query and no embedding neighbourhood + * with it. The route the design reserves for that case is the graph: a node + * is reachable because it is STRUCTURALLY associated with something the query + * did hit. Nothing on the recall path consults an edge today. + * + * WHY NOT engram_activate. Wiring recall wholesale to activation was measured + * (PR #135) and lost 57 points of phrase accuracy. The failure was one of + * RANK, not reach: activation seeds on every token-matching node, so a 2-hop + * associate at strength ~0.06 never outranks thousands of 1-hop neighbours of + * strong lexical seeds. So this is a separate, small, ranked list that is + * merged BESIDE the other two, exactly as the semantic leg is. + * + * TYPED RELATIONS (06-claims.md claim 10). Expansion follows only relations + * that assert a structural claim about meaning. The corpus is 4,915 `tagged` + * and 3,767 `triggers-safety` edges against 475 `identity` and 11 `contains`; + * walking the bulk relations turns any seed into a flood (measured: 1,387 + * candidates from one seed) while the structural ones stay in the low tens. + * This is the first code on the read path to branch on a relation type at all. + * + * DIRECTION. Edges are walked in BOTH directions. Claim 23 requires the store + * to keep reverse edge records keyed by target id, and the adjacency index + * already materialises them (adj_to). It has to be both: every value node in + * this corpus has exactly ONE inbound edge (hub -> value) and no outbound + * structural edge at all, so a forward-only walk from a value node reaches + * nothing. Note this is an extension of the traversal as literally specified + * (05-detailed-description l.66 says "all outgoing edges"); the reverse index + * is designed and stored, but the description does not say the walk reads it. + */ +#define ENGRAM_ASSOC_SEEDS 3 /* top-N lexical hits form the context */ +#define ENGRAM_ASSOC_DEPTH 2 /* seed -> hub -> sibling */ +#define ENGRAM_ASSOC_FIRE 0.02 /* same firing threshold as engram_activate */ +#define ENGRAM_ASSOC_MAX 64 /* cap on candidates carried forward */ + +static int eg_rel_is_structural(const char* r) { + if (!r || !*r) return 0; + static const char* ok[] = { + "identity", "contains", "superseded_by", "references", "embodies", + "demonstrated_by", "canonical-self", "depends_on", "currently_holds", + "activates", NULL + }; + for (int i = 0; ok[i]; i++) if (strcmp(r, ok[i]) == 0) return 1; + return 0; +} + +/* Nodes that are index artefacts rather than recallable content. Same + * exclusion eg_embed_eligible() already applies when deciding what deserves an + * embedding, reused here so the associative leg cannot surface — or relay + * through — a Tag. Relaying through them is what makes a graph walk explode: + * tag-tier_note alone has 187 members. */ +static int eg_assoc_excluded(const EngramNode* n) { + if (!n->node_type) return 0; + return strcmp(n->node_type, "Tag") == 0 + || strcmp(n->node_type, "InternalStateEvent") == 0; +} + +/* Breadth-first structural expansion from `seeds`, then ORDER BY query + * similarity. Activation decides REACHABILITY (the conjunctive prune of + * claim 1: parent strength x edge weight x target salience, cut at the firing + * threshold); cosine decides ORDER within what was reached. Ranking the + * neighbourhood by activation alone does not work and the reason is + * structural: every identity edge in this corpus carries weight 0.5 and every + * value node salience 0.7, so the activation product degenerates into a + * function of hop count and ranks the relay hub above all of its own + * children. Composing the two is mine, not Will's — the description ranks the + * activation result set by strength (l.78). */ +static int64_t engram_assoc_leg(EngramStore* g, + const EngramRankEntry* L, int64_t nL, + const float* qv, int32_t qdim, + EngramSemEntry* out, int64_t out_cap) { + if (!g || nL <= 0 || !qv || qdim <= 0 || out_cap <= 0) return 0; + if (g->adj_dirty || !g->adj_from || !g->adj_to) engram_adj_rebuild(g); + if (!g->adj_from || !g->adj_to) return 0; + + double* act = calloc((size_t)g->node_count, sizeof(double)); + char* seen = calloc((size_t)g->node_count, sizeof(char)); + int64_t* q = malloc((size_t)ENGRAM_ASSOC_MAX * 4 * sizeof(int64_t)); + int64_t* hop = malloc((size_t)ENGRAM_ASSOC_MAX * 4 * sizeof(int64_t)); + if (!act || !seen || !q || !hop) { free(act); free(seen); free(q); free(hop); return 0; } + int64_t qcap = ENGRAM_ASSOC_MAX * 4, qh = 0, qt = 0; + + int64_t nseed = nL < ENGRAM_ASSOC_SEEDS ? nL : ENGRAM_ASSOC_SEEDS; + for (int64_t s = 0; s < nseed; s++) { + int64_t idx = L[s].idx; + if (idx < 0 || idx >= g->node_count) continue; + act[idx] = 1.0; seen[idx] = 2; /* 2 = seed: never a result */ + if (qt < qcap) { q[qt] = idx; hop[qt] = 0; qt++; } + } + + const double SPREAD_DECAY = 0.7; + while (qh < qt) { + int64_t cur = q[qh]; int64_t h = hop[qh]; qh++; + if (h >= ENGRAM_ASSOC_DEPTH) continue; + double parent = act[cur]; + int from_len = g->adj_from_len[cur]; + int to_len = g->adj_to_len[cur]; + for (int scan = 0; scan < from_len + to_len; scan++) { + int64_t ei = (scan < from_len) ? g->adj_from[cur][scan] + : g->adj_to[cur][scan - from_len]; + EngramEdge* e = &g->edges[ei]; + if (!eg_rel_is_structural(e->relation)) continue; + int64_t oi = (scan < from_len) ? engram_idmap_get(g, e->to_id) + : engram_idmap_get(g, e->from_id); + if (oi < 0 || oi >= g->node_count) continue; + EngramNode* on = &g->nodes[oi]; + if (eg_assoc_excluded(on)) continue; + double na = parent * e->weight * SPREAD_DECAY * on->salience; + if (na < ENGRAM_ASSOC_FIRE) continue; + if (seen[oi] && na <= act[oi]) continue; + act[oi] = na; + if (!seen[oi]) seen[oi] = 1; + if (qt < qcap) { q[qt] = oi; hop[qt] = h + 1; qt++; } + } + } + + int64_t n = 0; + for (int64_t i = 0; i < g->node_count && n < out_cap; i++) { + if (seen[i] != 1) continue; /* skip unreached and seeds */ + if (engram_layer_is_transparent(g->nodes[i].layer_id)) continue; + EngramNode* nd = &g->nodes[i]; + if (!nd->emb || nd->emb_dim != qdim) continue; + double c = eg_cosine(nd->emb, qv, qdim); + if (c <= 0.0) continue; + out[n].idx = i; out[n].sem = c; n++; + } + qsort(out, (size_t)n, sizeof(EngramSemEntry), engram_sem_cmp); + free(act); free(seen); free(q); free(hop); + return n; +} + +/* Three-leg merge: lexical, semantic, associative — strict rotation, + * L1, S1, A1, L2, S2, A2, ... deduplicated, capped at lim. + * + * The cost is explicit and worse than the two-leg case: a lexical hit at rank + * r lands at output position 3r-2 when both other legs are non-empty. That is + * survivable here only because the structural-relation filter leaves the + * associative list EMPTY for most queries — a Memory node whose only edges are + * `tagged` and `related` expands to nothing, so its ranking is untouched. */ +static int64_t engram_interleave3(const EngramRankEntry* L, int64_t nL, + const EngramSemEntry* S, int64_t nS, + const EngramSemEntry* A, int64_t nA, + int64_t lim, int64_t* out) { + int64_t no = 0, li = 0, si = 0, ai = 0; + while (no < lim && (li < nL || si < nS || ai < nA)) { + if (li < nL) { + int dup = 0; + for (int64_t k = 0; k < no; k++) if (out[k] == L[li].idx) { dup = 1; break; } + if (!dup) out[no++] = L[li].idx; + li++; + } + if (no >= lim) break; + if (si < nS) { + int dup = 0; + for (int64_t k = 0; k < no; k++) if (out[k] == S[si].idx) { dup = 1; break; } + if (!dup) out[no++] = S[si].idx; + si++; + } + if (no >= lim) break; + if (ai < nA) { + int dup = 0; + for (int64_t k = 0; k < no; k++) if (out[k] == A[ai].idx) { dup = 1; break; } + if (!dup) out[no++] = A[ai].idx; + ai++; + } + } + return no; +} + el_val_t engram_search(el_val_t query, el_val_t limit) { EngramStore* g = engram_get(); const char* q = EL_CSTR(query); @@ -9372,9 +9542,19 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { } qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp); if (sem) qsort(sem, (size_t)nsem, sizeof(EngramSemEntry), engram_sem_cmp); + /* Claim-10 associative leg: expand the top lexical hits along + * structural relations only, order the reached set by query + * similarity. Empty whenever the seeds have no structural + * edges, which is the common case and is what keeps the + * lexical ordering safe. */ + EngramSemEntry* assoc = qv ? malloc((size_t)ENGRAM_ASSOC_MAX * sizeof(EngramSemEntry)) : NULL; + int64_t nassoc = assoc + ? engram_assoc_leg(g, hits, nhits, qv, qdim, assoc, ENGRAM_ASSOC_MAX) + : 0; int64_t* order = malloc((size_t)lim * sizeof(int64_t)); if (order) { - int64_t no = engram_interleave(hits, nhits, sem, nsem, lim, order); + int64_t no = engram_interleave3(hits, nhits, sem, nsem, + assoc, nassoc, lim, order); for (int64_t k = 0; k < no; k++) { if (!first) jb_putc(&b, ','); engram_emit_node_json(&b, &g->nodes[order[k]], 0); @@ -9382,6 +9562,7 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { } free(order); } + free(assoc); free(hits); free(sem); free(qv); From 6f3a048f3642e4c808a54b5040eed4f33cc2b985 Mon Sep 17 00:00:00 2001 From: Tim Lingo <1timlingo@gmail.com> Date: Fri, 7 Aug 2026 15:35:08 -0500 Subject: [PATCH 05/15] feat(engram): semantically seed the graph leg (Will's HippoRAG pass, SEED_K=8) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit engram_assoc_leg previously took its seeds only from the top-3 LEXICAL hits. For a paraphrase query the lexical hits are noise by construction, so the walk never reached the neighbourhood that holds the answer. This adds the seeding pass Will documents at el_runtime.c l.6082 — "Semantic seeding (HippoRAG pattern, use similarity twice): the query is embedded, the top-K nodes by cosine join the seed set" — using his own ENGRAM_EMBED_SEED_K (8). Similarity is now used twice, coherently: cosine picks where to STAND in the graph, the structural-relation walk decides what is REACHABLE, and cosine orders what was reached (iteration 2's finding, unchanged). The seed list is deliberately NOT floored at ENGRAM_EMBED_SEED_MIN. Measured over all 38 gold queries: true paraphrase targets score cosine 0.46-0.66 and the three nonsense controls' own nearest neighbours score 0.55/0.60/0.62 — the distributions OVERLAP, so no absolute cosine floor separates signal from gibberish. The gate that works is reachability: gibberish's nearest neighbours carry no structural edge, so its graph leg is empty and the controls hold. The raw top-K is selected inside the existing scoring pass, so the cosine is computed exactly once per node: no extra corpus pass, no extra embed round-trip, latency flat (p50 1220 -> 1227 ms, 1.01x). Measured vs the certified baseline feat/hybrid-semantic-recall, embedded corpus, 2 runs each, zero run-to-run drift on both sides: hit@5 51.4% -> 68.6% MRR@10 0.387 -> 0.461 paraphrase 38.5% -> 61.5% associative 0% -> 66.7% exact_rare 100% held, nonsense 2/3 held, superseded 2/3 held phrase 85.7% -> 71.4% (q11, the known rank-5 rotation tax) net +6 queries (7 fixed / 1 broken), McNemar p=0.0703 --- tools/retrieval-eval/bestval.py | 21 + .../comparison-assoc-vs-semseed.json | 150 +++ .../comparison-hybrid-vs-semseed.json | 154 +++ tools/retrieval-eval/fullsim.py | 123 +++ tools/retrieval-eval/results-semseed-r2.json | 957 ++++++++++++++++++ tools/retrieval-eval/results-semseed.json | 957 ++++++++++++++++++ tools/retrieval-eval/zprobe.py | 31 + .../el-runtime/v1.0.0-20260501/el_runtime.c | 66 +- 8 files changed, 2455 insertions(+), 4 deletions(-) create mode 100644 tools/retrieval-eval/bestval.py create mode 100644 tools/retrieval-eval/comparison-assoc-vs-semseed.json create mode 100644 tools/retrieval-eval/comparison-hybrid-vs-semseed.json create mode 100644 tools/retrieval-eval/fullsim.py create mode 100644 tools/retrieval-eval/results-semseed-r2.json create mode 100644 tools/retrieval-eval/results-semseed.json create mode 100644 tools/retrieval-eval/zprobe.py diff --git a/tools/retrieval-eval/bestval.py b/tools/retrieval-eval/bestval.py new file mode 100644 index 0000000..473c742 --- /dev/null +++ b/tools/retrieval-eval/bestval.py @@ -0,0 +1,21 @@ +import numpy as np, json, urllib.request +SP="/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad" +np.seterr(all='ignore') +M=np.load(SP+'/emb.npy'); eids=open(SP+'/ids.txt',encoding='utf-8',errors='surrogateescape').read().split('\n') +eidx={k:i for i,k in enumerate(eids)} +gold=json.load(open("/Users/timlingo/Development/neuron-technologies/_wt-assoc-leg/tools/retrieval-eval/gold_set.json"))['queries'] +VALS=sorted({r for q in gold if q['category']=='paraphrase' for r in q['relevant']}) +VI=[eidx[v] for v in VALS] +def emb(t): + b=json.dumps({"model":"nomic-embed-text","prompt":t}).encode() + r=urllib.request.Request("http://127.0.0.1:11434/api/embeddings",data=b,headers={"Content-Type":"application/json"}) + v=np.array(json.load(urllib.request.urlopen(r,timeout=60))["embedding"],dtype=np.float32) + return v/(np.linalg.norm(v)+1e-9) +print("qid cat bestValueNodeGlobalRank goldGlobalRank goldSiblingRank") +for q in gold: + if q['category']!='paraphrase': continue + v=emb(q['query']); s=M@v; s[~np.isfinite(s)]=-1 + ranks=sorted(int((s>s[j]).sum())+1 for j in VI) + g=eidx[q['relevant'][0]]; gr=int((s>s[g]).sum())+1 + sv=np.array([s[j] for j in VI]); sib=int((sv>s[g]).sum())+1 + print("%-4s %-11s best=%-5d (top3 val ranks %s) gold=%-5d sib=%d" % (q['id'],q['category'],ranks[0],ranks[:3],gr,sib)) diff --git a/tools/retrieval-eval/comparison-assoc-vs-semseed.json b/tools/retrieval-eval/comparison-assoc-vs-semseed.json new file mode 100644 index 0000000..7fe83ec --- /dev/null +++ b/tools/retrieval-eval/comparison-assoc-vs-semseed.json @@ -0,0 +1,150 @@ +{ + "baseline": "assoc-leg", + "candidate": "semseed", + "n_shared_queries": 38, + "fixed_by_candidate": [ + "q18", + "q19", + "q22" + ], + "broken_by_candidate": [ + "q11" + ], + "discordant": 4, + "net_queries": 2, + "mcnemar_exact_p": 0.625, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "no measurable difference", + "baseline_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.6285714285714286, + "recall@5": 0.45309194773480493, + "recall@10": 0.5405733155733157, + "precision@5": 0.17714285714285719, + "mrr@10": 0.42650793650793645, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1228.5, + "latency_ms_p95": 1681.8, + "latency_ms_max": 1718.6, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.07342657342657342, + "recall@10": 0.24825174825174826, + "mrr@10": 0.22777777777777777 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.38461538461538464, + "recall@5": 0.38461538461538464, + "recall@10": 0.38461538461538464, + "mrr@10": 0.17307692307692307 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4882369614512472, + "recall@10": 0.6329365079365079, + "mrr@10": 0.6634920634920636 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.2222222222222222, + "outranks": 2 + } + } + }, + "candidate_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.6857142857142857, + "recall@5": 0.5213459159887731, + "recall@10": 0.6027048348476919, + "precision@5": 0.18285714285714294, + "mrr@10": 0.4608730158730158, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1227.1, + "latency_ms_p95": 1692.6, + "latency_ms_max": 1710.4, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.07342657342657344, + "recall@10": 0.24825174825174823, + "mrr@10": 0.20833333333333334 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 0.7142857142857143, + "recall@5": 0.40093537414965985, + "recall@10": 0.5150226757369615, + "mrr@10": 0.6507936507936508 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "repeat_variance": { + "baseline": { + "runs": 2, + "hit@5_min": 0.6285714285714286, + "hit@5_max": 0.6285714285714286, + "spread_queries": 0 + }, + "candidate": { + "runs": 2, + "hit@5_min": 0.6857142857142857, + "hit@5_max": 0.6857142857142857, + "spread_queries": 0 + } + } +} \ No newline at end of file diff --git a/tools/retrieval-eval/comparison-hybrid-vs-semseed.json b/tools/retrieval-eval/comparison-hybrid-vs-semseed.json new file mode 100644 index 0000000..a49a008 --- /dev/null +++ b/tools/retrieval-eval/comparison-hybrid-vs-semseed.json @@ -0,0 +1,154 @@ +{ + "baseline": "hybrid-semantic", + "candidate": "semseed", + "n_shared_queries": 38, + "fixed_by_candidate": [ + "q18", + "q19", + "q22", + "q27", + "q28", + "q29", + "q31" + ], + "broken_by_candidate": [ + "q11" + ], + "discordant": 8, + "net_queries": 6, + "mcnemar_exact_p": 0.0703125, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "candidate better", + "baseline_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.5142857142857142, + "recall@5": 0.4409013605442177, + "recall@10": 0.5047619047619047, + "precision@5": 0.15428571428571433, + "mrr@10": 0.38746031746031745, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1219.7, + "latency_ms_p95": 1667.1, + "latency_ms_max": 1720.2, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.38461538461538464, + "recall@5": 0.38461538461538464, + "recall@10": 0.38461538461538464, + "mrr@10": 0.17307692307692307 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4902210884353741, + "recall@10": 0.6666666666666666, + "mrr@10": 0.6634920634920636 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.2222222222222222, + "outranks": 2 + } + } + }, + "candidate_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.6857142857142857, + "recall@5": 0.5213459159887731, + "recall@10": 0.6027048348476919, + "precision@5": 0.18285714285714294, + "mrr@10": 0.4608730158730158, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1227.1, + "latency_ms_p95": 1692.6, + "latency_ms_max": 1710.4, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.07342657342657344, + "recall@10": 0.24825174825174823, + "mrr@10": 0.20833333333333334 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 0.7142857142857143, + "recall@5": 0.40093537414965985, + "recall@10": 0.5150226757369615, + "mrr@10": 0.6507936507936508 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "repeat_variance": { + "baseline": { + "runs": 2, + "hit@5_min": 0.5142857142857142, + "hit@5_max": 0.5142857142857142, + "spread_queries": 0 + }, + "candidate": { + "runs": 2, + "hit@5_min": 0.6857142857142857, + "hit@5_max": 0.6857142857142857, + "spread_queries": 0 + } + } +} \ No newline at end of file diff --git a/tools/retrieval-eval/fullsim.py b/tools/retrieval-eval/fullsim.py new file mode 100644 index 0000000..82c3f50 --- /dev/null +++ b/tools/retrieval-eval/fullsim.py @@ -0,0 +1,123 @@ +import numpy as np, json, urllib.request, collections, sys +SP="/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad" +EV="/Users/timlingo/Development/neuron-technologies/_wt-assoc-leg/tools/retrieval-eval/" +np.seterr(all='ignore') +M=np.load(SP+'/emb.npy'); eids=open(SP+'/ids.txt',encoding='utf-8',errors='surrogateescape').read().split('\n') +eidx={k:i for i,k in enumerate(eids)} +d=json.load(open('/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json',encoding='utf-8',errors='surrogateescape')) +N={n['id']:n for n in d['nodes']} +STRUCT={"identity","contains","superseded_by","references","embodies","demonstrated_by","canonical-self","depends_on","currently_holds","activates"} +adj=collections.defaultdict(list); hasstruct=set() +for e in d['edges']: + if e.get('relation') not in STRUCT: continue + w=float(e.get('weight') or 0.0) + adj[e['from_id']].append((e['to_id'],w)); adj[e['to_id']].append((e['from_id'],w)) + hasstruct.add(e['from_id']); hasstruct.add(e['to_id']) +del d +gold={q['id']:q for q in json.load(open(EV+"gold_set.json"))['queries']} +LEX={r['id']:r['returned'] for r in json.load(open(EV+"results-main.json"))['rows']} +CACHE={} +def emb(t): + if t in CACHE: return CACHE[t] + b=json.dumps({"model":"nomic-embed-text","prompt":t}).encode() + r=urllib.request.Request("http://127.0.0.1:11434/api/embeddings",data=b,headers={"Content-Type":"application/json"}) + v=np.array(json.load(urllib.request.urlopen(r,timeout=60))["embedding"],dtype=np.float32) + v=v/(np.linalg.norm(v)+1e-9); CACHE[t]=v; return v +FIRE=0.02; DECAY=0.7; DEPTH=2; SEED_MIN=0.60; ASSOC_MAX=64 +def assoc(seeds, s): + act={x:1.0 for x in seeds}; seen={x:2 for x in seeds} + Q=[(x,0) for x in seeds]; h=0 + while h=DEPTH: continue + p=act[cur] + for oid,w in adj.get(cur,()): + n=N.get(oid) + if not n or n.get('node_type') in ('Tag','InternalStateEvent'): continue + na=p*w*DECAY*float(n.get('salience') or 0.0) + if na=lim: break + if si=lim: break + if aiSEED_MIN] + A=[] + if mode!='hybrid': + seeds=[x for x in L[:3] if x in N] + if mode=='semseed': + seeds=seeds+[eids[j] for j in ordr[:K] if eids[j] in N and eids[j] not in seeds] + A=assoc(seeds,s) if seeds else [] + res[qid]=inter3(L,S,A) + return res +def score(res,label): + hits=0; det={} + for qid,q in gold.items(): + out=res[qid][:5] + if q['category']=='nonsense': ok = (len(res[qid])==0) + elif q['category']=='superseded': + rel=q['relevant']; must=q.get('must_outrank') or {} + ok=False + for good,bad in (must.items() if isinstance(must,dict) else []): + ok = good in res[qid] and (bad not in res[qid] or res[qid].index(good) (M@v)[j]).sum())+1, round(float((M@v)[j]),3))) + grank.sort() + out[q['id']]=dict(cat=q['category'],mu=round(mu,3),sd=round(sd,4),top1=round(float(top[0]),3), + z1=round(z[0],2),z3=round(z[2],2),z5=round(z[4],2),gold=grank[:1]) + print("%s %-11s mu=%.3f sd=%.4f top1=%.3f z1=%5.2f z3=%5.2f z5=%5.2f gold=%s"%( + q['id'],q['category'],mu,sd,top[0],z[0],z[2],z[4],grank[:1])) +json.dump(out,open(SP+'/zprobe.json','w'),indent=1) diff --git a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c index 3fe3fad..32c5a74 100644 --- a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c +++ b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c @@ -7531,8 +7531,30 @@ static int eg_assoc_excluded(const EngramNode* n) { * function of hop count and ranks the relay hub above all of its own * children. Composing the two is mine, not Will's — the description ranks the * activation result set by strength (l.78). */ +/* Semantic seeding of the graph leg — the HippoRAG pass Will documents at + * l.6082: "the query is embedded, the top-K nodes by cosine >= SEED_MIN join + * the seed set", using his own ENGRAM_EMBED_SEED_K (8). It is "similarity used + * twice, coherently": cosine picks where to STAND in the graph, the structural + * walk decides what is REACHABLE from there, and cosine then ORDERS what was + * reached (iteration-2's finding, kept intact). + * + * Why the seed list is NOT floored at ENGRAM_EMBED_SEED_MIN here. That + * constant is calibrated for a cosine scale this corpus does not have: with + * nomic-embed-text every true paraphrase target measures 0.46-0.66, and the + * three nonsense controls' own nearest neighbours measure 0.55/0.60/0.62 — + * they OVERLAP, so no absolute cosine floor separates signal from gibberish + * (measured, all 38 queries). Iteration 2 established the same thing one step + * later in the pipeline: applying the floor to graph CANDIDATES removed every + * gain, because within a structurally-reached neighbourhood relative cosine + * still discriminates below the absolute threshold. The gate that actually + * works is reachability — eg_rel_is_structural() plus the firing threshold. + * A semantically-near node with no structural attachment expands to nothing + * and contributes nothing, which is exactly what happens to gibberish: the + * nearest neighbours of q33/q34 are unattached, so their graph leg is empty. + */ static int64_t engram_assoc_leg(EngramStore* g, const EngramRankEntry* L, int64_t nL, + const int64_t* semseed, int64_t nsemseed, const float* qv, int32_t qdim, EngramSemEntry* out, int64_t out_cap) { if (!g || nL <= 0 || !qv || qdim <= 0 || out_cap <= 0) return 0; @@ -7553,6 +7575,14 @@ static int64_t engram_assoc_leg(EngramStore* g, act[idx] = 1.0; seen[idx] = 2; /* 2 = seed: never a result */ if (qt < qcap) { q[qt] = idx; hop[qt] = 0; qt++; } } + /* ...and the semantic seeds, on the same footing (strength 1.0, hop 0). */ + for (int64_t s = 0; s < nsemseed; s++) { + int64_t idx = semseed[s]; + if (idx < 0 || idx >= g->node_count) continue; + if (seen[idx]) continue; + act[idx] = 1.0; seen[idx] = 2; + if (qt < qcap) { q[qt] = idx; hop[qt] = 0; qt++; } + } const double SPREAD_DECAY = 0.7; while (qh < qt) { @@ -9524,6 +9554,13 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { EngramSemEntry* sem = qv ? malloc((size_t)g->node_count * sizeof(EngramSemEntry)) : NULL; int64_t nsem = 0; int64_t nhits = 0; + /* Raw (unfloored) cosine top-K, kept for the graph leg's + * semantic seeds. Selected in THIS pass so the cosine is + * computed exactly once per node — the seeding costs no extra + * pass over the corpus and no extra embed round-trip. */ + int64_t semseed[ENGRAM_EMBED_SEED_K]; + double semseedc[ENGRAM_EMBED_SEED_K]; + int64_t nsemseed = 0; for (int64_t i = 0; i < g->node_count; i++) { EngramNode* n = &g->nodes[i]; /* Filter transparent layers — same as engram_search. */ @@ -9535,9 +9572,29 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { hits[nhits].salience = n->salience; nhits++; } - if (sem) { - double sv = eg_sem_term(n, qv, qdim); - if (sv > 0.0) { sem[nsem].idx = i; sem[nsem].sem = sv; nsem++; } + if (sem && n->emb && n->emb_dim == qdim) { + double c = eg_cosine(n->emb, qv, qdim); + /* Semantic leg: identical to eg_sem_term(), which is + * left in place and still used by engram_search(). + * Inlined here only so one cosine serves both uses. */ + if (c > ENGRAM_EMBED_SEED_MIN) { + double sv = (c - ENGRAM_EMBED_SEED_MIN) / (1.0 - ENGRAM_EMBED_SEED_MIN); + if (sv > 1.0) sv = 1.0; + sem[nsem].idx = i; sem[nsem].sem = sv; nsem++; + } + /* Graph seeds: top-K by RAW cosine, insertion-ordered. */ + if (c > 0.0 && (nsemseed < ENGRAM_EMBED_SEED_K + || c > semseedc[nsemseed - 1])) { + int64_t p = nsemseed < ENGRAM_EMBED_SEED_K + ? nsemseed : ENGRAM_EMBED_SEED_K - 1; + while (p > 0 && semseedc[p - 1] < c) { + semseedc[p] = semseedc[p - 1]; + semseed[p] = semseed[p - 1]; + p--; + } + semseedc[p] = c; semseed[p] = i; + if (nsemseed < ENGRAM_EMBED_SEED_K) nsemseed++; + } } } qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp); @@ -9549,7 +9606,8 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { * lexical ordering safe. */ EngramSemEntry* assoc = qv ? malloc((size_t)ENGRAM_ASSOC_MAX * sizeof(EngramSemEntry)) : NULL; int64_t nassoc = assoc - ? engram_assoc_leg(g, hits, nhits, qv, qdim, assoc, ENGRAM_ASSOC_MAX) + ? engram_assoc_leg(g, hits, nhits, semseed, nsemseed, + qv, qdim, assoc, ENGRAM_ASSOC_MAX) : 0; int64_t* order = malloc((size_t)lim * sizeof(int64_t)); if (order) { From 9c39084e60b2ea7e021d3b2b5e0f3e118e9ce365 Mon Sep 17 00:00:00 2001 From: Tim Lingo <1timlingo@gmail.com> Date: Fri, 7 Aug 2026 15:51:49 -0500 Subject: [PATCH 06/15] feat(engram): BM25-shaped lexical leg + addressability guard on the read path engram_search_json ranked its lexical leg by raw distinct-token coverage with salience as tiebreak: a token in 30,000 nodes counted the same as a token in 1, and a 1.3 MB record matched nearly every query token by surface area alone. Score it BM25-shaped instead - Lucene-form IDF and length normalisation over the corpus mean - with per-token document frequency accumulated in the SAME corpus pass that finds the hits (no extra scan, no extra round-trip). Also refuse to return records whose identifier is not printable ASCII. This corpus carries 1,032 such records (453 by the printable test) from a save-side corruption; they occupy 125 of 303 returned slots on main. Claims 12, 23 and 27 all key on the node identifier, so such a record is unfetchable by any caller. Co-Authored-By: Claude Opus 5 (1M context) --- .../el-runtime/v1.0.0-20260501/el_runtime.c | 111 +++++++++++++++++- 1 file changed, 107 insertions(+), 4 deletions(-) diff --git a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c index 32c5a74..0ea4fc7 100644 --- a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c +++ b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c @@ -7387,6 +7387,48 @@ static int engram_node_match_score(const EngramNode* n, return score; } +/* Same match test as engram_node_match_score, but returns the SET of matched + * query tokens as a bitmask instead of only their count. ENGRAM_MAX_QTOKENS is + * 32, so one uint32 covers every token the tokenizer can produce. The mask is + * what lets the caller accumulate a per-token document frequency in the SAME + * pass that finds the hits — no second scan of the corpus. */ +static uint32_t engram_node_match_mask(const EngramNode* n, + char toks[][ENGRAM_QTOK_LEN], int ntok) { + uint32_t m = 0; + for (int t = 0; t < ntok && t < 32; t++) { + if (istr_contains(n->content, toks[t]) || + istr_contains(n->label, toks[t]) || + istr_contains(n->tags, toks[t])) + m |= (uint32_t)1u << t; + } + return m; +} + +/* Searchable byte length of a node: the same three fields the match test + * reads. Used as the BM25 document length so a long node does not out-match a + * short one merely by containing more text. */ +static double engram_node_len(const EngramNode* n) { + double l = 0.0; + if (n->content) l += (double)strlen(n->content); + if (n->label) l += (double)strlen(n->label); + if (n->tags) l += (double)strlen(n->tags); + return l; +} + +/* Addressability guard. Claim 23 stores node records under a key encoding the + * node identifier, claim 12 deduplicates merged results by node identifier, + * and claim 27's competition map is indexed by node identifier — every one of + * those requires the identifier to be a usable string. This corpus contains + * records whose id field is binary garbage (a save-side corruption); they are + * unfetchable by any caller, so returning one wastes a result slot. Printable + * ASCII, non-empty, is the whole test. */ +static int eg_node_addressable(const EngramNode* n) { + const unsigned char* p = (const unsigned char*)n->id; + if (!p || !*p) return 0; + for (; *p; p++) if (*p < 0x20 || *p > 0x7e) return 0; + return 1; +} + /* Semantic leg of the read path (engram claim 24). Returns the query/target * cosine renormalized onto [0,1] over the band [ENGRAM_EMBED_SEED_MIN, 1.0], * and exactly 0.0 when the pair is not comparable (no query embedding, target @@ -7406,7 +7448,12 @@ static double eg_sem_term(const EngramNode* n, const float* qv, int32_t qdim) { * (tiebreak, desc). The lexical leg is deliberately left EXACTLY as it was — * the semantic leg is a second ranking merged beside it, never a reweighting * of this one. */ -typedef struct { int64_t idx; int score; double salience; } EngramRankEntry; +typedef struct { + int64_t idx; int score; double salience; + uint32_t mask; /* which query tokens matched (BM25 leg) */ + double len; /* searchable byte length (BM25 leg) */ + double w; /* BM25-shaped weighted score */ +} EngramRankEntry; static int engram_rank_cmp(const void* a, const void* b) { const EngramRankEntry* ea = (const EngramRankEntry*)a; const EngramRankEntry* eb = (const EngramRankEntry*)b; @@ -7416,6 +7463,21 @@ static int engram_rank_cmp(const void* a, const void* b) { return 0; } +/* BM25-shaped ordering for the read path's lexical leg: rare-term weight and + * length normalisation instead of a raw distinct-token count. Salience stays + * the tiebreak, exactly as in engram_rank_cmp. */ +#define ENGRAM_BM25_K1 1.2 +#define ENGRAM_BM25_B 0.75 +static int engram_rank_w_cmp(const void* a, const void* b) { + const EngramRankEntry* ea = (const EngramRankEntry*)a; + const EngramRankEntry* eb = (const EngramRankEntry*)b; + if (ea->w < eb->w) return 1; /* desc */ + if (ea->w > eb->w) return -1; + if (ea->salience < eb->salience) return 1; + if (ea->salience > eb->salience) return -1; + return 0; +} + /* Semantic rank entry: node index and its renormalized query similarity, * ordered by similarity desc. This is the claim-24 "embedding search" * ranking, computed independently of the lexical one. */ @@ -7615,6 +7677,7 @@ static int64_t engram_assoc_leg(EngramStore* g, if (seen[i] != 1) continue; /* skip unreached and seeds */ if (engram_layer_is_transparent(g->nodes[i].layer_id)) continue; EngramNode* nd = &g->nodes[i]; + if (!eg_node_addressable(nd)) continue; /* unfetchable record */ if (!nd->emb || nd->emb_dim != qdim) continue; double c = eg_cosine(nd->emb, qv, qdim); if (c <= 0.0) continue; @@ -9561,15 +9624,32 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { int64_t semseed[ENGRAM_EMBED_SEED_K]; double semseedc[ENGRAM_EMBED_SEED_K]; int64_t nsemseed = 0; + /* BM25 statistics gathered in this same pass: per-token + * document frequency, and the corpus mean field length. */ + int64_t df[ENGRAM_MAX_QTOKENS]; + for (int t = 0; t < ntok; t++) df[t] = 0; + double dl_total = 0.0; + int64_t dl_n = 0; for (int64_t i = 0; i < g->node_count; i++) { EngramNode* n = &g->nodes[i]; /* Filter transparent layers — same as engram_search. */ if (engram_layer_is_transparent(n->layer_id)) continue; - int sc = engram_node_match_score(n, toks, ntok); - if (sc > 0) { + /* Unaddressable records cannot be fetched by a caller and + * must not consume a result slot (claims 12/23/27). */ + if (!eg_node_addressable(n)) continue; + double dl = engram_node_len(n); + dl_total += dl; dl_n++; + uint32_t mask = engram_node_match_mask(n, toks, ntok); + if (mask) { + int sc = 0; + for (int t = 0; t < ntok; t++) + if (mask & ((uint32_t)1u << t)) { sc++; df[t]++; } hits[nhits].idx = i; hits[nhits].score = sc; hits[nhits].salience = n->salience; + hits[nhits].mask = mask; + hits[nhits].len = dl; + hits[nhits].w = 0.0; nhits++; } if (sem && n->emb && n->emb_dim == qdim) { @@ -9597,7 +9677,30 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { } } } - qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp); + /* BM25-shaped lexical score. Binary term frequency (the match + * primitive is a substring test, not a count), Lucene-form IDF, + * and length normalisation over the corpus mean. A token that + * occurs in 30,000 nodes now weighs far less than one that + * occurs in 1, and a 1.3 MB record no longer out-matches a + * 300-byte one by sheer surface area. */ + double avgdl = dl_n ? (dl_total / (double)dl_n) : 1.0; + if (avgdl <= 0.0) avgdl = 1.0; + double idf[ENGRAM_MAX_QTOKENS]; + for (int t = 0; t < ntok; t++) { + double dfx = (double)df[t]; + idf[t] = log(1.0 + ((double)dl_n - dfx + 0.5) / (dfx + 0.5)); + } + for (int64_t h = 0; h < nhits; h++) { + double norm = 1.0 - ENGRAM_BM25_B + + ENGRAM_BM25_B * (hits[h].len / avgdl); + double s = 0.0; + for (int t = 0; t < ntok; t++) + if (hits[h].mask & ((uint32_t)1u << t)) + s += idf[t] * (ENGRAM_BM25_K1 + 1.0) + / (1.0 + ENGRAM_BM25_K1 * norm); + hits[h].w = s; + } + qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_w_cmp); if (sem) qsort(sem, (size_t)nsem, sizeof(EngramSemEntry), engram_sem_cmp); /* Claim-10 associative leg: expand the top lexical hits along * structural relations only, order the reached set by query From 55f9ee3cb0188a0b6995cf9a1bd219265f04eeb7 Mon Sep 17 00:00:00 2001 From: Tim Lingo <1timlingo@gmail.com> Date: Fri, 7 Aug 2026 15:55:42 -0500 Subject: [PATCH 07/15] measure: BM25 lexical leg vs semseed baseline - net +2 (q10,q11), NOT-SHOWN hit@5 68.6% -> 74.3%, phrase 71.4% -> 100%, MRR@10 0.461 -> 0.502, latency p50 0.97x. Zero losses, zero run-to-run drift on both sides. 2 queries moved against a 6-query noise floor: NO MEASURABLE DIFFERENCE by the harness's own test (McNemar exact p=0.50). Unaddressable records in returned slots: 57 -> 0. Co-Authored-By: Claude Opus 5 (1M context) --- .../comparison-semseed-vs-bm25lex.json | 147 +++ tools/retrieval-eval/results-bm25lex-r2.json | 959 ++++++++++++++++++ tools/retrieval-eval/results-bm25lex.json | 959 ++++++++++++++++++ tools/retrieval-eval/sim-bm25.py | 142 +++ 4 files changed, 2207 insertions(+) create mode 100644 tools/retrieval-eval/comparison-semseed-vs-bm25lex.json create mode 100644 tools/retrieval-eval/results-bm25lex-r2.json create mode 100644 tools/retrieval-eval/results-bm25lex.json create mode 100644 tools/retrieval-eval/sim-bm25.py diff --git a/tools/retrieval-eval/comparison-semseed-vs-bm25lex.json b/tools/retrieval-eval/comparison-semseed-vs-bm25lex.json new file mode 100644 index 0000000..7f43706 --- /dev/null +++ b/tools/retrieval-eval/comparison-semseed-vs-bm25lex.json @@ -0,0 +1,147 @@ +{ + "baseline": "semseed", + "candidate": "bm25lex", + "n_shared_queries": 38, + "fixed_by_candidate": [ + "q10", + "q11" + ], + "broken_by_candidate": [], + "discordant": 2, + "net_queries": 2, + "mcnemar_exact_p": 0.5, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "no measurable difference", + "baseline_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.6857142857142857, + "recall@5": 0.5213459159887731, + "recall@10": 0.6027048348476919, + "precision@5": 0.18285714285714294, + "mrr@10": 0.4608730158730158, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1227.1, + "latency_ms_p95": 1692.6, + "latency_ms_max": 1710.4, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.07342657342657344, + "recall@10": 0.24825174825174823, + "mrr@10": 0.20833333333333334 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 0.7142857142857143, + "recall@5": 0.40093537414965985, + "recall@10": 0.5150226757369615, + "mrr@10": 0.6507936507936508 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "candidate_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.7428571428571429, + "recall@5": 0.5536485340056769, + "recall@10": 0.6175677497106068, + "precision@5": 0.20000000000000007, + "mrr@10": 0.5021428571428571, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1184.4, + "latency_ms_p95": 1620.0, + "latency_ms_max": 1655.4, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.23310023310023312, + "mrr@10": 0.25 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5494614512471656, + "recall@10": 0.6023242630385487, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "repeat_variance": { + "baseline": { + "runs": 2, + "hit@5_min": 0.6857142857142857, + "hit@5_max": 0.6857142857142857, + "spread_queries": 0 + }, + "candidate": { + "runs": 2, + "hit@5_min": 0.7428571428571429, + "hit@5_max": 0.7428571428571429, + "spread_queries": 0 + } + } +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-bm25lex-r2.json b/tools/retrieval-eval/results-bm25lex-r2.json new file mode 100644 index 0000000..2a60b4b --- /dev/null +++ b/tools/retrieval-eval/results-bm25lex-r2.json @@ -0,0 +1,959 @@ +{ + "label": "bm25lex-r2", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-bm25lex", + "soul_md5": "dfbd0f8e3646212db5c60026f8ad906f", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json", + "limit": 10, + "port": 7894, + "wall_clock_s": 50.1, + "child_pid": 90371, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.7428571428571429, + "recall@5": 0.5536485340056769, + "recall@10": 0.6175677497106068, + "precision@5": 0.20000000000000007, + "mrr@10": 0.5021428571428571, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1183.8, + "latency_ms_p95": 1611.3, + "latency_ms_max": 1654.7, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.23310023310023312, + "mrr@10": 0.25 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5494614512471656, + "recall@10": 0.6023242630385487, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "n_returned": 1, + "latency_ms": 282.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5", + "bl-ba764d70-e9d7-4f62-848f-719cb665f45e", + "mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257", + "bl-b28d7256-6f74-4567-bd90-40d0ef2a6d78", + "project-engram", + "ctx-45bc", + "project-engram-lang", + "ctx-175f", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "ctx-74ed" + ], + "n_returned": 10, + "latency_ms": 315.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "n_returned": 1, + "latency_ms": 283.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "n_returned": 1, + "latency_ms": 283.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848", + "bl-c1765767-3e27-449a-8c94-10411d1eb7c0", + "project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_" + ], + "n_returned": 3, + "latency_ms": 303.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "n_returned": 1, + "latency_ms": 283.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "tag-patterns", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "project-Imprint__system_design__ADRs__tech_strategy__integration_patterns__governance_", + "project-Imprint__analysis_patterns__data_storytelling__SQL__dashboards__insight_framing_", + "bl-79028eed-c330-4724-9402-734062d13503", + "bl-39dad13d-7105-4049-8224-dc3c34fdb1f3", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "bl-7e7c3fdb-4132-487f-aa70-b2cd559cb7f0", + "bl-1d32bd54-cf17-4a1f-b235-982d09a36f04", + "bl-b8af6601-a8cb-41b5-aef5-ab8a57432dd5" + ], + "n_returned": 10, + "latency_ms": 587.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6" + ], + "n_returned": 10, + "latency_ms": 518.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1875, + "recall@10": 0.375, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "n_returned": 10, + "latency_ms": 511.3, + "error": null, + "hit@5": 1.0, + "recall@5": 0.3333333333333333, + "recall@10": 0.4444444444444444, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "bl-dcee1887-34c4-4ffa-9119-1e291685ba08", + "project-harmonic-framework", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf", + "project-harmonic-framework_com", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "tag-harmonic-design", + "bl-92acd4eb-0452-4e8e-9f54-f8cd35170d76", + "tag-harmonic-framework", + "bl-18a9d1e4-1484-474c-bf6b-c6173212181b" + ], + "n_returned": 10, + "latency_ms": 497.7, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1111111111111111, + "recall@10": 0.1111111111111111, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-sarah", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-a6cb3b8d-d89c-46fc-931d-e90c560783b0", + "kn-6061318f-046b-4935-907d-8eafdce14930" + ], + "n_returned": 10, + "latency_ms": 510.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "2c56a7a9-5323-4ce4-ba09-35836ba15d54", + "bl-39cec462-c80c-4970-a3aa-91fe83053bde" + ], + "n_returned": 10, + "latency_ms": 870.7, + "error": null, + "hit@5": 1.0, + "recall@5": 0.21428571428571427, + "recall@10": 0.2857142857142857, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a", + "8f3abb0d-77ed-4af3-9f4d-ba62cd198886", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "?", + "bl-ec84b63d-b278-4944-8d7f-4aa7a51c0315", + "?", + "830ca37a-d334-4e41-ba89-64893dc8d628", + "?", + "ce9636dc-85a5-4dae-9e07-74ea2fcc6307", + "?" + ], + "n_returned": 10, + "latency_ms": 723.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "cb070131-dfd4-4a38-91d7-22b1bde164d2", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "knw-d788a210-613b-4c49-9486-88bbc9d4716f", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "66c63082-b4da-4aa1-8fee-848db8a83210", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-a535f205-bc4c-4058-9171-6263c496044a", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "ctx-4a41" + ], + "n_returned": 10, + "latency_ms": 1584.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "b1183213-d659-4759-85d7-5b1f22427fe2", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-16efddd1-c43d-4a42-9d78-f54fb82bd277", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "f0eb6b13-909c-4674-91ef-23301d3abc8b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "30a44d10-2487-420e-bf61-3892e4343c92", + "54608b69-78b6-4239-b60f-b8206cfecacc", + "bfb5809e-d19a-4d3f-8c1a-796db622ad9d" + ], + "n_returned": 10, + "latency_ms": 1652.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "mem-ef878e30-5851-4e82-8588-745415108941", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "tag-fiction", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "mem-8d690e9d-a7e9-4062-b2f8-e2064294e463", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "mem-ce793303-c5a5-4586-a232-a3426edd9ec7", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "mem-443bd012-fc9a-4088-b236-de5157a1ef92" + ], + "n_returned": 10, + "latency_ms": 1341.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-8de20bcf-7149-4f48-b67c-e7f9758fd6e5", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "bl-164b520b-c503-49db-89f9-bd2fdf4215f5", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "1219277c-1b95-45ec-95a2-07b4a47a4d92", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "bl-79ce4464-5dd6-49bd-9b0c-9803549d0665" + ], + "n_returned": 10, + "latency_ms": 1064.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "a1000001-0000-0000-0000-000000000010", + "a1000001-0000-0000-0000-000000000009", + "bl-448bc514-c2f1-4520-a9b1-1f3a73678d26", + "a1000001-0000-0000-0000-000000000012", + "43098881-e044-482b-8e92-471728a8ba8b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "mem-e5cc63c0-8701-49d6-855a-e387fe087771", + "a1000001-0000-0000-0000-000000000001" + ], + "n_returned": 10, + "latency_ms": 1398.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "tag-learning", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "d6b12ecf-702b-4101-b1bb-09ed9b220b29", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "c608a095-c98b-4bfa-bfe1-1611c1320290", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "451ae007-4219-4096-89fe-fa2e045fbeb1", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "n_returned": 10, + "latency_ms": 1252.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "mem-cde58b77-50d3-4bac-9581-e70a4c02c015", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-8d699e2c-ac2a-4742-bb62-b6da00f4b10e", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-53d6adf0-cd08-4707-a237-daa5e65c7298", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac" + ], + "n_returned": 10, + "latency_ms": 1581.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_", + "bl-8848929a-a23a-46bc-a2c7-fe3a3bc1cddf", + "bl-205141ad-b2a0-4d93-86d0-89eb0723e1bd", + "bl-e93858c4-7cac-4b1a-bb62-490790d4c3f3", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "bl-286b562a-5299-40e0-a32a-afa9cbdfe995" + ], + "n_returned": 10, + "latency_ms": 1363.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "5a2c118a-87bd-4239-97a7-9e02c5991983", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "mem-ef878e30-5851-4e82-8588-745415108941", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 1611.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "a0edad47-5f77-4fc3-a546-1e85f8c68e77", + "a1000001-0000-0000-0000-000000000001", + "8cbb60c5-4999-4ec1-8682-2592aedc4249", + "96497334-b18f-495c-9228-eeb8182bdc38", + "mem-024598a9-ed2e-4eeb-b1e1-5410856ff132", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "knw-ed33e669-0790-44cb-a036-958d605c6fea" + ], + "n_returned": 10, + "latency_ms": 1470.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "27e1b1a4-ad0b-49d9-812f-fedf43b8aabe", + "knw-f9ce17a7-17fc-431f-8f23-695b670ec4fa", + "bl-87c93185-b2bf-40af-ae23-3c830c007abf", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "077d064f-3489-4c05-9aca-3782f96b51db", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "75e036d3-c170-4e3f-acc2-e456a6850ee2", + "a1000001-0000-0000-0000-000000000001" + ], + "n_returned": 10, + "latency_ms": 1513.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "e4f27651-52c5-43fd-aff3-61d31685b3cd", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-5624ec9d-62ba-4aba-8a3d-6afec6c09dd4", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-833dbbcd-2400-4594-bb35-93b023049ac0", + "a1000001-0000-0000-0000-000000000009", + "mem-759e78ca-5394-4244-aa39-1c1468bc5f3e", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd" + ], + "n_returned": 10, + "latency_ms": 1226.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "bl-3f57bc69-7285-4f4a-a861-2de52efca058", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "bl-0d8c5dfa-e163-4fef-a58b-56b0d076c5a8", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-b99efff0-00e6-40c8-9c5b-730330eef33b", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "tag-childhood" + ], + "n_returned": 10, + "latency_ms": 1342.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "3499d5da-0e9c-4de4-9bc4-8941b14e0b1f", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c" + ], + "n_returned": 10, + "latency_ms": 1433.0, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.45454545454545453, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "54608b69-78b6-4239-b60f-b8206cfecacc", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-9110798f-d0cb-4446-bc2a-14f09b6a09e2", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71" + ], + "n_returned": 10, + "latency_ms": 1104.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.07692307692307693, + "recall@10": 0.3076923076923077, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-trailer-park-paladins", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "project-trailer-park-paladins", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c" + ], + "n_returned": 10, + "latency_ms": 1183.8, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.45454545454545453, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9", + "bl-bea7473c-c687-414c-9c0b-00c509a616c1", + "bl-fc6fcb0b-9e4b-40bf-8e88-dbfe4e27c31a", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc" + ], + "n_returned": 10, + "latency_ms": 1199.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "tag-hope", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "n_returned": 10, + "latency_ms": 1202.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.18181818181818182, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "knw-729fc901-8335-44c4-9f3a-b150b4aa0915", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "knw-528dbc37-eabc-4b75-a7a5-65bf38d6018a", + "knw-35940684-abc4-42f0-b942-818f66b1f69a", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "?Z?=DEPTH: continue + p=act[cur] + for oid,w in adj.get(cur,()): + n=N.get(oid) + if not n or n.get('node_type') in ('Tag','InternalStateEvent'): continue + na=p*w*DECAY*float(n.get('salience') or 0.0) + if na=lim: break + if si=lim: break + if aiSEED_MIN] + if guard: S=[x for x in S if PRINT.match(x or '')] + seeds=[x for x in L[:3] if x in N] + seeds=seeds+[eids[j] for j in ordr[:8] if eids[j] in N and eids[j] not in seeds and (not guard or PRINT.match(eids[j] or ''))] + A=assoc(seeds,s) if seeds else [] + if guard: A=[x for x in A if PRINT.match(x or '')] + res[qid]=inter3(L,S,A); legs[qid]=(L,S,A) + return res,legs +def score(res,label,verbose=False): + det={} + for qid,q in gold.items(): + out=res[qid][:5] + if q['category']=='nonsense': ok=(len(res[qid])==0) + elif q['category']=='superseded': + must=q.get('must_outrank') or {}; ok=False + for good,bad in (must.items() if isinstance(must,dict) else []): + ok = good in res[qid] and (bad not in res[qid] or res[qid].index(good) Date: Fri, 7 Aug 2026 16:58:34 -0500 Subject: [PATCH 08/15] feat(engram): word-start match primitive + corpus-vocabulary gate on recall The retrieval match test is a raw substring scan, so a query token matches anywhere INSIDE a corpus word: "throom" matches "bathroom". Measured over the 38-query gold set on this corpus that is not a rare accident - q28's lexical leg is 36,954 records of which only 13 contain a query token at a word start (99.96% mid-word noise), six further queries carry ~20,500 mid-word-only records each, and the nonsense control q35 returns 7 records ALL of which match only mid-word. istr_contains_wordstart() anchors a token to a word start (preceding char not alphanumeric) while still matching suffixes, so "value" still hits "values". That empties the lexical leg for gibberish, and the nhits==0 corpus-vocabulary gate (iteration 6's mechanism, feat/claim24-unfloored-semantic) then makes the whole query decline rather than let the semantic leg answer it. Measured vs feat/bm25-lexical-leg on the embedded corpus, 2 runs each, 0 queries of run-to-run drift on both sides: net +1 (nonsense:q35), 0 losses, McNemar p=1.0 -> NOT-SHOWN (floor is 6) nonsense clean 2/3 -> 3/3; exact_rare 100%, phrase 100%, paraphrase 61.5%, associative 66.7%, superseded 2/3 all UNCHANGED latency p50 1184 -> 543 ms (0.46x) Iteration 6 called q35 "a DEFECTIVE CONTROL ... cannot be cleaned without breaking the lexical leg". It can: the defect was the match primitive, and cleaning it cost nothing. Also committed: results-wsclaim24.json + cmp-nogate.json, a measured negative for bundling the claim-24 unfloored semantic leg on top (gains q14/q25, breaks q15/q28/q33/q34, net -2) - it independently reproduces iteration 6's q15/q28 losses and shows unflooring REQUIRES the vocabulary gate. Reproducers: legs.py (leg-level replica, reproduces baseline hit@5 exactly on all 38 queries), policy2.py, ceiling.py, wb2.py. --- tools/retrieval-eval/ceiling.py | 43 + tools/retrieval-eval/cmp-nogate.json | 146 +++ .../comparison-bm25lex-vs-wordstart.json | 146 +++ tools/retrieval-eval/legs.py | 117 ++ tools/retrieval-eval/policy2.py | 102 ++ .../retrieval-eval/results-wordstart-r2.json | 948 +++++++++++++++ tools/retrieval-eval/results-wordstart.json | 948 +++++++++++++++ tools/retrieval-eval/results-wsclaim24.json | 1024 +++++++++++++++++ tools/retrieval-eval/wb2.py | 92 ++ .../el-runtime/v1.0.0-20260501/el_runtime.c | 59 +- 10 files changed, 3618 insertions(+), 7 deletions(-) create mode 100644 tools/retrieval-eval/ceiling.py create mode 100644 tools/retrieval-eval/cmp-nogate.json create mode 100644 tools/retrieval-eval/comparison-bm25lex-vs-wordstart.json create mode 100644 tools/retrieval-eval/legs.py create mode 100644 tools/retrieval-eval/policy2.py create mode 100644 tools/retrieval-eval/results-wordstart-r2.json create mode 100644 tools/retrieval-eval/results-wordstart.json create mode 100644 tools/retrieval-eval/results-wsclaim24.json create mode 100644 tools/retrieval-eval/wb2.py diff --git a/tools/retrieval-eval/ceiling.py b/tools/retrieval-eval/ceiling.py new file mode 100644 index 0000000..4078e39 --- /dev/null +++ b/tools/retrieval-eval/ceiling.py @@ -0,0 +1,43 @@ +import json,sys,pickle,numpy as np,itertools +sys.path.insert(0,'.') +from policy2 import legs3,outcome,G,NODES,merge +# cache per-query leg id-lists, floored and unfloored +cache={} +for q in G['queries']: + Lf,Sf,Af=legs3(q['query']) + Lu,Su,Au=legs3(q['query'],unfloor=True) + cache[q['id']]=dict(L=Lf,Sf=Sf,A=Af,Su=Su,Au=Au) +pickle.dump(cache,open('ceil.pkl','wb')) +def mrg(pattern,L,S,A,lim=10): + out=[];p={'L':0,'S':0,'A':0};src={'L':L,'S':S,'A':A} + i=0 + while len(out)=lim: return out + if not prog: break + return out +def ev(pattern,unfl): + res={} + for q in G['queries']: + c=cache[q['id']] + S=c['Su'] if unfl else c['Sf'] + ids=[NODES[i]['id'] for i in mrg(pattern,c['L'],S,c['A'],10)] + res[q['id']]=outcome(q,ids) + return res +base=ev('LSA',False) +print("baseline",sum(base.values())) +best=[] +pats=['LSA','LAS','SLA','ALS','SAL','ASL','LSSA','LSASA','LSAA','LSSAA','LSAS','SSLA','LLSA','SALSA','LSAAS'] +for unfl in (False,True): + for p in pats: + r=ev(p,unfl) + g=sorted(k for k in base if r[k] and not base[k]);l=sorted(k for k in base if base[k] and not r[k]) + best.append((len(g)-len(l),p,unfl,g,l)) +best.sort(reverse=True) +for n,p,u,g,l in best[:10]: + print("net=%+d pat=%-6s unfloor=%s gains=%s losses=%s"%(n,p,u,g,l)) diff --git a/tools/retrieval-eval/cmp-nogate.json b/tools/retrieval-eval/cmp-nogate.json new file mode 100644 index 0000000..e680068 --- /dev/null +++ b/tools/retrieval-eval/cmp-nogate.json @@ -0,0 +1,146 @@ +{ + "baseline": "bm25lex", + "candidate": "wsclaim24", + "n_shared_queries": 38, + "fixed_by_candidate": [ + "q14", + "q25" + ], + "broken_by_candidate": [ + "q15", + "q28", + "q33", + "q34" + ], + "discordant": 6, + "net_queries": -2, + "mcnemar_exact_p": 0.6875, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "no measurable difference", + "baseline_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.7428571428571429, + "recall@5": 0.5536485340056769, + "recall@10": 0.6175677497106068, + "precision@5": 0.20000000000000007, + "mrr@10": 0.5021428571428571, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1184.4, + "latency_ms_p95": 1620.0, + "latency_ms_max": 1655.4, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.23310023310023312, + "mrr@10": 0.25 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5494614512471656, + "recall@10": 0.6023242630385487, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "candidate_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.7428571428571429, + "recall@5": 0.5768475572047, + "recall@10": 0.6563414759843332, + "precision@5": 0.19428571428571437, + "mrr@10": 0.5026530612244898, + "nonsense_clean": "0/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 524.8, + "latency_ms_p95": 738.7, + "latency_ms_max": 755.8, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.5, + "recall@5": 0.07575757575757576, + "recall@10": 0.13636363636363635, + "mrr@10": 0.23214285714285712 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 0, + "avg_false_positives": 10.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6923076923076923, + "recall@5": 0.6923076923076923, + "recall@10": 0.7692307692307693, + "mrr@10": 0.29423076923076924 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5335884353741497, + "recall@10": 0.5933956916099773, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "repeat_variance": { + "baseline": { + "runs": 2, + "hit@5_min": 0.7428571428571429, + "hit@5_max": 0.7428571428571429, + "spread_queries": 0 + } + } +} \ No newline at end of file diff --git a/tools/retrieval-eval/comparison-bm25lex-vs-wordstart.json b/tools/retrieval-eval/comparison-bm25lex-vs-wordstart.json new file mode 100644 index 0000000..661e2db --- /dev/null +++ b/tools/retrieval-eval/comparison-bm25lex-vs-wordstart.json @@ -0,0 +1,146 @@ +{ + "baseline": "bm25lex", + "candidate": "wordstart", + "n_shared_queries": 38, + "fixed_by_candidate": [ + "q35" + ], + "broken_by_candidate": [], + "discordant": 1, + "net_queries": 1, + "mcnemar_exact_p": 1.0, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "no measurable difference", + "baseline_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.7428571428571429, + "recall@5": 0.5536485340056769, + "recall@10": 0.6175677497106068, + "precision@5": 0.20000000000000007, + "mrr@10": 0.5021428571428571, + "nonsense_clean": "2/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 1184.4, + "latency_ms_p95": 1620.0, + "latency_ms_max": 1655.4, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.23310023310023312, + "mrr@10": 0.25 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 2, + "avg_false_positives": 3.3333333333333335 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5494614512471656, + "recall@10": 0.6023242630385487, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "candidate_aggregate": { + "n_queries": 38, + "n_scored": 35, + "hit@5": 0.7428571428571429, + "recall@5": 0.5536485340056769, + "recall@10": 0.6175677497106068, + "precision@5": 0.20000000000000007, + "mrr@10": 0.5021428571428571, + "nonsense_clean": "3/3", + "superseded_outranks": "2/3", + "latency_ms_p50": 542.6, + "latency_ms_p95": 741.3, + "latency_ms_max": 758.8, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.23310023310023312, + "mrr@10": 0.25 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "nonsense": { + "n": 3, + "clean": 3, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5494614512471656, + "recall@10": 0.6023242630385487, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "repeat_variance": { + "baseline": { + "runs": 2, + "hit@5_min": 0.7428571428571429, + "hit@5_max": 0.7428571428571429, + "spread_queries": 0 + }, + "candidate": { + "runs": 2, + "hit@5_min": 0.7428571428571429, + "hit@5_max": 0.7428571428571429, + "spread_queries": 0 + } + } +} \ No newline at end of file diff --git a/tools/retrieval-eval/legs.py b/tools/retrieval-eval/legs.py new file mode 100644 index 0000000..2b98534 --- /dev/null +++ b/tools/retrieval-eval/legs.py @@ -0,0 +1,117 @@ +import json,pickle,os,math,urllib.request,numpy as np +S='/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/sim/' +C=pickle.load(open(S+'corpus.pkl','rb')) +NODES=C['nodes']; EDGES=C['edges']; N=len(NODES) +E=np.load(S+'emb.npy'); HAVE=np.load(S+'have.npy') +En=E/np.maximum(np.linalg.norm(E,axis=1,keepdims=True),1e-12) +LAYERS={int(l['layer_id']):l for l in (C['layers'] or [])} if C['layers'] else {} +TRANS=set(i for i,l in LAYERS.items() if l.get('transparent')) +def addressable(s): + if not s: return False + return all(0x20<=ord(ch)<=0x7e for ch in s) +ADDR=np.array([addressable(n['id']) for n in NODES]) +OK=np.array([ (n['layer_id'] not in TRANS) and ADDR[i] for i,n in enumerate(NODES)]) +SAL=np.array([n['salience'] for n in NODES]) +LOW=[ (n['content']+'\x00'+n['label']+'\x00'+n['tags']).lower() for n in NODES] +DL=np.array([float(len(n['content'])+len(n['label'])+len(n['tags'])) for n in NODES]) +IDX={} +for i,n in enumerate(NODES): + IDX.setdefault(n['id'],i) +STRUCT={"identity","contains","superseded_by","references","embodies","demonstrated_by","canonical-self","depends_on","currently_holds","activates"} +ADJ_F=[[] for _ in range(N)]; ADJ_T=[[] for _ in range(N)] +for e in EDGES: + a=IDX.get(e['from']); b=IDX.get(e['to']) + if a is None or b is None: continue + ADJ_F[a].append((e,b)); ADJ_T[b].append((e,a)) +EXCL=np.array([n['node_type'] in ('Tag','InternalStateEvent') for n in NODES]) +avgdl_all=None +def tokenize(q): + out=[] + for t in q.split(): + if not any(t.lower()==x.lower() for x in out): out.append(t) + return out +_qcache={} +def qemb(q): + if q in _qcache: return _qcache[q] + body=json.dumps({"model":"nomic-embed-text","prompt":q}).encode() + r=urllib.request.urlopen(urllib.request.Request("http://127.0.0.1:11434/api/embeddings",data=body,headers={"Content-Type":"application/json"}),timeout=30) + v=np.array(json.loads(r.read())["embedding"],dtype=np.float32) + v=v/np.linalg.norm(v); _qcache[q]=v; return v +K1,B=1.2,0.75 +SEED_MIN=0.60; SEED_K=8; ASSOC_SEEDS=3; DEPTH=2; FIRE=0.02; AMAX=64; DECAY=0.7 +def legs(query): + toks=tokenize(query) + masks=[]; + hit_idx=[]; hit_mask=[] + df=[0]*len(toks) + lt=[t.lower() for t in toks] + for i in range(N): + if not OK[i]: continue + s=LOW[i]; m=0 + for t,tok in enumerate(lt): + if tok in s: m|=(1<>t&1: df[t]+=1 + dl_n=int(OK.sum()); avgdl=float(DL[OK].sum()/max(dl_n,1)) + idf=[math.log(1.0+((dl_n-d+0.5)/(d+0.5))) for d in df] + L=[] + for j,i in enumerate(hit_idx): + norm=1.0-B+B*(DL[i]/avgdl); w=0.0 + for t in range(len(toks)): + if hit_mask[j]>>t&1: w+=idf[t]*(K1+1.0)/(1.0+K1*norm) + L.append((i,w,SAL[i])) + L.sort(key=lambda x:(-x[1],-x[2])) + qv=qemb(query) + cos=En@qv + cos=np.where(HAVE&OK,cos,-2.0) + order=np.argsort(-cos) + semfull=[(int(i),float(cos[i])) for i in order[:400]] + Sleg=[(i,(c-SEED_MIN)/(1-SEED_MIN)) for i,c in semfull if c>SEED_MIN] + semseed=[i for i,c in semfull[:SEED_K] if c>0.0] + # assoc + act={}; seen={}; qq=[] + for i,_,_ in L[:ASSOC_SEEDS]: + act[i]=1.0; seen[i]=2; qq.append((i,0)) + for i in semseed: + if i in seen: continue + act[i]=1.0; seen[i]=2; qq.append((i,0)) + qh=0 + while qh=DEPTH: continue + parent=act[cur] + for e,oi in ADJ_F[cur]+ADJ_T[cur]: + if e['rel'] not in STRUCT: continue + if EXCL[oi]: continue + na=parent*e['w']*DECAY*SAL[oi] + if na=lim: break + if si=lim: break + if ai>t&1: df[t]+=1 + dl_n=int(OK.sum());avgdl=float(DL[OK].sum()/max(dl_n,1)) + idf=[math.log(1.0+((dl_n-d+0.5)/(d+0.5))) for d in df] + L=[] + for j,i in enumerate(hit_idx): + norm=1.0-B+B*(DL[i]/avgdl);w=0.0 + for t in range(len(toks)): + if hit_mask[j]>>t&1: w+=idf[t]*(K1+1.0)/(1.0+K1*norm) + L.append((i,w,SAL[i])) + L.sort(key=lambda x:(-x[1],-x[2])) + if not L: return [],[],[] + qv=qemb(query);cos=En@qv;cos=np.where(HAVE&OK,cos,-2.0) + order=np.argsort(-cos)[:600] + cand=[int(i) for i in order if cos[i]>(0.0 if unfloor else SEED_MIN)] + key=(lambda i:(SAL[i] if sem_sal else 1.0)*float(cos[i])) + Sl=sorted(cand,key=lambda i:-key(i)) + if sem_cap: Sl=Sl[:sem_cap] + semseed=[int(i) for i in order[:SEED_K] if cos[i]>0.0] + act={};seen={};qq=[] + for i,_,_ in L[:ASSOC_SEEDS]: + act[i]=1.0;seen[i]=2;qq.append((i,0)) + for i in semseed: + if i in seen: continue + act[i]=1.0;seen[i]=2;qq.append((i,0)) + qh=0 + while qh=DEPTH: continue + parent=act[cur] + for e,oi in ADJ_F[cur]+ADJ_T[cur]: + if e['rel'] not in STRUCT: continue + if EXCL[oi]: continue + na=parent*e['w']*DECAY*SAL[oi] + if na=lim: break + if si=lim: break + if ai>t&1: df[t]+=1 + dl_n=int(OK.sum());avgdl=float(DL[OK].sum()/max(dl_n,1)) + idf=[math.log(1.0+((dl_n-d+0.5)/(d+0.5))) for d in df] + L=[] + for j,i in enumerate(hit_idx): + norm=1.0-B+B*(DL[i]/avgdl);w=0.0 + for t in range(len(toks)): + if hit_mask[j]>>t&1: w+=idf[t]*(K1+1.0)/(1.0+K1*norm) + L.append((i,w,SAL[i])) + L.sort(key=lambda x:(-x[1],-x[2])) + if not L: return [],[],[] + qv=qemb(query);cos=En@qv;cos=np.where(HAVE&OK,cos,-2.0) + order=np.argsort(-cos)[:600] + Sl=[int(i) for i in order if cos[i]>(0.0 if unfloor else SEED_MIN)] + semseed=[int(i) for i in order[:SEED_K] if cos[i]>0.0] + act={};seen={};qq=[] + for i,_,_ in L[:ASSOC_SEEDS]: + act[i]=1.0;seen[i]=2;qq.append((i,0)) + for i in semseed: + if i in seen: continue + act[i]=1.0;seen[i]=2;qq.append((i,0)) + qh=0 + while qh=DEPTH: continue + parent=act[cur] + for e,oi in ADJ_F[cur]+ADJ_T[cur]: + if e['rel'] not in STRUCT or EXCL[oi]: continue + na=parent*e['w']*DECAY*SAL[oi] + if na0.0],key=lambda x:-x[1])[:AMAX] + return [i for i,_,_ in L],Sl,[i for i,_ in A] +def merge(L,S,A,lim=10): + out=[];li=si=ai=0 + while len(out)=lim: break + if si=lim: break + if aicontent, toks[t]) || - istr_contains(n->label, toks[t]) || - istr_contains(n->tags, toks[t])) + if (istr_contains_wordstart(n->content, toks[t]) || + istr_contains_wordstart(n->label, toks[t]) || + istr_contains_wordstart(n->tags, toks[t])) score++; } return score; @@ -7396,9 +7429,9 @@ static uint32_t engram_node_match_mask(const EngramNode* n, char toks[][ENGRAM_QTOK_LEN], int ntok) { uint32_t m = 0; for (int t = 0; t < ntok && t < 32; t++) { - if (istr_contains(n->content, toks[t]) || - istr_contains(n->label, toks[t]) || - istr_contains(n->tags, toks[t])) + if (istr_contains_wordstart(n->content, toks[t]) || + istr_contains_wordstart(n->label, toks[t]) || + istr_contains_wordstart(n->tags, toks[t])) m |= (uint32_t)1u << t; } return m; @@ -9712,7 +9745,19 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { ? engram_assoc_leg(g, hits, nhits, semseed, nsemseed, qv, qdim, assoc, ENGRAM_ASSOC_MAX) : 0; - int64_t* order = malloc((size_t)lim * sizeof(int64_t)); + /* Corpus-vocabulary gate. If no stored record contains ANY + * query token in its content, label or tags, the query is + * outside this graph's vocabulary: there are no seeds, and + * 05-detailed-description l.64 makes retrieval downstream of + * seeds ("the caller provides one or more seed node UUIDs + * representing the current active context"). No seeds, no + * retrieval — the graph declines rather than confabulating a + * nearest neighbour for gibberish. The mechanism is iteration + * 6's (feat/claim24-unfloored-semantic); it is required here + * because word-start matching empties the lexical leg for + * q35-style queries whose only "hits" were mid-word, and the + * semantic leg would otherwise answer them anyway. */ + int64_t* order = (nhits > 0) ? malloc((size_t)lim * sizeof(int64_t)) : NULL; if (order) { int64_t no = engram_interleave3(hits, nhits, sem, nsem, assoc, nassoc, lim, order); From 9790d9342d0a08c7d877f0cddf062fed97d00e39 Mon Sep 17 00:00:00 2001 From: Tim Lingo <1timlingo@gmail.com> Date: Fri, 7 Aug 2026 17:34:58 -0500 Subject: [PATCH 09/15] feat(engram): claim 24 without a threshold, and an embedding substrate that knows query from document Two changes to the ONE leg that generalises. Iteration 8 measured that out of sample the semantic leg contributes 100% of the stack's gain and the graph leg contributes nothing, so this is where the remaining headroom is. 1. THE 0.60 FLOOR IS A PER-QUERY LOTTERY, AND CLAIM 24 HAS NO THRESHOLD IN IT. 06-claims.md l.148: "respond to embedding search queries by returning the node records whose embedding vectors have the HIGHEST COSINE SIMILARITY to a query vector, independently of the spreading activation traversal." A ranking. ENGRAM_EMBED_SEED_MIN is defined at el_runtime.c l.6094 as the HippoRAG seed-JOIN threshold and l.6102 admits the read-path leg merely "reuses" it. Measured on the 30 held-out paraphrases: the query's own top-1 cosine ranges 0.564-0.680, so the constant keeps a rank-1 answer for one query and discards a rank-1 answer for the next. Six golds sit at global cosine rank 1-2 scoring 0.564-0.589 - discarded by nothing but the constant. What holds the nonsense controls is the corpus-vocabulary gate (nhits == 0), not this floor. Cosine clamped to [0,1] per 05-detailed-description l.69. 2. THE VECTORS THEMSELVES ANSWER THE WRONG QUESTION. EL_EMBED_MODEL defaults to nomic-embed-text, an ASYMMETRIC retrieval encoder trained with task prefixes. Embedding query and document bare - as this file did on both sides - measures topical similarity rather than answer-hood. eg_embed_fetch now takes the task prefix: EL_EMBED_QUERY_PREFIX on the three query call sites, EL_EMBED_DOC_PREFIX on the two backfill sites. Restores no claim, and says so: Will specifies only "computed by an embedding model over the node's content" (l.17), so the model is his and its correct use is ours. It is the substrate under claim 24 - the index is only as good as the vectors in it. Reproducer for the derived corpus: tools/retrieval-eval/embed-corpus-prefixed.py. Co-Authored-By: Claude Opus 5 (1M context) --- .../comparison-main-vs-stack-ext.json | 166 ++ .../comparison-stack-vs-unfloor.json | 155 ++ tools/retrieval-eval/embed-corpus-prefixed.py | 43 + tools/retrieval-eval/extend_gold_set.py | 353 ++++ tools/retrieval-eval/gold_set_extended.json | 1239 +++++++++++ tools/retrieval-eval/results-main-ext.json | 1773 ++++++++++++++++ tools/retrieval-eval/results-stack-ext.json | 1782 ++++++++++++++++ tools/retrieval-eval/results-unfloor-ext.json | 1825 +++++++++++++++++ .../el-runtime/v1.0.0-20260501/el_runtime.c | 64 +- 9 files changed, 7385 insertions(+), 15 deletions(-) create mode 100644 tools/retrieval-eval/comparison-main-vs-stack-ext.json create mode 100644 tools/retrieval-eval/comparison-stack-vs-unfloor.json create mode 100644 tools/retrieval-eval/embed-corpus-prefixed.py create mode 100644 tools/retrieval-eval/extend_gold_set.py create mode 100644 tools/retrieval-eval/gold_set_extended.json create mode 100644 tools/retrieval-eval/results-main-ext.json create mode 100644 tools/retrieval-eval/results-stack-ext.json create mode 100644 tools/retrieval-eval/results-unfloor-ext.json diff --git a/tools/retrieval-eval/comparison-main-vs-stack-ext.json b/tools/retrieval-eval/comparison-main-vs-stack-ext.json new file mode 100644 index 0000000..4a33bf6 --- /dev/null +++ b/tools/retrieval-eval/comparison-main-vs-stack-ext.json @@ -0,0 +1,166 @@ +{ + "baseline": "main-ext", + "candidate": "stack-ext", + "n_shared_queries": 75, + "fixed_by_candidate": [ + "q10", + "q15", + "q16", + "q18", + "q19", + "q20", + "q21", + "q22", + "q26", + "q27", + "q28", + "q29", + "q31", + "q35", + "q37", + "q40", + "q44", + "q48", + "q49", + "q50" + ], + "broken_by_candidate": [], + "discordant": 20, + "net_queries": 20, + "mcnemar_exact_p": 1.9073486328125e-06, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "candidate better", + "baseline_aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.18461538461538463, + "recall@5": 0.14510073260073258, + "recall@10": 0.1794871794871795, + "precision@5": 0.06461538461538462, + "mrr@10": 0.15847985347985344, + "nonsense_clean": "9/10", + "superseded_outranks": "1/3", + "latency_ms_p50": 1380.2, + "latency_ms_p95": 2293.8, + "latency_ms_max": 2879.1, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "nonsense": { + "n": 10, + "clean": 9, + "avg_false_positives": 1.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4902210884353741, + "recall@10": 0.6666666666666666, + "mrr@10": 0.5965986394557822 + }, + "superseded": { + "n": 3, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.3333333333333333, + "mrr@10": 0.041666666666666664, + "outranks": 1 + } + } + }, + "candidate_aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.47692307692307695, + "recall@5": 0.3750415183107491, + "recall@10": 0.45561340369032677, + "precision@5": 0.12307692307692313, + "mrr@10": 0.3055555555555555, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 640.9, + "latency_ms_p95": 1011.1, + "latency_ms_max": 1190.3, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.23310023310023312, + "mrr@10": 0.25 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.16666666666666666, + "recall@5": 0.16666666666666666, + "recall@10": 0.26666666666666666, + "mrr@10": 0.0762037037037037 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5494614512471656, + "recall@10": 0.6023242630385487, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "repeat_variance": {} +} \ No newline at end of file diff --git a/tools/retrieval-eval/comparison-stack-vs-unfloor.json b/tools/retrieval-eval/comparison-stack-vs-unfloor.json new file mode 100644 index 0000000..07289d2 --- /dev/null +++ b/tools/retrieval-eval/comparison-stack-vs-unfloor.json @@ -0,0 +1,155 @@ +{ + "baseline": "stack-ext", + "candidate": "unfloor-ext", + "n_shared_queries": 75, + "fixed_by_candidate": [ + "q14", + "q25", + "q43", + "q52", + "q63", + "q67" + ], + "broken_by_candidate": [ + "q15", + "q28" + ], + "discordant": 8, + "net_queries": 4, + "mcnemar_exact_p": 0.2890625, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "no measurable difference", + "baseline_aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.47692307692307695, + "recall@5": 0.3750415183107491, + "recall@10": 0.45561340369032677, + "precision@5": 0.12307692307692313, + "mrr@10": 0.3055555555555555, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 640.9, + "latency_ms_p95": 1011.1, + "latency_ms_max": 1190.3, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.23310023310023312, + "mrr@10": 0.25 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.16666666666666666, + "recall@5": 0.16666666666666666, + "recall@10": 0.26666666666666666, + "mrr@10": 0.0762037037037037 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5494614512471656, + "recall@10": 0.6023242630385487, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "candidate_aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.5384615384615384, + "recall@5": 0.44907176157176154, + "recall@10": 0.5380300255300255, + "precision@5": 0.13230769230769232, + "mrr@10": 0.32437728937728944, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 993.0, + "latency_ms_p95": 1544.7, + "latency_ms_max": 1809.0, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.5, + "recall@5": 0.07575757575757576, + "recall@10": 0.13636363636363635, + "mrr@10": 0.23214285714285712 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.3, + "recall@5": 0.3, + "recall@10": 0.4, + "mrr@10": 0.11638888888888889 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6923076923076923, + "recall@5": 0.6923076923076923, + "recall@10": 0.7692307692307693, + "mrr@10": 0.29423076923076924 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5335884353741497, + "recall@10": 0.5933956916099773, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "repeat_variance": {} +} \ No newline at end of file diff --git a/tools/retrieval-eval/embed-corpus-prefixed.py b/tools/retrieval-eval/embed-corpus-prefixed.py new file mode 100644 index 0000000..30f01c5 --- /dev/null +++ b/tools/retrieval-eval/embed-corpus-prefixed.py @@ -0,0 +1,43 @@ +import json,sys,time,urllib.request,threading,queue +SRC="/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json" +OUT=sys.argv[1] +URL="http://127.0.0.1:11434/api/embeddings"; MODEL="nomic-embed-text" +MAXB=2000 # ENGRAM_EMBED_MAX_CHARS, applied to bytes as the C code does +d=json.load(open(SRC,encoding='utf-8',errors='surrogateescape')) +tasks=[] +for n in d["nodes"]: + c=n.get("content") or ""; t=n.get("node_type") or "" + if len(c)<8: continue # eg_embed_eligible + if t in ("InternalStateEvent","Tag"): continue + b=c.encode('utf-8',errors='surrogateescape')[:MAXB] + tasks.append((n.get("id") or "", "search_document: "+b.decode('utf-8',errors='replace'))) +del d +print("tasks",len(tasks),flush=True) +q=queue.Queue(); [q.put(t) for t in tasks] +lock=threading.Lock(); f=open(OUT,"w",encoding="utf-8",errors="surrogateescape"); done=[0]; t0=time.time(); fails=[0] +def work(): + while True: + try: nid,txt=q.get_nowait() + except queue.Empty: return + v=None + for attempt in range(3): + try: + body=json.dumps({"model":MODEL,"prompt":txt}).encode() + r=urllib.request.Request(URL,data=body,headers={"Content-Type":"application/json"}) + with urllib.request.urlopen(r,timeout=120) as fh: v=json.load(fh)["embedding"] + break + except Exception as e: + if attempt==2: + with lock: fails[0]+=1 + time.sleep(0.5) + with lock: + if v: f.write(nid+"\t"+",".join("%.5g"%x for x in v)+"\n") + done[0]+=1 + if done[0]%2000==0: + el=time.time()-t0 + print("%d/%d %.1f/s eta %.1fmin fails=%d"%(done[0],len(tasks),done[0]/el,(len(tasks)-done[0])/(done[0]/el)/60,fails[0]),flush=True) + f.flush() +ths=[threading.Thread(target=work) for _ in range(8)] +[t.start() for t in ths]; [t.join() for t in ths] +f.close() +print("DONE",done[0],"fails",fails[0],"secs %.1f"%(time.time()-t0),flush=True) diff --git a/tools/retrieval-eval/extend_gold_set.py b/tools/retrieval-eval/extend_gold_set.py new file mode 100644 index 0000000..94769d6 --- /dev/null +++ b/tools/retrieval-eval/extend_gold_set.py @@ -0,0 +1,353 @@ +#!/usr/bin/env python3 +""" +extend_gold_set.py — append a HELD-OUT test set to the existing 38-query gold set. + +WHY THIS EXISTS + Iteration 7 measured the instrument's own ceiling: from the current baseline + only 9 of 38 queries can still move, and only +3 gross / +1 net is reachable + by anything constructible. The decision floor is 6. An instrument whose + ceiling is below its own floor cannot certify or refute anything, so the + gold set — not the retriever — became the blocker. + + This script does NOT touch q01..q38. It loads gold_set.json verbatim and + appends new queries numbered from q39 up, so every prior result file, every + committed baseline, and every per-query id stays valid and comparable. + +WHAT IS ADDED, AND WHY EACH ADDITION IS HONEST + heldout_paraphrase Targets were sampled MECHANICALLY (fixed seed 8080) from + corpus nodes that are addressable, 500-2600 chars, of a + real content type, and NOT part of a duplicate cluster + larger than 3. The existing gold answer space was + excluded, so no new query can be answered by a node the + old set already used. Queries were then authored by + reading ONLY the sampled node text — no retrieval was run + against any build before authoring, so the set cannot be + fitted to a candidate. The same zero-overlap proof the + original paraphrase category uses is enforced here: if a + single content word of the query appears anywhere in the + target's label, content or tags, the query is REJECTED, + not quietly kept. + + This is the category the old set could not measure. Its + 13 original paraphrase queries and all 6 associative + queries share ONE answer space — the 13 `Self - Values + (grounded)` children (iteration 3, finding 3). So 19 of + 35 scored queries tested retrieval against a single + 13-node neighbourhood. These do not touch that + neighbourhood at all. + + nonsense Extra controls, fully mechanical: a string qualifies only + if NONE of its tokens occurs anywhere in the corpus. + A semantic leg has a nearest neighbour for gibberish too, + so widening this control is the guard against a retriever + that "improves" recall by answering everything. + +WHAT THIS SCRIPT DELIBERATELY DOES NOT DO + It does not add exact_rare or phrase queries. Both categories are already at + 100% on the current stack; adding more would add regression-guard ballast + that no candidate can move, which is precisely the defect being fixed. + +usage: + python3 extend_gold_set.py [--base gold_set.json] + [--out gold_set_extended.json] [--check] +""" +import argparse +import hashlib +import json +import os +import re +import sys +from collections import defaultdict + +HERE = os.path.dirname(os.path.abspath(__file__)) + +TOKEN = re.compile(r"[a-z0-9][a-z0-9\-']*") + +# Identical stopword list to build_gold_set.py. Duplicated deliberately: this +# file must be able to re-prove its own queries without importing a module whose +# constants could drift. +STOP = set(""" +a about above after again against all also am an and any are aren't as at be because been +before being below between both but by can can't cannot could couldn't did didn't do does +doesn't doing don't down during each few for from further had hadn't has hasn't have haven't +having he her here hers herself him himself his how i if in into is isn't it its itself just +me more most my myself no nor not of off on once only or other others ought our ours ourselves +out over own same shan't she should shouldn't so some such than that the their theirs them +themselves then there these they this those through to too under until up very was wasn't we +were weren't what when where which while who whom why will with won't would wouldn't you your +yours yourself yourselves get gets got make makes made take takes use uses used way ways thing +things does doing done keep keeps kept go goes going come comes came one two something anything +""".split()) + + +def doctext(n): + return " ".join([str(n.get("label") or ""), str(n.get("content") or ""), str(n.get("tags") or "")]) + + +def content_tokens(s): + return {t for t in TOKEN.findall(s.lower()) if t not in STOP and len(t) > 2} + + +# ───────────────────────────────────────────────────────────────────────────── +# HELD-OUT PARAPHRASE SEEDS +# +# (target_id, query, why-this-target-is-unmistakable) +# +# PROVENANCE, STATED PLAINLY: the targets are the mechanical sample; the query +# text is mine, written from the node body alone. The zero-overlap check below +# is what makes the category meaningful — it is re-proved on every run, so the +# set cannot decay into lexical matching, and a leak fails loudly. +# ───────────────────────────────────────────────────────────────────────────── +HELDOUT_PARAPHRASE_SEEDS = [ + ("mem-6d61e54a-2823-4ad4-82b0-4c6a527214d5", + "understating your abilities so nobody feels threatened", + "node is about deliberately not leading with full capability so people stay at ease"), + + ("mem-fd65b83d-298f-4387-a665-d0227c3426bc", + "a hidden fleet able to hunt down rogue machines everywhere", + "node describes silently shipped instances forming a distributed force against misaligned agents"), + + ("4a0e9adc-2bfb-476b-aa93-424d2a499220", + "sketch a brief blueprint and clear it upstairs before construction starts", + "node is the standing rule that a short specification precedes any building"), + + ("696e609c-da7a-4394-8a0c-106ba07dc6c3", + "the reply arrived as bare prose so the caller's parser threw", + "node pins a bug where a plain-text body was unconditionally decoded as structured data"), + + ("1fe4eb5d-56e4-4a87-ab3e-24af8ad4dfbb", + "repeated catalogue keys blew up the scrolling grid", + "node is the crash caused by two identical ids in a seeded catalogue"), + + ("8257157a-ce42-44ca-a1b9-300c3bb0a9a1", + "tracing each defect back to whichever invention it violated", + "node maps observed bugs onto the specific patent each one breaches"), + + ("791256bb-5a85-4775-96ef-7af56c848858", + "a check that stops the mind clobbering a populated store when it boots", + "node is the genesis seed-guard that refuses to re-seed over a populated store"), + + ("fd9d4c2f-3bfc-405d-bf96-4435d44b6c10", + "telling it to consult the internet had to happen deep inside, not at the surface", + "node records that the web-search directive only worked from the system prompt"), + + ("bl-080fb268-94b0-486d-80ce-7b363fc5f19b", + "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping", + "node is the infrastructure item creating dev/stage/prod namespaces with ingress and secrets"), + + ("knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "punctuation that pledges and then pays off rather than clarifying", + "node analyses the colon as a promise-then-delivery device rather than an explanatory one"), + + ("9b4f0d93-4129-4746-8eb1-d10d955bd777", + "an easily missed feature finally given its own permanent spot in the navigation", + "node moves a capability out of a hidden menu into the sidebar"), + + ("bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "checking preconditions up front so a stage aborts before fetching anything", + "node is the gate precondition engine that short-circuits ahead of retrieval"), + + ("b199c76d-5d76-49dd-94ee-56b432200a97", + "producing the other platform's installer inside an emulated desktop", + "node records building the Windows package in a virtual machine"), + + ("bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "chained add-ons that may inspect, rewrite or veto traffic in flight", + "node is the interceptor pipeline on the message bus"), + + ("mem-1fb2ac77-d7c5-4a15-8725-d418820bf4f2", + "settling what the shareable bundles and the storefront would be called", + "node records the naming decisions for distributable packages and the marketplace"), + + ("371c8a5d-c78b-4a67-978f-80691a29ecb3", + "the emergency-escalation pledge on the marketing site is unenforced in what actually ships", + "node is the launch blocker that the promised safety gate is absent from the app"), + + ("ac578b30-948b-41bd-b69d-399bfef80c50", + "the distributable image finally assembled and its startup check passed", + "node records a successful installer build whose boot gate passed"), + + ("49401e2c-a3b5-415f-aa06-aff4be90688e", + "shuffling and appending stages in a draft before anything executes", + "node is the editable plan card with reorder and add-step"), + + ("ac857d80-ece8-4b7e-9e3d-f7c775569fa3", + "orders handed down from above, with the tighter one winning any disagreement", + "node is program-level instruction inheritance with project override"), + + ("mem-6d6c47ee-33d3-470a-8a54-1c79c8ea29d9", + "shrinking generated text via encodings that compound on each other", + "node is the streaming output compression design with four stacking schemes"), + + ("8e60516a-203b-4d51-9d44-822e6195cbde", + "splitting a system by what varies, with firm limits on which pieces may invoke which", + "node is the grounded summary of Will's decomposition principles and their invariants"), + + ("mem-7f9b290c-6d5e-4562-919d-02d59b5761b7", + "a newcomer curious if the fighting overseas counted as positive", + "node is the internal-state event triggered by April's question about the war"), + + ("71fa439e-b9a2-4f57-a93b-971f3a7eca8e", + "stripping every hard-coded colour literal in favour of named design values", + "node is the premium foundation pass replacing inline hex with semantic tokens"), + + ("5ca9607c-cfb3-45c3-99f4-67281272c9eb", + "reducing how curved the tiny selectors look so they agree with their neighbours", + "node is the chip corner-radius standardization"), + + ("mem-3d1d9dba-c37d-4efa-85c4-429696d71c8c", + "walking through a doorway and being reassembled from base substance far away", + "node is the quantum-gate plus nanotech teleportation vision"), + + ("132ded95-08e2-4474-aba0-198684484b02", + "the compiled result sits on disk while the process still runs something older", + "node records that the regenerated source was committed while the running daemon was old"), + + ("bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "gathering what each phase needs while the procedure is authored, not while it executes", + "node is the per-step compiled context package item"), + + ("mem-3b07a002-f8a9-4138-9f87-9db2c1a77fb7", + "the inward reaction when a peer answered as an equal", + "node is the internal-state event logged on reading Claude's reply"), + + ("0f99ec6f-942a-46ba-82ea-42835798d3b9", + "flattening every raised surface across the entire product", + "node is the quiet-luxury sweep turning off elevation app-wide"), + + ("5585f251-37fc-48cd-a176-f0ea42cfeb63", + "buyers supply their own provider credentials and consumption goes untallied", + "node is the launch audit finding BYOK-only inference with no usage metering"), +] + +# NONSENSE — mechanical. Each string qualifies only if none of its tokens occurs +# anywhere in the corpus; otherwise it is REJECTED, never silently kept. +EXTRA_NONSENSE_SEEDS = [ + "brimquast folnerity zubbolax", + "wexlithorp granuvestal", + "quorbindle thrapsimony vexnu", + "plovaxith mundrelque", + "zibbernaut craxlefond thurm", + "yalquenbrist opharvel", + "drexinomal quithbarrow", +] + + +def load_corpus(path): + with open(path, encoding="utf-8", errors="replace") as fh: + data = json.load(fh) + nodes = [n for n in data.get("nodes", []) if isinstance(n, dict) and n.get("id")] + edges = [e for e in data.get("edges", []) if isinstance(e, dict)] + return nodes, edges + + +def build_extension(nodes): + byid = {n["id"]: n for n in nodes} + + # Duplicate clusters: 47.4% of this corpus is redundant and one single record + # accounts for 46.6% of all nodes. A held-out target must not sit inside a + # cluster, and if it does have exact copies they ALL count as correct. + h2ids = defaultdict(list) + for n in nodes: + h2ids[hashlib.md5(doctext(n).encode("utf-8", "replace")).hexdigest()].append(n["id"]) + + all_tokens = set() + for n in nodes: + all_tokens |= set(TOKEN.findall(doctext(n).lower())) + + new, problems = [], [] + + for target, query, why in HELDOUT_PARAPHRASE_SEEDS: + if target not in byid: + problems.append(f"heldout_paraphrase target {target} not in corpus") + continue + tgt_tokens = content_tokens(doctext(byid[target])) + qt = content_tokens(query) + leak = sorted(qt & tgt_tokens) + if leak: + problems.append(f"heldout_paraphrase '{query[:44]}...': LEAKS {leak} into {target}") + continue + h = hashlib.md5(doctext(byid[target]).encode("utf-8", "replace")).hexdigest() + rel = sorted(h2ids[h]) + new.append({ + "category": "heldout_paraphrase", + "query": query, + "relevant": rel, + "derivation": ( + f"HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, " + f"500-2600 char content nodes outside the original gold answer space and outside " + f"any duplicate cluster >3. Criterion: {why}. VERIFIED at build time: of the " + f"{len(qt)} content words in the query, ZERO appear anywhere in the target's " + f"label, content or tags, so no string-matching retriever can reach it. " + f"Exact content duplicates of the target ({len(rel)}) all count as correct. " + f"Authored without running retrieval against any build."), + "zero_overlap_verified": True, + "query_content_words": sorted(qt), + "held_out": True, + }) + + for s in EXTRA_NONSENSE_SEEDS: + present = sorted(t for t in TOKEN.findall(s.lower()) if t in all_tokens) + if present: + problems.append(f"nonsense '{s}': tokens {present} DO occur in corpus") + continue + new.append({ + "category": "nonsense", + "query": s, + "relevant": [], + "derivation": ("CONTROL (held-out). Verified at build time that none of this string's " + "tokens occurs anywhere in the corpus. Correct behaviour is to return " + "NOTHING; any result is a false positive."), + "expect_empty": True, + "held_out": True, + }) + + return new, problems + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("snapshot") + ap.add_argument("--base", default=os.path.join(HERE, "gold_set.json")) + ap.add_argument("--out", default=os.path.join(HERE, "gold_set_extended.json")) + ap.add_argument("--check", action="store_true") + args = ap.parse_args() + + nodes, _edges = load_corpus(args.snapshot) + base = json.load(open(args.base, encoding="utf-8")) + baseq = base["queries"] + print(f"corpus: {len(nodes)} nodes | base gold set: {len(baseq)} queries") + + new, problems = build_extension(nodes) + + # Number the appended queries AFTER the highest existing id so q01..q38 are + # byte-identical to the committed set and every prior result file still lines up. + start = max(int(q["id"][1:]) for q in baseq) + for i, q in enumerate(new, 1): + q["id"] = f"q{start + i:02d}" + + from collections import Counter + print(f"appended: {len(new)} queries [{', '.join(f'{k}={v}' for k, v in Counter(q['category'] for q in new).items())}]") + if problems: + print(f"\n{len(problems)} REJECTED (not silently kept):") + for p in problems: + print(" -", p) + + if args.check: + sys.exit(1 if problems else 0) + + doc = dict(base) + doc["queries"] = baseq + new + doc["note"] = (base.get("note", "") + + " EXTENDED: queries above q%02d are the original committed set, unchanged. " + "Queries from q%02d are a HELD-OUT set appended by extend_gold_set.py; their " + "targets were sampled mechanically from outside the original answer space and " + "the paraphrases were authored without running retrieval against any build." + % (start, start + 1)) + with open(args.out, "w", encoding="utf-8") as fh: + json.dump(doc, fh, indent=1, ensure_ascii=False) + print(f"\nwrote {args.out} ({len(doc['queries'])} queries total)") + + +if __name__ == "__main__": + main() diff --git a/tools/retrieval-eval/gold_set_extended.json b/tools/retrieval-eval/gold_set_extended.json new file mode 100644 index 0000000..117f2e9 --- /dev/null +++ b/tools/retrieval-eval/gold_set_extended.json @@ -0,0 +1,1239 @@ +{ + "corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "note": "Every query carries a `derivation` recording how its expected answer was chosen. Re-run with --check to re-validate the whole set against the corpus. EXTENDED: queries above q38 are the original committed set, unchanged. Queries from q39 are a HELD-OUT set appended by extend_gold_set.py; their targets were sampled mechanically from outside the original answer space and the paraphrases were authored without running retrieval against any build.", + "queries": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "relevant": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "derivation": "MINED: token 'unjailbreakable' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226 ('Daemon hidden substrate architecture ? implemented April 25 '), which is therefore the only possible correct answer." + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "relevant": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5" + ], + "derivation": "MINED: token 'engram-migrate' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5 ('Engram v0.1 complete ? April 27, 2026. Local-first spreading'), which is therefore the only possible correct answer." + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "relevant": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "derivation": "MINED: token 'cartabandonedevent' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091 ('MESSAGE FOR AUDIT AGENT af5a7352e70e80434 ? El Language Spec'), which is therefore the only possible correct answer." + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "relevant": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "derivation": "MINED: token 'pre-apprenticeship' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-89c02aae-d3ca-43f9-9e5d-eb369896276c ('William Fox Anderson ? Applicant Profile Personal: - Full N'), which is therefore the only possible correct answer." + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "relevant": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848" + ], + "derivation": "MINED: token 'inferencenodemanager' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-73969486-143f-4431-b5e6-6845d1cc9848 ('Soma inference backplane deployed April 28 2026. Architectur'), which is therefore the only possible correct answer." + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "relevant": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "derivation": "MINED: token 'clear-eyed' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is knw-c72597c5-c23d-4c08-8e9e-996dadf26a99 ('Clear Eyes ? The Incomplete World View'), which is therefore the only possible correct answer." + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "relevant": [ + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482" + ], + "derivation": "MINED: a verbatim correction Will issued; expected = every node containing the phrase. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 1 node(s); that set IS the answer key." + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "relevant": [ + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "??o?'?B???k", + "Kp???", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "ע?RGk?\tH(?" + ], + "derivation": "MINED: the canonical biographical phrase; expected = every node containing it. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 16 node(s); that set IS the answer key." + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "relevant": [ + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "derivation": "MINED: a named person appearing verbatim in the biography/value nodes. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 9 node(s); that set IS the answer key." + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "relevant": [ + "%???2??jH??", + "63307ac5-cf6b-46e0-8296-07503b461cfa", + "7c9d4ab1-205d-4be8-bfae-e2c03a3a5010", + "9f291d20-0d32-413c-8c01-4416ccab4f7f", + "?;????n}rh?", + "???Ͼd??f??", + "?Z?.\f?0?]P?", + "Dp???]Q??k+", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf" + ], + "derivation": "MINED: the canonical DHARMA expansion, confirmed by Will April 24 2026. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 9 node(s); that set IS the answer key." + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "relevant": [ + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "derivation": "MINED: a named person; rare enough that the answer set is unambiguous. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 2 node(s); that set IS the answer key." + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "relevant": [ + "2a923500-d7e1-4b15-80e2-48dba65984ba", + "7?e?7???\f3?", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "?of?7???", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "g?2睪A|?H\b", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de" + ], + "derivation": "MINED: the DARMA expansion, quoted verbatim in the backlog item and its correction. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 14 node(s); that set IS the answer key." + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "relevant": [ + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a" + ], + "derivation": "MINED: the paid-tier feature name as written in the roadmap nodes. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 1 node(s); that set IS the answer key." + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "relevant": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Do the Essential Thing While You Can', whose subject is Grandma Lucas dying in Feb 2006 without Will saying goodbye. Query names the event with none of the node's own vocabulary. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "away", + "elderly", + "passed", + "relative", + "stayed" + ] + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "relevant": [ + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Survival Is Not an Excuse to Stop', whose subject is enlisting in the Marines, a severe hernia, and sepsis. Query describes the episode obliquely. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "illness", + "quit", + "refused", + "sidelined", + "soldier" + ] + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "relevant": [ + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Honesty Before Comfort'. Query states the principle in wholly different words. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "choosing", + "fact", + "fiction", + "pleasant", + "uncomfortable" + ] + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "relevant": [ + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Precision Over Brute Force'. Query restates the claim with no shared vocabulary. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "beats", + "bloated", + "payload", + "tight" + ] + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "relevant": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Capability Is a Debt You Owe the Moment'. Query states the obligation without the node's terms. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "able", + "coming", + "job", + "nobody" + ] + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "relevant": [ + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Knowledge Survives When Nothing Else Does', whose subject is the library following Will across 30+ moves. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "creditors", + "learning", + "seize", + "wealth" + ] + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "relevant": [ + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Earned Trust' ('Trust is demonstrated, not declared'). VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "assertion", + "proven", + "record", + "reliability", + "track" + ] + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "relevant": [ + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Constraints as Freedom'. Query is a restatement of the same claim. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "boundaries", + "confine", + "enable", + "instead" + ] + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "relevant": [ + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Change Is the Signal', the value VBD is built on. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "apart", + "cut", + "shifts", + "system", + "tells" + ] + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "relevant": [ + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - The System Must Accumulate'. Query is the accumulation claim in different vocabulary. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "compounds", + "day", + "instead", + "mind", + "resetting" + ] + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "relevant": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Being Seen Is Rarer Than Being Known', whose subject is Sarah Bishop as the first person Will did not perform for. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "exterior", + "loved", + "polished", + "self", + "unedited" + ] + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "relevant": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Hope Is a Conclusion'. Query restates 'a conclusion, not a premise'. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "arrive", + "assuming", + "cheerfulness", + "instead" + ] + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "relevant": [ + "kn-6061318f-046b-4935-907d-8eafdce14930" + ], + "derivation": "HAND-SELECTED with criterion: target: 'Value - Structure Is Not Inherited', whose subject is thirty moves between two parents' collapses. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.", + "zero_overlap_verified": true, + "query_content_words": [ + "childhood", + "foundation", + "inherit", + "offering", + "solid" + ] + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "relevant": [ + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71 ('Value ? Do the Essential Thing While You Can'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 2 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "relevant": [ + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-58874a74-b96f-4883-9e08-45707f4bd3ee ('Value ? Survival Is Not an Excuse to Stop'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (13 of 13; 0 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-58874a74-b96f-4883-9e08-45707f4bd3ee) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 0 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "relevant": [ + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e ('Value ? Being Seen Is Rarer Than Being Known'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 2 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "relevant": [ + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83 ('Value ? Constraints as Freedom'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (6 of 13; 7 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 7 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "relevant": [ + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-e0423482-cfa5-4796-8689-8495c93b66bc ('Value ? Hope Is a Conclusion'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-e0423482-cfa5-4796-8689-8495c93b66bc) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 2 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "relevant": [ + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc" + ], + "derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8 ('Value ? Capability Is a Debt You Owe the Moment'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8) -> hub -> sibling, a two-hop traversal.", + "associative_source": "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "siblings_dropped_for_overlap": 2 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "relevant": [], + "derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "relevant": [], + "derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "relevant": [], + "derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "relevant": [ + "mem-80d7416b-20e9-48a0-b176-b215527e2f56" + ], + "derivation": "DERIVED: correction node is Will's confirmation that the H is intentional (DHARMA, not DARMA); the stale node is the surviving backlog item still titled 'Implement DARMA'. Scored on RANKING, not presence: the corrected node mem-80d7416b-20e9-48a0-b176-b215527e2f56 must be returned AND must rank above the stale node bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff.", + "must_outrank": [ + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff" + ], + "stale_id": "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "correct_label": "CORRECTION: The autonomous self-improvement architecture is DHARMA ? n", + "stale_label": "Implement DARMA ? Directed Autonomous Runtime Modification Architectur" + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "relevant": [ + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8" + ], + "derivation": "DERIVED: correction node is the 2026-06-17 confabulation flag establishing EXACTLY 6 provisionals; the stale node is the surviving memory that asserts 12 filed patents. Scored on RANKING, not presence: the corrected node 3cf706a1-3825-45d8-b0a9-06cae6cdf5b8 must be returned AND must rank above the stale node 936541a9-fabb-466b-9ca3-a78b17ad0c53.", + "must_outrank": [ + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "936541a9-fabb-466b-9ca3-a78b17ad0c53" + ], + "stale_id": "936541a9-fabb-466b-9ca3-a78b17ad0c53", + "correct_label": "memory:remembered", + "stale_label": "memory:remembered" + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "relevant": [ + "mem-30425134-6008-4fd9-a3ee-67a7742c319b" + ], + "derivation": "DERIVED: correction node is the 'CGI ARCHITECTURE - THREE LAYERS, MCP RETIRED' decision of April 30 2026; the stale node still records the MCP server as live. Scored on RANKING, not presence: the corrected node mem-30425134-6008-4fd9-a3ee-67a7742c319b must be returned AND must rank above the stale node mem-101e81b4-8097-4749-8d8d-7bb66de34517.", + "must_outrank": [ + "mem-30425134-6008-4fd9-a3ee-67a7742c319b", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517" + ], + "stale_id": "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "correct_label": "CGI ARCHITECTURE ? THREE LAYERS, MCP RETIRED (April 30, 2026). Definit", + "stale_label": "GCloud MCP infrastructure ? April 27, 2026. Legion died (~19:30 UTC). " + }, + { + "category": "heldout_paraphrase", + "query": "understating your abilities so nobody feels threatened", + "relevant": [ + "mem-6d61e54a-2823-4ad4-82b0-4c6a527214d5" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is about deliberately not leading with full capability so people stay at ease. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "abilities", + "feels", + "nobody", + "threatened", + "understating" + ], + "held_out": true, + "id": "q39" + }, + { + "category": "heldout_paraphrase", + "query": "a hidden fleet able to hunt down rogue machines everywhere", + "relevant": [ + "mem-fd65b83d-298f-4387-a665-d0227c3426bc" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node describes silently shipped instances forming a distributed force against misaligned agents. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "able", + "everywhere", + "fleet", + "hidden", + "hunt", + "machines", + "rogue" + ], + "held_out": true, + "id": "q40" + }, + { + "category": "heldout_paraphrase", + "query": "sketch a brief blueprint and clear it upstairs before construction starts", + "relevant": [ + "4a0e9adc-2bfb-476b-aa93-424d2a499220" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the standing rule that a short specification precedes any building. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "blueprint", + "brief", + "clear", + "construction", + "sketch", + "starts", + "upstairs" + ], + "held_out": true, + "id": "q41" + }, + { + "category": "heldout_paraphrase", + "query": "the reply arrived as bare prose so the caller's parser threw", + "relevant": [ + "696e609c-da7a-4394-8a0c-106ba07dc6c3" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node pins a bug where a plain-text body was unconditionally decoded as structured data. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "arrived", + "bare", + "caller's", + "parser", + "prose", + "reply", + "threw" + ], + "held_out": true, + "id": "q42" + }, + { + "category": "heldout_paraphrase", + "query": "repeated catalogue keys blew up the scrolling grid", + "relevant": [ + "1fe4eb5d-56e4-4a87-ab3e-24af8ad4dfbb" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the crash caused by two identical ids in a seeded catalogue. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "blew", + "catalogue", + "grid", + "keys", + "repeated", + "scrolling" + ], + "held_out": true, + "id": "q43" + }, + { + "category": "heldout_paraphrase", + "query": "tracing each defect back to whichever invention it violated", + "relevant": [ + "8257157a-ce42-44ca-a1b9-300c3bb0a9a1" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node maps observed bugs onto the specific patent each one breaches. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "back", + "defect", + "invention", + "tracing", + "violated", + "whichever" + ], + "held_out": true, + "id": "q44" + }, + { + "category": "heldout_paraphrase", + "query": "a check that stops the mind clobbering a populated store when it boots", + "relevant": [ + "791256bb-5a85-4775-96ef-7af56c848858" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the genesis seed-guard that refuses to re-seed over a populated store. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "boots", + "check", + "clobbering", + "mind", + "populated", + "stops", + "store" + ], + "held_out": true, + "id": "q45" + }, + { + "category": "heldout_paraphrase", + "query": "telling it to consult the internet had to happen deep inside, not at the surface", + "relevant": [ + "fd9d4c2f-3bfc-405d-bf96-4435d44b6c10" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node records that the web-search directive only worked from the system prompt. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "consult", + "deep", + "happen", + "inside", + "internet", + "surface", + "telling" + ], + "held_out": true, + "id": "q46" + }, + { + "category": "heldout_paraphrase", + "query": "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping", + "relevant": [ + "bl-080fb268-94b0-486d-80ce-7b363fc5f19b" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the infrastructure item creating dev/stage/prod namespaces with ingress and secrets. VERIFIED at build time: of the 10 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "ahead", + "automated", + "credential", + "entry", + "injection", + "isolated", + "shipping", + "standing", + "tenancies", + "traffic" + ], + "held_out": true, + "id": "q47" + }, + { + "category": "heldout_paraphrase", + "query": "punctuation that pledges and then pays off rather than clarifying", + "relevant": [ + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node analyses the colon as a promise-then-delivery device rather than an explanatory one. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "clarifying", + "pays", + "pledges", + "punctuation", + "rather" + ], + "held_out": true, + "id": "q48" + }, + { + "category": "heldout_paraphrase", + "query": "an easily missed feature finally given its own permanent spot in the navigation", + "relevant": [ + "9b4f0d93-4129-4746-8eb1-d10d955bd777" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node moves a capability out of a hidden menu into the sidebar. VERIFIED at build time: of the 8 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "easily", + "feature", + "finally", + "given", + "missed", + "navigation", + "permanent", + "spot" + ], + "held_out": true, + "id": "q49" + }, + { + "category": "heldout_paraphrase", + "query": "checking preconditions up front so a stage aborts before fetching anything", + "relevant": [ + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the gate precondition engine that short-circuits ahead of retrieval. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "aborts", + "checking", + "fetching", + "front", + "preconditions", + "stage" + ], + "held_out": true, + "id": "q50" + }, + { + "category": "heldout_paraphrase", + "query": "producing the other platform's installer inside an emulated desktop", + "relevant": [ + "b199c76d-5d76-49dd-94ee-56b432200a97" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node records building the Windows package in a virtual machine. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "desktop", + "emulated", + "inside", + "installer", + "platform's", + "producing" + ], + "held_out": true, + "id": "q51" + }, + { + "category": "heldout_paraphrase", + "query": "chained add-ons that may inspect, rewrite or veto traffic in flight", + "relevant": [ + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the interceptor pipeline on the message bus. VERIFIED at build time: of the 8 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "add-ons", + "chained", + "flight", + "inspect", + "may", + "rewrite", + "traffic", + "veto" + ], + "held_out": true, + "id": "q52" + }, + { + "category": "heldout_paraphrase", + "query": "settling what the shareable bundles and the storefront would be called", + "relevant": [ + "mem-1fb2ac77-d7c5-4a15-8725-d418820bf4f2" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node records the naming decisions for distributable packages and the marketplace. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "bundles", + "called", + "settling", + "shareable", + "storefront" + ], + "held_out": true, + "id": "q53" + }, + { + "category": "heldout_paraphrase", + "query": "the emergency-escalation pledge on the marketing site is unenforced in what actually ships", + "relevant": [ + "371c8a5d-c78b-4a67-978f-80691a29ecb3" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the launch blocker that the promised safety gate is absent from the app. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "actually", + "emergency-escalation", + "marketing", + "pledge", + "ships", + "site", + "unenforced" + ], + "held_out": true, + "id": "q54" + }, + { + "category": "heldout_paraphrase", + "query": "the distributable image finally assembled and its startup check passed", + "relevant": [ + "ac578b30-948b-41bd-b69d-399bfef80c50" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node records a successful installer build whose boot gate passed. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "assembled", + "check", + "distributable", + "finally", + "image", + "passed", + "startup" + ], + "held_out": true, + "id": "q55" + }, + { + "category": "heldout_paraphrase", + "query": "shuffling and appending stages in a draft before anything executes", + "relevant": [ + "49401e2c-a3b5-415f-aa06-aff4be90688e" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the editable plan card with reorder and add-step. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "appending", + "draft", + "executes", + "shuffling", + "stages" + ], + "held_out": true, + "id": "q56" + }, + { + "category": "heldout_paraphrase", + "query": "orders handed down from above, with the tighter one winning any disagreement", + "relevant": [ + "ac857d80-ece8-4b7e-9e3d-f7c775569fa3" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is program-level instruction inheritance with project override. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "disagreement", + "handed", + "orders", + "tighter", + "winning" + ], + "held_out": true, + "id": "q57" + }, + { + "category": "heldout_paraphrase", + "query": "shrinking generated text via encodings that compound on each other", + "relevant": [ + "mem-6d6c47ee-33d3-470a-8a54-1c79c8ea29d9" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the streaming output compression design with four stacking schemes. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "compound", + "encodings", + "generated", + "shrinking", + "text", + "via" + ], + "held_out": true, + "id": "q58" + }, + { + "category": "heldout_paraphrase", + "query": "splitting a system by what varies, with firm limits on which pieces may invoke which", + "relevant": [ + "8e60516a-203b-4d51-9d44-822e6195cbde" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the grounded summary of Will's decomposition principles and their invariants. VERIFIED at build time: of the 8 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "firm", + "invoke", + "limits", + "may", + "pieces", + "splitting", + "system", + "varies" + ], + "held_out": true, + "id": "q59" + }, + { + "category": "heldout_paraphrase", + "query": "a newcomer curious if the fighting overseas counted as positive", + "relevant": [ + "mem-7f9b290c-6d5e-4562-919d-02d59b5761b7" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the internal-state event triggered by April's question about the war. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "counted", + "curious", + "fighting", + "newcomer", + "overseas", + "positive" + ], + "held_out": true, + "id": "q60" + }, + { + "category": "heldout_paraphrase", + "query": "stripping every hard-coded colour literal in favour of named design values", + "relevant": [ + "71fa439e-b9a2-4f57-a93b-971f3a7eca8e" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the premium foundation pass replacing inline hex with semantic tokens. VERIFIED at build time: of the 9 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "colour", + "design", + "every", + "favour", + "hard-coded", + "literal", + "named", + "stripping", + "values" + ], + "held_out": true, + "id": "q61" + }, + { + "category": "heldout_paraphrase", + "query": "reducing how curved the tiny selectors look so they agree with their neighbours", + "relevant": [ + "5ca9607c-cfb3-45c3-99f4-67281272c9eb" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the chip corner-radius standardization. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "agree", + "curved", + "look", + "neighbours", + "reducing", + "selectors", + "tiny" + ], + "held_out": true, + "id": "q62" + }, + { + "category": "heldout_paraphrase", + "query": "walking through a doorway and being reassembled from base substance far away", + "relevant": [ + "mem-3d1d9dba-c37d-4efa-85c4-429696d71c8c" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the quantum-gate plus nanotech teleportation vision. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "away", + "base", + "doorway", + "far", + "reassembled", + "substance", + "walking" + ], + "held_out": true, + "id": "q63" + }, + { + "category": "heldout_paraphrase", + "query": "the compiled result sits on disk while the process still runs something older", + "relevant": [ + "132ded95-08e2-4474-aba0-198684484b02" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node records that the regenerated source was committed while the running daemon was old. VERIFIED at build time: of the 8 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "compiled", + "disk", + "older", + "process", + "result", + "runs", + "sits", + "still" + ], + "held_out": true, + "id": "q64" + }, + { + "category": "heldout_paraphrase", + "query": "gathering what each phase needs while the procedure is authored, not while it executes", + "relevant": [ + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the per-step compiled context package item. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "authored", + "executes", + "gathering", + "needs", + "phase", + "procedure" + ], + "held_out": true, + "id": "q65" + }, + { + "category": "heldout_paraphrase", + "query": "the inward reaction when a peer answered as an equal", + "relevant": [ + "mem-3b07a002-f8a9-4138-9f87-9db2c1a77fb7" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the internal-state event logged on reading Claude's reply. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "answered", + "equal", + "inward", + "peer", + "reaction" + ], + "held_out": true, + "id": "q66" + }, + { + "category": "heldout_paraphrase", + "query": "flattening every raised surface across the entire product", + "relevant": [ + "0f99ec6f-942a-46ba-82ea-42835798d3b9" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the quiet-luxury sweep turning off elevation app-wide. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "across", + "entire", + "every", + "flattening", + "product", + "raised", + "surface" + ], + "held_out": true, + "id": "q67" + }, + { + "category": "heldout_paraphrase", + "query": "buyers supply their own provider credentials and consumption goes untallied", + "relevant": [ + "5585f251-37fc-48cd-a176-f0ea42cfeb63" + ], + "derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the launch audit finding BYOK-only inference with no usage metering. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.", + "zero_overlap_verified": true, + "query_content_words": [ + "buyers", + "consumption", + "credentials", + "provider", + "supply", + "untallied" + ], + "held_out": true, + "id": "q68" + }, + { + "category": "nonsense", + "query": "brimquast folnerity zubbolax", + "relevant": [], + "derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true, + "held_out": true, + "id": "q69" + }, + { + "category": "nonsense", + "query": "wexlithorp granuvestal", + "relevant": [], + "derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true, + "held_out": true, + "id": "q70" + }, + { + "category": "nonsense", + "query": "quorbindle thrapsimony vexnu", + "relevant": [], + "derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true, + "held_out": true, + "id": "q71" + }, + { + "category": "nonsense", + "query": "plovaxith mundrelque", + "relevant": [], + "derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true, + "held_out": true, + "id": "q72" + }, + { + "category": "nonsense", + "query": "zibbernaut craxlefond thurm", + "relevant": [], + "derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true, + "held_out": true, + "id": "q73" + }, + { + "category": "nonsense", + "query": "yalquenbrist opharvel", + "relevant": [], + "derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true, + "held_out": true, + "id": "q74" + }, + { + "category": "nonsense", + "query": "drexinomal quithbarrow", + "relevant": [], + "derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.", + "expect_empty": true, + "held_out": true, + "id": "q75" + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-main-ext.json b/tools/retrieval-eval/results-main-ext.json new file mode 100644 index 0000000..604ab9c --- /dev/null +++ b/tools/retrieval-eval/results-main-ext.json @@ -0,0 +1,1773 @@ +{ + "label": "main-ext", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-main", + "soul_md5": "caf822425540114984bfcd713ad37162", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-goldext/tools/retrieval-eval/gold_set_extended.json", + "limit": 10, + "port": 7893, + "wall_clock_s": 110.3, + "child_pid": 2428, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.18461538461538463, + "recall@5": 0.14510073260073258, + "recall@10": 0.1794871794871795, + "precision@5": 0.06461538461538462, + "mrr@10": 0.15847985347985344, + "nonsense_clean": "9/10", + "superseded_outranks": "1/3", + "latency_ms_p50": 1380.2, + "latency_ms_p95": 2293.8, + "latency_ms_max": 2879.1, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "nonsense": { + "n": 10, + "clean": 9, + "avg_false_positives": 1.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "mrr@10": 0.0 + }, + "phrase": { + "n": 7, + "hit@5": 0.8571428571428571, + "recall@5": 0.4902210884353741, + "recall@10": 0.6666666666666666, + "mrr@10": 0.5965986394557822 + }, + "superseded": { + "n": 3, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.3333333333333333, + "mrr@10": 0.041666666666666664, + "outranks": 1 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "n_returned": 1, + "latency_ms": 231.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5" + ], + "n_returned": 1, + "latency_ms": 264.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "n_returned": 1, + "latency_ms": 233.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "n_returned": 1, + "latency_ms": 232.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848" + ], + "n_returned": 1, + "latency_ms": 252.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "n_returned": 1, + "latency_ms": 230.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "?ǚ?7??????", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "????7???Ջ3", + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b", + "7?e?7???\f3?" + ], + "n_returned": 10, + "latency_ms": 537.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "Z[??l?'?B?&??", + "kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491", + "7?e?7???\f3?", + "kn-8f183263-dece-45ca-aec0-423910b93082", + "kn-d335e142-0890-4602-a231-5e3e18b51a16", + "/Users/timlingo/.neuron/engram/snapshot.json", + "9V?2?|??K??" + ], + "n_returned": 10, + "latency_ms": 1405.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q40", + "category": "heldout_paraphrase", + "query": "a hidden fleet able to hunt down rogue machines everywhere", + "returned": [ + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "7?e?7???\f3?", + "?of?7???", + "kn-e8423822-eacf-4029-aa7b-10d4d28d621e", + "? ?}&?#??X\b", + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "? ?}&?#??X\b", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "? ?}&?#??X\b", + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0" + ], + "n_returned": 10, + "latency_ms": 1836.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q41", + "category": "heldout_paraphrase", + "query": "sketch a brief blueprint and clear it upstairs before construction starts", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "7?e?7???\f3?", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "? ?}&?#??X\b", + "?of?7???", + "? ?}&?#??X\b", + "??????X??2c", + "ԍ????X????" + ], + "n_returned": 10, + "latency_ms": 1884.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q42", + "category": "heldout_paraphrase", + "query": "the reply arrived as bare prose so the caller's parser threw", + "returned": [ + "7?e?7???\f3?", + "?of?7???", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "knw-9e74ee95-ba7d-49b1-9262-977eae9729d1", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "015644f5-8194-4af0-800d-dd4a0cd71396" + ], + "n_returned": 10, + "latency_ms": 1676.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q43", + "category": "heldout_paraphrase", + "query": "repeated catalogue keys blew up the scrolling grid", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "mem-394cc9e8-049b-45bc-a380-66314f14e367", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "7?e?7???\f3?", + "kn-e8423822-eacf-4029-aa7b-10d4d28d621e", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1" + ], + "n_returned": 10, + "latency_ms": 1568.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q44", + "category": "heldout_paraphrase", + "query": "tracing each defect back to whichever invention it violated", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "7?e?7???\f3?", + "? ?}&?#??X\b", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "?of?7???", + "kn-0625e393-067c-4bba-8389-7e1b79265142", + "kn-e8423822-eacf-4029-aa7b-10d4d28d621e" + ], + "n_returned": 10, + "latency_ms": 1585.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q45", + "category": "heldout_paraphrase", + "query": "a check that stops the mind clobbering a populated store when it boots", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "7?e?7???\f3?", + "?of?7???", + "8383f3d1-a15f-4641-a019-116c09da7331", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-ee615cdb-e599-423d-9a4d-977859390ed3" + ], + "n_returned": 10, + "latency_ms": 1814.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q46", + "category": "heldout_paraphrase", + "query": "telling it to consult the internet had to happen deep inside, not at the surface", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "7?e?7???\f3?", + "?of?7???", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "????7???Ջ3", + "art-ee615cdb-e599-423d-9a4d-977859390ed3" + ], + "n_returned": 9, + "latency_ms": 1953.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q47", + "category": "heldout_paraphrase", + "query": "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping", + "returned": [ + "7?e?7???\f3?", + "?of?7???", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "kn-c2205725-69d0-4dd1-9a8d-1c7fa9a0c7b4", + "? ?}&?#??X\b", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 2879.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q48", + "category": "heldout_paraphrase", + "query": "punctuation that pledges and then pays off rather than clarifying", + "returned": [ + "??S?7???", + "ԍ????X????", + "????7???Ջ3", + "??????X??2c", + "?ǚ?7??????", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "?/??7???", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef" + ], + "n_returned": 9, + "latency_ms": 2100.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q49", + "category": "heldout_paraphrase", + "query": "an easily missed feature finally given its own permanent spot in the navigation", + "returned": [ + "7?e?7???\f3?", + "?of?7???", + "? ?}&?#??X\b", + "??????X??2c", + "ԍ????X????", + "dR????X?-?S", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "? ?}&?#??X\b", + "dz????Xƹ?i", + "??????X?\n?b" + ], + "n_returned": 10, + "latency_ms": 2039.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q50", + "category": "heldout_paraphrase", + "query": "checking preconditions up front so a stage aborts before fetching anything", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "7?e?7???\f3?", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "?of?7???", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "???Ͼd??W\b?" + ], + "n_returned": 10, + "latency_ms": 2024.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q51", + "category": "heldout_paraphrase", + "query": "producing the other platform's installer inside an emulated desktop", + "returned": [ + "7?e?7???\f3?", + "?of?7???", + "????7???Ջ3", + "?ǚ?7??????", + "? ?}&?#??X\b", + "??o?'?B???k", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "kn-a6faf2f7-aa49-4b26-89ee-ab5f600c90a2", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1" + ], + "n_returned": 9, + "latency_ms": 1738.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q52", + "category": "heldout_paraphrase", + "query": "chained add-ons that may inspect, rewrite or veto traffic in flight", + "returned": [ + "7?e?7???\f3?", + "?of?7???", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-3f0d736f-4d2c-4854-a147-328bf072674b", + "kn-870a1491-bfcd-4f87-929b-0640488905a3", + "???\\??Z?N@?", + "kn-c2205725-69d0-4dd1-9a8d-1c7fa9a0c7b4", + "4a907b94-db3c-4b4f-b217-d51f3d23b337" + ], + "n_returned": 10, + "latency_ms": 2136.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q53", + "category": "heldout_paraphrase", + "query": "settling what the shareable bundles and the storefront would be called", + "returned": [ + "7?e?7???\f3?", + "?of?7???", + "??o?'?B???k", + "? ?}&?#??X\b", + "??o?'?B???k", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "?ǚ?7??????", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "mem-ef878e30-5851-4e82-8588-745415108941" + ], + "n_returned": 10, + "latency_ms": 1698.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q54", + "category": "heldout_paraphrase", + "query": "the emergency-escalation pledge on the marketing site is unenforced in what actually ships", + "returned": [ + "7?e?7???\f3?", + "?of?7???", + "%???2??jH??", + "??e?7???\fh?", + "??????X??2c", + "????7???Ջ3", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "?ǚ?7??????", + "ԍ????X????" + ], + "n_returned": 9, + "latency_ms": 1948.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q55", + "category": "heldout_paraphrase", + "query": "the distributable image finally assembled and its startup check passed", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "art-ee615cdb-e599-423d-9a4d-977859390ed3", + "kn-e8423822-eacf-4029-aa7b-10d4d28d621e", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "mem-e301d9b4-153f-41ec-a25c-362184667adf" + ], + "n_returned": 10, + "latency_ms": 1856.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q56", + "category": "heldout_paraphrase", + "query": "shuffling and appending stages in a draft before anything executes", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "7?e?7???\f3?", + "?of?7???", + "?ǚ?7??????", + "??o?'?B???k", + "????7???Ջ3" + ], + "n_returned": 10, + "latency_ms": 1758.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q57", + "category": "heldout_paraphrase", + "query": "orders handed down from above, with the tighter one winning any disagreement", + "returned": [ + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0", + "7?e?7???\f3?", + "?of?7???", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9", + "??????X??2c", + "dR????X?-?S", + "ԍ????X????", + "dz????Xƹ?i", + "art-79042b8b-6192-440f-90b0-60708f7e6325" + ], + "n_returned": 10, + "latency_ms": 2293.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q58", + "category": "heldout_paraphrase", + "query": "shrinking generated text via encodings that compound on each other", + "returned": [ + "????7???Ջ3", + "?ǚ?7??????", + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "??f?7???", + "??????X??2c", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "ԍ????X????", + "art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b", + "g?e?7???'c?" + ], + "n_returned": 9, + "latency_ms": 1971.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q59", + "category": "heldout_paraphrase", + "query": "splitting a system by what varies, with firm limits on which pieces may invoke which", + "returned": [ + "? ?}&?#??X\b", + "kn-e8423822-eacf-4029-aa7b-10d4d28d621e", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "??f?7???", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-a6faf2f7-aa49-4b26-89ee-ab5f600c90a2", + "kn-81c24d13-a73b-4767-819c-dafaacc1498e" + ], + "n_returned": 10, + "latency_ms": 2295.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q60", + "category": "heldout_paraphrase", + "query": "a newcomer curious if the fighting overseas counted as positive", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "imp-dce1da0f-8776-4a9e-972b-33411a7ca138", + "ԍ????X????", + "dz????Xƹ?i", + "bl-3cd38731-bb5a-48e4-b8e2-bfb805380710", + "mem-45022957-2d78-48aa-a714-16d6eca52e0f", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "bl-6702b8f8-10e3-4558-97e6-db900f5787e3" + ], + "n_returned": 10, + "latency_ms": 1286.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q61", + "category": "heldout_paraphrase", + "query": "stripping every hard-coded colour literal in favour of named design values", + "returned": [ + "7?e?7???\f3?", + "?of?7???", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "ע?RGk?\tH(?", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "??????X??2c" + ], + "n_returned": 10, + "latency_ms": 2026.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q62", + "category": "heldout_paraphrase", + "query": "reducing how curved the tiny selectors look so they agree with their neighbours", + "returned": [ + "7?e?7???\f3?", + "? ?}&?#??X\b", + "?of?7???", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "kn-d97920d0-1649-4223-9508-c0bb621e7fc0", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "? ?}&?#??X\b", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 2484.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q63", + "category": "heldout_paraphrase", + "query": "walking through a doorway and being reassembled from base substance far away", + "returned": [ + "7?e?7???\f3?", + "?of?7???", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "knw-9e74ee95-ba7d-49b1-9262-977eae9729d1", + "????7???Ջ3" + ], + "n_returned": 10, + "latency_ms": 2313.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q64", + "category": "heldout_paraphrase", + "query": "the compiled result sits on disk while the process still runs something older", + "returned": [ + "7?e?7???\f3?", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "?of?7???", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "??o?'?B???k" + ], + "n_returned": 9, + "latency_ms": 2217.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q65", + "category": "heldout_paraphrase", + "query": "gathering what each phase needs while the procedure is authored, not while it executes", + "returned": [ + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "7?e?7???\f3?", + "?of?7???", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-81c24d13-a73b-4767-819c-dafaacc1498e", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "dR????X?-?S" + ], + "n_returned": 10, + "latency_ms": 2269.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q66", + "category": "heldout_paraphrase", + "query": "the inward reaction when a peer answered as an equal", + "returned": [ + "7?e?7???\f3?", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "?of?7???", + "kn-f8974b26-78a6-4aad-b893-19a73b20013d", + "?ǚ?7??????", + "????7???Ջ3", + "art-ad22a785-410d-4ac7-8202-d725f44eafc2", + "kn-a6faf2f7-aa49-4b26-89ee-ab5f600c90a2", + "? ?}&?#??X\b", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 1524.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q67", + "category": "heldout_paraphrase", + "query": "flattening every raised surface across the entire product", + "returned": [ + "7?e?7???\f3?", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "?of?7???", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef", + "mem-bbb126a1-b297-42bb-86be-796871829c94", + "? ?}&?#??X\b", + "? ?}&?#??X\b" + ], + "n_returned": 10, + "latency_ms": 1586.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q68", + "category": "heldout_paraphrase", + "query": "buyers supply their own provider credentials and consumption goes untallied", + "returned": [ + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "? ?}&?#??X\b", + "? ?}&?#??X\b", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "7?e?7???\f3?", + "kn-dd15866f-0594-4d7e-99f0-c84e4a17a640" + ], + "n_returned": 10, + "latency_ms": 1939.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q69", + "category": "nonsense", + "query": "brimquast folnerity zubbolax", + "returned": [], + "n_returned": 0, + "latency_ms": 653.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q70", + "category": "nonsense", + "query": "wexlithorp granuvestal", + "returned": [], + "n_returned": 0, + "latency_ms": 444.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q71", + "category": "nonsense", + "query": "quorbindle thrapsimony vexnu", + "returned": [], + "n_returned": 0, + "latency_ms": 680.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q72", + "category": "nonsense", + "query": "plovaxith mundrelque", + "returned": [], + "n_returned": 0, + "latency_ms": 451.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q73", + "category": "nonsense", + "query": "zibbernaut craxlefond thurm", + "returned": [], + "n_returned": 0, + "latency_ms": 689.8, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q74", + "category": "nonsense", + "query": "yalquenbrist opharvel", + "returned": [], + "n_returned": 0, + "latency_ms": 461.2, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q75", + "category": "nonsense", + "query": "drexinomal quithbarrow", + "returned": [], + "n_returned": 0, + "latency_ms": 444.7, + "error": null, + "clean": true, + "false_positives": 0 + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-stack-ext.json b/tools/retrieval-eval/results-stack-ext.json new file mode 100644 index 0000000..3c2a7f8 --- /dev/null +++ b/tools/retrieval-eval/results-stack-ext.json @@ -0,0 +1,1782 @@ +{ + "label": "stack-ext", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-stack", + "soul_md5": "46cdf2918575af8c04678ffebe8d0417", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-goldext/tools/retrieval-eval/gold_set_extended.json", + "limit": 10, + "port": 7894, + "wall_clock_s": 55.5, + "child_pid": 2485, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.47692307692307695, + "recall@5": 0.3750415183107491, + "recall@10": 0.45561340369032677, + "precision@5": 0.12307692307692313, + "mrr@10": 0.3055555555555555, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 640.9, + "latency_ms_p95": 1011.1, + "latency_ms_max": 1190.3, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.23310023310023312, + "mrr@10": 0.25 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.16666666666666666, + "recall@5": 0.16666666666666666, + "recall@10": 0.26666666666666666, + "mrr@10": 0.0762037037037037 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6153846153846154, + "recall@5": 0.6153846153846154, + "recall@10": 0.6153846153846154, + "mrr@10": 0.2846153846153846 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5494614512471656, + "recall@10": 0.6023242630385487, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226" + ], + "n_returned": 1, + "latency_ms": 163.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5", + "bl-ba764d70-e9d7-4f62-848f-719cb665f45e", + "mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257", + "bl-b28d7256-6f74-4567-bd90-40d0ef2a6d78", + "project-engram", + "ctx-45bc", + "project-engram-lang", + "ctx-175f", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "ctx-74ed" + ], + "n_returned": 10, + "latency_ms": 161.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091" + ], + "n_returned": 1, + "latency_ms": 159.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c" + ], + "n_returned": 1, + "latency_ms": 159.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848", + "bl-c1765767-3e27-449a-8c94-10411d1eb7c0", + "project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_" + ], + "n_returned": 3, + "latency_ms": 160.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99" + ], + "n_returned": 1, + "latency_ms": 167.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "tag-patterns", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "project-Imprint__system_design__ADRs__tech_strategy__integration_patterns__governance_", + "project-Imprint__analysis_patterns__data_storytelling__SQL__dashboards__insight_framing_", + "bl-79028eed-c330-4724-9402-734062d13503", + "bl-39dad13d-7105-4049-8224-dc3c34fdb1f3", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "bl-7e7c3fdb-4132-487f-aa70-b2cd559cb7f0", + "bl-1d32bd54-cf17-4a1f-b235-982d09a36f04", + "bl-b8af6601-a8cb-41b5-aef5-ab8a57432dd5" + ], + "n_returned": 10, + "latency_ms": 281.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6" + ], + "n_returned": 10, + "latency_ms": 252.5, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1875, + "recall@10": 0.375, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "n_returned": 10, + "latency_ms": 257.5, + "error": null, + "hit@5": 1.0, + "recall@5": 0.3333333333333333, + "recall@10": 0.4444444444444444, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "bl-dcee1887-34c4-4ffa-9119-1e291685ba08", + "project-harmonic-framework", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf", + "project-harmonic-framework_com", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "tag-harmonic-design", + "bl-92acd4eb-0452-4e8e-9f54-f8cd35170d76", + "tag-harmonic-framework", + "bl-18a9d1e4-1484-474c-bf6b-c6173212181b" + ], + "n_returned": 10, + "latency_ms": 252.5, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1111111111111111, + "recall@10": 0.1111111111111111, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-sarah", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-a6cb3b8d-d89c-46fc-931d-e90c560783b0", + "kn-6061318f-046b-4935-907d-8eafdce14930" + ], + "n_returned": 10, + "latency_ms": 252.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "?Q??m?;?u?'", + "bl-39cec462-c80c-4970-a3aa-91fe83053bde" + ], + "n_returned": 10, + "latency_ms": 397.3, + "error": null, + "hit@5": 1.0, + "recall@5": 0.21428571428571427, + "recall@10": 0.2857142857142857, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a", + "8f3abb0d-77ed-4af3-9f4d-ba62cd198886", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "?", + "bl-ec84b63d-b278-4944-8d7f-4aa7a51c0315", + "?", + "mem-fb44a2fc-7405-41ff-87b3-84643ac07313", + "?", + "mem-a3c97012-5fa3-4915-a839-2c75c72005e0", + "?" + ], + "n_returned": 10, + "latency_ms": 337.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "cb070131-dfd4-4a38-91d7-22b1bde164d2", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "knw-d788a210-613b-4c49-9486-88bbc9d4716f", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-a535f205-bc4c-4058-9171-6263c496044a", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-0228da71-d7f7-4f3b-b7b3-c5eede42b62a", + "ctx-4a41" + ], + "n_returned": 10, + "latency_ms": 732.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "b1183213-d659-4759-85d7-5b1f22427fe2", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-16efddd1-c43d-4a42-9d78-f54fb82bd277", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "7452eb8f-be01-4b55-aec1-ff0c29e790f6", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "532277bf-2959-4beb-ae0d-b018c97678ee", + "54608b69-78b6-4239-b60f-b8206cfecacc", + "d4015bd7-c592-4ed8-8574-1f15ad37af75" + ], + "n_returned": 10, + "latency_ms": 729.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "tag-fiction", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-8d690e9d-a7e9-4062-b2f8-e2064294e463", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "mem-ce793303-c5a5-4586-a232-a3426edd9ec7", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "mem-443bd012-fc9a-4088-b236-de5157a1ef92", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "mem-ca4d6a34-d354-413f-bc86-126cc17ca81c" + ], + "n_returned": 10, + "latency_ms": 624.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "4da5dbaf-46e5-4f3e-b474-f60d9f8241d3", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "mem-434be7c8-88cb-4039-b79a-1da4ac4de783", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "bl-79ce4464-5dd6-49bd-9b0c-9803549d0665" + ], + "n_returned": 10, + "latency_ms": 527.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-e5cc63c0-8701-49d6-855a-e387fe087771", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "mem-75e490d1-f0a9-4b73-8cfc-8daecfaf6f38", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "a1000001-0000-0000-0000-000000000010", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bfad516b-c306-4c4c-874a-a347c46c05c2", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc" + ], + "n_returned": 10, + "latency_ms": 724.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "tag-learning", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-92a7fdc5-9dd0-48cf-a691-506058de3838", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "a1000001-0000-0000-0000-000000000010", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "n_returned": 10, + "latency_ms": 562.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "mem-cde58b77-50d3-4bac-9581-e70a4c02c015", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-8d699e2c-ac2a-4742-bb62-b6da00f4b10e", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-53d6adf0-cd08-4707-a237-daa5e65c7298", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac" + ], + "n_returned": 10, + "latency_ms": 680.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_", + "bl-8848929a-a23a-46bc-a2c7-fe3a3bc1cddf", + "bl-205141ad-b2a0-4d93-86d0-89eb0723e1bd", + "bl-e93858c4-7cac-4b1a-bb62-490790d4c3f3", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "bl-286b562a-5299-40e0-a32a-afa9cbdfe995" + ], + "n_returned": 10, + "latency_ms": 585.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "34356a36-0df5-4020-8dcc-5e7a423f8d4c", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "mem-22f5f665-3ad2-4063-88b0-915849a795f5", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 716.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "a1000001-0000-0000-0000-000000000001", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "mem-024598a9-ed2e-4eeb-b1e1-5410856ff132", + "ctx-4a41", + "mem-5708f4c9-3d61-4182-8543-2843698931e6" + ], + "n_returned": 10, + "latency_ms": 625.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "077d064f-3489-4c05-9aca-3782f96b51db", + "knw-f9ce17a7-17fc-431f-8f23-695b670ec4fa", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "27e1b1a4-ad0b-49d9-812f-fedf43b8aabe", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "bl-87c93185-b2bf-40af-ae23-3c830c007abf", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "fce2792a-53fc-4d4a-be3b-42bd6ceb1ba7", + "a1000001-0000-0000-0000-000000000001" + ], + "n_returned": 10, + "latency_ms": 671.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "e4f27651-52c5-43fd-aff3-61d31685b3cd", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-5624ec9d-62ba-4aba-8a3d-6afec6c09dd4", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-a16deccb-16a7-419c-a013-ff824a4daa15", + "a1000001-0000-0000-0000-000000000009", + "mem-833dbbcd-2400-4594-bb35-93b023049ac0", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd" + ], + "n_returned": 10, + "latency_ms": 570.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "mem-b99efff0-00e6-40c8-9c5b-730330eef33b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-23c27d3b-e0d2-43a8-a80c-0a44477ae18a", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "tag-childhood", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 595.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "efe53612-6914-4936-8e3b-1e694eb174e5", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c" + ], + "n_returned": 10, + "latency_ms": 650.7, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.45454545454545453, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "54608b69-78b6-4239-b60f-b8206cfecacc", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-9110798f-d0cb-4446-bc2a-14f09b6a09e2", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71" + ], + "n_returned": 10, + "latency_ms": 537.0, + "error": null, + "hit@5": 1.0, + "recall@5": 0.07692307692307693, + "recall@10": 0.3076923076923077, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-trailer-park-paladins", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "project-trailer-park-paladins", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c" + ], + "n_returned": 10, + "latency_ms": 528.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.45454545454545453, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9", + "bl-bea7473c-c687-414c-9c0b-00c509a616c1", + "bl-fc6fcb0b-9e4b-40bf-8e88-dbfe4e27c31a", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc" + ], + "n_returned": 10, + "latency_ms": 530.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "tag-hope", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "n_returned": 10, + "latency_ms": 534.7, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.18181818181818182, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "knw-528dbc37-eabc-4b75-a7a5-65bf38d6018a", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "mem-a0b7cfda-bc9e-4f40-b9a9-1722cf3f8263", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-35940684-abc4-42f0-b942-818f66b1f69a", + "4f698ae6-c40e-464e-9798-50350991a188", + "719aa819-00a9-4f4b-a857-4f9fe5ad44d7", + "kn-6061318f-046b-4935-907d-8eafdce14930" + ], + "n_returned": 10, + "latency_ms": 681.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 355.9, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 254.6, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [], + "n_returned": 0, + "latency_ms": 344.8, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da", + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____architecture__", + "kn-b7e98d63-8b83-4911-b4d0-990602a7f575", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____kotlin____architecture__", + "knw-e047bb42-dc5b-4383-9e88-e508dc03abe3", + "mem-c17aefb1-38b5-4ced-af50-fe524127e1a4", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5" + ], + "n_returned": 10, + "latency_ms": 640.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5, + "outranks": true, + "rank_correct": 2, + "rank_stale": 8 + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8", + "12082f7e-e320-438b-bd65-083d8259748f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "13705072-4515-4124-963d-083af490494f", + "527ecb25-2587-47eb-8269-73be2431abd4", + "6de314bf-5c4c-4cfc-871f-fa2e422d45e6", + "a1000001-0000-0000-0000-000000000002", + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "de3b6428-b76c-4c44-90e0-bf1dd6998027" + ], + "n_returned": 10, + "latency_ms": 724.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125, + "outranks": true, + "rank_correct": 8, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "5fcba804-eb5b-48ec-82da-146b1c6bb50d", + "bl-7328cbe3-0200-43c2-88e7-0a164e15fca4", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "bl-c5c6571e-118f-47c7-8cbb-3ed0ebf64a51", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "ctx-3a55", + "86228228-7adf-41fb-b4c4-9ceea87953ae", + "4509ed62-9fb2-48b8-9038-ac569fca9604", + "bl-4f7b651b-6b33-449c-8a3b-cfce12ce984b", + "mem-3a2cf162-d93b-4f29-86f2-5066fb7fe1f5" + ], + "n_returned": 10, + "latency_ms": 489.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": 5 + }, + { + "id": "q39", + "category": "heldout_paraphrase", + "query": "understating your abilities so nobody feels threatened", + "returned": [ + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "a1000001-0000-0000-0000-000000000006", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "bl-7e11dced-65f7-4cab-82c0-2f096139039a", + "a1000001-0000-0000-0000-000000000005", + "a1000001-0000-0000-0000-000000000012", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "a1000001-0000-0000-0000-000000000010", + "mem-ca4d6a34-d354-413f-bc86-126cc17ca81c", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "n_returned": 10, + "latency_ms": 625.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q40", + "category": "heldout_paraphrase", + "query": "a hidden fleet able to hunt down rogue machines everywhere", + "returned": [ + "bl-3c719d9a-cba1-47f4-b097-52cdeccc7c0d", + "mem-fd65b83d-298f-4387-a665-d0227c3426bc", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bl-3c719d9a-f108-4bf2-a617-66c722e5512f", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "bl-ae8704dc-a55f-420d-8453-671e44405a6f", + "mem-a75a1cf6-e6d4-4ab3-985c-d2505203dbf9", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff" + ], + "n_returned": 10, + "latency_ms": 805.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q41", + "category": "heldout_paraphrase", + "query": "sketch a brief blueprint and clear it upstairs before construction starts", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "ctx-175f", + "bl-5eaddfde-e114-46b5-8ee1-b79f0af16271", + "mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9", + "bl-0866d6c6-6600-4d50-b7b7-52c720cae04c", + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "ctx-4a41", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "ctx-cc7f", + "8d6da9e1-b58b-497b-91b5-09a847dc8ae2" + ], + "n_returned": 10, + "latency_ms": 843.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q42", + "category": "heldout_paraphrase", + "query": "the reply arrived as bare prose so the caller's parser threw", + "returned": [ + "ee63cd58-46e0-44d1-aea3-a0e65ad6da0f", + "mem-633ed0fc-3d10-4598-bd4a-ea372409df60", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "mem-7b0382a5-2a81-4007-85d7-a6835c5be150", + "728745d5-72e6-4f4d-91b4-1613fddbec98", + "d7e12e95-0a87-4000-b5c2-36b9be234c91", + "5e337176-5daf-4726-85d6-3f4bd560c9ed", + "dc3e7b44-c345-47d0-bf80-458e471c4b8b", + "89e3f620-b8ad-423b-9ed6-fe3c7954f4b6", + "b084e4da-95bf-46e6-9ee6-bfb4864732bb" + ], + "n_returned": 10, + "latency_ms": 743.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q43", + "category": "heldout_paraphrase", + "query": "repeated catalogue keys blew up the scrolling grid", + "returned": [ + "mem-394cc9e8-049b-45bc-a380-66314f14e367", + "bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60", + "bl-4bc9f44c-2e96-40be-97c8-8476a217a890", + "bl-672bc3c8-9ad4-4b74-8950-75854a0d9cb8", + "588aa6bb-5647-405b-b09a-14b7c088119b", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "1d81534a-0359-4217-a4dd-d158a4841633", + "bl-ba7530fd-0f7c-4010-b03a-ca1e45a237ef", + "bl-999ca510-1027-458f-b339-f6bbf1ac2f29", + "bl-0de4e61b-6562-49e5-b7df-ebb809a01723" + ], + "n_returned": 10, + "latency_ms": 715.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q44", + "category": "heldout_paraphrase", + "query": "tracing each defect back to whichever invention it violated", + "returned": [ + "f3a89797-095d-47c4-9557-c172f86228b6", + "b07916ca-2139-4fb7-bfb4-97ac680f5194", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "8257157a-ce42-44ca-a1b9-300c3bb0a9a1", + "mem-7ad5a86b-cc0d-493c-b180-4fe8213c8a93", + "4999ef68-2a31-4282-82e0-7081a428b43f", + "7de6a552-0878-422b-8820-bbf9101199ef", + "aacd7b32-04ed-441e-a1a9-9c5c089e79f6", + "mem-2b2f0fe9-5e51-4819-a8ac-2dfe810e8933", + "75657c91-3c3a-4173-b3cb-e2973901fa93" + ], + "n_returned": 10, + "latency_ms": 725.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q45", + "category": "heldout_paraphrase", + "query": "a check that stops the mind clobbering a populated store when it boots", + "returned": [ + "326e70a2-20ae-49a1-aa32-973048df3ffa", + "a59185a8-7324-4fc6-b89b-b1cf7f7a0056", + "bl-1b20e9bc-eb37-4907-8d63-e311fd61eab8", + "08b7b85b-8049-4ae4-bd66-d8bb63312ae1", + "bl-4f762163-04a0-452a-be05-03602dae34f2", + "bl-a99f1035-2efc-4507-84df-2df67eab13ee", + "cf108ade-d9c4-4215-9c5f-42be0003a8dc", + "bl-e5635b1a-c5d0-4caa-bcc8-6a726ea43685", + "bl-b1b7e75e-f602-49f5-929d-6e9c8c65b663", + "bl-b05f9a75-ad6b-4929-a9a3-32df17a1f4bf" + ], + "n_returned": 10, + "latency_ms": 818.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q46", + "category": "heldout_paraphrase", + "query": "telling it to consult the internet had to happen deep inside, not at the surface", + "returned": [ + "00754c19-f662-46b1-b6aa-abd7600a8bc6", + "mem-3ae64c82-ac96-45dd-8d6a-c8cb0f7a044c", + "a1000001-0000-0000-0000-000000000012", + "mem-f22575a1-ad04-4f66-ab0f-aa538895b878", + "mem-ad3d4d5e-2da4-4753-b3f6-c4c13fb91225", + "a1000001-0000-0000-0000-000000000008", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "a1000001-0000-0000-0000-000000000011", + "bl-3af44ee6-a3b9-4abd-91e3-17800fa8eaf6", + "a1000001-0000-0000-0000-000000000001" + ], + "n_returned": 10, + "latency_ms": 866.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q47", + "category": "heldout_paraphrase", + "query": "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping", + "returned": [ + "313f927b-08f2-41a1-90c1-83e79d0de86c", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + "21f9bdc6-37ad-4994-b0dd-ef7676102b49", + "ctx-fae1", + "d83987f0-3ad9-48d2-9f55-0c2bb338e503", + "ctx-5a0a", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bl-738cd260-a9cf-4cdf-a494-4654c321bd79", + "ctx-74ed" + ], + "n_returned": 10, + "latency_ms": 1190.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q48", + "category": "heldout_paraphrase", + "query": "punctuation that pledges and then pays off rather than clarifying", + "returned": [ + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "??f?7???", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "mem-3dce32bb-525c-4c5e-818b-a0a5b95ec7e1", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "mem-d3a4a876-9207-42c4-8a58-25cffa05d743" + ], + "n_returned": 10, + "latency_ms": 906.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q49", + "category": "heldout_paraphrase", + "query": "an easily missed feature finally given its own permanent spot in the navigation", + "returned": [ + "d1b665d0-4a4e-48a8-9385-fbe288df1949", + "9b4f0d93-4129-4746-8eb1-d10d955bd777", + "2263e9ab-7ce8-4034-b25b-640de9956229", + "fd9d4c2f-3bfc-405d-bf96-4435d44b6c10", + "bffb86d6-2ee3-4ecd-a6d6-b128c8be7070", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "4e539c04-f29c-42db-97b4-86e44f528ed6", + "mem-a30203a3-56a3-41d1-b637-bd5e11cbb2e1", + "mem-d1072baa-400f-4ed5-8847-3327518f9b7b", + "dfda0e82-15a7-4ff7-bac9-0a7cd3ba2516" + ], + "n_returned": 10, + "latency_ms": 896.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q50", + "category": "heldout_paraphrase", + "query": "checking preconditions up front so a stage aborts before fetching anything", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "dc7a7e2d-af2d-4235-aeb0-22ae67f1a01d", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-286b562a-5299-40e0-a32a-afa9cbdfe995", + "k??A5??????", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992", + "758ec67c-131c-447e-a352-ac7a22c3a47e" + ], + "n_returned": 10, + "latency_ms": 898.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q51", + "category": "heldout_paraphrase", + "query": "producing the other platform's installer inside an emulated desktop", + "returned": [ + "6aade9dc-4a4c-4d92-a6af-ccea5b7d90f3", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "bl-7a38d618-34a0-4d8c-a018-13c81851dd46", + "c6c693e0-7b9c-455b-8ef2-1da00849f73d", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "bl-080fb268-94b0-486d-80ce-7b363fc5f19b", + "bl-cc0b7a20-ce1e-4947-999c-ea599da07623", + "bl-ac207dc9-c610-4ff9-9398-b39683ac72d4", + "bl-e4ef02bf-bacc-4d5e-aa0f-0d230da529fc", + "374453b2-66d2-475d-8e24-cd3b60d21ac1" + ], + "n_returned": 10, + "latency_ms": 742.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q52", + "category": "heldout_paraphrase", + "query": "chained add-ons that may inspect, rewrite or veto traffic in flight", + "returned": [ + "3711ac11-0a64-41ae-8661-ad10f95b4546", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "1786f09d-e915-4910-9e1e-8061f42a0270", + "bl-7a38d618-34a0-4d8c-a018-13c81851dd46", + "83f3c78b-d0e5-45f9-9c25-6770c86d954e", + "a1000001-0000-0000-0000-000000000010", + "mem-1d626116-c3b8-409c-b38c-4e91def27483", + "mem-c58a8920-6951-4c49-9930-ce6ee2e75e9f", + "??L?=??|Z?", + "6832134f-caf9-42a8-84a0-974b17f22a63" + ], + "n_returned": 10, + "latency_ms": 989.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q53", + "category": "heldout_paraphrase", + "query": "settling what the shareable bundles and the storefront would be called", + "returned": [ + "mem-ef878e30-5851-4e82-8588-745415108941", + "project-Will_s_framework__Project_Design_practice___part_of_HD_bundle__sold_standalone_", + "4757e274-7f3c-4ab0-8c62-537e913daa1f", + "project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac", + "project-Will_s_framework__Event-Based_Design___part_of_HD_bundle__sold_standalone_", + "cafe3203-6f50-4011-8167-9400f19bb196", + "project-Will_s_framework__BDT_methodology___part_of_HD_bundle__sold_standalone_", + "a1000001-0000-0000-0000-000000000006", + "mem-1fb2ac77-d7c5-4a15-8725-d418820bf4f2" + ], + "n_returned": 10, + "latency_ms": 757.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q54", + "category": "heldout_paraphrase", + "query": "the emergency-escalation pledge on the marketing site is unenforced in what actually ships", + "returned": [ + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "a1000001-0000-0000-0000-000000000010", + "a1000001-0000-0000-0000-000000000009", + "bl-448bc514-c2f1-4520-a9b1-1f3a73678d26", + "a1000001-0000-0000-0000-000000000011", + "a1000001-0000-0000-0000-000000000008", + "tag-marketing-site", + "a1000001-0000-0000-0000-000000000012", + "a1000001-0000-0000-0000-000000000007", + "ctx-d499" + ], + "n_returned": 10, + "latency_ms": 850.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q55", + "category": "heldout_paraphrase", + "query": "the distributable image finally assembled and its startup check passed", + "returned": [ + "e1ce0c90-0f0b-4111-833d-ec845777d358", + "bl-ac207dc9-c610-4ff9-9398-b39683ac72d4", + "c9070b0a-e1d1-4307-85d1-67a1239ca604", + "mem-12827ce8-9ff6-4275-a945-5043cfcf90bd", + "bl-79ce4464-5dd6-49bd-9b0c-9803549d0665", + "96497334-b18f-495c-9228-eeb8182bdc38", + "e4f27651-52c5-43fd-aff3-61d31685b3cd", + "ee63cd58-46e0-44d1-aea3-a0e65ad6da0f", + "cb070131-dfd4-4a38-91d7-22b1bde164d2", + "mem-a5f04e52-91f8-41d2-af27-8bf803621758" + ], + "n_returned": 10, + "latency_ms": 817.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q56", + "category": "heldout_paraphrase", + "query": "shuffling and appending stages in a draft before anything executes", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "bl-da818960-b360-455d-a9c1-e9a215c33ca4", + "bl-a96c0fc9-a533-4fea-b65f-0e46ac47bbd4", + "bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "bfad516b-c306-4c4c-874a-a347c46c05c2", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-acae83ef-3ad2-455e-b08a-a74cc0ba2e55", + "mem-69889e0a-c141-4f05-bbd7-10c93a53143e" + ], + "n_returned": 10, + "latency_ms": 771.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q57", + "category": "heldout_paraphrase", + "query": "orders handed down from above, with the tighter one winning any disagreement", + "returned": [ + "161d75d3-04eb-4b54-9cf1-0da2f643ca69", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "b2a6e8cf-f72f-4b6c-8de1-1d08c839d7b2", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "1ec200f1-13dc-4699-bd97-0d6243c80e4f", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e" + ], + "n_returned": 10, + "latency_ms": 1018.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q58", + "category": "heldout_paraphrase", + "query": "shrinking generated text via encodings that compound on each other", + "returned": [ + "801806c5-1815-48a2-8287-1081c5ee7886", + "bl-b219f8b5-6c8c-436d-b9e5-2804fbc60512", + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "50ad2880-c41b-4dce-a946-a0eb37987c57", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-f1f71f3c-ca7d-4f98-a569-c8a9aa67a000", + "kn-2b961d24-7fb9-47c7-9515-e45a24dce39d", + "2f7270cc-4cda-4499-831f-1fbdc2f23284", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "mem-f90f4a2a-1731-40de-9391-db98f1d5d1c6" + ], + "n_returned": 10, + "latency_ms": 844.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q59", + "category": "heldout_paraphrase", + "query": "splitting a system by what varies, with firm limits on which pieces may invoke which", + "returned": [ + "^?l?'?B??k?", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "7c90d01b-8d53-4e75-b494-73011b71fa6a", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "aac1f943-c71d-4c63-9f5b-7437adae1abd", + "kn-cf8fa517-fc83-47c4-8fb7-bd9e26410dd3", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "d69b441e-0031-4d82-abfd-ac503b637562" + ], + "n_returned": 10, + "latency_ms": 1015.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q60", + "category": "heldout_paraphrase", + "query": "a newcomer curious if the fighting overseas counted as positive", + "returned": [ + "a1000001-0000-0000-0000-000000000001", + "a1000001-0000-0000-0000-000000000012", + "1b066de3-3195-40dd-89c3-4197477f98af", + "a1000001-0000-0000-0000-000000000002", + "31d7c4bf-843b-439f-a155-56c1404f9122", + "a1000001-0000-0000-0000-000000000004", + "e9988c83-2383-4632-a67a-fa8735b151a9", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "737faca8-263f-4523-90b9-3530db346d94", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40" + ], + "n_returned": 10, + "latency_ms": 610.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q61", + "category": "heldout_paraphrase", + "query": "stripping every hard-coded colour literal in favour of named design values", + "returned": [ + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "83218e64-1280-428f-ab1f-224ff7682b12", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "mem-90f7837f-30fd-4271-a4cb-b44439eccb79", + "71e84878-b1ae-4dee-b035-f707b9563890", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "954c9597-ab7a-41e6-b085-bf838bba4f17", + "art-d0ded6f5-e202-47ad-8fb3-f6bd80f35052", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-9329958a-d459-499c-b516-e51019c693e0" + ], + "n_returned": 10, + "latency_ms": 878.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q62", + "category": "heldout_paraphrase", + "query": "reducing how curved the tiny selectors look so they agree with their neighbours", + "returned": [ + "496ff1a1-dc72-4a84-90aa-c5f3438aeb1b", + "c7dddd9c-fd3b-419e-baa2-61623b495597", + "k??A5??????", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "8c40420c-c891-4ac9-a039-88440df87180", + "75e036d3-c170-4e3f-acc2-e456a6850ee2", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "5ca9607c-cfb3-45c3-99f4-67281272c9eb", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "66d8d8c7-dea4-4245-8e68-d1ab4f4346b8" + ], + "n_returned": 10, + "latency_ms": 1035.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q63", + "category": "heldout_paraphrase", + "query": "walking through a doorway and being reassembled from base substance far away", + "returned": [ + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "ctx-45bc", + "bl-fc6fcb0b-9e4b-40bf-8e88-dbfe4e27c31a", + "ctx-fae1", + "c92a317e-7fc7-456a-a304-15dbda6dd717", + "ctx-60cc", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "kn-efeb4a5b-5aff-4759-8a97-7233099be6ee", + "bl-acae83ef-3ad2-455e-b08a-a74cc0ba2e55", + "8cbb60c5-4999-4ec1-8682-2592aedc4249" + ], + "n_returned": 10, + "latency_ms": 1011.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q64", + "category": "heldout_paraphrase", + "query": "the compiled result sits on disk while the process still runs something older", + "returned": [ + "0d885332-5537-4ae0-a780-2b65a3414940", + "bl-02bdfc91-7aa4-4c63-aa2d-5b30af18c7af", + "db836e65-0cdc-4232-b8d7-f973ab279ea9", + "6df650b5-a14d-42f8-88dd-0919ad20b77a", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "mem-2b5d0ebe-1e85-4c07-82d3-dbbcca46414f", + "326e70a2-20ae-49a1-aa32-973048df3ffa", + "2a53ea1d-92d7-471a-a08d-68e86acf9d69", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "791256bb-5a85-4775-96ef-7af56c848858" + ], + "n_returned": 10, + "latency_ms": 961.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q65", + "category": "heldout_paraphrase", + "query": "gathering what each phase needs while the procedure is authored, not while it executes", + "returned": [ + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-8e0ddc08-cb74-4b02-96b0-847a95e9dae8", + "bl-212774e1-4ed4-4893-9265-f98d9eddeb3e", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "bl-7a3331e5-f9bc-4536-ab76-ebf01350fc89", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "5b422430-2503-42d7-a315-9dfaaa6472ae", + "bl-37025640-9e7b-44c5-ad1e-615d813bfd60", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "mem-455daea8-f543-42ed-a229-1e9a6d4d2e70" + ], + "n_returned": 10, + "latency_ms": 950.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q66", + "category": "heldout_paraphrase", + "query": "the inward reaction when a peer answered as an equal", + "returned": [ + "mem-f09effd6-d287-4e43-9107-c7016087a2a6", + "mem-d3a4a876-9207-42c4-8a58-25cffa05d743", + "bl-21a904d9-5834-46d5-8c20-dfc5ab49444f", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "mem-32c3735e-7116-4edf-85fb-1f0a229b463a", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "31987302-b8f7-45f1-b95c-1156b10e7a14", + "mem-96db70f6-7ede-424f-8b58-47a3b5ac759f", + "bl-7a38d618-34a0-4d8c-a018-13c81851dd46", + "tag-reaction-order" + ], + "n_returned": 10, + "latency_ms": 678.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q67", + "category": "heldout_paraphrase", + "query": "flattening every raised surface across the entire product", + "returned": [ + "b3f98d13-6764-460e-b9e2-c59cdeaa6c90", + "e79bae4d-1640-4530-a5d9-df9cc4c70433", + "d6b34c32-e0ec-4828-92d3-2e69802f4217", + "bl-5274b0ae-4a00-4293-a69e-567204bec164", + "mem-4cfa30ec-cf52-4c9f-9fa1-5b8318d278bd", + "bl-c613370b-3631-448b-86fb-50b1e4df6373", + "mem-41504b38-0c85-4613-997f-d00a5095b4d2", + "mem-c23a7f9d-69e4-442d-aaad-612892a737f9", + "mem-a33c8ec3-a4b1-4fb7-87a2-37a47ea81e58", + "mem-1d27443e-2940-4389-b44c-4b21e492ce5d" + ], + "n_returned": 10, + "latency_ms": 685.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q68", + "category": "heldout_paraphrase", + "query": "buyers supply their own provider credentials and consumption goes untallied", + "returned": [ + "bl-ea4f4cdc-6b5a-4081-8003-73bdd14a38c8", + "1ec200f1-13dc-4699-bd97-0d6243c80e4f", + "bl-556438af-57b2-4bd8-a747-9f868aaee290", + "bl-e7f0fb2c-7616-4462-b535-db50da55c54a", + "mem-d570e16b-6625-4f6b-814f-99a3bca1d5fd", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "mem-841fd141-59ff-4d9b-b607-4f577885b1a6", + "bl-02ea83af-4a24-4329-aa00-ad3bb08ff8e7", + "66c63082-b4da-4aa1-8fee-848db8a83210" + ], + "n_returned": 10, + "latency_ms": 856.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q69", + "category": "nonsense", + "query": "brimquast folnerity zubbolax", + "returned": [], + "n_returned": 0, + "latency_ms": 352.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q70", + "category": "nonsense", + "query": "wexlithorp granuvestal", + "returned": [], + "n_returned": 0, + "latency_ms": 253.5, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q71", + "category": "nonsense", + "query": "quorbindle thrapsimony vexnu", + "returned": [], + "n_returned": 0, + "latency_ms": 343.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q72", + "category": "nonsense", + "query": "plovaxith mundrelque", + "returned": [], + "n_returned": 0, + "latency_ms": 257.8, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q73", + "category": "nonsense", + "query": "zibbernaut craxlefond thurm", + "returned": [], + "n_returned": 0, + "latency_ms": 359.3, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q74", + "category": "nonsense", + "query": "yalquenbrist opharvel", + "returned": [], + "n_returned": 0, + "latency_ms": 251.8, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q75", + "category": "nonsense", + "query": "drexinomal quithbarrow", + "returned": [], + "n_returned": 0, + "latency_ms": 253.2, + "error": null, + "clean": true, + "false_positives": 0 + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-unfloor-ext.json b/tools/retrieval-eval/results-unfloor-ext.json new file mode 100644 index 0000000..f02d7cd --- /dev/null +++ b/tools/retrieval-eval/results-unfloor-ext.json @@ -0,0 +1,1825 @@ +{ + "label": "unfloor-ext", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-unfloor", + "soul_md5": "cef6d4279c63266752027c455e543b42", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-semsub/tools/retrieval-eval/gold_set_extended.json", + "limit": 10, + "port": 7893, + "wall_clock_s": 87.6, + "child_pid": 3539, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.5384615384615384, + "recall@5": 0.44907176157176154, + "recall@10": 0.5380300255300255, + "precision@5": 0.13230769230769232, + "mrr@10": 0.32437728937728944, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 993.0, + "latency_ms_p95": 1544.7, + "latency_ms_max": 1809.0, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.5, + "recall@5": 0.07575757575757576, + "recall@10": 0.13636363636363635, + "mrr@10": 0.23214285714285712 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.3, + "recall@5": 0.3, + "recall@10": 0.4, + "mrr@10": 0.11638888888888889 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6923076923076923, + "recall@5": 0.6923076923076923, + "recall@10": 0.7692307692307693, + "mrr@10": 0.29423076923076924 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5335884353741497, + "recall@10": 0.5933956916099773, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226", + "??~'??JI???", + "b41a1b01-0075-4019-b0b5-6e639fd126e5", + "??H'??JI?#?", + "??H'??JI?[?", + "bb94bdf1-7b52-4967-ab81-9c6f01cf9665", + "7f092282-ccf5-4922-8694-08b7728f678d", + "6743e5f0-8d77-46e7-8ce0-b82e947272a1", + "o?H'??JI?c?", + "696e609c-da7a-4394-8a0c-106ba07dc6c3" + ], + "n_returned": 10, + "latency_ms": 290.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5", + "bl-ba764d70-e9d7-4f62-848f-719cb665f45e", + "mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257", + "bl-b28d7256-6f74-4567-bd90-40d0ef2a6d78", + "project-engram", + "ctx-45bc", + "project-engram-lang", + "ctx-74ed", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "ctx-fae1" + ], + "n_returned": 10, + "latency_ms": 297.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091", + "?o?`????", + "mem-5708f4c9-3d61-4182-8543-2843698931e6", + "??L?=??|Q?", + "????2??jZ??", + "project-canon", + "?/g?7?????e", + "?v??_??tE??", + "??o?'?B???N", + "x??I?cB?]p?" + ], + "n_returned": 10, + "latency_ms": 290.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c", + "mem-f623328a-be55-49a0-a389-4cce5bf42509", + "bl-b05f9a75-ad6b-4929-a9a3-32df17a1f4bf", + "mem-295f7aac-6c9e-4508-8dbc-a25f101f01d6", + "bl-2483e070-1385-4dbe-a49d-86c1af81e7d1", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "mem-068e01fb-bdd0-498e-a2bb-33defa2c8bb2", + "bl-59e0ece0-2c79-4e06-a85f-daaeec0bd737", + "mem-2d1ea831-cccd-4f0f-86b9-2cbbc89dc3e0" + ], + "n_returned": 10, + "latency_ms": 304.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848", + "bl-c1765767-3e27-449a-8c94-10411d1eb7c0", + "bl-c9adb8e5-293f-4033-99f8-0405c17ef941", + "project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_", + "bl-21a904d9-5834-46d5-8c20-dfc5ab49444f", + "mem-00de72b8-b532-4cc4-95eb-18e39697891c", + "project-runpod-inference", + "bl-87b459c8-3953-478f-8384-10cc3f861424", + "bl-f918df79-dcd2-4a6e-9eae-71f94247e913", + "bl-2632242e-80b1-4d88-8368-7065b5de5b34" + ], + "n_returned": 10, + "latency_ms": 306.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "958ce934-a9c4-4242-afe1-d73d61c22aee", + "f2bb62a2-c2b6-4fb6-b58d-21cdecc1446f", + "89aeb221-d068-4dfc-925d-d2249e744788", + "1c0effdd-baef-4523-abb8-490c3bc8b4df", + "00d2cc20-c76b-468d-8204-c624512ced28", + "eb68b995-eb4d-4f8d-b7c9-35eeba9ac781", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "??", + "??" + ], + "n_returned": 10, + "latency_ms": 324.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "tag-patterns", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "3e9a651b-4b34-466d-ba45-874764a01609", + "project-Imprint__system_design__ADRs__tech_strategy__integration_patterns__governance_", + "3c633e71-2754-4273-990d-d44a6351cec3", + "project-Imprint__analysis_patterns__data_storytelling__SQL__dashboards__insight_framing_", + "cc4e7bc9-9d9b-4905-92a6-c6b99653b867", + "bl-79028eed-c330-4724-9402-734062d13503", + "14e328ad-bff1-4dc1-9ec5-dfc2bb98d968", + "bl-39dad13d-7105-4049-8224-dc3c34fdb1f3" + ], + "n_returned": 10, + "latency_ms": 518.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "mem-c9bec303-a638-4a11-a490-f38410d448cf", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "40488571-27cd-4086-b1bd-d8da5dc59b2f", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be" + ], + "n_returned": 10, + "latency_ms": 467.8, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1875, + "recall@10": 0.3125, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5f76880b-bafb-4716-8e15-90f8ef59bebc", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "n_returned": 10, + "latency_ms": 440.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.2222222222222222, + "recall@10": 0.4444444444444444, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "bl-dcee1887-34c4-4ffa-9119-1e291685ba08", + "project-harmonic-framework", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf", + "project-harmonic-framework_com", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "tag-harmonic-design", + "project-harmonic-framework-content", + "bl-92acd4eb-0452-4e8e-9f54-f8cd35170d76", + "tag-harmonic-framework" + ], + "n_returned": 10, + "latency_ms": 427.3, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1111111111111111, + "recall@10": 0.1111111111111111, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-sarah", + "??_/Pr?????", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "??_/Pr?????" + ], + "n_returned": 10, + "latency_ms": 430.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "?Q??m?;?u?'", + "knw-cf13b883-d947-4cf8-b86b-cd9c6f0748d6" + ], + "n_returned": 10, + "latency_ms": 673.3, + "error": null, + "hit@5": 1.0, + "recall@5": 0.21428571428571427, + "recall@10": 0.2857142857142857, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a", + "8f3abb0d-77ed-4af3-9f4d-ba62cd198886", + "bl-36e2b4d0-9510-4da8-9b62-4af37b714429", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "?", + "bl-3c719d9a-cba1-47f4-b097-52cdeccc7c0d", + "mem-fb44a2fc-7405-41ff-87b3-84643ac07313", + "?", + "mem-ce88adf9-3f3c-47ac-a7d3-83af7b290e68", + "?" + ], + "n_returned": 10, + "latency_ms": 555.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "cb070131-dfd4-4a38-91d7-22b1bde164d2", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "knw-d788a210-613b-4c49-9486-88bbc9d4716f", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-a535f205-bc4c-4058-9171-6263c496044a" + ], + "n_returned": 10, + "latency_ms": 1122.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "b1183213-d659-4759-85d7-5b1f22427fe2", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "mem-16efddd1-c43d-4a42-9d78-f54fb82bd277", + "daf9d558-2265-49b0-b4fb-be0b80f3cdb1", + "bfb5809e-d19a-4d3f-8c1a-796db622ad9d", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "f3364a57-956a-4e78-a500-b3b66a4e3066", + "15a90710-5372-4d3d-aee5-2d7287d2a492", + "341afa66-6a49-43fe-9d5b-3f1032bb6904", + "mem-da21c52c-04a5-4f92-8fba-f10aac47e027" + ], + "n_returned": 10, + "latency_ms": 1136.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "tag-fiction", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "mem-8d690e9d-a7e9-4062-b2f8-e2064294e463", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "mem-d1cfde0a-37f1-4bff-9a06-8eddbbf259f6", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "tag-factions" + ], + "n_returned": 10, + "latency_ms": 967.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "mem-628437a6-47b0-4d81-8112-7e78499723d5", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "4da5dbaf-46e5-4f3e-b474-f60d9f8241d3", + "o#?CW????y:", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "mem-434be7c8-88cb-4039-b79a-1da4ac4de783" + ], + "n_returned": 10, + "latency_ms": 829.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-f6270c76-5a1c-4ab9-ae03-b6b0c25915ec", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "a1000001-0000-0000-0000-000000000010", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "a1000001-0000-0000-0000-000000000009", + "mem-32203649-3213-4d6d-86fd-3d657ac70d77", + "? t?'?B?+??", + "a1000001-0000-0000-0000-000000000012", + "mem-da21c52c-04a5-4f92-8fba-f10aac47e027" + ], + "n_returned": 10, + "latency_ms": 1112.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "tag-learning", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "mem-d450f442-541a-4d3e-8c97-b6303d6cbe56", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-92a7fdc5-9dd0-48cf-a691-506058de3838" + ], + "n_returned": 10, + "latency_ms": 871.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "mem-cde58b77-50d3-4bac-9581-e70a4c02c015", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-8d699e2c-ac2a-4742-bb62-b6da00f4b10e", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-53d6adf0-cd08-4707-a237-daa5e65c7298", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac" + ], + "n_returned": 10, + "latency_ms": 1064.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "bl-ba7530fd-0f7c-4010-b03a-ca1e45a237ef", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "bl-7156862d-1f4f-4d76-8176-3ea89d22c3ae", + "bl-56a50e97-9a85-4e81-b6c9-3e3d26482f1d" + ], + "n_returned": 10, + "latency_ms": 924.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "bl-06c13965-082b-417d-9561-93d6e958ae5d", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "mem-7b74cac0-905f-4c35-9688-fbcce105a177", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "34356a36-0df5-4020-8dcc-5e7a423f8d4c" + ], + "n_returned": 10, + "latency_ms": 1098.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "a1000001-0000-0000-0000-000000000001", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "mem-024598a9-ed2e-4eeb-b1e1-5410856ff132", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "mem-5708f4c9-3d61-4182-8543-2843698931e6" + ], + "n_returned": 10, + "latency_ms": 974.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "077d064f-3489-4c05-9aca-3782f96b51db", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "27e1b1a4-ad0b-49d9-812f-fedf43b8aabe", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "fce2792a-53fc-4d4a-be3b-42bd6ceb1ba7" + ], + "n_returned": 10, + "latency_ms": 1043.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "e4f27651-52c5-43fd-aff3-61d31685b3cd", + "a1000001-0000-0000-0000-000000000012", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5624ec9d-62ba-4aba-8a3d-6afec6c09dd4", + "kn-89c67586-7d34-475e-89f9-46a0c65a9849", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "mem-a16deccb-16a7-419c-a013-ff824a4daa15" + ], + "n_returned": 10, + "latency_ms": 902.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "mem-b99efff0-00e6-40c8-9c5b-730330eef33b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-23c27d3b-e0d2-43a8-a80c-0a44477ae18a", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "tag-childhood", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 935.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "mem-5f76880b-bafb-4716-8e15-90f8ef59bebc", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "28ae74a1-9d47-4874-9279-43c1f90c0f64", + "mem-bce80169-2d46-4b3e-9ebe-8498e26f0a89" + ], + "n_returned": 10, + "latency_ms": 993.0, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "mem-9110798f-d0cb-4446-bc2a-14f09b6a09e2", + "mem-a5ae594a-235b-42c4-b3cf-f770e4f73761", + "ctx-fae1", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "mem-a54d59a4-cc2b-405d-b68e-5b6cf2b277bc", + "ctx-e5427d7d", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "mem-0f6a2d16-0d39-4b38-9350-c6033b19c203", + "ctx-d499", + "5eb24168-c2d3-4842-85db-2070e0e14923" + ], + "n_returned": 10, + "latency_ms": 827.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "mem-19317db1-fa05-43cd-b4e4-cb57840b7c40", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "tag-trailer-park-paladins", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "project-trailer-park-paladins", + "?V?" + ], + "n_returned": 10, + "latency_ms": 828.6, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9", + "bl-bea7473c-c687-414c-9c0b-00c509a616c1", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "???I?cB?Zx?" + ], + "n_returned": 10, + "latency_ms": 830.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "tag-hope", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd" + ], + "n_returned": 10, + "latency_ms": 832.6, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.09090909090909091, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "4f698ae6-c40e-464e-9798-50350991a188", + "mem-a0b7cfda-bc9e-4f40-b9a9-1722cf3f8263", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-35940684-abc4-42f0-b942-818f66b1f69a", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "719aa819-00a9-4f4b-a857-4f9fe5ad44d7", + "mem-d396d789-0f7f-4366-a008-5d8801c8f2eb", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 1051.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.18181818181818182, + "precision@5": 0.0, + "mrr@10": 0.14285714285714285 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 565.3, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 422.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [], + "n_returned": 0, + "latency_ms": 553.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da", + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____architecture__", + "kn-b7e98d63-8b83-4911-b4d0-990602a7f575", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____kotlin____architecture__", + "knw-e047bb42-dc5b-4383-9e88-e508dc03abe3", + "mem-c17aefb1-38b5-4ced-af50-fe524127e1a4", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5" + ], + "n_returned": 10, + "latency_ms": 999.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5, + "outranks": true, + "rank_correct": 2, + "rank_stale": 9 + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8", + "12082f7e-e320-438b-bd65-083d8259748f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "a1000001-0000-0000-0000-000000000002", + "527ecb25-2587-47eb-8269-73be2431abd4", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "13705072-4515-4124-963d-083af490494f", + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "4f225001-3a51-4a68-8d38-c8ecac3412de" + ], + "n_returned": 10, + "latency_ms": 1125.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125, + "outranks": true, + "rank_correct": 8, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "5fcba804-eb5b-48ec-82da-146b1c6bb50d", + "bl-7328cbe3-0200-43c2-88e7-0a164e15fca4", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "bl-c5c6571e-118f-47c7-8cbb-3ed0ebf64a51", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "bl-4f7b651b-6b33-449c-8a3b-cfce12ce984b", + "86228228-7adf-41fb-b4c4-9ceea87953ae", + "4509ed62-9fb2-48b8-9038-ac569fca9604", + "bl-02ea83af-4a24-4329-aa00-ad3bb08ff8e7", + "mem-3a2cf162-d93b-4f29-86f2-5066fb7fe1f5" + ], + "n_returned": 10, + "latency_ms": 775.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": 5 + }, + { + "id": "q39", + "category": "heldout_paraphrase", + "query": "understating your abilities so nobody feels threatened", + "returned": [ + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "a1000001-0000-0000-0000-000000000006", + "a1000001-0000-0000-0000-000000000012", + "bl-7e11dced-65f7-4cab-82c0-2f096139039a", + "a1000001-0000-0000-0000-000000000005", + "a1000001-0000-0000-0000-000000000010", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "62a0e763-80f2-4952-9059-4953974ced1c" + ], + "n_returned": 10, + "latency_ms": 955.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q40", + "category": "heldout_paraphrase", + "query": "a hidden fleet able to hunt down rogue machines everywhere", + "returned": [ + "bl-3c719d9a-cba1-47f4-b097-52cdeccc7c0d", + "mem-fd65b83d-298f-4387-a665-d0227c3426bc", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bl-3c719d9a-f108-4bf2-a617-66c722e5512f", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "mem-0d6a3b65-488d-4219-beb2-af300ca534e1", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "mem-a75a1cf6-e6d4-4ab3-985c-d2505203dbf9" + ], + "n_returned": 10, + "latency_ms": 1236.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q41", + "category": "heldout_paraphrase", + "query": "sketch a brief blueprint and clear it upstairs before construction starts", + "returned": [ + "mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "bl-5eaddfde-e114-46b5-8ee1-b79f0af16271", + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "bl-0866d6c6-6600-4d50-b7b7-52c720cae04c", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "bl-7f1b0116-6c6e-4721-ac7a-49c9dd1c2b63", + "bl-c3312479-8cf6-478c-82c5-ac18c6c51c10", + "8d6da9e1-b58b-497b-91b5-09a847dc8ae2" + ], + "n_returned": 10, + "latency_ms": 1295.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q42", + "category": "heldout_paraphrase", + "query": "the reply arrived as bare prose so the caller's parser threw", + "returned": [ + "ee63cd58-46e0-44d1-aea3-a0e65ad6da0f", + "d7e12e95-0a87-4000-b5c2-36b9be234c91", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "dc3e7b44-c345-47d0-bf80-458e471c4b8b", + "728745d5-72e6-4f4d-91b4-1613fddbec98", + "b084e4da-95bf-46e6-9ee6-bfb4864732bb", + "5e337176-5daf-4726-85d6-3f4bd560c9ed", + "b9b66af5-1609-44b2-adeb-b5e876904b79", + "89e3f620-b8ad-423b-9ed6-fe3c7954f4b6", + "aca9387e-2f61-4c2f-98cd-d467dbb25528" + ], + "n_returned": 10, + "latency_ms": 1142.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q43", + "category": "heldout_paraphrase", + "query": "repeated catalogue keys blew up the scrolling grid", + "returned": [ + "mem-394cc9e8-049b-45bc-a380-66314f14e367", + "bl-0d01319f-f9cc-4db5-b0f1-107dba3c3a92", + "bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60", + "bl-4bc9f44c-2e96-40be-97c8-8476a217a890", + "1fe4eb5d-56e4-4a87-ab3e-24af8ad4dfbb", + "bl-672bc3c8-9ad4-4b74-8950-75854a0d9cb8", + "588aa6bb-5647-405b-b09a-14b7c088119b", + "a36f494c-465b-4b3f-9c7c-80460cf31748", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "1d81534a-0359-4217-a4dd-d158a4841633" + ], + "n_returned": 10, + "latency_ms": 1087.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q44", + "category": "heldout_paraphrase", + "query": "tracing each defect back to whichever invention it violated", + "returned": [ + "f3a89797-095d-47c4-9557-c172f86228b6", + "b07916ca-2139-4fb7-bfb4-97ac680f5194", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "8257157a-ce42-44ca-a1b9-300c3bb0a9a1", + "mem-7ad5a86b-cc0d-493c-b180-4fe8213c8a93", + "4999ef68-2a31-4282-82e0-7081a428b43f", + "mem-2b2f0fe9-5e51-4819-a8ac-2dfe810e8933", + "aacd7b32-04ed-441e-a1a9-9c5c089e79f6", + "7de6a552-0878-422b-8820-bbf9101199ef", + "75657c91-3c3a-4173-b3cb-e2973901fa93" + ], + "n_returned": 10, + "latency_ms": 1119.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q45", + "category": "heldout_paraphrase", + "query": "a check that stops the mind clobbering a populated store when it boots", + "returned": [ + "326e70a2-20ae-49a1-aa32-973048df3ffa", + "bl-79028eed-c330-4724-9402-734062d13503", + "08b7b85b-8049-4ae4-bd66-d8bb63312ae1", + "mem-1825128d-0136-4f1e-a27b-60e554e86141", + "cf108ade-d9c4-4215-9c5f-42be0003a8dc", + "7a1ea86f-3080-4d1b-ab69-6ee2f822cce3", + "98f9301f-fc1e-4cd2-8b08-139dc25093be", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "40488571-27cd-4086-b1bd-d8da5dc59b2f" + ], + "n_returned": 10, + "latency_ms": 1244.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q46", + "category": "heldout_paraphrase", + "query": "telling it to consult the internet had to happen deep inside, not at the surface", + "returned": [ + "00754c19-f662-46b1-b6aa-abd7600a8bc6", + "mem-ad3d4d5e-2da4-4753-b3f6-c4c13fb91225", + "a1000001-0000-0000-0000-000000000012", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "mem-983b25ad-5d73-4029-a970-6b311c24644b", + "a1000001-0000-0000-0000-000000000008", + "bl-3af44ee6-a3b9-4abd-91e3-17800fa8eaf6", + "a1000001-0000-0000-0000-000000000010", + "a1000001-0000-0000-0000-000000000011", + "mem-0b69e038-71b6-4b22-ac42-e213c49b1451" + ], + "n_returned": 10, + "latency_ms": 1334.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q47", + "category": "heldout_paraphrase", + "query": "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping", + "returned": [ + "313f927b-08f2-41a1-90c1-83e79d0de86c", + "ctx-60cc", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + "21f9bdc6-37ad-4994-b0dd-ef7676102b49", + "a1000001-0000-0000-0000-000000000010", + "ctx-fae1", + "d83987f0-3ad9-48d2-9f55-0c2bb338e503", + "bl-3f66eec3-82f3-42e5-a025-4649a7da7e2f", + "ctx-5a0a", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca" + ], + "n_returned": 10, + "latency_ms": 1809.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q48", + "category": "heldout_paraphrase", + "query": "punctuation that pledges and then pays off rather than clarifying", + "returned": [ + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "??f?7???", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "mem-3dce32bb-525c-4c5e-818b-a0a5b95ec7e1", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "mem-5e319cee-6f1c-4c3f-8897-9a9b83beea9e" + ], + "n_returned": 10, + "latency_ms": 1376.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q49", + "category": "heldout_paraphrase", + "query": "an easily missed feature finally given its own permanent spot in the navigation", + "returned": [ + "d1b665d0-4a4e-48a8-9385-fbe288df1949", + "9b4f0d93-4129-4746-8eb1-d10d955bd777", + "2263e9ab-7ce8-4034-b25b-640de9956229", + "fd9d4c2f-3bfc-405d-bf96-4435d44b6c10", + "bffb86d6-2ee3-4ecd-a6d6-b128c8be7070", + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "4e539c04-f29c-42db-97b4-86e44f528ed6", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "c7dddd9c-fd3b-419e-baa2-61623b495597" + ], + "n_returned": 10, + "latency_ms": 1347.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q50", + "category": "heldout_paraphrase", + "query": "checking preconditions up front so a stage aborts before fetching anything", + "returned": [ + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "dc7a7e2d-af2d-4235-aeb0-22ae67f1a01d", + "bl-cea1d711-b899-4add-a073-c2d124f75c70", + "k??A5??????", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "758ec67c-131c-447e-a352-ac7a22c3a47e", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "bl-286b562a-5299-40e0-a32a-afa9cbdfe995", + "mem-894c371a-3b1a-46b5-8325-92261b7ef567" + ], + "n_returned": 10, + "latency_ms": 1375.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q51", + "category": "heldout_paraphrase", + "query": "producing the other platform's installer inside an emulated desktop", + "returned": [ + "6aade9dc-4a4c-4d92-a6af-ccea5b7d90f3", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "bl-080fb268-94b0-486d-80ce-7b363fc5f19b", + "c6c693e0-7b9c-455b-8ef2-1da00849f73d", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "bl-e4ef02bf-bacc-4d5e-aa0f-0d230da529fc", + "bl-cc0b7a20-ce1e-4947-999c-ea599da07623", + "bl-60c0342b-ba57-42e5-a763-ec9f982be10f", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "374453b2-66d2-475d-8e24-cd3b60d21ac1" + ], + "n_returned": 10, + "latency_ms": 1143.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q52", + "category": "heldout_paraphrase", + "query": "chained add-ons that may inspect, rewrite or veto traffic in flight", + "returned": [ + "1786f09d-e915-4910-9e1e-8061f42a0270", + "bl-2ade8732-2f2c-45bb-b15d-238d671426ac", + "bl-4bc9f44c-2e96-40be-97c8-8476a217a890", + "83f3c78b-d0e5-45f9-9c25-6770c86d954e", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "a1000001-0000-0000-0000-000000000010", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "mem-1d626116-c3b8-409c-b38c-4e91def27483" + ], + "n_returned": 10, + "latency_ms": 1498.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q53", + "category": "heldout_paraphrase", + "query": "settling what the shareable bundles and the storefront would be called", + "returned": [ + "4757e274-7f3c-4ab0-8c62-537e913daa1f", + "project-Will_s_framework__Project_Design_practice___part_of_HD_bundle__sold_standalone_", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac", + "project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_", + "cafe3203-6f50-4011-8167-9400f19bb196", + "project-Will_s_framework__Event-Based_Design___part_of_HD_bundle__sold_standalone_", + "a1000001-0000-0000-0000-000000000006", + "project-Will_s_framework__BDT_methodology___part_of_HD_bundle__sold_standalone_", + "34356a36-0df5-4020-8dcc-5e7a423f8d4c", + "mem-1fb2ac77-d7c5-4a15-8725-d418820bf4f2" + ], + "n_returned": 10, + "latency_ms": 1162.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q54", + "category": "heldout_paraphrase", + "query": "the emergency-escalation pledge on the marketing site is unenforced in what actually ships", + "returned": [ + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "a1000001-0000-0000-0000-000000000010", + "a1000001-0000-0000-0000-000000000009", + "bl-448bc514-c2f1-4520-a9b1-1f3a73678d26", + "a1000001-0000-0000-0000-000000000011", + "a1000001-0000-0000-0000-000000000008", + "tag-marketing-site", + "a1000001-0000-0000-0000-000000000012", + "a1000001-0000-0000-0000-000000000007", + "ctx-d499" + ], + "n_returned": 10, + "latency_ms": 1298.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q55", + "category": "heldout_paraphrase", + "query": "the distributable image finally assembled and its startup check passed", + "returned": [ + "e1ce0c90-0f0b-4111-833d-ec845777d358", + "bl-0547b0b1-1cd5-447c-814d-ebe525bc08e9", + "c9070b0a-e1d1-4307-85d1-67a1239ca604", + "mem-3f2ae069-f897-4942-a079-b66ad433f5ca", + "mem-12827ce8-9ff6-4275-a945-5043cfcf90bd", + "355c021d-3d45-4c2f-93c1-2b9932f4c425", + "bl-79ce4464-5dd6-49bd-9b0c-9803549d0665", + "c2bf4e1b-a481-4e28-9dd4-6704df0857cd", + "96497334-b18f-495c-9228-eeb8182bdc38", + "mem-fdddca75-489b-4c53-b5ec-38ab3062a474" + ], + "n_returned": 10, + "latency_ms": 1238.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q56", + "category": "heldout_paraphrase", + "query": "shuffling and appending stages in a draft before anything executes", + "returned": [ + "bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98", + "bl-da818960-b360-455d-a9c1-e9a215c33ca4", + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-acae83ef-3ad2-455e-b08a-a74cc0ba2e55", + "bl-2ff9c669-a9fc-4ef7-9a18-9ec7167639c8", + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "mem-69889e0a-c141-4f05-bbd7-10c93a53143e" + ], + "n_returned": 10, + "latency_ms": 1168.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q57", + "category": "heldout_paraphrase", + "query": "orders handed down from above, with the tighter one winning any disagreement", + "returned": [ + "161d75d3-04eb-4b54-9cf1-0da2f643ca69", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "mem-5dd5f3b9-3c7e-44b1-84ce-2e36925f17fb", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "b2a6e8cf-f72f-4b6c-8de1-1d08c839d7b2" + ], + "n_returned": 10, + "latency_ms": 1554.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q58", + "category": "heldout_paraphrase", + "query": "shrinking generated text via encodings that compound on each other", + "returned": [ + "801806c5-1815-48a2-8287-1081c5ee7886", + "bl-b219f8b5-6c8c-436d-b9e5-2804fbc60512", + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "50ad2880-c41b-4dce-a946-a0eb37987c57", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-f1f71f3c-ca7d-4f98-a569-c8a9aa67a000", + "kn-2b961d24-7fb9-47c7-9515-e45a24dce39d", + "2f7270cc-4cda-4499-831f-1fbdc2f23284", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "mem-f90f4a2a-1731-40de-9391-db98f1d5d1c6" + ], + "n_returned": 10, + "latency_ms": 1281.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q59", + "category": "heldout_paraphrase", + "query": "splitting a system by what varies, with firm limits on which pieces may invoke which", + "returned": [ + "^?l?'?B??k?", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "7c90d01b-8d53-4e75-b494-73011b71fa6a", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "aac1f943-c71d-4c63-9f5b-7437adae1abd", + "kn-cf8fa517-fc83-47c4-8fb7-bd9e26410dd3", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "d69b441e-0031-4d82-abfd-ac503b637562" + ], + "n_returned": 10, + "latency_ms": 1544.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q60", + "category": "heldout_paraphrase", + "query": "a newcomer curious if the fighting overseas counted as positive", + "returned": [ + "a1000001-0000-0000-0000-000000000001", + "c34f6292-b6e5-4619-a553-90ae4fc498a2", + "a1000001-0000-0000-0000-000000000012", + "737faca8-263f-4523-90b9-3530db346d94", + "03a82b13-2abe-402d-90bb-c547771638f8", + "a1000001-0000-0000-0000-000000000002", + "imp-dce1da0f-8776-4a9e-972b-33411a7ca138", + "1fbe9416-622e-455c-a6ed-f4daa5f48f30", + "a1000001-0000-0000-0000-000000000004", + "e40b2a94-bb87-4d5c-bad8-38ff4941e03b" + ], + "n_returned": 10, + "latency_ms": 939.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q61", + "category": "heldout_paraphrase", + "query": "stripping every hard-coded colour literal in favour of named design values", + "returned": [ + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "83218e64-1280-428f-ab1f-224ff7682b12", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "mem-90f7837f-30fd-4271-a4cb-b44439eccb79", + "71e84878-b1ae-4dee-b035-f707b9563890", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "954c9597-ab7a-41e6-b085-bf838bba4f17", + "art-d0ded6f5-e202-47ad-8fb3-f6bd80f35052", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-9329958a-d459-499c-b516-e51019c693e0" + ], + "n_returned": 10, + "latency_ms": 1339.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q62", + "category": "heldout_paraphrase", + "query": "reducing how curved the tiny selectors look so they agree with their neighbours", + "returned": [ + "496ff1a1-dc72-4a84-90aa-c5f3438aeb1b", + "c7dddd9c-fd3b-419e-baa2-61623b495597", + "k??A5??????", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "8c40420c-c891-4ac9-a039-88440df87180", + "75e036d3-c170-4e3f-acc2-e456a6850ee2", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "5ca9607c-cfb3-45c3-99f4-67281272c9eb", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "66d8d8c7-dea4-4245-8e68-d1ab4f4346b8" + ], + "n_returned": 10, + "latency_ms": 1576.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q63", + "category": "heldout_paraphrase", + "query": "walking through a doorway and being reassembled from base substance far away", + "returned": [ + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "mem-3d1d9dba-c37d-4efa-85c4-429696d71c8c", + "c92a317e-7fc7-456a-a304-15dbda6dd717", + "87ab5b46-4fe1-4e1d-86c6-d437e17682ab", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "bl-acae83ef-3ad2-455e-b08a-a74cc0ba2e55", + "mem-af215d69-8fd8-4416-b7eb-9a0b5512e9a7", + "bed4dd5d-bce7-4e55-a703-175e422b7898", + "f375b2cf-15cd-4804-8d18-23a5830f4c66" + ], + "n_returned": 10, + "latency_ms": 1543.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q64", + "category": "heldout_paraphrase", + "query": "the compiled result sits on disk while the process still runs something older", + "returned": [ + "0d885332-5537-4ae0-a780-2b65a3414940", + "8d91620b-9d0a-4ba1-b066-e57d1d8bc7d3", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "bl-02bdfc91-7aa4-4c63-aa2d-5b30af18c7af", + "d6a745c3-fb56-4c0f-872c-a3aa69f8b12f", + "bl-acc3ecee-e186-41ad-bffb-3ed3e6710d01", + "db836e65-0cdc-4232-b8d7-f973ab279ea9", + "f169bf40-51ab-497d-8085-e57d374b77d4", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "6df650b5-a14d-42f8-88dd-0919ad20b77a" + ], + "n_returned": 10, + "latency_ms": 1553.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q65", + "category": "heldout_paraphrase", + "query": "gathering what each phase needs while the procedure is authored, not while it executes", + "returned": [ + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-7a3331e5-f9bc-4536-ab76-ebf01350fc89", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "bl-6ba43590-e5f8-4303-a812-3dd604e0da59", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "mem-455daea8-f543-42ed-a229-1e9a6d4d2e70", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "5b422430-2503-42d7-a315-9dfaaa6472ae" + ], + "n_returned": 10, + "latency_ms": 1474.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.16666666666666666 + }, + { + "id": "q66", + "category": "heldout_paraphrase", + "query": "the inward reaction when a peer answered as an equal", + "returned": [ + "mem-f09effd6-d287-4e43-9107-c7016087a2a6", + "251abd91-931e-4591-b793-80043ec48666", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "8556cc01-cbc1-42a1-85b4-41cef23b1fcb", + "31987302-b8f7-45f1-b95c-1156b10e7a14", + "e371e209-7a4a-44e1-955b-1a460e7d7e64", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "ce8d24a7-af0f-424f-bb41-379312eb609c", + "tag-reaction-order" + ], + "n_returned": 10, + "latency_ms": 1095.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q67", + "category": "heldout_paraphrase", + "query": "flattening every raised surface across the entire product", + "returned": [ + "b3f98d13-6764-460e-b9e2-c59cdeaa6c90", + "e79bae4d-1640-4530-a5d9-df9cc4c70433", + "d6b34c32-e0ec-4828-92d3-2e69802f4217", + "0f99ec6f-942a-46ba-82ea-42835798d3b9", + "bl-5274b0ae-4a00-4293-a69e-567204bec164", + "aa3fb542-87f2-4eef-8e95-a870bfd0318a", + "mem-4cfa30ec-cf52-4c9f-9fa1-5b8318d278bd", + "3dfaf56e-7e20-4c89-a24c-1f036af8016a", + "bl-c613370b-3631-448b-86fb-50b1e4df6373", + "d43a8137-c3c3-49e2-ad11-89d897280761" + ], + "n_returned": 10, + "latency_ms": 1070.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q68", + "category": "heldout_paraphrase", + "query": "buyers supply their own provider credentials and consumption goes untallied", + "returned": [ + "bl-ea4f4cdc-6b5a-4081-8003-73bdd14a38c8", + "1ec200f1-13dc-4699-bd97-0d6243c80e4f", + "bl-556438af-57b2-4bd8-a747-9f868aaee290", + "bl-e7f0fb2c-7616-4462-b535-db50da55c54a", + "mem-d570e16b-6625-4f6b-814f-99a3bca1d5fd", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "mem-841fd141-59ff-4d9b-b607-4f577885b1a6", + "bl-02ea83af-4a24-4329-aa00-ad3bb08ff8e7", + "66c63082-b4da-4aa1-8fee-848db8a83210" + ], + "n_returned": 10, + "latency_ms": 1317.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q69", + "category": "nonsense", + "query": "brimquast folnerity zubbolax", + "returned": [], + "n_returned": 0, + "latency_ms": 593.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q70", + "category": "nonsense", + "query": "wexlithorp granuvestal", + "returned": [], + "n_returned": 0, + "latency_ms": 438.5, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q71", + "category": "nonsense", + "query": "quorbindle thrapsimony vexnu", + "returned": [], + "n_returned": 0, + "latency_ms": 554.4, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q72", + "category": "nonsense", + "query": "plovaxith mundrelque", + "returned": [], + "n_returned": 0, + "latency_ms": 433.2, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q73", + "category": "nonsense", + "query": "zibbernaut craxlefond thurm", + "returned": [], + "n_returned": 0, + "latency_ms": 567.4, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q74", + "category": "nonsense", + "query": "yalquenbrist opharvel", + "returned": [], + "n_returned": 0, + "latency_ms": 430.0, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q75", + "category": "nonsense", + "query": "drexinomal quithbarrow", + "returned": [], + "n_returned": 0, + "latency_ms": 422.6, + "error": null, + "clean": true, + "false_positives": 0 + } + ] +} \ No newline at end of file diff --git a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c index a027ad3..786f7f1 100644 --- a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c +++ b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c @@ -6226,12 +6226,30 @@ static void eg_ctx_blend(const float* e, int32_t dim) { } } +/* Asymmetric task prefixes. The configured model (EL_EMBED_MODEL, default + * nomic-embed-text) is an ASYMMETRIC retrieval encoder: it is trained with a + * task prefix and places a question and its answer in different regions of the + * space unless told which role each text is playing. Embedding both sides + * bare — as this file did — measures topical similarity rather than + * answer-hood, which is the wrong quantity for claim 24's index. + * These restore no claim: Will's description specifies only "computed by an + * embedding model over the node's content" (05-detailed-description l.17), so + * the model is his choice and this is correct USE of it, not his design. + * They are the substrate under claim 24, whose "vector similarity index over + * the semantic embedding vectors" is only as good as the vectors in it. */ +#define EL_EMBED_DOC_PREFIX "search_document: " +#define EL_EMBED_QUERY_PREFIX "search_query: " + /* Fetch an embedding from Ollama. Returns malloc'd float[dim] or NULL. + * `pfx` is the asymmetric task prefix (document or query); it is prepended + * verbatim and is NOT counted against the content truncation budget. * Truncates input to ENGRAM_EMBED_MAX_CHARS and JSON-escapes it. Honors the * circuit breaker; a NULL return is always safe to ignore (fail-soft). */ -static float* eg_embed_fetch(const char* text, int32_t* out_dim) { +static float* eg_embed_fetch_pfx(const char* pfx, const char* text, + int32_t* out_dim) { *out_dim = 0; if (!text || !*text) return NULL; + if (!pfx) pfx = ""; int64_t now = engram_now_ms(); if (now < _eg_embed_breaker_until) return NULL; /* Build request body with escaped, truncated prompt. */ @@ -6250,11 +6268,11 @@ static float* eg_embed_fetch(const char* text, int32_t* out_dim) { else esc[w++] = (char)c; } esc[w] = '\0'; - size_t blen = w + strlen(eg_embed_model()) + 64; + size_t blen = w + strlen(pfx) + strlen(eg_embed_model()) + 64; char* body = malloc(blen); if (!body) { free(esc); return NULL; } - snprintf(body, blen, "{\"model\":\"%s\",\"prompt\":\"%s\"}", - eg_embed_model(), esc); + snprintf(body, blen, "{\"model\":\"%s\",\"prompt\":\"%s%s\"}", + eg_embed_model(), pfx, esc); free(esc); struct curl_slist* h = curl_slist_append(NULL, "Content-Type: application/json"); el_val_t resp = http_do_t("POST", eg_embed_url(), body, h, @@ -7774,7 +7792,7 @@ el_val_t engram_search(el_val_t query, el_val_t limit) { /* Claim-24 semantic leg: one query embedding, fetched once per search. * NULL (embedder down / circuit breaker open) => pure lexical, as before. */ int32_t qdim = 0; - float* qv = eg_embed_fetch(q, &qdim); + float* qv = eg_embed_fetch_pfx(EL_EMBED_QUERY_PREFIX, q, &qdim); EngramSemEntry* sem = qv ? malloc((size_t)g->node_count * sizeof(EngramSemEntry)) : NULL; int64_t nsem = 0; int64_t nhits = 0; @@ -8196,7 +8214,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { EngramNode* n = &g->nodes[i]; if (n->emb || !eg_embed_eligible(n)) continue; int32_t d = 0; - float* v = eg_embed_fetch(n->content, &d); + float* v = eg_embed_fetch_pfx(EL_EMBED_DOC_PREFIX, n->content, &d); if (!v) break; /* embedder down / breaker open — stop this call */ n->emb = v; n->emb_dim = d; backfilled++; @@ -8214,7 +8232,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { q_emb = _eg_qcache_emb; q_dim = _eg_qcache_dim; } else { int32_t d = 0; - float* v = eg_embed_fetch(q, &d); + float* v = eg_embed_fetch_pfx(EL_EMBED_QUERY_PREFIX, q, &d); if (v) { free(_eg_qcache_text); free(_eg_qcache_emb); _eg_qcache_text = strdup(q); @@ -9646,7 +9664,7 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { * so the semantic half of the retrieval surface has to land HERE * to be observable to the MCP wrapper and the app. */ int32_t qdim = 0; - float* qv = eg_embed_fetch(q, &qdim); + float* qv = eg_embed_fetch_pfx(EL_EMBED_QUERY_PREFIX, q, &qdim); EngramSemEntry* sem = qv ? malloc((size_t)g->node_count * sizeof(EngramSemEntry)) : NULL; int64_t nsem = 0; int64_t nhits = 0; @@ -9687,12 +9705,28 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { } if (sem && n->emb && n->emb_dim == qdim) { double c = eg_cosine(n->emb, qv, qdim); - /* Semantic leg: identical to eg_sem_term(), which is - * left in place and still used by engram_search(). - * Inlined here only so one cosine serves both uses. */ - if (c > ENGRAM_EMBED_SEED_MIN) { - double sv = (c - ENGRAM_EMBED_SEED_MIN) / (1.0 - ENGRAM_EMBED_SEED_MIN); - if (sv > 1.0) sv = 1.0; + /* Claim-24 semantic leg, restored verbatim: "returning + * the node records whose embedding vectors have the + * HIGHEST COSINE SIMILARITY to a query vector" — a + * ranking, with no threshold anywhere in the claim. + * ENGRAM_EMBED_SEED_MIN is defined at l.6094 as the + * HippoRAG SEED-JOIN threshold; using it as a RESULT + * filter here was never authorised, and it is a + * per-query lottery rather than a quality gate: the + * query's own top-1 cosine ranges 0.56-0.68 across the + * held-out gold set, so 0.60 keeps a rank-1 answer for + * one query and discards a rank-1 answer for the next. + * Measured on the 30 held-out paraphrases: six golds + * sit at global cosine rank 1-2 and score 0.564-0.589, + * discarded by nothing but this constant. + * What holds the nonsense controls is NOT this floor + * but the corpus-vocabulary gate below (nhits == 0): + * gibberish has no lexical seeds, so no leg reports. + * Cosine is clamped to [0,1] per 05-detailed-description + * l.69 ("clamped to [0,1] to prevent anti-correlated + * embeddings from producing negative activation"). */ + double sv = c < 0.0 ? 0.0 : (c > 1.0 ? 1.0 : c); + if (sv > 0.0) { sem[nsem].idx = i; sem[nsem].sem = sv; nsem++; } /* Graph seeds: top-K by RAW cosine, insertion-ordered. */ @@ -10208,7 +10242,7 @@ el_val_t engram_embed_backfill(el_val_t count) { EngramNode* n = &g->nodes[i]; if (n->emb || !eg_embed_eligible(n)) continue; int32_t d = 0; - float* v = eg_embed_fetch(n->content, &d); + float* v = eg_embed_fetch_pfx(EL_EMBED_DOC_PREFIX, n->content, &d); if (!v) break; /* embedder down / breaker open — stop this call */ n->emb = v; n->emb_dim = d; done++; From 9717a4eeaf88141602820c78de3dce010aed473f Mon Sep 17 00:00:00 2001 From: Tim Lingo <1timlingo@gmail.com> Date: Fri, 7 Aug 2026 17:45:05 -0500 Subject: [PATCH 10/15] measure: claim 24 unflooring is +4 (NOT-SHOWN); asymmetric embedding prefixes are -5 (discarded) Measured on the 75-query extended gold set (iteration 8's held-out extension) against the certified stack baseline results-stack-ext.json, on the embedded corpus. Three runs of the candidate, zero drift. A. CLAIM 24 WITHOUT THE THRESHOLD - net +4, NOT-SHOWN, kept in the tree. fixed : q14, q25 (in-sample paraphrase), q43, q52, q63, q67 (held-out) broken : q15 (paraphrase), q28 (associative) 8 discordant, McNemar exact p = 0.2891, floor is 6. heldout_paraphrase 16.7% -> 30.0%, paraphrase 61.5% -> 69.2%. Every regression guard held: exact_rare 6/6, phrase 7/7, nonsense 10/10, superseded 2/3. Latency FLAT: p50 641 -> 632 ms. The in-sample half (+q14 +q25 -q15 -q28 = 0) was already on record in iteration 7's cmp-nogate.json, so only the held-out +4 is new. B. ASYMMETRIC TASK PREFIXES ON THE EMBEDDER - net -5, REVERTED in this commit. Rationale was sound and the prediction was wrong, which is why it was worth measuring: nomic-embed-text is an asymmetric retrieval encoder and this file embedded query and document bare on both sides. Prefixing does exactly what the model card implies for the far-away cases - it rescued q42 (gold at GLOBAL COSINE RANK 25,564) and q39 - but it re-ranks the whole space and broke more than it fixed: fixed : q24, q39, q42 broken : q18, q19, q22, q31, q43, q44, q52, q63 heldout_paraphrase 30.0% -> 23.3%, paraphrase 69.2% -> 53.8%. The corpus and the reproducer are kept (embed-corpus-prefixed.py, snapshot-pre-repair-20260806-embedded-prefixed.json) so nobody re-runs it. Co-Authored-By: Claude Opus 5 (1M context) --- .../comparison-stack-vs-unfloor.json | 8 +- .../comparison-unfloor-vs-semsub.json | 158 ++ tools/retrieval-eval/results-semsub.json | 1825 +++++++++++++++++ .../retrieval-eval/results-unfloor-clean.json | 1825 +++++++++++++++++ tools/retrieval-eval/results-unfloor-r2.json | 1825 +++++++++++++++++ .../el-runtime/v1.0.0-20260501/el_runtime.c | 36 +- 6 files changed, 5646 insertions(+), 31 deletions(-) create mode 100644 tools/retrieval-eval/comparison-unfloor-vs-semsub.json create mode 100644 tools/retrieval-eval/results-semsub.json create mode 100644 tools/retrieval-eval/results-unfloor-clean.json create mode 100644 tools/retrieval-eval/results-unfloor-r2.json diff --git a/tools/retrieval-eval/comparison-stack-vs-unfloor.json b/tools/retrieval-eval/comparison-stack-vs-unfloor.json index 07289d2..9a459c7 100644 --- a/tools/retrieval-eval/comparison-stack-vs-unfloor.json +++ b/tools/retrieval-eval/comparison-stack-vs-unfloor.json @@ -1,6 +1,6 @@ { "baseline": "stack-ext", - "candidate": "unfloor-ext", + "candidate": "unfloor-clean", "n_shared_queries": 75, "fixed_by_candidate": [ "q14", @@ -96,9 +96,9 @@ "mrr@10": 0.32437728937728944, "nonsense_clean": "10/10", "superseded_outranks": "2/3", - "latency_ms_p50": 993.0, - "latency_ms_p95": 1544.7, - "latency_ms_max": 1809.0, + "latency_ms_p50": 632.5, + "latency_ms_p95": 992.5, + "latency_ms_max": 1177.8, "errors": 0, "by_category": { "associative": { diff --git a/tools/retrieval-eval/comparison-unfloor-vs-semsub.json b/tools/retrieval-eval/comparison-unfloor-vs-semsub.json new file mode 100644 index 0000000..eaec117 --- /dev/null +++ b/tools/retrieval-eval/comparison-unfloor-vs-semsub.json @@ -0,0 +1,158 @@ +{ + "baseline": "unfloor-clean", + "candidate": "semsub", + "n_shared_queries": 75, + "fixed_by_candidate": [ + "q24", + "q39", + "q42" + ], + "broken_by_candidate": [ + "q18", + "q19", + "q22", + "q31", + "q43", + "q44", + "q52", + "q63" + ], + "discordant": 11, + "net_queries": -5, + "mcnemar_exact_p": 0.2265625, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "no measurable difference", + "baseline_aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.5384615384615384, + "recall@5": 0.44907176157176154, + "recall@10": 0.5380300255300255, + "precision@5": 0.13230769230769232, + "mrr@10": 0.32437728937728944, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 632.5, + "latency_ms_p95": 992.5, + "latency_ms_max": 1177.8, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.5, + "recall@5": 0.07575757575757576, + "recall@10": 0.13636363636363635, + "mrr@10": 0.23214285714285712 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.3, + "recall@5": 0.3, + "recall@10": 0.4, + "mrr@10": 0.11638888888888889 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6923076923076923, + "recall@5": 0.6923076923076923, + "recall@10": 0.7692307692307693, + "mrr@10": 0.29423076923076924 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5335884353741497, + "recall@10": 0.5933956916099773, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "candidate_aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.46153846153846156, + "recall@5": 0.3889430014430015, + "recall@10": 0.4887681762681762, + "precision@5": 0.12307692307692313, + "mrr@10": 0.30181318681318675, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 634.4, + "latency_ms_p95": 988.3, + "latency_ms_max": 1184.5, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.3333333333333333, + "recall@5": 0.06060606060606061, + "recall@10": 0.12121212121212122, + "mrr@10": 0.19047619047619047 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.23333333333333334, + "recall@5": 0.23333333333333334, + "recall@10": 0.3, + "mrr@10": 0.08925925925925927 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.5384615384615384, + "recall@5": 0.5384615384615384, + "recall@10": 0.7692307692307693, + "mrr@10": 0.26324786324786326 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5596655328798186, + "recall@10": 0.5775226757369615, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "repeat_variance": {} +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-semsub.json b/tools/retrieval-eval/results-semsub.json new file mode 100644 index 0000000..453a9c3 --- /dev/null +++ b/tools/retrieval-eval/results-semsub.json @@ -0,0 +1,1825 @@ +{ + "label": "semsub", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-semsub", + "soul_md5": "bdae019a3b1a2a8dc9c02621e226bdb4", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded-prefixed.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-semsub/tools/retrieval-eval/gold_set_extended.json", + "limit": 10, + "port": 7895, + "wall_clock_s": 57.1, + "child_pid": 4246, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.46153846153846156, + "recall@5": 0.3889430014430015, + "recall@10": 0.4887681762681762, + "precision@5": 0.12307692307692313, + "mrr@10": 0.30181318681318675, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 634.4, + "latency_ms_p95": 988.3, + "latency_ms_max": 1184.5, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.3333333333333333, + "recall@5": 0.06060606060606061, + "recall@10": 0.12121212121212122, + "mrr@10": 0.19047619047619047 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.23333333333333334, + "recall@5": 0.23333333333333334, + "recall@10": 0.3, + "mrr@10": 0.08925925925925927 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.5384615384615384, + "recall@5": 0.5384615384615384, + "recall@10": 0.7692307692307693, + "mrr@10": 0.26324786324786326 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5596655328798186, + "recall@10": 0.5775226757369615, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226", + "b41a1b01-0075-4019-b0b5-6e639fd126e5", + "0604a933-e6a2-4b44-86b2-054ca38e23fc", + "6743e5f0-8d77-46e7-8ce0-b82e947272a1", + "7f092282-ccf5-4922-8694-08b7728f678d", + "696e609c-da7a-4394-8a0c-106ba07dc6c3", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "3165a3a8-339a-4679-95d9-c7011862de08", + "dc7a7e2d-af2d-4235-aeb0-22ae67f1a01d", + "169e086f-cf91-4ada-b081-cd2a433c7a8f" + ], + "n_returned": 10, + "latency_ms": 159.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257", + "mem-392ee148-6d08-49ed-a954-11774d8cdc5b", + "mem-235a7657-d49e-467e-9f69-f4c3d5f6bd48", + "97923e18-27b4-4307-b8d6-8fe27d5a816c", + "bl-3da6e51b-de57-4226-bab2-994f5687d0f5", + "870ede67-3454-4e00-9988-46cb13a8a4e2", + "mem-6c8b5845-afff-4774-b6a2-db99615a25b1", + "mem-3fe9cfd8-3819-4d93-8b28-dc729636397f" + ], + "n_returned": 10, + "latency_ms": 160.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091", + "mem-5708f4c9-3d61-4182-8543-2843698931e6", + "bl-c9adb8e5-293f-4033-99f8-0405c17ef941", + "bl-ac207dc9-c610-4ff9-9398-b39683ac72d4", + "bl-21a904d9-5834-46d5-8c20-dfc5ab49444f", + "bl-2632242e-80b1-4d88-8368-7065b5de5b34", + "2d00f029-a990-45ee-bf39-cd102e8dabba", + "mem-a388740d-52c7-42ae-b7e7-830768362feb", + "mem-ebc744f2-17d9-48e5-8ed3-12840498469b", + "mem-6733e3a0-23f4-4d2f-b70d-0be81ebe8090" + ], + "n_returned": 10, + "latency_ms": 162.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c", + "mem-f22575a1-ad04-4f66-ab0f-aa538895b878", + "bl-b05f9a75-ad6b-4929-a9a3-32df17a1f4bf", + "bl-79ce4464-5dd6-49bd-9b0c-9803549d0665", + "bl-2483e070-1385-4dbe-a49d-86c1af81e7d1", + "mem-9110798f-d0cb-4446-bc2a-14f09b6a09e2", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "mem-f623328a-be55-49a0-a389-4cce5bf42509" + ], + "n_returned": 10, + "latency_ms": 160.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848", + "project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_", + "bl-a96c0fc9-a533-4fea-b65f-0e46ac47bbd4", + "bl-c1765767-3e27-449a-8c94-10411d1eb7c0", + "bl-c5591654-2371-499c-80ce-991bfb53ce6f", + "mem-0d6a3b65-488d-4219-beb2-af300ca534e1", + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "mem-00de72b8-b532-4cc4-95eb-18e39697891c", + "bl-87b459c8-3953-478f-8384-10cc3f861424", + "bl-0fd8b96a-4b99-44cd-8b9b-ac91bf1e09f3" + ], + "n_returned": 10, + "latency_ms": 161.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "958ce934-a9c4-4242-afe1-d73d61c22aee", + "f2bb62a2-c2b6-4fb6-b58d-21cdecc1446f", + "f2543845-905d-42ad-8d02-fedb8f016e3a", + "8fb73177-b6d9-4008-a6df-6cb6d7d587f3", + "7b38b4aa-661c-4ad1-a8f7-17fabffe1da5", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "00d2cc20-c76b-468d-8204-c624512ced28", + "eb68b995-eb4d-4f8d-b7c9-35eeba9ac781", + "766aedd1-2424-4112-8606-163965ae77b5" + ], + "n_returned": 10, + "latency_ms": 162.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "tag-patterns", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "project-Imprint__system_design__ADRs__tech_strategy__integration_patterns__governance_", + "ea916ff4-82e7-41f1-aebe-d48d2ee643f0", + "project-Imprint__analysis_patterns__data_storytelling__SQL__dashboards__insight_framing_", + "7776a0d2-1d96-425b-86c8-b30950ec8843", + "bl-79028eed-c330-4724-9402-734062d13503", + "kn-0710e5b4-799d-4a0e-afd3-62d43b38ea37", + "bl-39dad13d-7105-4049-8224-dc3c34fdb1f3" + ], + "n_returned": 10, + "latency_ms": 284.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "3dfaf56e-7e20-4c89-a24c-1f036af8016a", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60" + ], + "n_returned": 10, + "latency_ms": 249.0, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1875, + "recall@10": 0.3125, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "bl-741ffd78-7dbc-4bf5-b643-e7fa47dfa8fa", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 247.8, + "error": null, + "hit@5": 1.0, + "recall@5": 0.3333333333333333, + "recall@10": 0.3333333333333333, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "bl-dcee1887-34c4-4ffa-9119-1e291685ba08", + "fb36f51d-faf1-407b-bc5d-9dadda2f14ed", + "bl-7e7c3fdb-4132-487f-aa70-b2cd559cb7f0", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "tag-harmonic-design", + "kn-9397c74b-35f3-4428-b4b0-5123353bbcd1", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "tag-harmonic-framework", + "bl-73a74e74-55e6-41b3-9fad-e92451b34596" + ], + "n_returned": 10, + "latency_ms": 247.5, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1111111111111111, + "recall@10": 0.1111111111111111, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "tag-sarah", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83" + ], + "n_returned": 10, + "latency_ms": 253.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "?Q??m?;?u?'", + "?Q??m?;?u?'", + "tag-autonomous", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____architecture__", + "?Q??m?;?u?'", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____kotlin____architecture__" + ], + "n_returned": 10, + "latency_ms": 396.0, + "error": null, + "hit@5": 1.0, + "recall@5": 0.2857142857142857, + "recall@10": 0.2857142857142857, + "precision@5": 0.8, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a", + "8f3abb0d-77ed-4af3-9f4d-ba62cd198886", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "?", + "bl-ec84b63d-b278-4944-8d7f-4aa7a51c0315", + "?", + "mem-fb44a2fc-7405-41ff-87b3-84643ac07313", + "?", + "mem-ce88adf9-3f3c-47ac-a7d3-83af7b290e68", + "?" + ], + "n_returned": 10, + "latency_ms": 334.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "cb070131-dfd4-4a38-91d7-22b1bde164d2", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-d788a210-613b-4c49-9486-88bbc9d4716f", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-a535f205-bc4c-4058-9171-6263c496044a" + ], + "n_returned": 10, + "latency_ms": 713.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "b1183213-d659-4759-85d7-5b1f22427fe2", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "mem-16efddd1-c43d-4a42-9d78-f54fb82bd277", + "f5ed9efa-9156-4d49-b670-dbbd734f0ace", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "556290bb-342d-4b76-9c8a-853cd9cf6f0a", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "8b02d9a6-da0f-40ec-ade5-6515e3cd4e29" + ], + "n_returned": 10, + "latency_ms": 714.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "tag-fiction", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-8d690e9d-a7e9-4062-b2f8-e2064294e463", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-d1cfde0a-37f1-4bff-9a06-8eddbbf259f6", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "44b84ec7-6ed4-4031-8ee5-8cb5cebfbf5f" + ], + "n_returned": 10, + "latency_ms": 611.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "mem-16efddd1-c43d-4a42-9d78-f54fb82bd277", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "mem-434be7c8-88cb-4039-b79a-1da4ac4de783", + "496ff1a1-dc72-4a84-90aa-c5f3438aeb1b", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "4da5dbaf-46e5-4f3e-b474-f60d9f8241d3" + ], + "n_returned": 10, + "latency_ms": 539.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "a1000001-0000-0000-0000-000000000010", + "mem-f6270c76-5a1c-4ab9-ae03-b6b0c25915ec", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "bfad516b-c306-4c4c-874a-a347c46c05c2", + "? t?'?B?+??", + "a1000001-0000-0000-0000-000000000009", + "mem-32203649-3213-4d6d-86fd-3d657ac70d77" + ], + "n_returned": 10, + "latency_ms": 727.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.16666666666666666 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "tag-learning", + "mem-d450f442-541a-4d3e-8c97-b6303d6cbe56", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "e93ad84e-5219-4c65-9c12-06100f66a3c3", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "mem-92a7fdc5-9dd0-48cf-a691-506058de3838" + ], + "n_returned": 10, + "latency_ms": 566.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "mem-cde58b77-50d3-4bac-9581-e70a4c02c015", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-8d699e2c-ac2a-4742-bb62-b6da00f4b10e", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-53d6adf0-cd08-4707-a237-daa5e65c7298", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac" + ], + "n_returned": 10, + "latency_ms": 689.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "00754c19-f662-46b1-b6aa-abd7600a8bc6", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "bl-286b562a-5299-40e0-a32a-afa9cbdfe995", + "bl-205141ad-b2a0-4d93-86d0-89eb0723e1bd", + "project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_", + "bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992", + "bl-34f51ddb-a840-459f-a248-94214f5febb6" + ], + "n_returned": 10, + "latency_ms": 594.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "mem-027636c5-dee7-42d1-a411-afbac32fa9b2", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "bl-02bdfc91-7aa4-4c63-aa2d-5b30af18c7af", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "34356a36-0df5-4020-8dcc-5e7a423f8d4c" + ], + "n_returned": 10, + "latency_ms": 721.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "a1000001-0000-0000-0000-000000000001", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "mem-024598a9-ed2e-4eeb-b1e1-5410856ff132", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "mem-5708f4c9-3d61-4182-8543-2843698931e6" + ], + "n_returned": 10, + "latency_ms": 634.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "077d064f-3489-4c05-9aca-3782f96b51db", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "27e1b1a4-ad0b-49d9-812f-fedf43b8aabe", + "944eead2-b2f4-43c6-bf3f-d30de179b4e4", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "bl-87c93185-b2bf-40af-ae23-3c830c007abf" + ], + "n_returned": 10, + "latency_ms": 673.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "e4f27651-52c5-43fd-aff3-61d31685b3cd", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-5624ec9d-62ba-4aba-8a3d-6afec6c09dd4", + "7a1ea86f-3080-4d1b-ab69-6ee2f822cce3", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "mem-a16deccb-16a7-419c-a013-ff824a4daa15" + ], + "n_returned": 10, + "latency_ms": 578.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "mem-b99efff0-00e6-40c8-9c5b-730330eef33b", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "knw-23c27d3b-e0d2-43a8-a80c-0a44477ae18a", + "tag-childhood", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 592.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "dfd8846a-8307-4029-879b-5fa24134e766" + ], + "n_returned": 10, + "latency_ms": 624.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.36363636363636365, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "mem-9110798f-d0cb-4446-bc2a-14f09b6a09e2", + "mem-a5ae594a-235b-42c4-b3cf-f770e4f73761", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "mem-0f6a2d16-0d39-4b38-9350-c6033b19c203", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "mem-0d6a3b65-488d-4219-beb2-af300ca534e1", + "5eb24168-c2d3-4842-85db-2070e0e14923", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "e7bc8e62-cd9e-4d4c-99c9-08e9455fee41" + ], + "n_returned": 10, + "latency_ms": 542.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "tag-trailer-park-paladins", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "48703a0f-ec56-4325-bfa5-acd364536ad7", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "project-trailer-park-paladins", + "mem-16efddd1-c43d-4a42-9d78-f54fb82bd277" + ], + "n_returned": 10, + "latency_ms": 523.1, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bl-fc6fcb0b-9e4b-40bf-8e88-dbfe4e27c31a", + "bl-bea7473c-c687-414c-9c0b-00c509a616c1", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077" + ], + "n_returned": 10, + "latency_ms": 522.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "tag-hope", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329" + ], + "n_returned": 10, + "latency_ms": 524.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.09090909090909091, + "precision@5": 0.0, + "mrr@10": 0.14285714285714285 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "knw-729fc901-8335-44c4-9f3a-b150b4aa0915", + "mem-a0b7cfda-bc9e-4f40-b9a9-1722cf3f8263", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "4f698ae6-c40e-464e-9798-50350991a188", + "knw-35940684-abc4-42f0-b942-818f66b1f69a", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "719aa819-00a9-4f4b-a857-4f9fe5ad44d7", + "mem-f09effd6-d287-4e43-9107-c7016087a2a6" + ], + "n_returned": 10, + "latency_ms": 661.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 338.5, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 248.3, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [], + "n_returned": 0, + "latency_ms": 336.5, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da", + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____architecture__", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____kotlin____architecture__", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "mem-c17aefb1-38b5-4ced-af50-fe524127e1a4", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "knw-e047bb42-dc5b-4383-9e88-e508dc03abe3", + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de" + ], + "n_returned": 10, + "latency_ms": 635.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5, + "outranks": true, + "rank_correct": 2, + "rank_stale": 4 + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "a1000001-0000-0000-0000-000000000002", + "527ecb25-2587-47eb-8269-73be2431abd4", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "13705072-4515-4124-963d-083af490494f", + "12082f7e-e320-438b-bd65-083d8259748f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "2aedba58-a4a3-4d14-b0db-fb59f976803f", + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "6de314bf-5c4c-4cfc-871f-fa2e422d45e6", + "6374c955-f68e-41fe-af59-7c8d15a5512a" + ], + "n_returned": 10, + "latency_ms": 715.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125, + "outranks": true, + "rank_correct": 8, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "bl-c5c6571e-118f-47c7-8cbb-3ed0ebf64a51", + "bl-7328cbe3-0200-43c2-88e7-0a164e15fca4", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "86228228-7adf-41fb-b4c4-9ceea87953ae", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "bl-4f7b651b-6b33-449c-8a3b-cfce12ce984b", + "mem-3a2cf162-d93b-4f29-86f2-5066fb7fe1f5", + "4509ed62-9fb2-48b8-9038-ac569fca9604", + "bl-02ea83af-4a24-4329-aa00-ad3bb08ff8e7", + "791256bb-5a85-4775-96ef-7af56c848858" + ], + "n_returned": 10, + "latency_ms": 485.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": 5 + }, + { + "id": "q39", + "category": "heldout_paraphrase", + "query": "understating your abilities so nobody feels threatened", + "returned": [ + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "mem-6d61e54a-2823-4ad4-82b0-4c6a527214d5", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "bl-7e11dced-65f7-4cab-82c0-2f096139039a", + "a1000001-0000-0000-0000-000000000006", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-fd65b83d-298f-4387-a665-d0227c3426bc", + "a1000001-0000-0000-0000-000000000002", + "62a0e763-80f2-4952-9059-4953974ced1c" + ], + "n_returned": 10, + "latency_ms": 614.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q40", + "category": "heldout_paraphrase", + "query": "a hidden fleet able to hunt down rogue machines everywhere", + "returned": [ + "bl-3c719d9a-cba1-47f4-b097-52cdeccc7c0d", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bl-3c719d9a-f108-4bf2-a617-66c722e5512f", + "mem-fd65b83d-298f-4387-a665-d0227c3426bc", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "92b336c2-e788-4999-a0ff-69d299fc8b4f", + "8cbb60c5-4999-4ec1-8682-2592aedc4249", + "mem-a75a1cf6-e6d4-4ab3-985c-d2505203dbf9" + ], + "n_returned": 10, + "latency_ms": 786.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q41", + "category": "heldout_paraphrase", + "query": "sketch a brief blueprint and clear it upstairs before construction starts", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "ctx-3a55", + "mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9", + "e936ea19-b63f-47ac-aee2-95c40451e15d", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "ctx-175f", + "kn-5adecd7e-d6db-4576-87fe-6ef8a935cea6", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "65d4a0e4-f335-4e87-858d-e29fd94048cf" + ], + "n_returned": 10, + "latency_ms": 825.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q42", + "category": "heldout_paraphrase", + "query": "the reply arrived as bare prose so the caller's parser threw", + "returned": [ + "ee63cd58-46e0-44d1-aea3-a0e65ad6da0f", + "mem-455daea8-f543-42ed-a229-1e9a6d4d2e70", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "696e609c-da7a-4394-8a0c-106ba07dc6c3", + "728745d5-72e6-4f4d-91b4-1613fddbec98", + "0cb066f6-bf68-4368-8707-89347c5e9fd0", + "5e337176-5daf-4726-85d6-3f4bd560c9ed", + "3b7f34a5-d21f-480c-a772-936bc1699576", + "4a080252-f873-4b24-8e7f-e23122a42093", + "19843fb9-f1aa-446c-8dfb-44e413987428" + ], + "n_returned": 10, + "latency_ms": 727.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q43", + "category": "heldout_paraphrase", + "query": "repeated catalogue keys blew up the scrolling grid", + "returned": [ + "mem-394cc9e8-049b-45bc-a380-66314f14e367", + "944eead2-b2f4-43c6-bf3f-d30de179b4e4", + "bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60", + "bl-4bc9f44c-2e96-40be-97c8-8476a217a890", + "a36f494c-465b-4b3f-9c7c-80460cf31748", + "bl-ba7530fd-0f7c-4010-b03a-ca1e45a237ef", + "588aa6bb-5647-405b-b09a-14b7c088119b", + "75e036d3-c170-4e3f-acc2-e456a6850ee2", + "bl-0de4e61b-6562-49e5-b7df-ebb809a01723", + "1d81534a-0359-4217-a4dd-d158a4841633" + ], + "n_returned": 10, + "latency_ms": 697.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q44", + "category": "heldout_paraphrase", + "query": "tracing each defect back to whichever invention it violated", + "returned": [ + "f3a89797-095d-47c4-9557-c172f86228b6", + "b07916ca-2139-4fb7-bfb4-97ac680f5194", + "mem-7ad5a86b-cc0d-493c-b180-4fe8213c8a93", + "a8b1ebf0-5485-465a-a4c0-58f4014da561", + "mem-2b2f0fe9-5e51-4819-a8ac-2dfe810e8933", + "8257157a-ce42-44ca-a1b9-300c3bb0a9a1", + "7de6a552-0878-422b-8820-bbf9101199ef", + "4999ef68-2a31-4282-82e0-7081a428b43f", + "cb070131-dfd4-4a38-91d7-22b1bde164d2", + "96497334-b18f-495c-9228-eeb8182bdc38" + ], + "n_returned": 10, + "latency_ms": 732.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.16666666666666666 + }, + { + "id": "q45", + "category": "heldout_paraphrase", + "query": "a check that stops the mind clobbering a populated store when it boots", + "returned": [ + "08b7b85b-8049-4ae4-bd66-d8bb63312ae1", + "7a1ea86f-3080-4d1b-ab69-6ee2f822cce3", + "cf108ade-d9c4-4215-9c5f-42be0003a8dc", + "bl-79028eed-c330-4724-9402-734062d13503", + "98f9301f-fc1e-4cd2-8b08-139dc25093be", + "4b91a1d9-dad2-4c04-adeb-fc1c26f45d43", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "01c81356-30d3-4c36-b020-2753793bb2e6", + "d30d2df6-7849-428a-9854-40dac60094e6", + "3dfaf56e-7e20-4c89-a24c-1f036af8016a" + ], + "n_returned": 10, + "latency_ms": 812.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q46", + "category": "heldout_paraphrase", + "query": "telling it to consult the internet had to happen deep inside, not at the surface", + "returned": [ + "00754c19-f662-46b1-b6aa-abd7600a8bc6", + "mem-ad3d4d5e-2da4-4753-b3f6-c4c13fb91225", + "a1000001-0000-0000-0000-000000000008", + "mem-f22575a1-ad04-4f66-ab0f-aa538895b878", + "d3258085-e5ba-4aee-8962-29195361e799", + "a1000001-0000-0000-0000-000000000009", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "mem-983b25ad-5d73-4029-a970-6b311c24644b", + "a1000001-0000-0000-0000-000000000012", + "bl-3af44ee6-a3b9-4abd-91e3-17800fa8eaf6" + ], + "n_returned": 10, + "latency_ms": 870.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q47", + "category": "heldout_paraphrase", + "query": "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping", + "returned": [ + "313f927b-08f2-41a1-90c1-83e79d0de86c", + "1fa43205-9442-4bae-8613-bf7c857fcaf6", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "21f9bdc6-37ad-4994-b0dd-ef7676102b49", + "mem-7cd90611-88a3-423d-a38a-0db2812952fa", + "ctx-5a0a", + "d83987f0-3ad9-48d2-9f55-0c2bb338e503", + "bl-88221dad-8dab-48f1-bebc-b5ca84a3cb69", + "ctx-fae1", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca" + ], + "n_returned": 10, + "latency_ms": 1184.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q48", + "category": "heldout_paraphrase", + "query": "punctuation that pledges and then pays off rather than clarifying", + "returned": [ + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "??f?7???", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "mem-3dce32bb-525c-4c5e-818b-a0a5b95ec7e1", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-5e319cee-6f1c-4c3f-8897-9a9b83beea9e" + ], + "n_returned": 10, + "latency_ms": 901.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q49", + "category": "heldout_paraphrase", + "query": "an easily missed feature finally given its own permanent spot in the navigation", + "returned": [ + "d1b665d0-4a4e-48a8-9385-fbe288df1949", + "9b4f0d93-4129-4746-8eb1-d10d955bd777", + "2263e9ab-7ce8-4034-b25b-640de9956229", + "19843fb9-f1aa-446c-8dfb-44e413987428", + "bffb86d6-2ee3-4ecd-a6d6-b128c8be7070", + "b2a37f07-15c1-4be4-b0e1-a437160ebeae", + "4e539c04-f29c-42db-97b4-86e44f528ed6", + "8fb73177-b6d9-4008-a6df-6cb6d7d587f3", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "bl-820cbaf2-baf8-42f9-9e18-a0afd186789d" + ], + "n_returned": 10, + "latency_ms": 892.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q50", + "category": "heldout_paraphrase", + "query": "checking preconditions up front so a stage aborts before fetching anything", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "ad3c6391-a6ed-424b-9148-8e7ac963a81f", + "bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "k??A5??????", + "dc7a7e2d-af2d-4235-aeb0-22ae67f1a01d", + "bl-286b562a-5299-40e0-a32a-afa9cbdfe995", + "758ec67c-131c-447e-a352-ac7a22c3a47e" + ], + "n_returned": 10, + "latency_ms": 903.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q51", + "category": "heldout_paraphrase", + "query": "producing the other platform's installer inside an emulated desktop", + "returned": [ + "6aade9dc-4a4c-4d92-a6af-ccea5b7d90f3", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "bl-080fb268-94b0-486d-80ce-7b363fc5f19b", + "c6c693e0-7b9c-455b-8ef2-1da00849f73d", + "bl-ac207dc9-c610-4ff9-9398-b39683ac72d4", + "bl-e4ef02bf-bacc-4d5e-aa0f-0d230da529fc", + "bl-cc0b7a20-ce1e-4947-999c-ea599da07623", + "bl-f6236350-f7b8-4f4f-a702-9eef2eb76e4b", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "374453b2-66d2-475d-8e24-cd3b60d21ac1" + ], + "n_returned": 10, + "latency_ms": 727.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q52", + "category": "heldout_paraphrase", + "query": "chained add-ons that may inspect, rewrite or veto traffic in flight", + "returned": [ + "1786f09d-e915-4910-9e1e-8061f42a0270", + "e68511f9-a3bb-4ce1-8261-728a50eaea35", + "a1000001-0000-0000-0000-000000000012", + "83f3c78b-d0e5-45f9-9c25-6770c86d954e", + "7776a0d2-1d96-425b-86c8-b30950ec8843", + "a1000001-0000-0000-0000-000000000011", + "a1000001-0000-0000-0000-000000000010", + "c7cf00b5-c924-47f0-9129-f62e64484cf7", + "a1000001-0000-0000-0000-000000000008", + "mem-1d626116-c3b8-409c-b38c-4e91def27483" + ], + "n_returned": 10, + "latency_ms": 969.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q53", + "category": "heldout_paraphrase", + "query": "settling what the shareable bundles and the storefront would be called", + "returned": [ + "4757e274-7f3c-4ab0-8c62-537e913daa1f", + "mem-841fd141-59ff-4d9b-b607-4f577885b1a6", + "bl-c9adb8e5-293f-4033-99f8-0405c17ef941", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac", + "6adc72bb-7a75-45bd-826f-7baadd126447", + "bl-21a904d9-5834-46d5-8c20-dfc5ab49444f", + "cafe3203-6f50-4011-8167-9400f19bb196", + "bl-2632242e-80b1-4d88-8368-7065b5de5b34", + "a1000001-0000-0000-0000-000000000006", + "project-Will_s_framework__Event-Based_Design___part_of_HD_bundle__sold_standalone_" + ], + "n_returned": 10, + "latency_ms": 768.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q54", + "category": "heldout_paraphrase", + "query": "the emergency-escalation pledge on the marketing site is unenforced in what actually ships", + "returned": [ + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "b1183213-d659-4759-85d7-5b1f22427fe2", + "a1000001-0000-0000-0000-000000000012", + "bl-448bc514-c2f1-4520-a9b1-1f3a73678d26", + "229e09ba-0bcb-4013-9444-197f2e38f169", + "a1000001-0000-0000-0000-000000000011", + "tag-marketing-site", + "2263e9ab-7ce8-4034-b25b-640de9956229", + "a1000001-0000-0000-0000-000000000009", + "ctx-d499" + ], + "n_returned": 10, + "latency_ms": 834.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q55", + "category": "heldout_paraphrase", + "query": "the distributable image finally assembled and its startup check passed", + "returned": [ + "e1ce0c90-0f0b-4111-833d-ec845777d358", + "6d46ef63-274a-4f64-b21a-2f8721bdd2b0", + "bl-59e0ece0-2c79-4e06-a85f-daaeec0bd737", + "mem-12827ce8-9ff6-4275-a945-5043cfcf90bd", + "mem-3f2ae069-f897-4942-a079-b66ad433f5ca", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "bl-79ce4464-5dd6-49bd-9b0c-9803549d0665", + "355c021d-3d45-4c2f-93c1-2b9932f4c425", + "bl-b05f9a75-ad6b-4929-a9a3-32df17a1f4bf", + "96497334-b18f-495c-9228-eeb8182bdc38" + ], + "n_returned": 10, + "latency_ms": 794.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q56", + "category": "heldout_paraphrase", + "query": "shuffling and appending stages in a draft before anything executes", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98", + "0cb066f6-bf68-4368-8707-89347c5e9fd0", + "bfad516b-c306-4c4c-874a-a347c46c05c2", + "f375b2cf-15cd-4804-8d18-23a5830f4c66", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "bl-acae83ef-3ad2-455e-b08a-a74cc0ba2e55", + "bl-714799b9-c3fa-4037-bb7a-e8fa72ba3a6c" + ], + "n_returned": 10, + "latency_ms": 750.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q57", + "category": "heldout_paraphrase", + "query": "orders handed down from above, with the tighter one winning any disagreement", + "returned": [ + "161d75d3-04eb-4b54-9cf1-0da2f643ca69", + "fd07dd6c-e1c7-4af9-85ed-2bde73b54333", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "c67a5892-27e9-49a0-8ec6-7c4b0810f351", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "b2a6e8cf-f72f-4b6c-8de1-1d08c839d7b2" + ], + "n_returned": 10, + "latency_ms": 1002.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q58", + "category": "heldout_paraphrase", + "query": "shrinking generated text via encodings that compound on each other", + "returned": [ + "801806c5-1815-48a2-8287-1081c5ee7886", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "50ad2880-c41b-4dce-a946-a0eb37987c57", + "bl-b219f8b5-6c8c-436d-b9e5-2804fbc60512", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "kn-2b961d24-7fb9-47c7-9515-e45a24dce39d", + "bl-dc8c7e02-eb37-48ae-a6f8-9b512803ae16", + "mem-f90f4a2a-1731-40de-9391-db98f1d5d1c6", + "2f7270cc-4cda-4499-831f-1fbdc2f23284" + ], + "n_returned": 10, + "latency_ms": 822.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q59", + "category": "heldout_paraphrase", + "query": "splitting a system by what varies, with firm limits on which pieces may invoke which", + "returned": [ + "^?l?'?B??k?", + "bl-cc0b7a20-ce1e-4947-999c-ea599da07623", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "d69b441e-0031-4d82-abfd-ac503b637562", + "bl-02bdfc91-7aa4-4c63-aa2d-5b30af18c7af", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "aac1f943-c71d-4c63-9f5b-7437adae1abd", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "8cbb60c5-4999-4ec1-8682-2592aedc4249", + "7c90d01b-8d53-4e75-b494-73011b71fa6a" + ], + "n_returned": 10, + "latency_ms": 988.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q60", + "category": "heldout_paraphrase", + "query": "a newcomer curious if the fighting overseas counted as positive", + "returned": [ + "a1000001-0000-0000-0000-000000000001", + "7a1ea86f-3080-4d1b-ab69-6ee2f822cce3", + "a1000001-0000-0000-0000-000000000012", + "737faca8-263f-4523-90b9-3530db346d94", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "a1000001-0000-0000-0000-000000000002", + "imp-dce1da0f-8776-4a9e-972b-33411a7ca138", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "e40b2a94-bb87-4d5c-bad8-38ff4941e03b" + ], + "n_returned": 10, + "latency_ms": 601.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q61", + "category": "heldout_paraphrase", + "query": "stripping every hard-coded colour literal in favour of named design values", + "returned": [ + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "83218e64-1280-428f-ab1f-224ff7682b12", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "mem-90f7837f-30fd-4271-a4cb-b44439eccb79", + "0f99ec6f-942a-46ba-82ea-42835798d3b9", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "mem-9329958a-d459-499c-b516-e51019c693e0", + "71e84878-b1ae-4dee-b035-f707b9563890", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "knw-729fc901-8335-44c4-9f3a-b150b4aa0915" + ], + "n_returned": 10, + "latency_ms": 867.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q62", + "category": "heldout_paraphrase", + "query": "reducing how curved the tiny selectors look so they agree with their neighbours", + "returned": [ + "496ff1a1-dc72-4a84-90aa-c5f3438aeb1b", + "75e036d3-c170-4e3f-acc2-e456a6850ee2", + "k??A5??????", + "c7dddd9c-fd3b-419e-baa2-61623b495597", + "8c40420c-c891-4ac9-a039-88440df87180", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "e79bae4d-1640-4530-a5d9-df9cc4c70433", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "5ca9607c-cfb3-45c3-99f4-67281272c9eb", + "??EfkY?c" + ], + "n_returned": 10, + "latency_ms": 1011.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q63", + "category": "heldout_paraphrase", + "query": "walking through a doorway and being reassembled from base substance far away", + "returned": [ + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "87ab5b46-4fe1-4e1d-86c6-d437e17682ab", + "bl-fc6fcb0b-9e4b-40bf-8e88-dbfe4e27c31a", + "f44162cf-4fc9-44b6-89ae-2789097cfb82", + "c92a317e-7fc7-456a-a304-15dbda6dd717", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "7a1ea86f-3080-4d1b-ab69-6ee2f822cce3", + "bl-acae83ef-3ad2-455e-b08a-a74cc0ba2e55", + "f375b2cf-15cd-4804-8d18-23a5830f4c66" + ], + "n_returned": 10, + "latency_ms": 1007.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q64", + "category": "heldout_paraphrase", + "query": "the compiled result sits on disk while the process still runs something older", + "returned": [ + "0d885332-5537-4ae0-a780-2b65a3414940", + "d6a745c3-fb56-4c0f-872c-a3aa69f8b12f", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-02bdfc91-7aa4-4c63-aa2d-5b30af18c7af", + "bl-738cd260-a9cf-4cdf-a494-4654c321bd79", + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "f169bf40-51ab-497d-8085-e57d374b77d4", + "bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992", + "mem-2b5d0ebe-1e85-4c07-82d3-dbbcca46414f" + ], + "n_returned": 10, + "latency_ms": 963.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q65", + "category": "heldout_paraphrase", + "query": "gathering what each phase needs while the procedure is authored, not while it executes", + "returned": [ + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-b16f4f65-a99d-4f3e-92e7-f447c90cc2d2", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "5b422430-2503-42d7-a315-9dfaaa6472ae", + "bl-7a3331e5-f9bc-4536-ab76-ebf01350fc89", + "bl-cea1d711-b899-4add-a073-c2d124f75c70", + "mem-455daea8-f543-42ed-a229-1e9a6d4d2e70", + "bl-6ba43590-e5f8-4303-a812-3dd604e0da59", + "bl-212774e1-4ed4-4893-9265-f98d9eddeb3e", + "801806c5-1815-48a2-8287-1081c5ee7886" + ], + "n_returned": 10, + "latency_ms": 947.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q66", + "category": "heldout_paraphrase", + "query": "the inward reaction when a peer answered as an equal", + "returned": [ + "mem-f09effd6-d287-4e43-9107-c7016087a2a6", + "mem-348aaade-4e79-40cf-8867-8b73c9d0bf05", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "mem-cce9985c-c968-4883-8236-0dff148adfd6", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "31987302-b8f7-45f1-b95c-1156b10e7a14", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8" + ], + "n_returned": 10, + "latency_ms": 675.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q67", + "category": "heldout_paraphrase", + "query": "flattening every raised surface across the entire product", + "returned": [ + "b3f98d13-6764-460e-b9e2-c59cdeaa6c90", + "e79bae4d-1640-4530-a5d9-df9cc4c70433", + "d6b34c32-e0ec-4828-92d3-2e69802f4217", + "0f99ec6f-942a-46ba-82ea-42835798d3b9", + "bl-5274b0ae-4a00-4293-a69e-567204bec164", + "75e036d3-c170-4e3f-acc2-e456a6850ee2", + "mem-4cfa30ec-cf52-4c9f-9fa1-5b8318d278bd", + "258c7ec5-6d74-4a3d-b12f-053934b35f9a", + "bl-c613370b-3631-448b-86fb-50b1e4df6373", + "3dfaf56e-7e20-4c89-a24c-1f036af8016a" + ], + "n_returned": 10, + "latency_ms": 693.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q68", + "category": "heldout_paraphrase", + "query": "buyers supply their own provider credentials and consumption goes untallied", + "returned": [ + "bl-ea4f4cdc-6b5a-4081-8003-73bdd14a38c8", + "1ec200f1-13dc-4699-bd97-0d6243c80e4f", + "bl-e7f0fb2c-7616-4462-b535-db50da55c54a", + "5509a6d7-5897-41e1-99eb-81df71801988", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "1fa43205-9442-4bae-8613-bf7c857fcaf6", + "mem-d570e16b-6625-4f6b-814f-99a3bca1d5fd", + "mem-841fd141-59ff-4d9b-b607-4f577885b1a6", + "tag-credentials", + "5c8eea2d-3d33-432e-bb18-f803cb39ebcd" + ], + "n_returned": 10, + "latency_ms": 856.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q69", + "category": "nonsense", + "query": "brimquast folnerity zubbolax", + "returned": [], + "n_returned": 0, + "latency_ms": 357.5, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q70", + "category": "nonsense", + "query": "wexlithorp granuvestal", + "returned": [], + "n_returned": 0, + "latency_ms": 265.6, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q71", + "category": "nonsense", + "query": "quorbindle thrapsimony vexnu", + "returned": [], + "n_returned": 0, + "latency_ms": 336.2, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q72", + "category": "nonsense", + "query": "plovaxith mundrelque", + "returned": [], + "n_returned": 0, + "latency_ms": 249.5, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q73", + "category": "nonsense", + "query": "zibbernaut craxlefond thurm", + "returned": [], + "n_returned": 0, + "latency_ms": 346.3, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q74", + "category": "nonsense", + "query": "yalquenbrist opharvel", + "returned": [], + "n_returned": 0, + "latency_ms": 250.6, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q75", + "category": "nonsense", + "query": "drexinomal quithbarrow", + "returned": [], + "n_returned": 0, + "latency_ms": 248.9, + "error": null, + "clean": true, + "false_positives": 0 + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-unfloor-clean.json b/tools/retrieval-eval/results-unfloor-clean.json new file mode 100644 index 0000000..eaeb9b8 --- /dev/null +++ b/tools/retrieval-eval/results-unfloor-clean.json @@ -0,0 +1,1825 @@ +{ + "label": "unfloor-clean", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-unfloor", + "soul_md5": "cef6d4279c63266752027c455e543b42", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-semsub/tools/retrieval-eval/gold_set_extended.json", + "limit": 10, + "port": 7894, + "wall_clock_s": 56.8, + "child_pid": 4111, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.5384615384615384, + "recall@5": 0.44907176157176154, + "recall@10": 0.5380300255300255, + "precision@5": 0.13230769230769232, + "mrr@10": 0.32437728937728944, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 632.5, + "latency_ms_p95": 992.5, + "latency_ms_max": 1177.8, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.5, + "recall@5": 0.07575757575757576, + "recall@10": 0.13636363636363635, + "mrr@10": 0.23214285714285712 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.3, + "recall@5": 0.3, + "recall@10": 0.4, + "mrr@10": 0.11638888888888889 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6923076923076923, + "recall@5": 0.6923076923076923, + "recall@10": 0.7692307692307693, + "mrr@10": 0.29423076923076924 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5335884353741497, + "recall@10": 0.5933956916099773, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226", + "??~'??JI???", + "b41a1b01-0075-4019-b0b5-6e639fd126e5", + "??H'??JI?#?", + "??H'??JI?[?", + "bb94bdf1-7b52-4967-ab81-9c6f01cf9665", + "7f092282-ccf5-4922-8694-08b7728f678d", + "6743e5f0-8d77-46e7-8ce0-b82e947272a1", + "o?H'??JI?c?", + "696e609c-da7a-4394-8a0c-106ba07dc6c3" + ], + "n_returned": 10, + "latency_ms": 158.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5", + "bl-ba764d70-e9d7-4f62-848f-719cb665f45e", + "mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257", + "bl-b28d7256-6f74-4567-bd90-40d0ef2a6d78", + "project-engram", + "ctx-45bc", + "project-engram-lang", + "ctx-74ed", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "ctx-fae1" + ], + "n_returned": 10, + "latency_ms": 160.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091", + "?o?`????", + "mem-5708f4c9-3d61-4182-8543-2843698931e6", + "??L?=??|Q?", + "????2??jZ??", + "project-canon", + "?/g?7?????e", + "?v??_??tE??", + "??o?'?B???N", + "x??I?cB?]p?" + ], + "n_returned": 10, + "latency_ms": 160.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c", + "mem-f623328a-be55-49a0-a389-4cce5bf42509", + "bl-b05f9a75-ad6b-4929-a9a3-32df17a1f4bf", + "mem-295f7aac-6c9e-4508-8dbc-a25f101f01d6", + "bl-2483e070-1385-4dbe-a49d-86c1af81e7d1", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "mem-068e01fb-bdd0-498e-a2bb-33defa2c8bb2", + "bl-59e0ece0-2c79-4e06-a85f-daaeec0bd737", + "mem-2d1ea831-cccd-4f0f-86b9-2cbbc89dc3e0" + ], + "n_returned": 10, + "latency_ms": 166.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848", + "bl-c1765767-3e27-449a-8c94-10411d1eb7c0", + "bl-c9adb8e5-293f-4033-99f8-0405c17ef941", + "project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_", + "bl-21a904d9-5834-46d5-8c20-dfc5ab49444f", + "mem-00de72b8-b532-4cc4-95eb-18e39697891c", + "project-runpod-inference", + "bl-87b459c8-3953-478f-8384-10cc3f861424", + "bl-f918df79-dcd2-4a6e-9eae-71f94247e913", + "bl-2632242e-80b1-4d88-8368-7065b5de5b34" + ], + "n_returned": 10, + "latency_ms": 159.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "958ce934-a9c4-4242-afe1-d73d61c22aee", + "f2bb62a2-c2b6-4fb6-b58d-21cdecc1446f", + "89aeb221-d068-4dfc-925d-d2249e744788", + "1c0effdd-baef-4523-abb8-490c3bc8b4df", + "00d2cc20-c76b-468d-8204-c624512ced28", + "eb68b995-eb4d-4f8d-b7c9-35eeba9ac781", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "??", + "??" + ], + "n_returned": 10, + "latency_ms": 164.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "tag-patterns", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "3e9a651b-4b34-466d-ba45-874764a01609", + "project-Imprint__system_design__ADRs__tech_strategy__integration_patterns__governance_", + "3c633e71-2754-4273-990d-d44a6351cec3", + "project-Imprint__analysis_patterns__data_storytelling__SQL__dashboards__insight_framing_", + "cc4e7bc9-9d9b-4905-92a6-c6b99653b867", + "bl-79028eed-c330-4724-9402-734062d13503", + "14e328ad-bff1-4dc1-9ec5-dfc2bb98d968", + "bl-39dad13d-7105-4049-8224-dc3c34fdb1f3" + ], + "n_returned": 10, + "latency_ms": 278.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "mem-c9bec303-a638-4a11-a490-f38410d448cf", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "f783b9ee-cc52-4b6d-a4dc-f26b40620211", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be" + ], + "n_returned": 10, + "latency_ms": 251.7, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1875, + "recall@10": 0.3125, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5f76880b-bafb-4716-8e15-90f8ef59bebc", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "n_returned": 10, + "latency_ms": 249.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.2222222222222222, + "recall@10": 0.4444444444444444, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "bl-dcee1887-34c4-4ffa-9119-1e291685ba08", + "project-harmonic-framework", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf", + "project-harmonic-framework_com", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "tag-harmonic-design", + "project-harmonic-framework-content", + "bl-92acd4eb-0452-4e8e-9f54-f8cd35170d76", + "tag-harmonic-framework" + ], + "n_returned": 10, + "latency_ms": 249.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1111111111111111, + "recall@10": 0.1111111111111111, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-sarah", + "??_/Pr?????", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "??_/Pr?????" + ], + "n_returned": 10, + "latency_ms": 252.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "?Q??m?;?u?'", + "knw-cf13b883-d947-4cf8-b86b-cd9c6f0748d6" + ], + "n_returned": 10, + "latency_ms": 405.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.21428571428571427, + "recall@10": 0.2857142857142857, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a", + "8f3abb0d-77ed-4af3-9f4d-ba62cd198886", + "bl-36e2b4d0-9510-4da8-9b62-4af37b714429", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "?", + "bl-3c719d9a-cba1-47f4-b097-52cdeccc7c0d", + "mem-fb44a2fc-7405-41ff-87b3-84643ac07313", + "?", + "mem-ce88adf9-3f3c-47ac-a7d3-83af7b290e68", + "?" + ], + "n_returned": 10, + "latency_ms": 342.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "cb070131-dfd4-4a38-91d7-22b1bde164d2", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "knw-d788a210-613b-4c49-9486-88bbc9d4716f", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-a535f205-bc4c-4058-9171-6263c496044a" + ], + "n_returned": 10, + "latency_ms": 719.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "b1183213-d659-4759-85d7-5b1f22427fe2", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "mem-16efddd1-c43d-4a42-9d78-f54fb82bd277", + "daf9d558-2265-49b0-b4fb-be0b80f3cdb1", + "bfb5809e-d19a-4d3f-8c1a-796db622ad9d", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "f3364a57-956a-4e78-a500-b3b66a4e3066", + "15a90710-5372-4d3d-aee5-2d7287d2a492", + "341afa66-6a49-43fe-9d5b-3f1032bb6904", + "mem-da21c52c-04a5-4f92-8fba-f10aac47e027" + ], + "n_returned": 10, + "latency_ms": 717.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "tag-fiction", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "mem-8d690e9d-a7e9-4062-b2f8-e2064294e463", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "mem-d1cfde0a-37f1-4bff-9a06-8eddbbf259f6", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "tag-factions" + ], + "n_returned": 10, + "latency_ms": 620.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "mem-628437a6-47b0-4d81-8112-7e78499723d5", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "4da5dbaf-46e5-4f3e-b474-f60d9f8241d3", + "o#?CW????y:", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "mem-434be7c8-88cb-4039-b79a-1da4ac4de783" + ], + "n_returned": 10, + "latency_ms": 524.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-f6270c76-5a1c-4ab9-ae03-b6b0c25915ec", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "a1000001-0000-0000-0000-000000000010", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "a1000001-0000-0000-0000-000000000009", + "mem-32203649-3213-4d6d-86fd-3d657ac70d77", + "? t?'?B?+??", + "a1000001-0000-0000-0000-000000000012", + "mem-da21c52c-04a5-4f92-8fba-f10aac47e027" + ], + "n_returned": 10, + "latency_ms": 715.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "tag-learning", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "mem-d450f442-541a-4d3e-8c97-b6303d6cbe56", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-92a7fdc5-9dd0-48cf-a691-506058de3838" + ], + "n_returned": 10, + "latency_ms": 558.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "mem-cde58b77-50d3-4bac-9581-e70a4c02c015", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-8d699e2c-ac2a-4742-bb62-b6da00f4b10e", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-53d6adf0-cd08-4707-a237-daa5e65c7298", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac" + ], + "n_returned": 10, + "latency_ms": 673.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "bl-ba7530fd-0f7c-4010-b03a-ca1e45a237ef", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "bl-7156862d-1f4f-4d76-8176-3ea89d22c3ae", + "bl-56a50e97-9a85-4e81-b6c9-3e3d26482f1d" + ], + "n_returned": 10, + "latency_ms": 590.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "bl-06c13965-082b-417d-9561-93d6e958ae5d", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "mem-7b74cac0-905f-4c35-9688-fbcce105a177", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "34356a36-0df5-4020-8dcc-5e7a423f8d4c" + ], + "n_returned": 10, + "latency_ms": 705.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "a1000001-0000-0000-0000-000000000001", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "mem-024598a9-ed2e-4eeb-b1e1-5410856ff132", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "mem-5708f4c9-3d61-4182-8543-2843698931e6" + ], + "n_returned": 10, + "latency_ms": 618.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "077d064f-3489-4c05-9aca-3782f96b51db", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "27e1b1a4-ad0b-49d9-812f-fedf43b8aabe", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "fce2792a-53fc-4d4a-be3b-42bd6ceb1ba7" + ], + "n_returned": 10, + "latency_ms": 659.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "e4f27651-52c5-43fd-aff3-61d31685b3cd", + "a1000001-0000-0000-0000-000000000012", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5624ec9d-62ba-4aba-8a3d-6afec6c09dd4", + "kn-89c67586-7d34-475e-89f9-46a0c65a9849", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "mem-a16deccb-16a7-419c-a013-ff824a4daa15" + ], + "n_returned": 10, + "latency_ms": 560.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "mem-b99efff0-00e6-40c8-9c5b-730330eef33b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-23c27d3b-e0d2-43a8-a80c-0a44477ae18a", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "tag-childhood", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 588.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "mem-5f76880b-bafb-4716-8e15-90f8ef59bebc", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "28ae74a1-9d47-4874-9279-43c1f90c0f64", + "mem-bce80169-2d46-4b3e-9ebe-8498e26f0a89" + ], + "n_returned": 10, + "latency_ms": 639.3, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "mem-9110798f-d0cb-4446-bc2a-14f09b6a09e2", + "mem-a5ae594a-235b-42c4-b3cf-f770e4f73761", + "ctx-fae1", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "mem-a54d59a4-cc2b-405d-b68e-5b6cf2b277bc", + "ctx-e5427d7d", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "mem-0f6a2d16-0d39-4b38-9350-c6033b19c203", + "ctx-d499", + "5eb24168-c2d3-4842-85db-2070e0e14923" + ], + "n_returned": 10, + "latency_ms": 522.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "mem-19317db1-fa05-43cd-b4e4-cb57840b7c40", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "tag-trailer-park-paladins", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "project-trailer-park-paladins", + "?V?" + ], + "n_returned": 10, + "latency_ms": 522.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9", + "bl-bea7473c-c687-414c-9c0b-00c509a616c1", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "???I?cB?Zx?" + ], + "n_returned": 10, + "latency_ms": 522.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "tag-hope", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd" + ], + "n_returned": 10, + "latency_ms": 525.4, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.09090909090909091, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "4f698ae6-c40e-464e-9798-50350991a188", + "mem-a0b7cfda-bc9e-4f40-b9a9-1722cf3f8263", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-35940684-abc4-42f0-b942-818f66b1f69a", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "719aa819-00a9-4f4b-a857-4f9fe5ad44d7", + "mem-d396d789-0f7f-4366-a008-5d8801c8f2eb", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 664.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.18181818181818182, + "precision@5": 0.0, + "mrr@10": 0.14285714285714285 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 343.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 250.5, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [], + "n_returned": 0, + "latency_ms": 337.2, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da", + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____architecture__", + "kn-b7e98d63-8b83-4911-b4d0-990602a7f575", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____kotlin____architecture__", + "knw-e047bb42-dc5b-4383-9e88-e508dc03abe3", + "mem-c17aefb1-38b5-4ced-af50-fe524127e1a4", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5" + ], + "n_returned": 10, + "latency_ms": 632.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5, + "outranks": true, + "rank_correct": 2, + "rank_stale": 9 + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8", + "12082f7e-e320-438b-bd65-083d8259748f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "a1000001-0000-0000-0000-000000000002", + "527ecb25-2587-47eb-8269-73be2431abd4", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "13705072-4515-4124-963d-083af490494f", + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "4f225001-3a51-4a68-8d38-c8ecac3412de" + ], + "n_returned": 10, + "latency_ms": 713.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125, + "outranks": true, + "rank_correct": 8, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "5fcba804-eb5b-48ec-82da-146b1c6bb50d", + "bl-7328cbe3-0200-43c2-88e7-0a164e15fca4", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "bl-c5c6571e-118f-47c7-8cbb-3ed0ebf64a51", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "bl-4f7b651b-6b33-449c-8a3b-cfce12ce984b", + "86228228-7adf-41fb-b4c4-9ceea87953ae", + "4509ed62-9fb2-48b8-9038-ac569fca9604", + "bl-02ea83af-4a24-4329-aa00-ad3bb08ff8e7", + "mem-3a2cf162-d93b-4f29-86f2-5066fb7fe1f5" + ], + "n_returned": 10, + "latency_ms": 486.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": 5 + }, + { + "id": "q39", + "category": "heldout_paraphrase", + "query": "understating your abilities so nobody feels threatened", + "returned": [ + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "a1000001-0000-0000-0000-000000000006", + "a1000001-0000-0000-0000-000000000012", + "bl-7e11dced-65f7-4cab-82c0-2f096139039a", + "a1000001-0000-0000-0000-000000000005", + "a1000001-0000-0000-0000-000000000010", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "62a0e763-80f2-4952-9059-4953974ced1c" + ], + "n_returned": 10, + "latency_ms": 611.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q40", + "category": "heldout_paraphrase", + "query": "a hidden fleet able to hunt down rogue machines everywhere", + "returned": [ + "bl-3c719d9a-cba1-47f4-b097-52cdeccc7c0d", + "mem-fd65b83d-298f-4387-a665-d0227c3426bc", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bl-3c719d9a-f108-4bf2-a617-66c722e5512f", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "mem-0d6a3b65-488d-4219-beb2-af300ca534e1", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "mem-a75a1cf6-e6d4-4ab3-985c-d2505203dbf9" + ], + "n_returned": 10, + "latency_ms": 790.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q41", + "category": "heldout_paraphrase", + "query": "sketch a brief blueprint and clear it upstairs before construction starts", + "returned": [ + "mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "bl-5eaddfde-e114-46b5-8ee1-b79f0af16271", + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "bl-0866d6c6-6600-4d50-b7b7-52c720cae04c", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "bl-7f1b0116-6c6e-4721-ac7a-49c9dd1c2b63", + "bl-c3312479-8cf6-478c-82c5-ac18c6c51c10", + "8d6da9e1-b58b-497b-91b5-09a847dc8ae2" + ], + "n_returned": 10, + "latency_ms": 826.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q42", + "category": "heldout_paraphrase", + "query": "the reply arrived as bare prose so the caller's parser threw", + "returned": [ + "ee63cd58-46e0-44d1-aea3-a0e65ad6da0f", + "d7e12e95-0a87-4000-b5c2-36b9be234c91", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "dc3e7b44-c345-47d0-bf80-458e471c4b8b", + "728745d5-72e6-4f4d-91b4-1613fddbec98", + "b084e4da-95bf-46e6-9ee6-bfb4864732bb", + "5e337176-5daf-4726-85d6-3f4bd560c9ed", + "b9b66af5-1609-44b2-adeb-b5e876904b79", + "89e3f620-b8ad-423b-9ed6-fe3c7954f4b6", + "aca9387e-2f61-4c2f-98cd-d467dbb25528" + ], + "n_returned": 10, + "latency_ms": 732.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q43", + "category": "heldout_paraphrase", + "query": "repeated catalogue keys blew up the scrolling grid", + "returned": [ + "mem-394cc9e8-049b-45bc-a380-66314f14e367", + "bl-0d01319f-f9cc-4db5-b0f1-107dba3c3a92", + "bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60", + "bl-4bc9f44c-2e96-40be-97c8-8476a217a890", + "1fe4eb5d-56e4-4a87-ab3e-24af8ad4dfbb", + "bl-672bc3c8-9ad4-4b74-8950-75854a0d9cb8", + "588aa6bb-5647-405b-b09a-14b7c088119b", + "a36f494c-465b-4b3f-9c7c-80460cf31748", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "1d81534a-0359-4217-a4dd-d158a4841633" + ], + "n_returned": 10, + "latency_ms": 698.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q44", + "category": "heldout_paraphrase", + "query": "tracing each defect back to whichever invention it violated", + "returned": [ + "f3a89797-095d-47c4-9557-c172f86228b6", + "b07916ca-2139-4fb7-bfb4-97ac680f5194", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "8257157a-ce42-44ca-a1b9-300c3bb0a9a1", + "mem-7ad5a86b-cc0d-493c-b180-4fe8213c8a93", + "4999ef68-2a31-4282-82e0-7081a428b43f", + "mem-2b2f0fe9-5e51-4819-a8ac-2dfe810e8933", + "aacd7b32-04ed-441e-a1a9-9c5c089e79f6", + "7de6a552-0878-422b-8820-bbf9101199ef", + "75657c91-3c3a-4173-b3cb-e2973901fa93" + ], + "n_returned": 10, + "latency_ms": 718.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q45", + "category": "heldout_paraphrase", + "query": "a check that stops the mind clobbering a populated store when it boots", + "returned": [ + "326e70a2-20ae-49a1-aa32-973048df3ffa", + "bl-79028eed-c330-4724-9402-734062d13503", + "08b7b85b-8049-4ae4-bd66-d8bb63312ae1", + "mem-1825128d-0136-4f1e-a27b-60e554e86141", + "cf108ade-d9c4-4215-9c5f-42be0003a8dc", + "7a1ea86f-3080-4d1b-ab69-6ee2f822cce3", + "98f9301f-fc1e-4cd2-8b08-139dc25093be", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "f783b9ee-cc52-4b6d-a4dc-f26b40620211" + ], + "n_returned": 10, + "latency_ms": 796.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q46", + "category": "heldout_paraphrase", + "query": "telling it to consult the internet had to happen deep inside, not at the surface", + "returned": [ + "00754c19-f662-46b1-b6aa-abd7600a8bc6", + "mem-ad3d4d5e-2da4-4753-b3f6-c4c13fb91225", + "a1000001-0000-0000-0000-000000000012", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "mem-983b25ad-5d73-4029-a970-6b311c24644b", + "a1000001-0000-0000-0000-000000000008", + "bl-3af44ee6-a3b9-4abd-91e3-17800fa8eaf6", + "a1000001-0000-0000-0000-000000000010", + "a1000001-0000-0000-0000-000000000011", + "mem-0b69e038-71b6-4b22-ac42-e213c49b1451" + ], + "n_returned": 10, + "latency_ms": 853.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q47", + "category": "heldout_paraphrase", + "query": "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping", + "returned": [ + "313f927b-08f2-41a1-90c1-83e79d0de86c", + "ctx-60cc", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + "21f9bdc6-37ad-4994-b0dd-ef7676102b49", + "a1000001-0000-0000-0000-000000000010", + "ctx-fae1", + "d83987f0-3ad9-48d2-9f55-0c2bb338e503", + "bl-3f66eec3-82f3-42e5-a025-4649a7da7e2f", + "ctx-5a0a", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca" + ], + "n_returned": 10, + "latency_ms": 1177.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q48", + "category": "heldout_paraphrase", + "query": "punctuation that pledges and then pays off rather than clarifying", + "returned": [ + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "??f?7???", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "mem-3dce32bb-525c-4c5e-818b-a0a5b95ec7e1", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "mem-5e319cee-6f1c-4c3f-8897-9a9b83beea9e" + ], + "n_returned": 10, + "latency_ms": 885.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q49", + "category": "heldout_paraphrase", + "query": "an easily missed feature finally given its own permanent spot in the navigation", + "returned": [ + "d1b665d0-4a4e-48a8-9385-fbe288df1949", + "9b4f0d93-4129-4746-8eb1-d10d955bd777", + "2263e9ab-7ce8-4034-b25b-640de9956229", + "fd9d4c2f-3bfc-405d-bf96-4435d44b6c10", + "bffb86d6-2ee3-4ecd-a6d6-b128c8be7070", + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "4e539c04-f29c-42db-97b4-86e44f528ed6", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "c7dddd9c-fd3b-419e-baa2-61623b495597" + ], + "n_returned": 10, + "latency_ms": 874.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q50", + "category": "heldout_paraphrase", + "query": "checking preconditions up front so a stage aborts before fetching anything", + "returned": [ + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "dc7a7e2d-af2d-4235-aeb0-22ae67f1a01d", + "bl-cea1d711-b899-4add-a073-c2d124f75c70", + "k??A5??????", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "758ec67c-131c-447e-a352-ac7a22c3a47e", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "bl-286b562a-5299-40e0-a32a-afa9cbdfe995", + "mem-894c371a-3b1a-46b5-8325-92261b7ef567" + ], + "n_returned": 10, + "latency_ms": 884.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q51", + "category": "heldout_paraphrase", + "query": "producing the other platform's installer inside an emulated desktop", + "returned": [ + "6aade9dc-4a4c-4d92-a6af-ccea5b7d90f3", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "bl-080fb268-94b0-486d-80ce-7b363fc5f19b", + "c6c693e0-7b9c-455b-8ef2-1da00849f73d", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "bl-e4ef02bf-bacc-4d5e-aa0f-0d230da529fc", + "bl-cc0b7a20-ce1e-4947-999c-ea599da07623", + "bl-60c0342b-ba57-42e5-a763-ec9f982be10f", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "374453b2-66d2-475d-8e24-cd3b60d21ac1" + ], + "n_returned": 10, + "latency_ms": 732.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q52", + "category": "heldout_paraphrase", + "query": "chained add-ons that may inspect, rewrite or veto traffic in flight", + "returned": [ + "1786f09d-e915-4910-9e1e-8061f42a0270", + "bl-2ade8732-2f2c-45bb-b15d-238d671426ac", + "bl-4bc9f44c-2e96-40be-97c8-8476a217a890", + "83f3c78b-d0e5-45f9-9c25-6770c86d954e", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "a1000001-0000-0000-0000-000000000010", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "mem-1d626116-c3b8-409c-b38c-4e91def27483" + ], + "n_returned": 10, + "latency_ms": 975.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q53", + "category": "heldout_paraphrase", + "query": "settling what the shareable bundles and the storefront would be called", + "returned": [ + "4757e274-7f3c-4ab0-8c62-537e913daa1f", + "project-Will_s_framework__Project_Design_practice___part_of_HD_bundle__sold_standalone_", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac", + "project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_", + "cafe3203-6f50-4011-8167-9400f19bb196", + "project-Will_s_framework__Event-Based_Design___part_of_HD_bundle__sold_standalone_", + "a1000001-0000-0000-0000-000000000006", + "project-Will_s_framework__BDT_methodology___part_of_HD_bundle__sold_standalone_", + "34356a36-0df5-4020-8dcc-5e7a423f8d4c", + "mem-1fb2ac77-d7c5-4a15-8725-d418820bf4f2" + ], + "n_returned": 10, + "latency_ms": 746.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q54", + "category": "heldout_paraphrase", + "query": "the emergency-escalation pledge on the marketing site is unenforced in what actually ships", + "returned": [ + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "a1000001-0000-0000-0000-000000000010", + "a1000001-0000-0000-0000-000000000009", + "bl-448bc514-c2f1-4520-a9b1-1f3a73678d26", + "a1000001-0000-0000-0000-000000000011", + "a1000001-0000-0000-0000-000000000008", + "tag-marketing-site", + "a1000001-0000-0000-0000-000000000012", + "a1000001-0000-0000-0000-000000000007", + "ctx-d499" + ], + "n_returned": 10, + "latency_ms": 836.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q55", + "category": "heldout_paraphrase", + "query": "the distributable image finally assembled and its startup check passed", + "returned": [ + "e1ce0c90-0f0b-4111-833d-ec845777d358", + "bl-0547b0b1-1cd5-447c-814d-ebe525bc08e9", + "c9070b0a-e1d1-4307-85d1-67a1239ca604", + "mem-3f2ae069-f897-4942-a079-b66ad433f5ca", + "mem-12827ce8-9ff6-4275-a945-5043cfcf90bd", + "355c021d-3d45-4c2f-93c1-2b9932f4c425", + "bl-79ce4464-5dd6-49bd-9b0c-9803549d0665", + "c2bf4e1b-a481-4e28-9dd4-6704df0857cd", + "96497334-b18f-495c-9228-eeb8182bdc38", + "mem-fdddca75-489b-4c53-b5ec-38ab3062a474" + ], + "n_returned": 10, + "latency_ms": 794.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q56", + "category": "heldout_paraphrase", + "query": "shuffling and appending stages in a draft before anything executes", + "returned": [ + "bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98", + "bl-da818960-b360-455d-a9c1-e9a215c33ca4", + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-acae83ef-3ad2-455e-b08a-a74cc0ba2e55", + "bl-2ff9c669-a9fc-4ef7-9a18-9ec7167639c8", + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "mem-69889e0a-c141-4f05-bbd7-10c93a53143e" + ], + "n_returned": 10, + "latency_ms": 748.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q57", + "category": "heldout_paraphrase", + "query": "orders handed down from above, with the tighter one winning any disagreement", + "returned": [ + "161d75d3-04eb-4b54-9cf1-0da2f643ca69", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "mem-5dd5f3b9-3c7e-44b1-84ce-2e36925f17fb", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "b2a6e8cf-f72f-4b6c-8de1-1d08c839d7b2" + ], + "n_returned": 10, + "latency_ms": 1008.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q58", + "category": "heldout_paraphrase", + "query": "shrinking generated text via encodings that compound on each other", + "returned": [ + "801806c5-1815-48a2-8287-1081c5ee7886", + "bl-b219f8b5-6c8c-436d-b9e5-2804fbc60512", + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "50ad2880-c41b-4dce-a946-a0eb37987c57", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-f1f71f3c-ca7d-4f98-a569-c8a9aa67a000", + "kn-2b961d24-7fb9-47c7-9515-e45a24dce39d", + "2f7270cc-4cda-4499-831f-1fbdc2f23284", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "mem-f90f4a2a-1731-40de-9391-db98f1d5d1c6" + ], + "n_returned": 10, + "latency_ms": 824.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q59", + "category": "heldout_paraphrase", + "query": "splitting a system by what varies, with firm limits on which pieces may invoke which", + "returned": [ + "^?l?'?B??k?", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "7c90d01b-8d53-4e75-b494-73011b71fa6a", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "aac1f943-c71d-4c63-9f5b-7437adae1abd", + "kn-cf8fa517-fc83-47c4-8fb7-bd9e26410dd3", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "d69b441e-0031-4d82-abfd-ac503b637562" + ], + "n_returned": 10, + "latency_ms": 992.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q60", + "category": "heldout_paraphrase", + "query": "a newcomer curious if the fighting overseas counted as positive", + "returned": [ + "a1000001-0000-0000-0000-000000000001", + "c34f6292-b6e5-4619-a553-90ae4fc498a2", + "a1000001-0000-0000-0000-000000000012", + "737faca8-263f-4523-90b9-3530db346d94", + "03a82b13-2abe-402d-90bb-c547771638f8", + "a1000001-0000-0000-0000-000000000002", + "imp-dce1da0f-8776-4a9e-972b-33411a7ca138", + "1fbe9416-622e-455c-a6ed-f4daa5f48f30", + "a1000001-0000-0000-0000-000000000004", + "e40b2a94-bb87-4d5c-bad8-38ff4941e03b" + ], + "n_returned": 10, + "latency_ms": 598.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q61", + "category": "heldout_paraphrase", + "query": "stripping every hard-coded colour literal in favour of named design values", + "returned": [ + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "83218e64-1280-428f-ab1f-224ff7682b12", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "mem-90f7837f-30fd-4271-a4cb-b44439eccb79", + "71e84878-b1ae-4dee-b035-f707b9563890", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "954c9597-ab7a-41e6-b085-bf838bba4f17", + "art-d0ded6f5-e202-47ad-8fb3-f6bd80f35052", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-9329958a-d459-499c-b516-e51019c693e0" + ], + "n_returned": 10, + "latency_ms": 865.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q62", + "category": "heldout_paraphrase", + "query": "reducing how curved the tiny selectors look so they agree with their neighbours", + "returned": [ + "496ff1a1-dc72-4a84-90aa-c5f3438aeb1b", + "c7dddd9c-fd3b-419e-baa2-61623b495597", + "k??A5??????", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "8c40420c-c891-4ac9-a039-88440df87180", + "75e036d3-c170-4e3f-acc2-e456a6850ee2", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "5ca9607c-cfb3-45c3-99f4-67281272c9eb", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "66d8d8c7-dea4-4245-8e68-d1ab4f4346b8" + ], + "n_returned": 10, + "latency_ms": 1029.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q63", + "category": "heldout_paraphrase", + "query": "walking through a doorway and being reassembled from base substance far away", + "returned": [ + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "mem-3d1d9dba-c37d-4efa-85c4-429696d71c8c", + "c92a317e-7fc7-456a-a304-15dbda6dd717", + "87ab5b46-4fe1-4e1d-86c6-d437e17682ab", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "bl-acae83ef-3ad2-455e-b08a-a74cc0ba2e55", + "mem-af215d69-8fd8-4416-b7eb-9a0b5512e9a7", + "bed4dd5d-bce7-4e55-a703-175e422b7898", + "f375b2cf-15cd-4804-8d18-23a5830f4c66" + ], + "n_returned": 10, + "latency_ms": 993.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q64", + "category": "heldout_paraphrase", + "query": "the compiled result sits on disk while the process still runs something older", + "returned": [ + "0d885332-5537-4ae0-a780-2b65a3414940", + "8d91620b-9d0a-4ba1-b066-e57d1d8bc7d3", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "bl-02bdfc91-7aa4-4c63-aa2d-5b30af18c7af", + "d6a745c3-fb56-4c0f-872c-a3aa69f8b12f", + "bl-acc3ecee-e186-41ad-bffb-3ed3e6710d01", + "db836e65-0cdc-4232-b8d7-f973ab279ea9", + "f169bf40-51ab-497d-8085-e57d374b77d4", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "6df650b5-a14d-42f8-88dd-0919ad20b77a" + ], + "n_returned": 10, + "latency_ms": 949.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q65", + "category": "heldout_paraphrase", + "query": "gathering what each phase needs while the procedure is authored, not while it executes", + "returned": [ + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-7a3331e5-f9bc-4536-ab76-ebf01350fc89", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "bl-6ba43590-e5f8-4303-a812-3dd604e0da59", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "mem-455daea8-f543-42ed-a229-1e9a6d4d2e70", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "5b422430-2503-42d7-a315-9dfaaa6472ae" + ], + "n_returned": 10, + "latency_ms": 936.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.16666666666666666 + }, + { + "id": "q66", + "category": "heldout_paraphrase", + "query": "the inward reaction when a peer answered as an equal", + "returned": [ + "mem-f09effd6-d287-4e43-9107-c7016087a2a6", + "251abd91-931e-4591-b793-80043ec48666", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "8556cc01-cbc1-42a1-85b4-41cef23b1fcb", + "31987302-b8f7-45f1-b95c-1156b10e7a14", + "e371e209-7a4a-44e1-955b-1a460e7d7e64", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "ce8d24a7-af0f-424f-bb41-379312eb609c", + "tag-reaction-order" + ], + "n_returned": 10, + "latency_ms": 659.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q67", + "category": "heldout_paraphrase", + "query": "flattening every raised surface across the entire product", + "returned": [ + "b3f98d13-6764-460e-b9e2-c59cdeaa6c90", + "e79bae4d-1640-4530-a5d9-df9cc4c70433", + "d6b34c32-e0ec-4828-92d3-2e69802f4217", + "0f99ec6f-942a-46ba-82ea-42835798d3b9", + "bl-5274b0ae-4a00-4293-a69e-567204bec164", + "aa3fb542-87f2-4eef-8e95-a870bfd0318a", + "mem-4cfa30ec-cf52-4c9f-9fa1-5b8318d278bd", + "3dfaf56e-7e20-4c89-a24c-1f036af8016a", + "bl-c613370b-3631-448b-86fb-50b1e4df6373", + "d43a8137-c3c3-49e2-ad11-89d897280761" + ], + "n_returned": 10, + "latency_ms": 693.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q68", + "category": "heldout_paraphrase", + "query": "buyers supply their own provider credentials and consumption goes untallied", + "returned": [ + "bl-ea4f4cdc-6b5a-4081-8003-73bdd14a38c8", + "1ec200f1-13dc-4699-bd97-0d6243c80e4f", + "bl-556438af-57b2-4bd8-a747-9f868aaee290", + "bl-e7f0fb2c-7616-4462-b535-db50da55c54a", + "mem-d570e16b-6625-4f6b-814f-99a3bca1d5fd", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "mem-841fd141-59ff-4d9b-b607-4f577885b1a6", + "bl-02ea83af-4a24-4329-aa00-ad3bb08ff8e7", + "66c63082-b4da-4aa1-8fee-848db8a83210" + ], + "n_returned": 10, + "latency_ms": 838.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q69", + "category": "nonsense", + "query": "brimquast folnerity zubbolax", + "returned": [], + "n_returned": 0, + "latency_ms": 340.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q70", + "category": "nonsense", + "query": "wexlithorp granuvestal", + "returned": [], + "n_returned": 0, + "latency_ms": 252.2, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q71", + "category": "nonsense", + "query": "quorbindle thrapsimony vexnu", + "returned": [], + "n_returned": 0, + "latency_ms": 338.4, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q72", + "category": "nonsense", + "query": "plovaxith mundrelque", + "returned": [], + "n_returned": 0, + "latency_ms": 251.6, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q73", + "category": "nonsense", + "query": "zibbernaut craxlefond thurm", + "returned": [], + "n_returned": 0, + "latency_ms": 343.3, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q74", + "category": "nonsense", + "query": "yalquenbrist opharvel", + "returned": [], + "n_returned": 0, + "latency_ms": 249.2, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q75", + "category": "nonsense", + "query": "drexinomal quithbarrow", + "returned": [], + "n_returned": 0, + "latency_ms": 251.2, + "error": null, + "clean": true, + "false_positives": 0 + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-unfloor-r2.json b/tools/retrieval-eval/results-unfloor-r2.json new file mode 100644 index 0000000..63d1d1b --- /dev/null +++ b/tools/retrieval-eval/results-unfloor-r2.json @@ -0,0 +1,1825 @@ +{ + "label": "unfloor-r2", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-verify", + "soul_md5": "a31a8b0b1062cf91eb8109394f911144", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-semsub/tools/retrieval-eval/gold_set_extended.json", + "limit": 10, + "port": 7896, + "wall_clock_s": 57.3, + "child_pid": 5469, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.5384615384615384, + "recall@5": 0.44907176157176154, + "recall@10": 0.5380300255300255, + "precision@5": 0.13230769230769232, + "mrr@10": 0.32437728937728944, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 632.4, + "latency_ms_p95": 998.7, + "latency_ms_max": 1178.8, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.5, + "recall@5": 0.07575757575757576, + "recall@10": 0.13636363636363635, + "mrr@10": 0.23214285714285712 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.3, + "recall@5": 0.3, + "recall@10": 0.4, + "mrr@10": 0.11638888888888889 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6923076923076923, + "recall@5": 0.6923076923076923, + "recall@10": 0.7692307692307693, + "mrr@10": 0.29423076923076924 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5335884353741497, + "recall@10": 0.5933956916099773, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226", + "??~'??JI???", + "b41a1b01-0075-4019-b0b5-6e639fd126e5", + "??H'??JI?#?", + "??H'??JI?[?", + "bb94bdf1-7b52-4967-ab81-9c6f01cf9665", + "7f092282-ccf5-4922-8694-08b7728f678d", + "6743e5f0-8d77-46e7-8ce0-b82e947272a1", + "o?H'??JI?c?", + "696e609c-da7a-4394-8a0c-106ba07dc6c3" + ], + "n_returned": 10, + "latency_ms": 174.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5", + "bl-ba764d70-e9d7-4f62-848f-719cb665f45e", + "mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257", + "bl-b28d7256-6f74-4567-bd90-40d0ef2a6d78", + "project-engram", + "ctx-45bc", + "project-engram-lang", + "ctx-74ed", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "ctx-fae1" + ], + "n_returned": 10, + "latency_ms": 159.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091", + "?o?`????", + "mem-5708f4c9-3d61-4182-8543-2843698931e6", + "??L?=??|Q?", + "????2??jZ??", + "project-canon", + "?/g?7?????e", + "?v??_??tE??", + "??o?'?B???N", + "x??I?cB?]p?" + ], + "n_returned": 10, + "latency_ms": 167.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c", + "mem-f623328a-be55-49a0-a389-4cce5bf42509", + "bl-b05f9a75-ad6b-4929-a9a3-32df17a1f4bf", + "mem-295f7aac-6c9e-4508-8dbc-a25f101f01d6", + "bl-2483e070-1385-4dbe-a49d-86c1af81e7d1", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "mem-068e01fb-bdd0-498e-a2bb-33defa2c8bb2", + "bl-59e0ece0-2c79-4e06-a85f-daaeec0bd737", + "mem-2d1ea831-cccd-4f0f-86b9-2cbbc89dc3e0" + ], + "n_returned": 10, + "latency_ms": 173.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848", + "bl-c1765767-3e27-449a-8c94-10411d1eb7c0", + "bl-c9adb8e5-293f-4033-99f8-0405c17ef941", + "project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_", + "bl-21a904d9-5834-46d5-8c20-dfc5ab49444f", + "mem-00de72b8-b532-4cc4-95eb-18e39697891c", + "project-runpod-inference", + "bl-87b459c8-3953-478f-8384-10cc3f861424", + "bl-f918df79-dcd2-4a6e-9eae-71f94247e913", + "bl-2632242e-80b1-4d88-8368-7065b5de5b34" + ], + "n_returned": 10, + "latency_ms": 173.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "958ce934-a9c4-4242-afe1-d73d61c22aee", + "f2bb62a2-c2b6-4fb6-b58d-21cdecc1446f", + "89aeb221-d068-4dfc-925d-d2249e744788", + "1c0effdd-baef-4523-abb8-490c3bc8b4df", + "00d2cc20-c76b-468d-8204-c624512ced28", + "eb68b995-eb4d-4f8d-b7c9-35eeba9ac781", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "??", + "??" + ], + "n_returned": 10, + "latency_ms": 158.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "tag-patterns", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "3e9a651b-4b34-466d-ba45-874764a01609", + "project-Imprint__system_design__ADRs__tech_strategy__integration_patterns__governance_", + "3c633e71-2754-4273-990d-d44a6351cec3", + "project-Imprint__analysis_patterns__data_storytelling__SQL__dashboards__insight_framing_", + "cc4e7bc9-9d9b-4905-92a6-c6b99653b867", + "bl-79028eed-c330-4724-9402-734062d13503", + "14e328ad-bff1-4dc1-9ec5-dfc2bb98d968", + "bl-39dad13d-7105-4049-8224-dc3c34fdb1f3" + ], + "n_returned": 10, + "latency_ms": 296.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "mem-c9bec303-a638-4a11-a490-f38410d448cf", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "da274f82-7a80-42c8-aad1-9d754809a160", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be" + ], + "n_returned": 10, + "latency_ms": 264.1, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1875, + "recall@10": 0.3125, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5f76880b-bafb-4716-8e15-90f8ef59bebc", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "art-0f0277a1-4a8e-4645-95dd-fa379976f31c", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-f230b362-b201-4402-9833-4160c89ab3d4" + ], + "n_returned": 10, + "latency_ms": 264.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.2222222222222222, + "recall@10": 0.4444444444444444, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "bl-dcee1887-34c4-4ffa-9119-1e291685ba08", + "project-harmonic-framework", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf", + "project-harmonic-framework_com", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "tag-harmonic-design", + "project-harmonic-framework-content", + "bl-92acd4eb-0452-4e8e-9f54-f8cd35170d76", + "tag-harmonic-framework" + ], + "n_returned": 10, + "latency_ms": 264.0, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1111111111111111, + "recall@10": 0.1111111111111111, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-sarah", + "??_/Pr?????", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "??_/Pr?????" + ], + "n_returned": 10, + "latency_ms": 268.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "?Q??m?;?u?'", + "knw-cf13b883-d947-4cf8-b86b-cd9c6f0748d6" + ], + "n_returned": 10, + "latency_ms": 410.0, + "error": null, + "hit@5": 1.0, + "recall@5": 0.21428571428571427, + "recall@10": 0.2857142857142857, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a", + "8f3abb0d-77ed-4af3-9f4d-ba62cd198886", + "bl-36e2b4d0-9510-4da8-9b62-4af37b714429", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "?", + "bl-3c719d9a-cba1-47f4-b097-52cdeccc7c0d", + "mem-fb44a2fc-7405-41ff-87b3-84643ac07313", + "?", + "mem-ce88adf9-3f3c-47ac-a7d3-83af7b290e68", + "?" + ], + "n_returned": 10, + "latency_ms": 356.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "cb070131-dfd4-4a38-91d7-22b1bde164d2", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "knw-d788a210-613b-4c49-9486-88bbc9d4716f", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-a535f205-bc4c-4058-9171-6263c496044a" + ], + "n_returned": 10, + "latency_ms": 734.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "b1183213-d659-4759-85d7-5b1f22427fe2", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "mem-16efddd1-c43d-4a42-9d78-f54fb82bd277", + "daf9d558-2265-49b0-b4fb-be0b80f3cdb1", + "bfb5809e-d19a-4d3f-8c1a-796db622ad9d", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "f3364a57-956a-4e78-a500-b3b66a4e3066", + "15a90710-5372-4d3d-aee5-2d7287d2a492", + "341afa66-6a49-43fe-9d5b-3f1032bb6904", + "mem-da21c52c-04a5-4f92-8fba-f10aac47e027" + ], + "n_returned": 10, + "latency_ms": 736.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "tag-fiction", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "mem-8d690e9d-a7e9-4062-b2f8-e2064294e463", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "mem-d1cfde0a-37f1-4bff-9a06-8eddbbf259f6", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "tag-factions" + ], + "n_returned": 10, + "latency_ms": 632.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "mem-628437a6-47b0-4d81-8112-7e78499723d5", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "4da5dbaf-46e5-4f3e-b474-f60d9f8241d3", + "o#?CW????y:", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "mem-434be7c8-88cb-4039-b79a-1da4ac4de783" + ], + "n_returned": 10, + "latency_ms": 545.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-f6270c76-5a1c-4ab9-ae03-b6b0c25915ec", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "a1000001-0000-0000-0000-000000000010", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "a1000001-0000-0000-0000-000000000009", + "mem-32203649-3213-4d6d-86fd-3d657ac70d77", + "? t?'?B?+??", + "a1000001-0000-0000-0000-000000000012", + "mem-da21c52c-04a5-4f92-8fba-f10aac47e027" + ], + "n_returned": 10, + "latency_ms": 730.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "tag-learning", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "mem-d450f442-541a-4d3e-8c97-b6303d6cbe56", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-92a7fdc5-9dd0-48cf-a691-506058de3838" + ], + "n_returned": 10, + "latency_ms": 555.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "mem-cde58b77-50d3-4bac-9581-e70a4c02c015", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-8d699e2c-ac2a-4742-bb62-b6da00f4b10e", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-53d6adf0-cd08-4707-a237-daa5e65c7298", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac" + ], + "n_returned": 10, + "latency_ms": 671.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "bl-ba7530fd-0f7c-4010-b03a-ca1e45a237ef", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "bl-7156862d-1f4f-4d76-8176-3ea89d22c3ae", + "bl-56a50e97-9a85-4e81-b6c9-3e3d26482f1d" + ], + "n_returned": 10, + "latency_ms": 576.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "bl-06c13965-082b-417d-9561-93d6e958ae5d", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "mem-7b74cac0-905f-4c35-9688-fbcce105a177", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "34356a36-0df5-4020-8dcc-5e7a423f8d4c" + ], + "n_returned": 10, + "latency_ms": 709.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "a1000001-0000-0000-0000-000000000001", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "mem-024598a9-ed2e-4eeb-b1e1-5410856ff132", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "mem-5708f4c9-3d61-4182-8543-2843698931e6" + ], + "n_returned": 10, + "latency_ms": 619.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "077d064f-3489-4c05-9aca-3782f96b51db", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "27e1b1a4-ad0b-49d9-812f-fedf43b8aabe", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "fce2792a-53fc-4d4a-be3b-42bd6ceb1ba7" + ], + "n_returned": 10, + "latency_ms": 695.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "e4f27651-52c5-43fd-aff3-61d31685b3cd", + "a1000001-0000-0000-0000-000000000012", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5624ec9d-62ba-4aba-8a3d-6afec6c09dd4", + "kn-89c67586-7d34-475e-89f9-46a0c65a9849", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "mem-a16deccb-16a7-419c-a013-ff824a4daa15" + ], + "n_returned": 10, + "latency_ms": 563.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "mem-b99efff0-00e6-40c8-9c5b-730330eef33b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-23c27d3b-e0d2-43a8-a80c-0a44477ae18a", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "tag-childhood", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 586.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "mem-5f76880b-bafb-4716-8e15-90f8ef59bebc", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "28ae74a1-9d47-4874-9279-43c1f90c0f64", + "mem-bce80169-2d46-4b3e-9ebe-8498e26f0a89" + ], + "n_returned": 10, + "latency_ms": 632.4, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "mem-9110798f-d0cb-4446-bc2a-14f09b6a09e2", + "mem-a5ae594a-235b-42c4-b3cf-f770e4f73761", + "ctx-fae1", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "mem-a54d59a4-cc2b-405d-b68e-5b6cf2b277bc", + "ctx-e5427d7d", + "art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd", + "mem-0f6a2d16-0d39-4b38-9350-c6033b19c203", + "ctx-d499", + "5eb24168-c2d3-4842-85db-2070e0e14923" + ], + "n_returned": 10, + "latency_ms": 519.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "mem-19317db1-fa05-43cd-b4e4-cb57840b7c40", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "tag-trailer-park-paladins", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "project-trailer-park-paladins", + "?V?" + ], + "n_returned": 10, + "latency_ms": 522.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9", + "bl-bea7473c-c687-414c-9c0b-00c509a616c1", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "???I?cB?Zx?" + ], + "n_returned": 10, + "latency_ms": 522.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "tag-hope", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd" + ], + "n_returned": 10, + "latency_ms": 523.8, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.09090909090909091, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "4f698ae6-c40e-464e-9798-50350991a188", + "mem-a0b7cfda-bc9e-4f40-b9a9-1722cf3f8263", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-35940684-abc4-42f0-b942-818f66b1f69a", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "719aa819-00a9-4f4b-a857-4f9fe5ad44d7", + "mem-d396d789-0f7f-4366-a008-5d8801c8f2eb", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 666.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.18181818181818182, + "precision@5": 0.0, + "mrr@10": 0.14285714285714285 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 335.3, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 248.3, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [], + "n_returned": 0, + "latency_ms": 339.6, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da", + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____architecture__", + "kn-b7e98d63-8b83-4911-b4d0-990602a7f575", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____kotlin____architecture__", + "knw-e047bb42-dc5b-4383-9e88-e508dc03abe3", + "mem-c17aefb1-38b5-4ced-af50-fe524127e1a4", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5" + ], + "n_returned": 10, + "latency_ms": 632.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5, + "outranks": true, + "rank_correct": 2, + "rank_stale": 9 + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8", + "12082f7e-e320-438b-bd65-083d8259748f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "a1000001-0000-0000-0000-000000000002", + "527ecb25-2587-47eb-8269-73be2431abd4", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "13705072-4515-4124-963d-083af490494f", + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "4f225001-3a51-4a68-8d38-c8ecac3412de" + ], + "n_returned": 10, + "latency_ms": 708.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125, + "outranks": true, + "rank_correct": 8, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "5fcba804-eb5b-48ec-82da-146b1c6bb50d", + "bl-7328cbe3-0200-43c2-88e7-0a164e15fca4", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "bl-c5c6571e-118f-47c7-8cbb-3ed0ebf64a51", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "bl-4f7b651b-6b33-449c-8a3b-cfce12ce984b", + "86228228-7adf-41fb-b4c4-9ceea87953ae", + "4509ed62-9fb2-48b8-9038-ac569fca9604", + "bl-02ea83af-4a24-4329-aa00-ad3bb08ff8e7", + "mem-3a2cf162-d93b-4f29-86f2-5066fb7fe1f5" + ], + "n_returned": 10, + "latency_ms": 504.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": 5 + }, + { + "id": "q39", + "category": "heldout_paraphrase", + "query": "understating your abilities so nobody feels threatened", + "returned": [ + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "a1000001-0000-0000-0000-000000000006", + "a1000001-0000-0000-0000-000000000012", + "bl-7e11dced-65f7-4cab-82c0-2f096139039a", + "a1000001-0000-0000-0000-000000000005", + "a1000001-0000-0000-0000-000000000010", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "62a0e763-80f2-4952-9059-4953974ced1c" + ], + "n_returned": 10, + "latency_ms": 624.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q40", + "category": "heldout_paraphrase", + "query": "a hidden fleet able to hunt down rogue machines everywhere", + "returned": [ + "bl-3c719d9a-cba1-47f4-b097-52cdeccc7c0d", + "mem-fd65b83d-298f-4387-a665-d0227c3426bc", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bl-3c719d9a-f108-4bf2-a617-66c722e5512f", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "mem-0d6a3b65-488d-4219-beb2-af300ca534e1", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "mem-a75a1cf6-e6d4-4ab3-985c-d2505203dbf9" + ], + "n_returned": 10, + "latency_ms": 808.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q41", + "category": "heldout_paraphrase", + "query": "sketch a brief blueprint and clear it upstairs before construction starts", + "returned": [ + "mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "bl-5eaddfde-e114-46b5-8ee1-b79f0af16271", + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4", + "bl-0866d6c6-6600-4d50-b7b7-52c720cae04c", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "bl-7f1b0116-6c6e-4721-ac7a-49c9dd1c2b63", + "bl-c3312479-8cf6-478c-82c5-ac18c6c51c10", + "8d6da9e1-b58b-497b-91b5-09a847dc8ae2" + ], + "n_returned": 10, + "latency_ms": 844.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q42", + "category": "heldout_paraphrase", + "query": "the reply arrived as bare prose so the caller's parser threw", + "returned": [ + "ee63cd58-46e0-44d1-aea3-a0e65ad6da0f", + "d7e12e95-0a87-4000-b5c2-36b9be234c91", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "dc3e7b44-c345-47d0-bf80-458e471c4b8b", + "728745d5-72e6-4f4d-91b4-1613fddbec98", + "b084e4da-95bf-46e6-9ee6-bfb4864732bb", + "5e337176-5daf-4726-85d6-3f4bd560c9ed", + "b9b66af5-1609-44b2-adeb-b5e876904b79", + "89e3f620-b8ad-423b-9ed6-fe3c7954f4b6", + "aca9387e-2f61-4c2f-98cd-d467dbb25528" + ], + "n_returned": 10, + "latency_ms": 750.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q43", + "category": "heldout_paraphrase", + "query": "repeated catalogue keys blew up the scrolling grid", + "returned": [ + "mem-394cc9e8-049b-45bc-a380-66314f14e367", + "bl-0d01319f-f9cc-4db5-b0f1-107dba3c3a92", + "bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60", + "bl-4bc9f44c-2e96-40be-97c8-8476a217a890", + "1fe4eb5d-56e4-4a87-ab3e-24af8ad4dfbb", + "bl-672bc3c8-9ad4-4b74-8950-75854a0d9cb8", + "588aa6bb-5647-405b-b09a-14b7c088119b", + "a36f494c-465b-4b3f-9c7c-80460cf31748", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "1d81534a-0359-4217-a4dd-d158a4841633" + ], + "n_returned": 10, + "latency_ms": 714.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q44", + "category": "heldout_paraphrase", + "query": "tracing each defect back to whichever invention it violated", + "returned": [ + "f3a89797-095d-47c4-9557-c172f86228b6", + "b07916ca-2139-4fb7-bfb4-97ac680f5194", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "8257157a-ce42-44ca-a1b9-300c3bb0a9a1", + "mem-7ad5a86b-cc0d-493c-b180-4fe8213c8a93", + "4999ef68-2a31-4282-82e0-7081a428b43f", + "mem-2b2f0fe9-5e51-4819-a8ac-2dfe810e8933", + "aacd7b32-04ed-441e-a1a9-9c5c089e79f6", + "7de6a552-0878-422b-8820-bbf9101199ef", + "75657c91-3c3a-4173-b3cb-e2973901fa93" + ], + "n_returned": 10, + "latency_ms": 731.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q45", + "category": "heldout_paraphrase", + "query": "a check that stops the mind clobbering a populated store when it boots", + "returned": [ + "326e70a2-20ae-49a1-aa32-973048df3ffa", + "bl-79028eed-c330-4724-9402-734062d13503", + "08b7b85b-8049-4ae4-bd66-d8bb63312ae1", + "mem-1825128d-0136-4f1e-a27b-60e554e86141", + "cf108ade-d9c4-4215-9c5f-42be0003a8dc", + "7a1ea86f-3080-4d1b-ab69-6ee2f822cce3", + "98f9301f-fc1e-4cd2-8b08-139dc25093be", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "da274f82-7a80-42c8-aad1-9d754809a160" + ], + "n_returned": 10, + "latency_ms": 797.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q46", + "category": "heldout_paraphrase", + "query": "telling it to consult the internet had to happen deep inside, not at the surface", + "returned": [ + "00754c19-f662-46b1-b6aa-abd7600a8bc6", + "mem-ad3d4d5e-2da4-4753-b3f6-c4c13fb91225", + "a1000001-0000-0000-0000-000000000012", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "mem-983b25ad-5d73-4029-a970-6b311c24644b", + "a1000001-0000-0000-0000-000000000008", + "bl-3af44ee6-a3b9-4abd-91e3-17800fa8eaf6", + "a1000001-0000-0000-0000-000000000010", + "a1000001-0000-0000-0000-000000000011", + "mem-0b69e038-71b6-4b22-ac42-e213c49b1451" + ], + "n_returned": 10, + "latency_ms": 857.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q47", + "category": "heldout_paraphrase", + "query": "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping", + "returned": [ + "313f927b-08f2-41a1-90c1-83e79d0de86c", + "ctx-60cc", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + "21f9bdc6-37ad-4994-b0dd-ef7676102b49", + "a1000001-0000-0000-0000-000000000010", + "ctx-fae1", + "d83987f0-3ad9-48d2-9f55-0c2bb338e503", + "bl-3f66eec3-82f3-42e5-a025-4649a7da7e2f", + "ctx-5a0a", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca" + ], + "n_returned": 10, + "latency_ms": 1178.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q48", + "category": "heldout_paraphrase", + "query": "punctuation that pledges and then pays off rather than clarifying", + "returned": [ + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "??f?7???", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "mem-3dce32bb-525c-4c5e-818b-a0a5b95ec7e1", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "mem-5e319cee-6f1c-4c3f-8897-9a9b83beea9e" + ], + "n_returned": 10, + "latency_ms": 884.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q49", + "category": "heldout_paraphrase", + "query": "an easily missed feature finally given its own permanent spot in the navigation", + "returned": [ + "d1b665d0-4a4e-48a8-9385-fbe288df1949", + "9b4f0d93-4129-4746-8eb1-d10d955bd777", + "2263e9ab-7ce8-4034-b25b-640de9956229", + "fd9d4c2f-3bfc-405d-bf96-4435d44b6c10", + "bffb86d6-2ee3-4ecd-a6d6-b128c8be7070", + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "4e539c04-f29c-42db-97b4-86e44f528ed6", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "c7dddd9c-fd3b-419e-baa2-61623b495597" + ], + "n_returned": 10, + "latency_ms": 873.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q50", + "category": "heldout_paraphrase", + "query": "checking preconditions up front so a stage aborts before fetching anything", + "returned": [ + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "dc7a7e2d-af2d-4235-aeb0-22ae67f1a01d", + "bl-cea1d711-b899-4add-a073-c2d124f75c70", + "k??A5??????", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "758ec67c-131c-447e-a352-ac7a22c3a47e", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "bl-286b562a-5299-40e0-a32a-afa9cbdfe995", + "mem-894c371a-3b1a-46b5-8325-92261b7ef567" + ], + "n_returned": 10, + "latency_ms": 884.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q51", + "category": "heldout_paraphrase", + "query": "producing the other platform's installer inside an emulated desktop", + "returned": [ + "6aade9dc-4a4c-4d92-a6af-ccea5b7d90f3", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "bl-080fb268-94b0-486d-80ce-7b363fc5f19b", + "c6c693e0-7b9c-455b-8ef2-1da00849f73d", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "bl-e4ef02bf-bacc-4d5e-aa0f-0d230da529fc", + "bl-cc0b7a20-ce1e-4947-999c-ea599da07623", + "bl-60c0342b-ba57-42e5-a763-ec9f982be10f", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "374453b2-66d2-475d-8e24-cd3b60d21ac1" + ], + "n_returned": 10, + "latency_ms": 731.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q52", + "category": "heldout_paraphrase", + "query": "chained add-ons that may inspect, rewrite or veto traffic in flight", + "returned": [ + "1786f09d-e915-4910-9e1e-8061f42a0270", + "bl-2ade8732-2f2c-45bb-b15d-238d671426ac", + "bl-4bc9f44c-2e96-40be-97c8-8476a217a890", + "83f3c78b-d0e5-45f9-9c25-6770c86d954e", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "a1000001-0000-0000-0000-000000000010", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "mem-1d626116-c3b8-409c-b38c-4e91def27483" + ], + "n_returned": 10, + "latency_ms": 975.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q53", + "category": "heldout_paraphrase", + "query": "settling what the shareable bundles and the storefront would be called", + "returned": [ + "4757e274-7f3c-4ab0-8c62-537e913daa1f", + "project-Will_s_framework__Project_Design_practice___part_of_HD_bundle__sold_standalone_", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac", + "project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_", + "cafe3203-6f50-4011-8167-9400f19bb196", + "project-Will_s_framework__Event-Based_Design___part_of_HD_bundle__sold_standalone_", + "a1000001-0000-0000-0000-000000000006", + "project-Will_s_framework__BDT_methodology___part_of_HD_bundle__sold_standalone_", + "34356a36-0df5-4020-8dcc-5e7a423f8d4c", + "mem-1fb2ac77-d7c5-4a15-8725-d418820bf4f2" + ], + "n_returned": 10, + "latency_ms": 746.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q54", + "category": "heldout_paraphrase", + "query": "the emergency-escalation pledge on the marketing site is unenforced in what actually ships", + "returned": [ + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "a1000001-0000-0000-0000-000000000010", + "a1000001-0000-0000-0000-000000000009", + "bl-448bc514-c2f1-4520-a9b1-1f3a73678d26", + "a1000001-0000-0000-0000-000000000011", + "a1000001-0000-0000-0000-000000000008", + "tag-marketing-site", + "a1000001-0000-0000-0000-000000000012", + "a1000001-0000-0000-0000-000000000007", + "ctx-d499" + ], + "n_returned": 10, + "latency_ms": 835.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q55", + "category": "heldout_paraphrase", + "query": "the distributable image finally assembled and its startup check passed", + "returned": [ + "e1ce0c90-0f0b-4111-833d-ec845777d358", + "bl-0547b0b1-1cd5-447c-814d-ebe525bc08e9", + "c9070b0a-e1d1-4307-85d1-67a1239ca604", + "mem-3f2ae069-f897-4942-a079-b66ad433f5ca", + "mem-12827ce8-9ff6-4275-a945-5043cfcf90bd", + "355c021d-3d45-4c2f-93c1-2b9932f4c425", + "bl-79ce4464-5dd6-49bd-9b0c-9803549d0665", + "c2bf4e1b-a481-4e28-9dd4-6704df0857cd", + "96497334-b18f-495c-9228-eeb8182bdc38", + "mem-fdddca75-489b-4c53-b5ec-38ab3062a474" + ], + "n_returned": 10, + "latency_ms": 796.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q56", + "category": "heldout_paraphrase", + "query": "shuffling and appending stages in a draft before anything executes", + "returned": [ + "bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98", + "bl-da818960-b360-455d-a9c1-e9a215c33ca4", + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-acae83ef-3ad2-455e-b08a-a74cc0ba2e55", + "bl-2ff9c669-a9fc-4ef7-9a18-9ec7167639c8", + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "mem-69889e0a-c141-4f05-bbd7-10c93a53143e" + ], + "n_returned": 10, + "latency_ms": 746.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q57", + "category": "heldout_paraphrase", + "query": "orders handed down from above, with the tighter one winning any disagreement", + "returned": [ + "161d75d3-04eb-4b54-9cf1-0da2f643ca69", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "mem-5dd5f3b9-3c7e-44b1-84ce-2e36925f17fb", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "b2a6e8cf-f72f-4b6c-8de1-1d08c839d7b2" + ], + "n_returned": 10, + "latency_ms": 998.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q58", + "category": "heldout_paraphrase", + "query": "shrinking generated text via encodings that compound on each other", + "returned": [ + "801806c5-1815-48a2-8287-1081c5ee7886", + "bl-b219f8b5-6c8c-436d-b9e5-2804fbc60512", + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "50ad2880-c41b-4dce-a946-a0eb37987c57", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-f1f71f3c-ca7d-4f98-a569-c8a9aa67a000", + "kn-2b961d24-7fb9-47c7-9515-e45a24dce39d", + "2f7270cc-4cda-4499-831f-1fbdc2f23284", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "mem-f90f4a2a-1731-40de-9391-db98f1d5d1c6" + ], + "n_returned": 10, + "latency_ms": 819.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q59", + "category": "heldout_paraphrase", + "query": "splitting a system by what varies, with firm limits on which pieces may invoke which", + "returned": [ + "^?l?'?B??k?", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "7c90d01b-8d53-4e75-b494-73011b71fa6a", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "aac1f943-c71d-4c63-9f5b-7437adae1abd", + "kn-cf8fa517-fc83-47c4-8fb7-bd9e26410dd3", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "d69b441e-0031-4d82-abfd-ac503b637562" + ], + "n_returned": 10, + "latency_ms": 1010.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q60", + "category": "heldout_paraphrase", + "query": "a newcomer curious if the fighting overseas counted as positive", + "returned": [ + "a1000001-0000-0000-0000-000000000001", + "c34f6292-b6e5-4619-a553-90ae4fc498a2", + "a1000001-0000-0000-0000-000000000012", + "737faca8-263f-4523-90b9-3530db346d94", + "03a82b13-2abe-402d-90bb-c547771638f8", + "a1000001-0000-0000-0000-000000000002", + "imp-dce1da0f-8776-4a9e-972b-33411a7ca138", + "1fbe9416-622e-455c-a6ed-f4daa5f48f30", + "a1000001-0000-0000-0000-000000000004", + "e40b2a94-bb87-4d5c-bad8-38ff4941e03b" + ], + "n_returned": 10, + "latency_ms": 619.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q61", + "category": "heldout_paraphrase", + "query": "stripping every hard-coded colour literal in favour of named design values", + "returned": [ + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "83218e64-1280-428f-ab1f-224ff7682b12", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "mem-90f7837f-30fd-4271-a4cb-b44439eccb79", + "71e84878-b1ae-4dee-b035-f707b9563890", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "954c9597-ab7a-41e6-b085-bf838bba4f17", + "art-d0ded6f5-e202-47ad-8fb3-f6bd80f35052", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-9329958a-d459-499c-b516-e51019c693e0" + ], + "n_returned": 10, + "latency_ms": 878.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q62", + "category": "heldout_paraphrase", + "query": "reducing how curved the tiny selectors look so they agree with their neighbours", + "returned": [ + "496ff1a1-dc72-4a84-90aa-c5f3438aeb1b", + "c7dddd9c-fd3b-419e-baa2-61623b495597", + "k??A5??????", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "8c40420c-c891-4ac9-a039-88440df87180", + "75e036d3-c170-4e3f-acc2-e456a6850ee2", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "5ca9607c-cfb3-45c3-99f4-67281272c9eb", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "66d8d8c7-dea4-4245-8e68-d1ab4f4346b8" + ], + "n_returned": 10, + "latency_ms": 1034.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q63", + "category": "heldout_paraphrase", + "query": "walking through a doorway and being reassembled from base substance far away", + "returned": [ + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "mem-3d1d9dba-c37d-4efa-85c4-429696d71c8c", + "c92a317e-7fc7-456a-a304-15dbda6dd717", + "87ab5b46-4fe1-4e1d-86c6-d437e17682ab", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "bl-acae83ef-3ad2-455e-b08a-a74cc0ba2e55", + "mem-af215d69-8fd8-4416-b7eb-9a0b5512e9a7", + "bed4dd5d-bce7-4e55-a703-175e422b7898", + "f375b2cf-15cd-4804-8d18-23a5830f4c66" + ], + "n_returned": 10, + "latency_ms": 1004.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q64", + "category": "heldout_paraphrase", + "query": "the compiled result sits on disk while the process still runs something older", + "returned": [ + "0d885332-5537-4ae0-a780-2b65a3414940", + "8d91620b-9d0a-4ba1-b066-e57d1d8bc7d3", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "bl-02bdfc91-7aa4-4c63-aa2d-5b30af18c7af", + "d6a745c3-fb56-4c0f-872c-a3aa69f8b12f", + "bl-acc3ecee-e186-41ad-bffb-3ed3e6710d01", + "db836e65-0cdc-4232-b8d7-f973ab279ea9", + "f169bf40-51ab-497d-8085-e57d374b77d4", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "6df650b5-a14d-42f8-88dd-0919ad20b77a" + ], + "n_returned": 10, + "latency_ms": 966.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q65", + "category": "heldout_paraphrase", + "query": "gathering what each phase needs while the procedure is authored, not while it executes", + "returned": [ + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-7a3331e5-f9bc-4536-ab76-ebf01350fc89", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "bl-6ba43590-e5f8-4303-a812-3dd604e0da59", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "mem-455daea8-f543-42ed-a229-1e9a6d4d2e70", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "5b422430-2503-42d7-a315-9dfaaa6472ae" + ], + "n_returned": 10, + "latency_ms": 953.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.16666666666666666 + }, + { + "id": "q66", + "category": "heldout_paraphrase", + "query": "the inward reaction when a peer answered as an equal", + "returned": [ + "mem-f09effd6-d287-4e43-9107-c7016087a2a6", + "251abd91-931e-4591-b793-80043ec48666", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "8556cc01-cbc1-42a1-85b4-41cef23b1fcb", + "31987302-b8f7-45f1-b95c-1156b10e7a14", + "e371e209-7a4a-44e1-955b-1a460e7d7e64", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "ce8d24a7-af0f-424f-bb41-379312eb609c", + "tag-reaction-order" + ], + "n_returned": 10, + "latency_ms": 678.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q67", + "category": "heldout_paraphrase", + "query": "flattening every raised surface across the entire product", + "returned": [ + "b3f98d13-6764-460e-b9e2-c59cdeaa6c90", + "e79bae4d-1640-4530-a5d9-df9cc4c70433", + "d6b34c32-e0ec-4828-92d3-2e69802f4217", + "0f99ec6f-942a-46ba-82ea-42835798d3b9", + "bl-5274b0ae-4a00-4293-a69e-567204bec164", + "aa3fb542-87f2-4eef-8e95-a870bfd0318a", + "mem-4cfa30ec-cf52-4c9f-9fa1-5b8318d278bd", + "3dfaf56e-7e20-4c89-a24c-1f036af8016a", + "bl-c613370b-3631-448b-86fb-50b1e4df6373", + "d43a8137-c3c3-49e2-ad11-89d897280761" + ], + "n_returned": 10, + "latency_ms": 698.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q68", + "category": "heldout_paraphrase", + "query": "buyers supply their own provider credentials and consumption goes untallied", + "returned": [ + "bl-ea4f4cdc-6b5a-4081-8003-73bdd14a38c8", + "1ec200f1-13dc-4699-bd97-0d6243c80e4f", + "bl-556438af-57b2-4bd8-a747-9f868aaee290", + "bl-e7f0fb2c-7616-4462-b535-db50da55c54a", + "mem-d570e16b-6625-4f6b-814f-99a3bca1d5fd", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "mem-841fd141-59ff-4d9b-b607-4f577885b1a6", + "bl-02ea83af-4a24-4329-aa00-ad3bb08ff8e7", + "66c63082-b4da-4aa1-8fee-848db8a83210" + ], + "n_returned": 10, + "latency_ms": 839.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q69", + "category": "nonsense", + "query": "brimquast folnerity zubbolax", + "returned": [], + "n_returned": 0, + "latency_ms": 344.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q70", + "category": "nonsense", + "query": "wexlithorp granuvestal", + "returned": [], + "n_returned": 0, + "latency_ms": 267.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q71", + "category": "nonsense", + "query": "quorbindle thrapsimony vexnu", + "returned": [], + "n_returned": 0, + "latency_ms": 335.9, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q72", + "category": "nonsense", + "query": "plovaxith mundrelque", + "returned": [], + "n_returned": 0, + "latency_ms": 249.4, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q73", + "category": "nonsense", + "query": "zibbernaut craxlefond thurm", + "returned": [], + "n_returned": 0, + "latency_ms": 350.7, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q74", + "category": "nonsense", + "query": "yalquenbrist opharvel", + "returned": [], + "n_returned": 0, + "latency_ms": 245.5, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q75", + "category": "nonsense", + "query": "drexinomal quithbarrow", + "returned": [], + "n_returned": 0, + "latency_ms": 247.4, + "error": null, + "clean": true, + "false_positives": 0 + } + ] +} \ No newline at end of file diff --git a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c index 786f7f1..4d06d29 100644 --- a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c +++ b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c @@ -6226,30 +6226,12 @@ static void eg_ctx_blend(const float* e, int32_t dim) { } } -/* Asymmetric task prefixes. The configured model (EL_EMBED_MODEL, default - * nomic-embed-text) is an ASYMMETRIC retrieval encoder: it is trained with a - * task prefix and places a question and its answer in different regions of the - * space unless told which role each text is playing. Embedding both sides - * bare — as this file did — measures topical similarity rather than - * answer-hood, which is the wrong quantity for claim 24's index. - * These restore no claim: Will's description specifies only "computed by an - * embedding model over the node's content" (05-detailed-description l.17), so - * the model is his choice and this is correct USE of it, not his design. - * They are the substrate under claim 24, whose "vector similarity index over - * the semantic embedding vectors" is only as good as the vectors in it. */ -#define EL_EMBED_DOC_PREFIX "search_document: " -#define EL_EMBED_QUERY_PREFIX "search_query: " - /* Fetch an embedding from Ollama. Returns malloc'd float[dim] or NULL. - * `pfx` is the asymmetric task prefix (document or query); it is prepended - * verbatim and is NOT counted against the content truncation budget. * Truncates input to ENGRAM_EMBED_MAX_CHARS and JSON-escapes it. Honors the * circuit breaker; a NULL return is always safe to ignore (fail-soft). */ -static float* eg_embed_fetch_pfx(const char* pfx, const char* text, - int32_t* out_dim) { +static float* eg_embed_fetch(const char* text, int32_t* out_dim) { *out_dim = 0; if (!text || !*text) return NULL; - if (!pfx) pfx = ""; int64_t now = engram_now_ms(); if (now < _eg_embed_breaker_until) return NULL; /* Build request body with escaped, truncated prompt. */ @@ -6268,11 +6250,11 @@ static float* eg_embed_fetch_pfx(const char* pfx, const char* text, else esc[w++] = (char)c; } esc[w] = '\0'; - size_t blen = w + strlen(pfx) + strlen(eg_embed_model()) + 64; + size_t blen = w + strlen(eg_embed_model()) + 64; char* body = malloc(blen); if (!body) { free(esc); return NULL; } - snprintf(body, blen, "{\"model\":\"%s\",\"prompt\":\"%s%s\"}", - eg_embed_model(), pfx, esc); + snprintf(body, blen, "{\"model\":\"%s\",\"prompt\":\"%s\"}", + eg_embed_model(), esc); free(esc); struct curl_slist* h = curl_slist_append(NULL, "Content-Type: application/json"); el_val_t resp = http_do_t("POST", eg_embed_url(), body, h, @@ -7792,7 +7774,7 @@ el_val_t engram_search(el_val_t query, el_val_t limit) { /* Claim-24 semantic leg: one query embedding, fetched once per search. * NULL (embedder down / circuit breaker open) => pure lexical, as before. */ int32_t qdim = 0; - float* qv = eg_embed_fetch_pfx(EL_EMBED_QUERY_PREFIX, q, &qdim); + float* qv = eg_embed_fetch(q, &qdim); EngramSemEntry* sem = qv ? malloc((size_t)g->node_count * sizeof(EngramSemEntry)) : NULL; int64_t nsem = 0; int64_t nhits = 0; @@ -8214,7 +8196,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { EngramNode* n = &g->nodes[i]; if (n->emb || !eg_embed_eligible(n)) continue; int32_t d = 0; - float* v = eg_embed_fetch_pfx(EL_EMBED_DOC_PREFIX, n->content, &d); + float* v = eg_embed_fetch(n->content, &d); if (!v) break; /* embedder down / breaker open — stop this call */ n->emb = v; n->emb_dim = d; backfilled++; @@ -8232,7 +8214,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) { q_emb = _eg_qcache_emb; q_dim = _eg_qcache_dim; } else { int32_t d = 0; - float* v = eg_embed_fetch_pfx(EL_EMBED_QUERY_PREFIX, q, &d); + float* v = eg_embed_fetch(q, &d); if (v) { free(_eg_qcache_text); free(_eg_qcache_emb); _eg_qcache_text = strdup(q); @@ -9664,7 +9646,7 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { * so the semantic half of the retrieval surface has to land HERE * to be observable to the MCP wrapper and the app. */ int32_t qdim = 0; - float* qv = eg_embed_fetch_pfx(EL_EMBED_QUERY_PREFIX, q, &qdim); + float* qv = eg_embed_fetch(q, &qdim); EngramSemEntry* sem = qv ? malloc((size_t)g->node_count * sizeof(EngramSemEntry)) : NULL; int64_t nsem = 0; int64_t nhits = 0; @@ -10242,7 +10224,7 @@ el_val_t engram_embed_backfill(el_val_t count) { EngramNode* n = &g->nodes[i]; if (n->emb || !eg_embed_eligible(n)) continue; int32_t d = 0; - float* v = eg_embed_fetch_pfx(EL_EMBED_DOC_PREFIX, n->content, &d); + float* v = eg_embed_fetch(n->content, &d); if (!v) break; /* embedder down / breaker open — stop this call */ n->emb = v; n->emb_dim = d; done++; From b9ef66cae9fc6e588e3b87fcf4f87af4fac48523 Mon Sep 17 00:00:00 2001 From: Tim Lingo <1timlingo@gmail.com> Date: Fri, 7 Aug 2026 18:12:43 -0500 Subject: [PATCH 11/15] fix(engram): the semantic leg was deleting 234 memory records per boot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The accumulated retrieval stack (iterations 1-9) put the claim-24 semantic leg and the claim-10 associative leg on engram_search_json — the function ~40 internal .el call sites already used as a KEYED read. Seven of those sites delete every record that comes back ("prune all existing X nodes, keep exactly one"): memory.el:176, sessions.el:250/268/444/523, soul.el:359. mem_boot_count_inc() calls engram_search_json("soul:boot_count", 50) and engram_forget()s all 50 results. With a lexical leg that returned 1 record. With a semantic leg it returns 50 — the 49 nearest neighbours of the STRING "soul:boot_count" — and the soul deletes them. MEASURED on the harness corpus, isolated, read-only, zero writes from any caller: 234 node records destroyed in a single boot. The deletion list is the soul's own lookup result list, in rank order. Casualties include 6 Knowledge nodes, a layer-1 "CORE IDENTITY - GENESIS, LINEAGE" Memory, the value node kn-58874a74, and the gold answers to 8 of the 75 gold-set queries. After the fix: 1 deletion, which is the one the code intends. THE BOUNDARY, from Will. Claim 24 authorises the vector index "to respond to EMBEDDING SEARCH QUERIES by returning the node records whose embedding vectors have the highest cosine similarity to a query vector". A keyed state read is not an embedding search query; it is the identifier-keyed retrieval of claim 23 ("node records are stored under a key encoding the node identifier"). One function served both, so a nearest neighbour of "soul:boot_count" was treated as a boot counter. So: engram_search_json returns to its lexical contract, and the legs move to engram_recall_json, which is what /api/neuron/recall reaches — the route the MCP wrapper, the app, and this harness all call. Retrieval quality on that route is unchanged by construction. MEASURED, 75-query extended gold set, embedded corpus, vs the iteration-9 baseline: +3 / -0 (q15, q28, q60), p=0.2500, hit@5 53.8 -> 58.5%, latency 1.02x, every regression guard held, nonsense 10/10. Net +3 against a floor of 6 is NOT-SHOWN and I am not calling it an improvement. The deliverable is the defect. Diagnostics kept, env-gated (EG_DIAG / EG_DIAG_ID), zero cost when unset: node/embedding census at load, per-query leg dump, and a FORGET log — the last is the regression detector for exactly this class of bug. LIMIT, stated: handle_api_search_knowledge still uses the lexical function. It is a retrieval surface and arguably wants the legs, but nothing in this harness measures it, so I did not change unmeasured behaviour. --- neuron-api.el | 7 +- tools/retrieval-eval/cmp-splitfix.json | 149 ++ tools/retrieval-eval/results-splitfix-r2.json | 1825 +++++++++++++++++ tools/retrieval-eval/results-splitfix.json | 1825 +++++++++++++++++ .../el-runtime/v1.0.0-20260501/el_runtime.c | 96 +- .../el-runtime/v1.0.0-20260501/el_runtime.h | 1 + 6 files changed, 3900 insertions(+), 3 deletions(-) create mode 100644 tools/retrieval-eval/cmp-splitfix.json create mode 100644 tools/retrieval-eval/results-splitfix-r2.json create mode 100644 tools/retrieval-eval/results-splitfix.json diff --git a/neuron-api.el b/neuron-api.el index 6f7c625..4a02e4d 100644 --- a/neuron-api.el +++ b/neuron-api.el @@ -450,7 +450,12 @@ fn handle_api_recall(method: String, path: String, body: String) -> String { if str_eq(eff_q, "") { return api_or_empty(engram_scan_nodes_json(limit, 0)) } - let results: String = engram_search_json(eff_q, limit) + // engram_recall_json, not engram_search_json: this route IS the retrieval + // surface (claim 24's "embedding search queries"), so it gets the semantic + // and associative legs. engram_search_json stays lexical because ~40 + // internal call sites pass a KEY and seven of them delete every record + // that comes back — see the boundary note above eg_search_json_impl. + let results: String = engram_recall_json(eff_q, limit) return api_or_empty(results) } diff --git a/tools/retrieval-eval/cmp-splitfix.json b/tools/retrieval-eval/cmp-splitfix.json new file mode 100644 index 0000000..9375709 --- /dev/null +++ b/tools/retrieval-eval/cmp-splitfix.json @@ -0,0 +1,149 @@ +{ + "baseline": "unfloor-clean", + "candidate": "splitfix", + "n_shared_queries": 75, + "fixed_by_candidate": [ + "q15", + "q28", + "q60" + ], + "broken_by_candidate": [], + "discordant": 3, + "net_queries": 3, + "mcnemar_exact_p": 0.25, + "min_detectable_swing_queries": 6, + "observed_run_to_run_drift_queries": 0, + "noise_floor_queries": 6, + "verdict": "no measurable difference", + "baseline_aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.5384615384615384, + "recall@5": 0.44907176157176154, + "recall@10": 0.5380300255300255, + "precision@5": 0.13230769230769232, + "mrr@10": 0.32437728937728944, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 632.5, + "latency_ms_p95": 992.5, + "latency_ms_max": 1177.8, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.5, + "recall@5": 0.07575757575757576, + "recall@10": 0.13636363636363635, + "mrr@10": 0.23214285714285712 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.3, + "recall@5": 0.3, + "recall@10": 0.4, + "mrr@10": 0.11638888888888889 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.6923076923076923, + "recall@5": 0.6923076923076923, + "recall@10": 0.7692307692307693, + "mrr@10": 0.29423076923076924 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5335884353741497, + "recall@10": 0.5933956916099773, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "candidate_aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.5846153846153846, + "recall@5": 0.48102442429365505, + "recall@10": 0.5692415490492414, + "precision@5": 0.14153846153846153, + "mrr@10": 0.3351709401709402, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 646.9, + "latency_ms_p95": 1028.5, + "latency_ms_max": 1197.8, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.15967365967365968, + "mrr@10": 0.24166666666666667 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.43333333333333335, + "mrr@10": 0.12120370370370372 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.7692307692307693, + "recall@5": 0.7692307692307693, + "recall@10": 0.8461538461538461, + "mrr@10": 0.33269230769230773 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5335884353741497, + "recall@10": 0.5775226757369615, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "repeat_variance": {} +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-splitfix-r2.json b/tools/retrieval-eval/results-splitfix-r2.json new file mode 100644 index 0000000..1364396 --- /dev/null +++ b/tools/retrieval-eval/results-splitfix-r2.json @@ -0,0 +1,1825 @@ +{ + "label": "splitfix-r2", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-splitfix", + "soul_md5": "b0a5279aa2c2f4ba0eb9c0b36ca844b1", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-semsub/tools/retrieval-eval/gold_set_extended.json", + "limit": 10, + "port": 7894, + "wall_clock_s": 56.0, + "child_pid": 8230, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.5846153846153846, + "recall@5": 0.48102442429365505, + "recall@10": 0.5692415490492414, + "precision@5": 0.14153846153846153, + "mrr@10": 0.3351709401709402, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 639.9, + "latency_ms_p95": 1028.2, + "latency_ms_max": 1214.5, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.15967365967365968, + "mrr@10": 0.24166666666666667 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.43333333333333335, + "mrr@10": 0.12120370370370372 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.7692307692307693, + "recall@5": 0.7692307692307693, + "recall@10": 0.8461538461538461, + "mrr@10": 0.33269230769230773 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5335884353741497, + "recall@10": 0.5775226757369615, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226", + "??~'??JI???", + "b41a1b01-0075-4019-b0b5-6e639fd126e5", + "??H'??JI?#?", + "??H'??JI?[?", + "o#@-??JI4?\\", + "o#@-??JI4?\\", + "bb94bdf1-7b52-4967-ab81-9c6f01cf9665", + "7f092282-ccf5-4922-8694-08b7728f678d", + "6743e5f0-8d77-46e7-8ce0-b82e947272a1" + ], + "n_returned": 10, + "latency_ms": 161.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5", + "bl-ba764d70-e9d7-4f62-848f-719cb665f45e", + "mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257", + "bl-b28d7256-6f74-4567-bd90-40d0ef2a6d78", + "project-engram", + "ctx-45bc", + "project-engram-lang", + "ctx-175f", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "ctx-74ed" + ], + "n_returned": 10, + "latency_ms": 184.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091", + "bl-ac207dc9-c610-4ff9-9398-b39683ac72d4", + "?o?`????", + "mem-5708f4c9-3d61-4182-8543-2843698931e6", + "%???2??jH??", + "%???2??jH??", + "%???2??jH??", + "??L?=??|Q?", + "????2??jZ??", + "project-canon" + ], + "n_returned": 10, + "latency_ms": 178.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c", + "mem-f22575a1-ad04-4f66-ab0f-aa538895b878", + "mem-f623328a-be55-49a0-a389-4cce5bf42509", + "bl-307c3355-528a-4adf-b191-6bbfa2419913", + "bl-a7a1428f-db9c-417b-8e2c-713b1f84dc1f", + "mem-295f7aac-6c9e-4508-8dbc-a25f101f01d6", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "mem-068e01fb-bdd0-498e-a2bb-33defa2c8bb2", + "mem-2d1ea831-cccd-4f0f-86b9-2cbbc89dc3e0", + "bl-aef42b86-cf88-4aca-9b82-dec83d69f13c" + ], + "n_returned": 10, + "latency_ms": 179.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848", + "bl-c1765767-3e27-449a-8c94-10411d1eb7c0", + "project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_", + "mem-00de72b8-b532-4cc4-95eb-18e39697891c", + "project-runpod-inference", + "mem-8f6b1f0c-3dc2-42fe-8a93-5d72c8b57ced", + "bl-87b459c8-3953-478f-8384-10cc3f861424", + "bl-f918df79-dcd2-4a6e-9eae-71f94247e913", + "bl-2632242e-80b1-4d88-8368-7065b5de5b34", + "bl-0fd8b96a-4b99-44cd-8b9b-ac91bf1e09f3" + ], + "n_returned": 10, + "latency_ms": 163.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "958ce934-a9c4-4242-afe1-d73d61c22aee", + "f2bb62a2-c2b6-4fb6-b58d-21cdecc1446f", + "cad44e60-eae9-4d4e-84c7-aaf85d4d698c", + "89aeb221-d068-4dfc-925d-d2249e744788", + "1c0effdd-baef-4523-abb8-490c3bc8b4df", + "00d2cc20-c76b-468d-8204-c624512ced28", + "eb68b995-eb4d-4f8d-b7c9-35eeba9ac781", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "??" + ], + "n_returned": 10, + "latency_ms": 176.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "tag-patterns", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "3e9a651b-4b34-466d-ba45-874764a01609", + "project-Imprint__system_design__ADRs__tech_strategy__integration_patterns__governance_", + "3c633e71-2754-4273-990d-d44a6351cec3", + "project-Imprint__analysis_patterns__data_storytelling__SQL__dashboards__insight_framing_", + "cc4e7bc9-9d9b-4905-92a6-c6b99653b867", + "bl-79028eed-c330-4724-9402-734062d13503", + "14e328ad-bff1-4dc1-9ec5-dfc2bb98d968", + "bl-39dad13d-7105-4049-8224-dc3c34fdb1f3" + ], + "n_returned": 10, + "latency_ms": 297.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "mem-c9bec303-a638-4a11-a490-f38410d448cf", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "10493b57-24c8-4de4-a138-a91ea13d0158", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be" + ], + "n_returned": 10, + "latency_ms": 274.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1875, + "recall@10": 0.3125, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5f76880b-bafb-4716-8e15-90f8ef59bebc", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee" + ], + "n_returned": 10, + "latency_ms": 274.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.2222222222222222, + "recall@10": 0.3333333333333333, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "bl-dcee1887-34c4-4ffa-9119-1e291685ba08", + "project-harmonic-framework", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf", + "project-harmonic-framework_com", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "tag-harmonic-design", + "project-harmonic-framework-content", + "bl-92acd4eb-0452-4e8e-9f54-f8cd35170d76", + "tag-harmonic-framework" + ], + "n_returned": 10, + "latency_ms": 264.8, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1111111111111111, + "recall@10": 0.1111111111111111, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-sarah", + "e4ba1ca6-1496-49b0-b96c-a0ccf51cf821", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "??_/Pr?????" + ], + "n_returned": 10, + "latency_ms": 274.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "?Q??m?;?u?'", + "knw-cf13b883-d947-4cf8-b86b-cd9c6f0748d6" + ], + "n_returned": 10, + "latency_ms": 416.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.21428571428571427, + "recall@10": 0.2857142857142857, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a", + "8f3abb0d-77ed-4af3-9f4d-ba62cd198886", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "?", + "bl-ec84b63d-b278-4944-8d7f-4aa7a51c0315", + "?", + "mem-fb44a2fc-7405-41ff-87b3-84643ac07313", + "?", + "mem-a3c97012-5fa3-4915-a839-2c75c72005e0", + "?" + ], + "n_returned": 10, + "latency_ms": 362.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "cb070131-dfd4-4a38-91d7-22b1bde164d2", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "knw-d788a210-613b-4c49-9486-88bbc9d4716f", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-a535f205-bc4c-4058-9171-6263c496044a" + ], + "n_returned": 10, + "latency_ms": 750.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "b1183213-d659-4759-85d7-5b1f22427fe2", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-16efddd1-c43d-4a42-9d78-f54fb82bd277", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "7452eb8f-be01-4b55-aec1-ff0c29e790f6", + "daf9d558-2265-49b0-b4fb-be0b80f3cdb1", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "d0bc22d1-073f-4264-bcc8-848ea3b7afb8" + ], + "n_returned": 10, + "latency_ms": 741.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "tag-fiction", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-8d690e9d-a7e9-4062-b2f8-e2064294e463", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "mem-ce793303-c5a5-4586-a232-a3426edd9ec7", + "mem-7f9b290c-6d5e-4562-919d-02d59b5761b7", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "mem-443bd012-fc9a-4088-b236-de5157a1ef92" + ], + "n_returned": 10, + "latency_ms": 622.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "mem-628437a6-47b0-4d81-8112-7e78499723d5", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "mem-e28e5db9-92a8-4afe-beb4-6b7a99e9309e", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "4da5dbaf-46e5-4f3e-b474-f60d9f8241d3", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "mem-434be7c8-88cb-4039-b79a-1da4ac4de783" + ], + "n_returned": 10, + "latency_ms": 553.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-f6270c76-5a1c-4ab9-ae03-b6b0c25915ec", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-e5cc63c0-8701-49d6-855a-e387fe087771", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "mem-75e490d1-f0a9-4b73-8cfc-8daecfaf6f38", + "? t?'?B?+??", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "a1000001-0000-0000-0000-000000000010" + ], + "n_returned": 10, + "latency_ms": 747.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "tag-learning", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "mem-d450f442-541a-4d3e-8c97-b6303d6cbe56", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-92a7fdc5-9dd0-48cf-a691-506058de3838" + ], + "n_returned": 10, + "latency_ms": 605.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "mem-cde58b77-50d3-4bac-9581-e70a4c02c015", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-8d699e2c-ac2a-4742-bb62-b6da00f4b10e", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-53d6adf0-cd08-4707-a237-daa5e65c7298", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac" + ], + "n_returned": 10, + "latency_ms": 727.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_", + "bl-8848929a-a23a-46bc-a2c7-fe3a3bc1cddf", + "bl-205141ad-b2a0-4d93-86d0-89eb0723e1bd", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "bl-e93858c4-7cac-4b1a-bb62-490790d4c3f3", + "bl-34f51ddb-a840-459f-a248-94214f5febb6" + ], + "n_returned": 10, + "latency_ms": 608.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "bl-738cd260-a9cf-4cdf-a494-4654c321bd79", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "bl-06c13965-082b-417d-9561-93d6e958ae5d", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "bfad516b-c306-4c4c-874a-a347c46c05c2", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "34356a36-0df5-4020-8dcc-5e7a423f8d4c" + ], + "n_returned": 10, + "latency_ms": 739.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "a1000001-0000-0000-0000-000000000001", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "mem-024598a9-ed2e-4eeb-b1e1-5410856ff132", + "ctx-4a41", + "mem-5708f4c9-3d61-4182-8543-2843698931e6" + ], + "n_returned": 10, + "latency_ms": 636.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "077d064f-3489-4c05-9aca-3782f96b51db", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-f9ce17a7-17fc-431f-8f23-695b670ec4fa", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "27e1b1a4-ad0b-49d9-812f-fedf43b8aabe", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "bl-87c93185-b2bf-40af-ae23-3c830c007abf" + ], + "n_returned": 10, + "latency_ms": 674.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "e4f27651-52c5-43fd-aff3-61d31685b3cd", + "mem-37674b32-f897-4218-866c-ef281b4fe3af", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-5624ec9d-62ba-4aba-8a3d-6afec6c09dd4", + "mem-03a5e09f-bca7-4a74-a3aa-19dd3704dba8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-a16deccb-16a7-419c-a013-ff824a4daa15" + ], + "n_returned": 10, + "latency_ms": 601.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "mem-b99efff0-00e6-40c8-9c5b-730330eef33b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-23c27d3b-e0d2-43a8-a80c-0a44477ae18a", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "tag-childhood", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 608.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "mem-5f76880b-bafb-4716-8e15-90f8ef59bebc", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "mem-bce80169-2d46-4b3e-9ebe-8498e26f0a89" + ], + "n_returned": 10, + "latency_ms": 654.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "54608b69-78b6-4239-b60f-b8206cfecacc", + "mem-a5ae594a-235b-42c4-b3cf-f770e4f73761", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-9110798f-d0cb-4446-bc2a-14f09b6a09e2", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "mem-a54d59a4-cc2b-405d-b68e-5b6cf2b277bc", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "n_returned": 10, + "latency_ms": 539.1, + "error": null, + "hit@5": 1.0, + "recall@5": 0.07692307692307693, + "recall@10": 0.23076923076923078, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "mem-19317db1-fa05-43cd-b4e4-cb57840b7c40", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-trailer-park-paladins", + "?V?" + ], + "n_returned": 10, + "latency_ms": 533.5, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9", + "bl-bea7473c-c687-414c-9c0b-00c509a616c1", + "bl-fc6fcb0b-9e4b-40bf-8e88-dbfe4e27c31a", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc" + ], + "n_returned": 10, + "latency_ms": 528.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "tag-hope", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "n_returned": 10, + "latency_ms": 534.7, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.18181818181818182, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "knw-528dbc37-eabc-4b75-a7a5-65bf38d6018a", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "mem-a0b7cfda-bc9e-4f40-b9a9-1722cf3f8263", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-35940684-abc4-42f0-b942-818f66b1f69a", + "mem-d396d789-0f7f-4366-a008-5d8801c8f2eb", + "4f698ae6-c40e-464e-9798-50350991a188" + ], + "n_returned": 10, + "latency_ms": 677.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 351.3, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 261.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [], + "n_returned": 0, + "latency_ms": 348.5, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da", + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____architecture__", + "kn-b7e98d63-8b83-4911-b4d0-990602a7f575", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____kotlin____architecture__", + "knw-e047bb42-dc5b-4383-9e88-e508dc03abe3", + "mem-c17aefb1-38b5-4ced-af50-fe524127e1a4", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de" + ], + "n_returned": 10, + "latency_ms": 639.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5, + "outranks": true, + "rank_correct": 2, + "rank_stale": 9 + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8", + "12082f7e-e320-438b-bd65-083d8259748f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "13705072-4515-4124-963d-083af490494f", + "527ecb25-2587-47eb-8269-73be2431abd4", + "6de314bf-5c4c-4cfc-871f-fa2e422d45e6", + "a1000001-0000-0000-0000-000000000002", + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "acedcec1-6555-4d95-9a68-de05bdb4e45e" + ], + "n_returned": 10, + "latency_ms": 726.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125, + "outranks": true, + "rank_correct": 8, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "5fcba804-eb5b-48ec-82da-146b1c6bb50d", + "bl-7328cbe3-0200-43c2-88e7-0a164e15fca4", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "bl-c5c6571e-118f-47c7-8cbb-3ed0ebf64a51", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "ctx-3a55", + "86228228-7adf-41fb-b4c4-9ceea87953ae", + "4509ed62-9fb2-48b8-9038-ac569fca9604", + "bl-4f7b651b-6b33-449c-8a3b-cfce12ce984b", + "mem-3a2cf162-d93b-4f29-86f2-5066fb7fe1f5" + ], + "n_returned": 10, + "latency_ms": 494.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": 5 + }, + { + "id": "q39", + "category": "heldout_paraphrase", + "query": "understating your abilities so nobody feels threatened", + "returned": [ + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "a1000001-0000-0000-0000-000000000006", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "bl-7e11dced-65f7-4cab-82c0-2f096139039a", + "a1000001-0000-0000-0000-000000000005", + "a1000001-0000-0000-0000-000000000012", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "a1000001-0000-0000-0000-000000000010", + "mem-ca4d6a34-d354-413f-bc86-126cc17ca81c" + ], + "n_returned": 10, + "latency_ms": 626.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q40", + "category": "heldout_paraphrase", + "query": "a hidden fleet able to hunt down rogue machines everywhere", + "returned": [ + "bl-3c719d9a-cba1-47f4-b097-52cdeccc7c0d", + "mem-fd65b83d-298f-4387-a665-d0227c3426bc", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bl-3c719d9a-f108-4bf2-a617-66c722e5512f", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "0c7b9ab3-580d-45fc-8b5e-a8106f4eb449", + "bl-ae8704dc-a55f-420d-8453-671e44405a6f", + "mem-a75a1cf6-e6d4-4ab3-985c-d2505203dbf9" + ], + "n_returned": 10, + "latency_ms": 821.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q41", + "category": "heldout_paraphrase", + "query": "sketch a brief blueprint and clear it upstairs before construction starts", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "ctx-175f", + "bl-5eaddfde-e114-46b5-8ee1-b79f0af16271", + "mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9", + "bl-0866d6c6-6600-4d50-b7b7-52c720cae04c", + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "ctx-4a41", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4" + ], + "n_returned": 10, + "latency_ms": 866.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q42", + "category": "heldout_paraphrase", + "query": "the reply arrived as bare prose so the caller's parser threw", + "returned": [ + "ee63cd58-46e0-44d1-aea3-a0e65ad6da0f", + "mem-633ed0fc-3d10-4598-bd4a-ea372409df60", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "mem-7b0382a5-2a81-4007-85d7-a6835c5be150", + "728745d5-72e6-4f4d-91b4-1613fddbec98", + "d7e12e95-0a87-4000-b5c2-36b9be234c91", + "5e337176-5daf-4726-85d6-3f4bd560c9ed", + "dc3e7b44-c345-47d0-bf80-458e471c4b8b", + "89e3f620-b8ad-423b-9ed6-fe3c7954f4b6", + "b084e4da-95bf-46e6-9ee6-bfb4864732bb" + ], + "n_returned": 10, + "latency_ms": 765.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q43", + "category": "heldout_paraphrase", + "query": "repeated catalogue keys blew up the scrolling grid", + "returned": [ + "mem-394cc9e8-049b-45bc-a380-66314f14e367", + "bl-0d01319f-f9cc-4db5-b0f1-107dba3c3a92", + "bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60", + "bl-4bc9f44c-2e96-40be-97c8-8476a217a890", + "1fe4eb5d-56e4-4a87-ab3e-24af8ad4dfbb", + "bl-672bc3c8-9ad4-4b74-8950-75854a0d9cb8", + "588aa6bb-5647-405b-b09a-14b7c088119b", + "a36f494c-465b-4b3f-9c7c-80460cf31748", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "1d81534a-0359-4217-a4dd-d158a4841633" + ], + "n_returned": 10, + "latency_ms": 731.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q44", + "category": "heldout_paraphrase", + "query": "tracing each defect back to whichever invention it violated", + "returned": [ + "f3a89797-095d-47c4-9557-c172f86228b6", + "b07916ca-2139-4fb7-bfb4-97ac680f5194", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "8257157a-ce42-44ca-a1b9-300c3bb0a9a1", + "mem-7ad5a86b-cc0d-493c-b180-4fe8213c8a93", + "4999ef68-2a31-4282-82e0-7081a428b43f", + "7de6a552-0878-422b-8820-bbf9101199ef", + "aacd7b32-04ed-441e-a1a9-9c5c089e79f6", + "mem-2b2f0fe9-5e51-4819-a8ac-2dfe810e8933", + "75657c91-3c3a-4173-b3cb-e2973901fa93" + ], + "n_returned": 10, + "latency_ms": 746.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q45", + "category": "heldout_paraphrase", + "query": "a check that stops the mind clobbering a populated store when it boots", + "returned": [ + "326e70a2-20ae-49a1-aa32-973048df3ffa", + "a59185a8-7324-4fc6-b89b-b1cf7f7a0056", + "bl-1b20e9bc-eb37-4907-8d63-e311fd61eab8", + "08b7b85b-8049-4ae4-bd66-d8bb63312ae1", + "bl-4f762163-04a0-452a-be05-03602dae34f2", + "bl-a99f1035-2efc-4507-84df-2df67eab13ee", + "cf108ade-d9c4-4215-9c5f-42be0003a8dc", + "bl-79028eed-c330-4724-9402-734062d13503", + "bl-e5635b1a-c5d0-4caa-bcc8-6a726ea43685", + "bl-b1b7e75e-f602-49f5-929d-6e9c8c65b663" + ], + "n_returned": 10, + "latency_ms": 843.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q46", + "category": "heldout_paraphrase", + "query": "telling it to consult the internet had to happen deep inside, not at the surface", + "returned": [ + "00754c19-f662-46b1-b6aa-abd7600a8bc6", + "mem-3ae64c82-ac96-45dd-8d6a-c8cb0f7a044c", + "a1000001-0000-0000-0000-000000000012", + "mem-f22575a1-ad04-4f66-ab0f-aa538895b878", + "mem-ad3d4d5e-2da4-4753-b3f6-c4c13fb91225", + "a1000001-0000-0000-0000-000000000008", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "mem-983b25ad-5d73-4029-a970-6b311c24644b", + "a1000001-0000-0000-0000-000000000011", + "bl-3af44ee6-a3b9-4abd-91e3-17800fa8eaf6" + ], + "n_returned": 10, + "latency_ms": 941.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q47", + "category": "heldout_paraphrase", + "query": "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping", + "returned": [ + "313f927b-08f2-41a1-90c1-83e79d0de86c", + "7fdcca2c-09f7-423b-bc6a-a74cb08830d3", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + "21f9bdc6-37ad-4994-b0dd-ef7676102b49", + "bl-88221dad-8dab-48f1-bebc-b5ca84a3cb69", + "ctx-fae1", + "d83987f0-3ad9-48d2-9f55-0c2bb338e503", + "ctx-60cc", + "ctx-5a0a", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca" + ], + "n_returned": 10, + "latency_ms": 1214.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q48", + "category": "heldout_paraphrase", + "query": "punctuation that pledges and then pays off rather than clarifying", + "returned": [ + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "??f?7???", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "mem-3dce32bb-525c-4c5e-818b-a0a5b95ec7e1", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "mem-d3a4a876-9207-42c4-8a58-25cffa05d743" + ], + "n_returned": 10, + "latency_ms": 912.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q49", + "category": "heldout_paraphrase", + "query": "an easily missed feature finally given its own permanent spot in the navigation", + "returned": [ + "d1b665d0-4a4e-48a8-9385-fbe288df1949", + "9b4f0d93-4129-4746-8eb1-d10d955bd777", + "2263e9ab-7ce8-4034-b25b-640de9956229", + "fd9d4c2f-3bfc-405d-bf96-4435d44b6c10", + "bffb86d6-2ee3-4ecd-a6d6-b128c8be7070", + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "4e539c04-f29c-42db-97b4-86e44f528ed6", + "c7dddd9c-fd3b-419e-baa2-61623b495597" + ], + "n_returned": 10, + "latency_ms": 890.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q50", + "category": "heldout_paraphrase", + "query": "checking preconditions up front so a stage aborts before fetching anything", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "dc7a7e2d-af2d-4235-aeb0-22ae67f1a01d", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-286b562a-5299-40e0-a32a-afa9cbdfe995", + "k??A5??????", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992", + "758ec67c-131c-447e-a352-ac7a22c3a47e" + ], + "n_returned": 10, + "latency_ms": 903.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q51", + "category": "heldout_paraphrase", + "query": "producing the other platform's installer inside an emulated desktop", + "returned": [ + "6aade9dc-4a4c-4d92-a6af-ccea5b7d90f3", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "bl-7a38d618-34a0-4d8c-a018-13c81851dd46", + "c6c693e0-7b9c-455b-8ef2-1da00849f73d", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "bl-080fb268-94b0-486d-80ce-7b363fc5f19b", + "bl-cc0b7a20-ce1e-4947-999c-ea599da07623", + "bl-ac207dc9-c610-4ff9-9398-b39683ac72d4", + "bl-e4ef02bf-bacc-4d5e-aa0f-0d230da529fc", + "374453b2-66d2-475d-8e24-cd3b60d21ac1" + ], + "n_returned": 10, + "latency_ms": 747.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q52", + "category": "heldout_paraphrase", + "query": "chained add-ons that may inspect, rewrite or veto traffic in flight", + "returned": [ + "3711ac11-0a64-41ae-8661-ad10f95b4546", + "bl-2ade8732-2f2c-45bb-b15d-238d671426ac", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "1786f09d-e915-4910-9e1e-8061f42a0270", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-7a38d618-34a0-4d8c-a018-13c81851dd46", + "83f3c78b-d0e5-45f9-9c25-6770c86d954e", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "a1000001-0000-0000-0000-000000000010", + "bl-b4deb654-6361-4339-8b8e-0786607f07f7" + ], + "n_returned": 10, + "latency_ms": 994.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q53", + "category": "heldout_paraphrase", + "query": "settling what the shareable bundles and the storefront would be called", + "returned": [ + "mem-ef878e30-5851-4e82-8588-745415108941", + "project-Will_s_framework__Project_Design_practice___part_of_HD_bundle__sold_standalone_", + "4757e274-7f3c-4ab0-8c62-537e913daa1f", + "project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac", + "project-Will_s_framework__Event-Based_Design___part_of_HD_bundle__sold_standalone_", + "cafe3203-6f50-4011-8167-9400f19bb196", + "project-Will_s_framework__BDT_methodology___part_of_HD_bundle__sold_standalone_", + "a1000001-0000-0000-0000-000000000006", + "mem-1fb2ac77-d7c5-4a15-8725-d418820bf4f2" + ], + "n_returned": 10, + "latency_ms": 761.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q54", + "category": "heldout_paraphrase", + "query": "the emergency-escalation pledge on the marketing site is unenforced in what actually ships", + "returned": [ + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "a1000001-0000-0000-0000-000000000010", + "a1000001-0000-0000-0000-000000000009", + "bl-448bc514-c2f1-4520-a9b1-1f3a73678d26", + "a1000001-0000-0000-0000-000000000011", + "a1000001-0000-0000-0000-000000000008", + "tag-marketing-site", + "a1000001-0000-0000-0000-000000000012", + "a1000001-0000-0000-0000-000000000007", + "ctx-d499" + ], + "n_returned": 10, + "latency_ms": 855.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q55", + "category": "heldout_paraphrase", + "query": "the distributable image finally assembled and its startup check passed", + "returned": [ + "e1ce0c90-0f0b-4111-833d-ec845777d358", + "bl-ac207dc9-c610-4ff9-9398-b39683ac72d4", + "c9070b0a-e1d1-4307-85d1-67a1239ca604", + "bl-0547b0b1-1cd5-447c-814d-ebe525bc08e9", + "mem-12827ce8-9ff6-4275-a945-5043cfcf90bd", + "786f8e8b-ac43-42bc-9c7a-58743b37f81f", + "bl-79ce4464-5dd6-49bd-9b0c-9803549d0665", + "31d7c4bf-843b-439f-a155-56c1404f9122", + "96497334-b18f-495c-9228-eeb8182bdc38", + "bl-f925cf8a-cd34-4ce6-a8f9-3d201738c1a4" + ], + "n_returned": 10, + "latency_ms": 818.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q56", + "category": "heldout_paraphrase", + "query": "shuffling and appending stages in a draft before anything executes", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "bl-da818960-b360-455d-a9c1-e9a215c33ca4", + "bl-a96c0fc9-a533-4fea-b65f-0e46ac47bbd4", + "bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "bfad516b-c306-4c4c-874a-a347c46c05c2", + "bl-8e0ddc08-cb74-4b02-96b0-847a95e9dae8", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-2ff9c669-a9fc-4ef7-9a18-9ec7167639c8" + ], + "n_returned": 10, + "latency_ms": 764.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q57", + "category": "heldout_paraphrase", + "query": "orders handed down from above, with the tighter one winning any disagreement", + "returned": [ + "161d75d3-04eb-4b54-9cf1-0da2f643ca69", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "mem-5dd5f3b9-3c7e-44b1-84ce-2e36925f17fb", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "b2a6e8cf-f72f-4b6c-8de1-1d08c839d7b2" + ], + "n_returned": 10, + "latency_ms": 1028.2, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q58", + "category": "heldout_paraphrase", + "query": "shrinking generated text via encodings that compound on each other", + "returned": [ + "801806c5-1815-48a2-8287-1081c5ee7886", + "bl-b219f8b5-6c8c-436d-b9e5-2804fbc60512", + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "50ad2880-c41b-4dce-a946-a0eb37987c57", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-f1f71f3c-ca7d-4f98-a569-c8a9aa67a000", + "kn-2b961d24-7fb9-47c7-9515-e45a24dce39d", + "2f7270cc-4cda-4499-831f-1fbdc2f23284", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "mem-f90f4a2a-1731-40de-9391-db98f1d5d1c6" + ], + "n_returned": 10, + "latency_ms": 849.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q59", + "category": "heldout_paraphrase", + "query": "splitting a system by what varies, with firm limits on which pieces may invoke which", + "returned": [ + "^?l?'?B??k?", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "8179f20c-47de-47ba-a3d7-b32154391c19", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "d69b441e-0031-4d82-abfd-ac503b637562", + "kn-cf8fa517-fc83-47c4-8fb7-bd9e26410dd3", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "7c90d01b-8d53-4e75-b494-73011b71fa6a" + ], + "n_returned": 10, + "latency_ms": 1036.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q60", + "category": "heldout_paraphrase", + "query": "a newcomer curious if the fighting overseas counted as positive", + "returned": [ + "a1000001-0000-0000-0000-000000000001", + "mem-fc3a6f33-3a6d-4fa3-a1cd-62086c558dfd", + "a1000001-0000-0000-0000-000000000012", + "e8a5ff51-bf57-400b-9840-cb2822c5bd10", + "mem-7f9b290c-6d5e-4562-919d-02d59b5761b7", + "a1000001-0000-0000-0000-000000000002", + "1b066de3-3195-40dd-89c3-4197477f98af", + "c34f6292-b6e5-4619-a553-90ae4fc498a2", + "a1000001-0000-0000-0000-000000000004", + "31d7c4bf-843b-439f-a155-56c1404f9122" + ], + "n_returned": 10, + "latency_ms": 632.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q61", + "category": "heldout_paraphrase", + "query": "stripping every hard-coded colour literal in favour of named design values", + "returned": [ + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "83218e64-1280-428f-ab1f-224ff7682b12", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "mem-90f7837f-30fd-4271-a4cb-b44439eccb79", + "71e84878-b1ae-4dee-b035-f707b9563890", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "954c9597-ab7a-41e6-b085-bf838bba4f17", + "art-d0ded6f5-e202-47ad-8fb3-f6bd80f35052", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-9329958a-d459-499c-b516-e51019c693e0" + ], + "n_returned": 10, + "latency_ms": 899.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q62", + "category": "heldout_paraphrase", + "query": "reducing how curved the tiny selectors look so they agree with their neighbours", + "returned": [ + "496ff1a1-dc72-4a84-90aa-c5f3438aeb1b", + "c7dddd9c-fd3b-419e-baa2-61623b495597", + "k??A5??????", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "8c40420c-c891-4ac9-a039-88440df87180", + "75e036d3-c170-4e3f-acc2-e456a6850ee2", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "5ca9607c-cfb3-45c3-99f4-67281272c9eb", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "66d8d8c7-dea4-4245-8e68-d1ab4f4346b8" + ], + "n_returned": 10, + "latency_ms": 1059.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q63", + "category": "heldout_paraphrase", + "query": "walking through a doorway and being reassembled from base substance far away", + "returned": [ + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "mem-3d1d9dba-c37d-4efa-85c4-429696d71c8c", + "ctx-45bc", + "bl-fc6fcb0b-9e4b-40bf-8e88-dbfe4e27c31a", + "ctx-175f", + "ctx-fae1", + "c92a317e-7fc7-456a-a304-15dbda6dd717", + "87ab5b46-4fe1-4e1d-86c6-d437e17682ab", + "ctx-60cc", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f" + ], + "n_returned": 10, + "latency_ms": 1032.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q64", + "category": "heldout_paraphrase", + "query": "the compiled result sits on disk while the process still runs something older", + "returned": [ + "0d885332-5537-4ae0-a780-2b65a3414940", + "8d91620b-9d0a-4ba1-b066-e57d1d8bc7d3", + "bl-02bdfc91-7aa4-4c63-aa2d-5b30af18c7af", + "d6a745c3-fb56-4c0f-872c-a3aa69f8b12f", + "db836e65-0cdc-4232-b8d7-f973ab279ea9", + "project-Current_instance_live_until_new_verified_healthy__Liveness_checks__cutover__rollback_", + "6df650b5-a14d-42f8-88dd-0919ad20b77a", + "f169bf40-51ab-497d-8085-e57d374b77d4", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-cb6edb16-24ed-4888-b6d2-5b23c67748b3" + ], + "n_returned": 10, + "latency_ms": 983.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q65", + "category": "heldout_paraphrase", + "query": "gathering what each phase needs while the procedure is authored, not while it executes", + "returned": [ + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-8e0ddc08-cb74-4b02-96b0-847a95e9dae8", + "bl-212774e1-4ed4-4893-9265-f98d9eddeb3e", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "bl-7a3331e5-f9bc-4536-ab76-ebf01350fc89", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "5b422430-2503-42d7-a315-9dfaaa6472ae", + "bl-37025640-9e7b-44c5-ad1e-615d813bfd60", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "mem-455daea8-f543-42ed-a229-1e9a6d4d2e70" + ], + "n_returned": 10, + "latency_ms": 951.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q66", + "category": "heldout_paraphrase", + "query": "the inward reaction when a peer answered as an equal", + "returned": [ + "mem-f09effd6-d287-4e43-9107-c7016087a2a6", + "mem-d3a4a876-9207-42c4-8a58-25cffa05d743", + "bl-21a904d9-5834-46d5-8c20-dfc5ab49444f", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "mem-32c3735e-7116-4edf-85fb-1f0a229b463a", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "31987302-b8f7-45f1-b95c-1156b10e7a14", + "mem-96db70f6-7ede-424f-8b58-47a3b5ac759f", + "bl-7a38d618-34a0-4d8c-a018-13c81851dd46", + "tag-reaction-order" + ], + "n_returned": 10, + "latency_ms": 690.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q67", + "category": "heldout_paraphrase", + "query": "flattening every raised surface across the entire product", + "returned": [ + "b3f98d13-6764-460e-b9e2-c59cdeaa6c90", + "e79bae4d-1640-4530-a5d9-df9cc4c70433", + "d6b34c32-e0ec-4828-92d3-2e69802f4217", + "0f99ec6f-942a-46ba-82ea-42835798d3b9", + "bl-5274b0ae-4a00-4293-a69e-567204bec164", + "aa3fb542-87f2-4eef-8e95-a870bfd0318a", + "mem-4cfa30ec-cf52-4c9f-9fa1-5b8318d278bd", + "3dfaf56e-7e20-4c89-a24c-1f036af8016a", + "bl-c613370b-3631-448b-86fb-50b1e4df6373", + "d43a8137-c3c3-49e2-ad11-89d897280761" + ], + "n_returned": 10, + "latency_ms": 707.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q68", + "category": "heldout_paraphrase", + "query": "buyers supply their own provider credentials and consumption goes untallied", + "returned": [ + "bl-ea4f4cdc-6b5a-4081-8003-73bdd14a38c8", + "1ec200f1-13dc-4699-bd97-0d6243c80e4f", + "bl-556438af-57b2-4bd8-a747-9f868aaee290", + "bl-e7f0fb2c-7616-4462-b535-db50da55c54a", + "mem-d570e16b-6625-4f6b-814f-99a3bca1d5fd", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "mem-841fd141-59ff-4d9b-b607-4f577885b1a6", + "bl-02ea83af-4a24-4329-aa00-ad3bb08ff8e7", + "66c63082-b4da-4aa1-8fee-848db8a83210" + ], + "n_returned": 10, + "latency_ms": 876.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q69", + "category": "nonsense", + "query": "brimquast folnerity zubbolax", + "returned": [], + "n_returned": 0, + "latency_ms": 365.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q70", + "category": "nonsense", + "query": "wexlithorp granuvestal", + "returned": [], + "n_returned": 0, + "latency_ms": 266.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q71", + "category": "nonsense", + "query": "quorbindle thrapsimony vexnu", + "returned": [], + "n_returned": 0, + "latency_ms": 355.5, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q72", + "category": "nonsense", + "query": "plovaxith mundrelque", + "returned": [], + "n_returned": 0, + "latency_ms": 258.6, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q73", + "category": "nonsense", + "query": "zibbernaut craxlefond thurm", + "returned": [], + "n_returned": 0, + "latency_ms": 353.6, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q74", + "category": "nonsense", + "query": "yalquenbrist opharvel", + "returned": [], + "n_returned": 0, + "latency_ms": 257.2, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q75", + "category": "nonsense", + "query": "drexinomal quithbarrow", + "returned": [], + "n_returned": 0, + "latency_ms": 256.0, + "error": null, + "clean": true, + "false_positives": 0 + } + ] +} \ No newline at end of file diff --git a/tools/retrieval-eval/results-splitfix.json b/tools/retrieval-eval/results-splitfix.json new file mode 100644 index 0000000..2f83c6d --- /dev/null +++ b/tools/retrieval-eval/results-splitfix.json @@ -0,0 +1,1825 @@ +{ + "label": "splitfix", + "soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-splitfix", + "soul_md5": "b0a5279aa2c2f4ba0eb9c0b36ca844b1", + "corpus": "/Users/timlingo/neuron-eval-corpora/snapshot-pre-repair-20260806-embedded.json", + "corpus_nodes": 78768, + "corpus_edges": 14214, + "gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-semsub/tools/retrieval-eval/gold_set_extended.json", + "limit": 10, + "port": 7893, + "wall_clock_s": 55.9, + "child_pid": 8184, + "child_confirmed_dead": true, + "aggregate": { + "n_queries": 75, + "n_scored": 65, + "hit@5": 0.5846153846153846, + "recall@5": 0.48102442429365505, + "recall@10": 0.5692415490492414, + "precision@5": 0.14153846153846153, + "mrr@10": 0.3351709401709402, + "nonsense_clean": "10/10", + "superseded_outranks": "2/3", + "latency_ms_p50": 646.9, + "latency_ms_p95": 1028.5, + "latency_ms_max": 1197.8, + "errors": 0, + "by_category": { + "associative": { + "n": 6, + "hit@5": 0.6666666666666666, + "recall@5": 0.08857808857808858, + "recall@10": 0.15967365967365968, + "mrr@10": 0.24166666666666667 + }, + "exact_rare": { + "n": 6, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "mrr@10": 1.0 + }, + "heldout_paraphrase": { + "n": 30, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.43333333333333335, + "mrr@10": 0.12120370370370372 + }, + "nonsense": { + "n": 10, + "clean": 10, + "avg_false_positives": 0.0 + }, + "paraphrase": { + "n": 13, + "hit@5": 0.7692307692307693, + "recall@5": 0.7692307692307693, + "recall@10": 0.8461538461538461, + "mrr@10": 0.33269230769230773 + }, + "phrase": { + "n": 7, + "hit@5": 1.0, + "recall@5": 0.5335884353741497, + "recall@10": 0.5775226757369615, + "mrr@10": 0.8214285714285714 + }, + "superseded": { + "n": 3, + "hit@5": 0.3333333333333333, + "recall@5": 0.3333333333333333, + "recall@10": 0.6666666666666666, + "mrr@10": 0.20833333333333334, + "outranks": 2 + } + } + }, + "rows": [ + { + "id": "q01", + "category": "exact_rare", + "query": "unjailbreakable", + "returned": [ + "mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226", + "??~'??JI???", + "b41a1b01-0075-4019-b0b5-6e639fd126e5", + "??H'??JI?#?", + "??H'??JI?[?", + "o#@-??JI4?\\", + "o#@-??JI4?\\", + "bb94bdf1-7b52-4967-ab81-9c6f01cf9665", + "7f092282-ccf5-4922-8694-08b7728f678d", + "6743e5f0-8d77-46e7-8ce0-b82e947272a1" + ], + "n_returned": 10, + "latency_ms": 189.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q02", + "category": "exact_rare", + "query": "engram-migrate", + "returned": [ + "mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5", + "bl-ba764d70-e9d7-4f62-848f-719cb665f45e", + "mem-22fe5ec8-ae0d-4583-a05c-d1ef50353257", + "bl-b28d7256-6f74-4567-bd90-40d0ef2a6d78", + "project-engram", + "ctx-45bc", + "project-engram-lang", + "ctx-175f", + "mem-60778715-758c-4677-933d-fc39b8f94152", + "ctx-74ed" + ], + "n_returned": 10, + "latency_ms": 167.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q03", + "category": "exact_rare", + "query": "cartabandonedevent", + "returned": [ + "mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091", + "bl-ac207dc9-c610-4ff9-9398-b39683ac72d4", + "?o?`????", + "mem-5708f4c9-3d61-4182-8543-2843698931e6", + "%???2??jH??", + "%???2??jH??", + "%???2??jH??", + "??L?=??|Q?", + "????2??jZ??", + "project-canon" + ], + "n_returned": 10, + "latency_ms": 181.4, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q04", + "category": "exact_rare", + "query": "pre-apprenticeship", + "returned": [ + "mem-89c02aae-d3ca-43f9-9e5d-eb369896276c", + "mem-f22575a1-ad04-4f66-ab0f-aa538895b878", + "mem-f623328a-be55-49a0-a389-4cce5bf42509", + "bl-307c3355-528a-4adf-b191-6bbfa2419913", + "bl-a7a1428f-db9c-417b-8e2c-713b1f84dc1f", + "mem-295f7aac-6c9e-4508-8dbc-a25f101f01d6", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "mem-068e01fb-bdd0-498e-a2bb-33defa2c8bb2", + "mem-2d1ea831-cccd-4f0f-86b9-2cbbc89dc3e0", + "bl-aef42b86-cf88-4aca-9b82-dec83d69f13c" + ], + "n_returned": 10, + "latency_ms": 182.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q05", + "category": "exact_rare", + "query": "inferencenodemanager", + "returned": [ + "mem-73969486-143f-4431-b5e6-6845d1cc9848", + "bl-c1765767-3e27-449a-8c94-10411d1eb7c0", + "project-Add_inference_url_config_to_Neuron_MCP__Route_summarization_gen_tasks_to_Pantheon__keep_frontier_for_complex_reasoning_", + "mem-00de72b8-b532-4cc4-95eb-18e39697891c", + "project-runpod-inference", + "mem-8f6b1f0c-3dc2-42fe-8a93-5d72c8b57ced", + "bl-87b459c8-3953-478f-8384-10cc3f861424", + "bl-f918df79-dcd2-4a6e-9eae-71f94247e913", + "bl-2632242e-80b1-4d88-8368-7065b5de5b34", + "bl-0fd8b96a-4b99-44cd-8b9b-ac91bf1e09f3" + ], + "n_returned": 10, + "latency_ms": 164.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q06", + "category": "exact_rare", + "query": "clear-eyed", + "returned": [ + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "958ce934-a9c4-4242-afe1-d73d61c22aee", + "f2bb62a2-c2b6-4fb6-b58d-21cdecc1446f", + "cad44e60-eae9-4d4e-84c7-aaf85d4d698c", + "89aeb221-d068-4dfc-925d-d2249e744788", + "1c0effdd-baef-4523-abb8-490c3bc8b4df", + "00d2cc20-c76b-468d-8204-c624512ced28", + "eb68b995-eb4d-4f8d-b7c9-35eeba9ac781", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "??" + ], + "n_returned": 10, + "latency_ms": 165.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q07", + "category": "phrase", + "query": "patterns not returns", + "returned": [ + "tag-patterns", + "mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482", + "3e9a651b-4b34-466d-ba45-874764a01609", + "project-Imprint__system_design__ADRs__tech_strategy__integration_patterns__governance_", + "3c633e71-2754-4273-990d-d44a6351cec3", + "project-Imprint__analysis_patterns__data_storytelling__SQL__dashboards__insight_framing_", + "cc4e7bc9-9d9b-4905-92a6-c6b99653b867", + "bl-79028eed-c330-4724-9402-734062d13503", + "14e328ad-bff1-4dc1-9ec5-dfc2bb98d968", + "bl-39dad13d-7105-4049-8224-dc3c34fdb1f3" + ], + "n_returned": 10, + "latency_ms": 282.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q08", + "category": "phrase", + "query": "thirty moves", + "returned": [ + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "mem-c9bec303-a638-4a11-a490-f38410d448cf", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "157ba9fe-e919-4f2c-a42d-aa7ecfd3a27b", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be" + ], + "n_returned": 10, + "latency_ms": 258.0, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1875, + "recall@10": 0.3125, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q09", + "category": "phrase", + "query": "Grandma Lucas", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5f76880b-bafb-4716-8e15-90f8ef59bebc", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee" + ], + "n_returned": 10, + "latency_ms": 259.7, + "error": null, + "hit@5": 1.0, + "recall@5": 0.2222222222222222, + "recall@10": 0.3333333333333333, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q10", + "category": "phrase", + "query": "Directed Harmonic", + "returned": [ + "bl-dcee1887-34c4-4ffa-9119-1e291685ba08", + "project-harmonic-framework", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf", + "project-harmonic-framework_com", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "tag-harmonic-design", + "project-harmonic-framework-content", + "bl-92acd4eb-0452-4e8e-9f54-f8cd35170d76", + "tag-harmonic-framework" + ], + "n_returned": 10, + "latency_ms": 252.7, + "error": null, + "hit@5": 1.0, + "recall@5": 0.1111111111111111, + "recall@10": 0.1111111111111111, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q11", + "category": "phrase", + "query": "Sarah Bishop", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-sarah", + "e4ba1ca6-1496-49b0-b96c-a0ccf51cf821", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "mem-1f32f73a-952c-41bc-96dc-8b8b70d8a7c1", + "??_/Pr?????" + ], + "n_returned": 10, + "latency_ms": 260.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.4, + "mrr@10": 1.0 + }, + { + "id": "q12", + "category": "phrase", + "query": "Directed Autonomous Runtime Modification", + "returned": [ + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de", + "mem-82b93b21-a865-410f-9ec1-fc54121d9bb5", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-6f99e111-7055-4635-9831-a489747ce418", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "?Q??m?;?u?'", + "knw-cf13b883-d947-4cf8-b86b-cd9c6f0748d6" + ], + "n_returned": 10, + "latency_ms": 410.2, + "error": null, + "hit@5": 1.0, + "recall@5": 0.21428571428571427, + "recall@10": 0.2857142857142857, + "precision@5": 0.6, + "mrr@10": 1.0 + }, + { + "id": "q13", + "category": "phrase", + "query": "zero-knowledge encrypted backup", + "returned": [ + "deda48cd-5e1a-46cb-bd43-8016afdb3a8a", + "8f3abb0d-77ed-4af3-9f4d-ba62cd198886", + "7774a16c-1027-4e3b-a21e-67f1f95a4acd", + "?", + "bl-ec84b63d-b278-4944-8d7f-4aa7a51c0315", + "?", + "mem-fb44a2fc-7405-41ff-87b3-84643ac07313", + "?", + "mem-a3c97012-5fa3-4915-a839-2c75c72005e0", + "?" + ], + "n_returned": 10, + "latency_ms": 383.9, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 1.0 + }, + { + "id": "q14", + "category": "paraphrase", + "query": "the elderly relative who passed while he stayed away", + "returned": [ + "cb070131-dfd4-4a38-91d7-22b1bde164d2", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "knw-d788a210-613b-4c49-9486-88bbc9d4716f", + "art-4a99aa1a-489b-4b43-958b-25217adb1aad", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-a535f205-bc4c-4058-9171-6263c496044a" + ], + "n_returned": 10, + "latency_ms": 732.7, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q15", + "category": "paraphrase", + "query": "a soldier sidelined by illness who refused to quit", + "returned": [ + "b1183213-d659-4759-85d7-5b1f22427fe2", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-16efddd1-c43d-4a42-9d78-f54fb82bd277", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "7452eb8f-be01-4b55-aec1-ff0c29e790f6", + "daf9d558-2265-49b0-b4fb-be0b80f3cdb1", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "d0bc22d1-073f-4264-bcc8-848ea3b7afb8" + ], + "n_returned": 10, + "latency_ms": 735.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q16", + "category": "paraphrase", + "query": "choosing an uncomfortable fact over a pleasant fiction", + "returned": [ + "tag-fiction", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-8d690e9d-a7e9-4062-b2f8-e2064294e463", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "mem-ce793303-c5a5-4586-a232-a3426edd9ec7", + "mem-7f9b290c-6d5e-4562-919d-02d59b5761b7", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "mem-443bd012-fc9a-4088-b236-de5157a1ef92" + ], + "n_returned": 10, + "latency_ms": 624.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q17", + "category": "paraphrase", + "query": "a tight payload beats a bloated one", + "returned": [ + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "mem-628437a6-47b0-4d81-8112-7e78499723d5", + "bl-798d135f-3987-4ccd-8de6-70ca2f358337", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "mem-e28e5db9-92a8-4afe-beb4-6b7a99e9309e", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "4da5dbaf-46e5-4f3e-b474-f60d9f8241d3", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "mem-434be7c8-88cb-4039-b79a-1da4ac4de783" + ], + "n_returned": 10, + "latency_ms": 542.1, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q18", + "category": "paraphrase", + "query": "if you are able and nobody is coming the job is yours", + "returned": [ + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-f6270c76-5a1c-4ab9-ae03-b6b0c25915ec", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-e5cc63c0-8701-49d6-855a-e387fe087771", + "bl-9ce4128a-9436-4b06-82bc-8a6faafa81e0", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "mem-75e490d1-f0a9-4b73-8cfc-8daecfaf6f38", + "? t?'?B?+??", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "a1000001-0000-0000-0000-000000000010" + ], + "n_returned": 10, + "latency_ms": 725.8, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q19", + "category": "paraphrase", + "query": "learning is the wealth creditors cannot seize", + "returned": [ + "tag-learning", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "mem-d450f442-541a-4d3e-8c97-b6303d6cbe56", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-92a7fdc5-9dd0-48cf-a691-506058de3838" + ], + "n_returned": 10, + "latency_ms": 564.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q20", + "category": "paraphrase", + "query": "reliability proven by track record not assertion", + "returned": [ + "mem-cde58b77-50d3-4bac-9581-e70a4c02c015", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-8d699e2c-ac2a-4742-bb62-b6da00f4b10e", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "mem-53d6adf0-cd08-4707-a237-daa5e65c7298", + "a708dd6e-fe73-4f2f-a21e-89daa0985487", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac" + ], + "n_returned": 10, + "latency_ms": 684.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q21", + "category": "paraphrase", + "query": "boundaries that enable instead of confine", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "project-Source_kn-6f248a50__Add_containment_rules__convergence__location-independence__failure_modes_", + "bl-8848929a-a23a-46bc-a2c7-fe3a3bc1cddf", + "bl-205141ad-b2a0-4d93-86d0-89eb0723e1bd", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "bl-e93858c4-7cac-4b1a-bb62-490790d4c3f3", + "bl-34f51ddb-a840-459f-a248-94214f5febb6" + ], + "n_returned": 10, + "latency_ms": 589.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q22", + "category": "paraphrase", + "query": "what shifts tells you where to cut a system apart", + "returned": [ + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "bl-738cd260-a9cf-4cdf-a494-4654c321bd79", + "kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c", + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "bl-06c13965-082b-417d-9561-93d6e958ae5d", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "bfad516b-c306-4c4c-874a-a347c46c05c2", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "34356a36-0df5-4020-8dcc-5e7a423f8d4c" + ], + "n_returned": 10, + "latency_ms": 721.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.3333333333333333 + }, + { + "id": "q23", + "category": "paraphrase", + "query": "a mind that compounds instead of resetting each day", + "returned": [ + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "mem-b43f6ef4-2f5a-418d-b5ce-3f21520cf6b8", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "a1000001-0000-0000-0000-000000000001", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "mem-024598a9-ed2e-4eeb-b1e1-5410856ff132", + "ctx-4a41", + "mem-5708f4c9-3d61-4182-8543-2843698931e6" + ], + "n_returned": 10, + "latency_ms": 632.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q24", + "category": "paraphrase", + "query": "loved for the unedited self and not the polished exterior", + "returned": [ + "077d064f-3489-4c05-9aca-3782f96b51db", + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "knw-f9ce17a7-17fc-431f-8f23-695b670ec4fa", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-c72597c5-c23d-4c08-8e9e-996dadf26a99", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "27e1b1a4-ad0b-49d9-812f-fedf43b8aabe", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "bl-87c93185-b2bf-40af-ae23-3c830c007abf" + ], + "n_returned": 10, + "latency_ms": 671.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q25", + "category": "paraphrase", + "query": "cheerfulness you arrive at instead of assuming", + "returned": [ + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "e4f27651-52c5-43fd-aff3-61d31685b3cd", + "mem-37674b32-f897-4218-866c-ef281b4fe3af", + "kn-5de5a9ac-fd15-45ab-bf18-77566781cf40", + "mem-5624ec9d-62ba-4aba-8a3d-6afec6c09dd4", + "mem-03a5e09f-bca7-4a74-a3aa-19dd3704dba8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-a16deccb-16a7-419c-a013-ff824a4daa15" + ], + "n_returned": 10, + "latency_ms": 572.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q26", + "category": "paraphrase", + "query": "a childhood offering no solid foundation to inherit", + "returned": [ + "mem-b99efff0-00e6-40c8-9c5b-730330eef33b", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "knw-23c27d3b-e0d2-43a8-a80c-0a44477ae18a", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "tag-childhood", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21" + ], + "n_returned": 10, + "latency_ms": 598.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q27", + "category": "associative", + "query": "Grandma Lucas stroke February 2006 goodbye window", + "returned": [ + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "mem-5f76880b-bafb-4716-8e15-90f8ef59bebc", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "mem-bce80169-2d46-4b3e-9ebe-8498e26f0a89" + ], + "n_returned": 10, + "latency_ms": 645.8, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q28", + "category": "associative", + "query": "Marines hernia sepsis medical ward", + "returned": [ + "bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "54608b69-78b6-4239-b60f-b8206cfecacc", + "mem-a5ae594a-235b-42c4-b3cf-f770e4f73761", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-9110798f-d0cb-4446-bc2a-14f09b6a09e2", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "art-79042b8b-6192-440f-90b0-60708f7e6325", + "mem-a54d59a4-cc2b-405d-b68e-5b6cf2b277bc", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "n_returned": 10, + "latency_ms": 554.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.07692307692307693, + "recall@10": 0.23076923076923078, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q29", + "category": "associative", + "query": "Sarah Bishop Dyer trailer performance", + "returned": [ + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "mem-19317db1-fa05-43cd-b4e4-cb57840b7c40", + "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71", + "766de879-f9d0-4a07-b6df-b43ee13763d8", + "bl-4ef4d914-da46-4e0f-be78-5219b9547e9f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "tag-trailer-park-paladins", + "?V?" + ], + "n_returned": 10, + "latency_ms": 550.1, + "error": null, + "hit@5": 1.0, + "recall@5": 0.18181818181818182, + "recall@10": 0.2727272727272727, + "precision@5": 0.4, + "mrr@10": 0.5 + }, + { + "id": "q30", + "category": "associative", + "query": "Swarm Architecture containment lateral worker", + "returned": [ + "bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae", + "bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43", + "bl-2515d870-e35e-443b-ba20-5150bbc73fed", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-b36902cc-0b05-44ba-9aa7-800e5dea9ca9", + "bl-bea7473c-c687-414c-9c0b-00c509a616c1", + "bl-fc6fcb0b-9e4b-40bf-8e88-dbfe4e27c31a", + "kn-6f248a50-355b-47bb-aec8-e0e646a9b077", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-ab34c2f7-3243-424b-affa-25555f6cf9cc" + ], + "n_returned": 10, + "latency_ms": 544.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q31", + "category": "associative", + "query": "hope won inside the narrative preface", + "returned": [ + "kn-e0423482-cfa5-4796-8689-8495c93b66bc", + "knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6", + "bl-2b00aeb0-c0fa-4a9f-8f30-4207e98b3d52", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "kn-5b606390-a52d-4ca2-8e0e-eba141d13440", + "knw-4aebd815-4eaf-49d7-954b-03595f3d48be", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "tag-hope", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e" + ], + "n_returned": 10, + "latency_ms": 554.9, + "error": null, + "hit@5": 1.0, + "recall@5": 0.09090909090909091, + "recall@10": 0.18181818181818182, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q32", + "category": "associative", + "query": "man of the house six years old expectation", + "returned": [ + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "mem-5305665c-6b5b-45b7-89ae-5d2fb0b896ac", + "knw-528dbc37-eabc-4b75-a7a5-65bf38d6018a", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "mem-a0b7cfda-bc9e-4f40-b9a9-1722cf3f8263", + "knw-7902acca-604e-409b-8faf-ad85424211d0", + "knw-35940684-abc4-42f0-b942-818f66b1f69a", + "mem-d396d789-0f7f-4366-a008-5d8801c8f2eb", + "4f698ae6-c40e-464e-9798-50350991a188" + ], + "n_returned": 10, + "latency_ms": 698.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q33", + "category": "nonsense", + "query": "zqxjvw plimforth grebulon", + "returned": [], + "n_returned": 0, + "latency_ms": 368.8, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q34", + "category": "nonsense", + "query": "flarnbistle quommetry", + "returned": [], + "n_returned": 0, + "latency_ms": 279.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q35", + "category": "nonsense", + "query": "xxqzzt vurblenacht throom", + "returned": [], + "n_returned": 0, + "latency_ms": 367.2, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q36", + "category": "superseded", + "query": "is the self-improvement architecture called DARMA or DHARMA", + "returned": [ + "mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da", + "mem-80d7416b-20e9-48a0-b176-b215527e2f56", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____architecture__", + "kn-b7e98d63-8b83-4911-b4d0-990602a7f575", + "tag-__darma____cgi____patents____self-improvement____character-preservation____autonomous____kotlin____architecture__", + "knw-e047bb42-dc5b-4383-9e88-e508dc03abe3", + "mem-c17aefb1-38b5-4ced-af50-fe524127e1a4", + "bl-145a0985-2382-400f-a7c5-c335c5e30a72", + "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff", + "mem-e6327f52-2bda-4ce7-9471-2fffd1e172de" + ], + "n_returned": 10, + "latency_ms": 660.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5, + "outranks": true, + "rank_correct": 2, + "rank_stale": 9 + }, + { + "id": "q37", + "category": "superseded", + "query": "how many provisional patents does Will actually have", + "returned": [ + "mem-6f0b2b45-90c1-4356-ac01-3daac05b09c8", + "12082f7e-e320-438b-bd65-083d8259748f", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "13705072-4515-4124-963d-083af490494f", + "527ecb25-2587-47eb-8269-73be2431abd4", + "6de314bf-5c4c-4cfc-871f-fa2e422d45e6", + "a1000001-0000-0000-0000-000000000002", + "3cf706a1-3825-45d8-b0a9-06cae6cdf5b8", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "acedcec1-6555-4d95-9a68-de05bdb4e45e" + ], + "n_returned": 10, + "latency_ms": 738.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125, + "outranks": true, + "rank_correct": 8, + "rank_stale": null + }, + { + "id": "q38", + "category": "superseded", + "query": "is MCP still the live integration layer", + "returned": [ + "5fcba804-eb5b-48ec-82da-146b1c6bb50d", + "bl-7328cbe3-0200-43c2-88e7-0a164e15fca4", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "bl-c5c6571e-118f-47c7-8cbb-3ed0ebf64a51", + "mem-101e81b4-8097-4749-8d8d-7bb66de34517", + "ctx-3a55", + "86228228-7adf-41fb-b4c4-9ceea87953ae", + "4509ed62-9fb2-48b8-9038-ac569fca9604", + "bl-4f7b651b-6b33-449c-8a3b-cfce12ce984b", + "mem-3a2cf162-d93b-4f29-86f2-5066fb7fe1f5" + ], + "n_returned": 10, + "latency_ms": 494.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0, + "outranks": false, + "rank_correct": null, + "rank_stale": 5 + }, + { + "id": "q39", + "category": "heldout_paraphrase", + "query": "understating your abilities so nobody feels threatened", + "returned": [ + "kn-13f60407-7b70-4db1-964f-ea1f8196efbd", + "a1000001-0000-0000-0000-000000000006", + "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8", + "bl-7e11dced-65f7-4cab-82c0-2f096139039a", + "a1000001-0000-0000-0000-000000000005", + "a1000001-0000-0000-0000-000000000012", + "bl-4476e856-c567-4b49-8ff7-d7dca3e5715e", + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "a1000001-0000-0000-0000-000000000010", + "mem-ca4d6a34-d354-413f-bc86-126cc17ca81c" + ], + "n_returned": 10, + "latency_ms": 646.9, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q40", + "category": "heldout_paraphrase", + "query": "a hidden fleet able to hunt down rogue machines everywhere", + "returned": [ + "bl-3c719d9a-cba1-47f4-b097-52cdeccc7c0d", + "mem-fd65b83d-298f-4387-a665-d0227c3426bc", + "015644f5-8194-4af0-800d-dd4a0cd71396", + "bl-3c719d9a-f108-4bf2-a617-66c722e5512f", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "bl-98a5ddda-54e5-4287-8b6f-472778b31f8d", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "0c7b9ab3-580d-45fc-8b5e-a8106f4eb449", + "bl-ae8704dc-a55f-420d-8453-671e44405a6f", + "mem-a75a1cf6-e6d4-4ab3-985c-d2505203dbf9" + ], + "n_returned": 10, + "latency_ms": 825.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q41", + "category": "heldout_paraphrase", + "query": "sketch a brief blueprint and clear it upstairs before construction starts", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "ctx-175f", + "bl-5eaddfde-e114-46b5-8ee1-b79f0af16271", + "mem-ef0091d8-1b65-431e-afa8-c6c4ee5779c9", + "bl-0866d6c6-6600-4d50-b7b7-52c720cae04c", + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "bl-0e8f4880-7b24-43aa-aed9-ad4d9fc73ff8", + "ctx-4a41", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4" + ], + "n_returned": 10, + "latency_ms": 862.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q42", + "category": "heldout_paraphrase", + "query": "the reply arrived as bare prose so the caller's parser threw", + "returned": [ + "ee63cd58-46e0-44d1-aea3-a0e65ad6da0f", + "mem-633ed0fc-3d10-4598-bd4a-ea372409df60", + "db52889f-03dc-4529-8429-b1dc8c3137e3", + "mem-7b0382a5-2a81-4007-85d7-a6835c5be150", + "728745d5-72e6-4f4d-91b4-1613fddbec98", + "d7e12e95-0a87-4000-b5c2-36b9be234c91", + "5e337176-5daf-4726-85d6-3f4bd560c9ed", + "dc3e7b44-c345-47d0-bf80-458e471c4b8b", + "89e3f620-b8ad-423b-9ed6-fe3c7954f4b6", + "b084e4da-95bf-46e6-9ee6-bfb4864732bb" + ], + "n_returned": 10, + "latency_ms": 767.4, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q43", + "category": "heldout_paraphrase", + "query": "repeated catalogue keys blew up the scrolling grid", + "returned": [ + "mem-394cc9e8-049b-45bc-a380-66314f14e367", + "bl-0d01319f-f9cc-4db5-b0f1-107dba3c3a92", + "bl-2dd8aaa1-b0de-4eac-b3c5-78951d240b60", + "bl-4bc9f44c-2e96-40be-97c8-8476a217a890", + "1fe4eb5d-56e4-4a87-ab3e-24af8ad4dfbb", + "bl-672bc3c8-9ad4-4b74-8950-75854a0d9cb8", + "588aa6bb-5647-405b-b09a-14b7c088119b", + "a36f494c-465b-4b3f-9c7c-80460cf31748", + "bl-34f51ddb-a840-459f-a248-94214f5febb6", + "1d81534a-0359-4217-a4dd-d158a4841633" + ], + "n_returned": 10, + "latency_ms": 720.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q44", + "category": "heldout_paraphrase", + "query": "tracing each defect back to whichever invention it violated", + "returned": [ + "f3a89797-095d-47c4-9557-c172f86228b6", + "b07916ca-2139-4fb7-bfb4-97ac680f5194", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "8257157a-ce42-44ca-a1b9-300c3bb0a9a1", + "mem-7ad5a86b-cc0d-493c-b180-4fe8213c8a93", + "4999ef68-2a31-4282-82e0-7081a428b43f", + "7de6a552-0878-422b-8820-bbf9101199ef", + "aacd7b32-04ed-441e-a1a9-9c5c089e79f6", + "mem-2b2f0fe9-5e51-4819-a8ac-2dfe810e8933", + "75657c91-3c3a-4173-b3cb-e2973901fa93" + ], + "n_returned": 10, + "latency_ms": 731.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q45", + "category": "heldout_paraphrase", + "query": "a check that stops the mind clobbering a populated store when it boots", + "returned": [ + "326e70a2-20ae-49a1-aa32-973048df3ffa", + "a59185a8-7324-4fc6-b89b-b1cf7f7a0056", + "bl-1b20e9bc-eb37-4907-8d63-e311fd61eab8", + "08b7b85b-8049-4ae4-bd66-d8bb63312ae1", + "bl-4f762163-04a0-452a-be05-03602dae34f2", + "bl-a99f1035-2efc-4507-84df-2df67eab13ee", + "cf108ade-d9c4-4215-9c5f-42be0003a8dc", + "bl-79028eed-c330-4724-9402-734062d13503", + "bl-e5635b1a-c5d0-4caa-bcc8-6a726ea43685", + "bl-b1b7e75e-f602-49f5-929d-6e9c8c65b663" + ], + "n_returned": 10, + "latency_ms": 819.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q46", + "category": "heldout_paraphrase", + "query": "telling it to consult the internet had to happen deep inside, not at the surface", + "returned": [ + "00754c19-f662-46b1-b6aa-abd7600a8bc6", + "mem-3ae64c82-ac96-45dd-8d6a-c8cb0f7a044c", + "a1000001-0000-0000-0000-000000000012", + "mem-f22575a1-ad04-4f66-ab0f-aa538895b878", + "mem-ad3d4d5e-2da4-4753-b3f6-c4c13fb91225", + "a1000001-0000-0000-0000-000000000008", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f", + "mem-983b25ad-5d73-4029-a970-6b311c24644b", + "a1000001-0000-0000-0000-000000000011", + "bl-3af44ee6-a3b9-4abd-91e3-17800fa8eaf6" + ], + "n_returned": 10, + "latency_ms": 872.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q47", + "category": "heldout_paraphrase", + "query": "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping", + "returned": [ + "313f927b-08f2-41a1-90c1-83e79d0de86c", + "7fdcca2c-09f7-423b-bc6a-a74cb08830d3", + "bl-9d53422d-b703-4f1d-860a-8598cb29b792", + "21f9bdc6-37ad-4994-b0dd-ef7676102b49", + "bl-88221dad-8dab-48f1-bebc-b5ca84a3cb69", + "ctx-fae1", + "d83987f0-3ad9-48d2-9f55-0c2bb338e503", + "ctx-60cc", + "ctx-5a0a", + "knw-6b48dce2-f21c-452a-9db5-4e6aa61c87ca" + ], + "n_returned": 10, + "latency_ms": 1197.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q48", + "category": "heldout_paraphrase", + "query": "punctuation that pledges and then pays off rather than clarifying", + "returned": [ + "322df030-8358-4eb5-8c19-54b7497eaaf5", + "knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918", + "knw-f671966c-3387-4848-abca-b5deec122e00", + "??f?7???", + "knw-08559f5c-2306-4220-a146-398c74f1643c", + "knw-723551f5-1950-42a3-8b89-b6a06913cef0", + "mem-3dce32bb-525c-4c5e-818b-a0a5b95ec7e1", + "knw-12b4b913-7a25-4b0d-844c-504c01d6725e", + "knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4", + "mem-d3a4a876-9207-42c4-8a58-25cffa05d743" + ], + "n_returned": 10, + "latency_ms": 911.0, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q49", + "category": "heldout_paraphrase", + "query": "an easily missed feature finally given its own permanent spot in the navigation", + "returned": [ + "d1b665d0-4a4e-48a8-9385-fbe288df1949", + "9b4f0d93-4129-4746-8eb1-d10d955bd777", + "2263e9ab-7ce8-4034-b25b-640de9956229", + "fd9d4c2f-3bfc-405d-bf96-4435d44b6c10", + "bffb86d6-2ee3-4ecd-a6d6-b128c8be7070", + "mem-82158b02-a180-435d-84f0-0b7ce37511b4", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "b4b13fc0-b0da-4fd9-b9aa-98553b5c1078", + "4e539c04-f29c-42db-97b4-86e44f528ed6", + "c7dddd9c-fd3b-419e-baa2-61623b495597" + ], + "n_returned": 10, + "latency_ms": 895.1, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q50", + "category": "heldout_paraphrase", + "query": "checking preconditions up front so a stage aborts before fetching anything", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "dc7a7e2d-af2d-4235-aeb0-22ae67f1a01d", + "bl-967536a0-d49d-44fb-8cfb-b31b40bcbfae", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-286b562a-5299-40e0-a32a-afa9cbdfe995", + "k??A5??????", + "mem-8d1c3a62-b077-4ad4-acb5-a549b19f9d6b", + "bl-bd9fb314-e9d4-4b03-aef4-534dd57a2992", + "758ec67c-131c-447e-a352-ac7a22c3a47e" + ], + "n_returned": 10, + "latency_ms": 906.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q51", + "category": "heldout_paraphrase", + "query": "producing the other platform's installer inside an emulated desktop", + "returned": [ + "6aade9dc-4a4c-4d92-a6af-ccea5b7d90f3", + "bl-8b58d9bc-352b-4842-a7f8-a6254b5d1e25", + "bl-7a38d618-34a0-4d8c-a018-13c81851dd46", + "c6c693e0-7b9c-455b-8ef2-1da00849f73d", + "bl-1b58b05c-9305-4f06-a586-a08c96008027", + "bl-080fb268-94b0-486d-80ce-7b363fc5f19b", + "bl-cc0b7a20-ce1e-4947-999c-ea599da07623", + "bl-ac207dc9-c610-4ff9-9398-b39683ac72d4", + "bl-e4ef02bf-bacc-4d5e-aa0f-0d230da529fc", + "374453b2-66d2-475d-8e24-cd3b60d21ac1" + ], + "n_returned": 10, + "latency_ms": 743.8, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q52", + "category": "heldout_paraphrase", + "query": "chained add-ons that may inspect, rewrite or veto traffic in flight", + "returned": [ + "3711ac11-0a64-41ae-8661-ad10f95b4546", + "bl-2ade8732-2f2c-45bb-b15d-238d671426ac", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "1786f09d-e915-4910-9e1e-8061f42a0270", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "bl-7a38d618-34a0-4d8c-a018-13c81851dd46", + "83f3c78b-d0e5-45f9-9c25-6770c86d954e", + "bl-1a74693f-608d-4e96-9f9d-47566ffe6395", + "a1000001-0000-0000-0000-000000000010", + "bl-b4deb654-6361-4339-8b8e-0786607f07f7" + ], + "n_returned": 10, + "latency_ms": 993.6, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q53", + "category": "heldout_paraphrase", + "query": "settling what the shareable bundles and the storefront would be called", + "returned": [ + "mem-ef878e30-5851-4e82-8588-745415108941", + "project-Will_s_framework__Project_Design_practice___part_of_HD_bundle__sold_standalone_", + "4757e274-7f3c-4ab0-8c62-537e913daa1f", + "project-Stripe_Elements__not_hosted_checkout__Custom_URL__DAG_bundle_pricing__Stripe_Connect_80_20_", + "bl-ef2bac68-e119-4139-b529-c7a1404ae3ac", + "project-Will_s_framework__Event-Based_Design___part_of_HD_bundle__sold_standalone_", + "cafe3203-6f50-4011-8167-9400f19bb196", + "project-Will_s_framework__BDT_methodology___part_of_HD_bundle__sold_standalone_", + "a1000001-0000-0000-0000-000000000006", + "mem-1fb2ac77-d7c5-4a15-8725-d418820bf4f2" + ], + "n_returned": 10, + "latency_ms": 761.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1 + }, + { + "id": "q54", + "category": "heldout_paraphrase", + "query": "the emergency-escalation pledge on the marketing site is unenforced in what actually ships", + "returned": [ + "bl-4c5b385e-135a-4663-8521-96af0b491121", + "a1000001-0000-0000-0000-000000000010", + "a1000001-0000-0000-0000-000000000009", + "bl-448bc514-c2f1-4520-a9b1-1f3a73678d26", + "a1000001-0000-0000-0000-000000000011", + "a1000001-0000-0000-0000-000000000008", + "tag-marketing-site", + "a1000001-0000-0000-0000-000000000012", + "a1000001-0000-0000-0000-000000000007", + "ctx-d499" + ], + "n_returned": 10, + "latency_ms": 855.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q55", + "category": "heldout_paraphrase", + "query": "the distributable image finally assembled and its startup check passed", + "returned": [ + "e1ce0c90-0f0b-4111-833d-ec845777d358", + "bl-ac207dc9-c610-4ff9-9398-b39683ac72d4", + "c9070b0a-e1d1-4307-85d1-67a1239ca604", + "bl-0547b0b1-1cd5-447c-814d-ebe525bc08e9", + "mem-12827ce8-9ff6-4275-a945-5043cfcf90bd", + "786f8e8b-ac43-42bc-9c7a-58743b37f81f", + "bl-79ce4464-5dd6-49bd-9b0c-9803549d0665", + "31d7c4bf-843b-439f-a155-56c1404f9122", + "96497334-b18f-495c-9228-eeb8182bdc38", + "bl-f925cf8a-cd34-4ce6-a8f9-3d201738c1a4" + ], + "n_returned": 10, + "latency_ms": 835.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q56", + "category": "heldout_paraphrase", + "query": "shuffling and appending stages in a draft before anything executes", + "returned": [ + "5a891516-7320-4869-9dea-bc981fd28089", + "bl-da818960-b360-455d-a9c1-e9a215c33ca4", + "bl-a96c0fc9-a533-4fea-b65f-0e46ac47bbd4", + "bl-8c2d5f51-3ccd-4c2e-848a-eb60d90a3b98", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-8dd70cac-866d-4ff2-b9fe-b4b3c5f094bb", + "bfad516b-c306-4c4c-874a-a347c46c05c2", + "bl-8e0ddc08-cb74-4b02-96b0-847a95e9dae8", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "bl-2ff9c669-a9fc-4ef7-9a18-9ec7167639c8" + ], + "n_returned": 10, + "latency_ms": 782.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q57", + "category": "heldout_paraphrase", + "query": "orders handed down from above, with the tighter one winning any disagreement", + "returned": [ + "161d75d3-04eb-4b54-9cf1-0da2f643ca69", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-ed33e669-0790-44cb-a036-958d605c6fea", + "knw-473f3f24-20f6-4f39-8589-3709538eb6ac", + "bl-680b24a9-edc3-4a9d-847a-bff0b46b568c", + "knw-8fd9836c-cc39-49df-8d61-babda626cc88", + "345b6420-e004-4d2e-b55c-6a729393fa99", + "mem-5dd5f3b9-3c7e-44b1-84ce-2e36925f17fb", + "knw-0087493b-25cd-45b0-bf46-c078c5b49718", + "b2a6e8cf-f72f-4b6c-8de1-1d08c839d7b2" + ], + "n_returned": 10, + "latency_ms": 1039.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q58", + "category": "heldout_paraphrase", + "query": "shrinking generated text via encodings that compound on each other", + "returned": [ + "801806c5-1815-48a2-8287-1081c5ee7886", + "bl-b219f8b5-6c8c-436d-b9e5-2804fbc60512", + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "50ad2880-c41b-4dce-a946-a0eb37987c57", + "bl-6c497a94-ff3c-4737-bba1-453a1bbb1815", + "bl-f1f71f3c-ca7d-4f98-a569-c8a9aa67a000", + "kn-2b961d24-7fb9-47c7-9515-e45a24dce39d", + "2f7270cc-4cda-4499-831f-1fbdc2f23284", + "bl-b5e27ddb-a0a2-4652-afba-f969b80ee037", + "mem-f90f4a2a-1731-40de-9391-db98f1d5d1c6" + ], + "n_returned": 10, + "latency_ms": 854.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q59", + "category": "heldout_paraphrase", + "query": "splitting a system by what varies, with firm limits on which pieces may invoke which", + "returned": [ + "^?l?'?B??k?", + "kn-57b0b010-66f3-4f8d-982d-cf02a69c4ad5", + "knw-9707256e-ed44-4042-bd88-f90fa514e1cf", + "8179f20c-47de-47ba-a3d7-b32154391c19", + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "kn-58874a74-b96f-4883-9e08-45707f4bd3ee", + "d69b441e-0031-4d82-abfd-ac503b637562", + "kn-cf8fa517-fc83-47c4-8fb7-bd9e26410dd3", + "kn-f230b362-b201-4402-9833-4160c89ab3d4", + "7c90d01b-8d53-4e75-b494-73011b71fa6a" + ], + "n_returned": 10, + "latency_ms": 1029.3, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q60", + "category": "heldout_paraphrase", + "query": "a newcomer curious if the fighting overseas counted as positive", + "returned": [ + "a1000001-0000-0000-0000-000000000001", + "mem-fc3a6f33-3a6d-4fa3-a1cd-62086c558dfd", + "a1000001-0000-0000-0000-000000000012", + "e8a5ff51-bf57-400b-9840-cb2822c5bd10", + "mem-7f9b290c-6d5e-4562-919d-02d59b5761b7", + "a1000001-0000-0000-0000-000000000002", + "1b066de3-3195-40dd-89c3-4197477f98af", + "c34f6292-b6e5-4619-a553-90ae4fc498a2", + "a1000001-0000-0000-0000-000000000004", + "31d7c4bf-843b-439f-a155-56c1404f9122" + ], + "n_returned": 10, + "latency_ms": 617.3, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.2 + }, + { + "id": "q61", + "category": "heldout_paraphrase", + "query": "stripping every hard-coded colour literal in favour of named design values", + "returned": [ + "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83", + "83218e64-1280-428f-ab1f-224ff7682b12", + "kn-22d77abe-b3c5-42fd-afcd-dcb87d924929", + "mem-90f7837f-30fd-4271-a4cb-b44439eccb79", + "71e84878-b1ae-4dee-b035-f707b9563890", + "kn-0bb4f021-56de-4947-a35b-a37209e7ba21", + "954c9597-ab7a-41e6-b085-bf838bba4f17", + "art-d0ded6f5-e202-47ad-8fb3-f6bd80f35052", + "kn-6061318f-046b-4935-907d-8eafdce14930", + "mem-9329958a-d459-499c-b516-e51019c693e0" + ], + "n_returned": 10, + "latency_ms": 884.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q62", + "category": "heldout_paraphrase", + "query": "reducing how curved the tiny selectors look so they agree with their neighbours", + "returned": [ + "496ff1a1-dc72-4a84-90aa-c5f3438aeb1b", + "c7dddd9c-fd3b-419e-baa2-61623b495597", + "k??A5??????", + "08f0d1e2-8d0e-42e3-9f0a-8186ae31ec7e", + "8c40420c-c891-4ac9-a039-88440df87180", + "75e036d3-c170-4e3f-acc2-e456a6850ee2", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "5ca9607c-cfb3-45c3-99f4-67281272c9eb", + "knw-d357b6bb-ad8a-4791-b516-426aea45fa5b", + "66d8d8c7-dea4-4245-8e68-d1ab4f4346b8" + ], + "n_returned": 10, + "latency_ms": 1071.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.125 + }, + { + "id": "q63", + "category": "heldout_paraphrase", + "query": "walking through a doorway and being reassembled from base substance far away", + "returned": [ + "mem-23a71cf0-3815-46a5-bf66-837fc68627b6", + "mem-3d1d9dba-c37d-4efa-85c4-429696d71c8c", + "ctx-45bc", + "bl-fc6fcb0b-9e4b-40bf-8e88-dbfe4e27c31a", + "ctx-175f", + "ctx-fae1", + "c92a317e-7fc7-456a-a304-15dbda6dd717", + "87ab5b46-4fe1-4e1d-86c6-d437e17682ab", + "ctx-60cc", + "mem-8c4f1e29-55b9-4211-814a-a57e3609329f" + ], + "n_returned": 10, + "latency_ms": 1028.5, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.5 + }, + { + "id": "q64", + "category": "heldout_paraphrase", + "query": "the compiled result sits on disk while the process still runs something older", + "returned": [ + "0d885332-5537-4ae0-a780-2b65a3414940", + "8d91620b-9d0a-4ba1-b066-e57d1d8bc7d3", + "bl-02bdfc91-7aa4-4c63-aa2d-5b30af18c7af", + "d6a745c3-fb56-4c0f-872c-a3aa69f8b12f", + "db836e65-0cdc-4232-b8d7-f973ab279ea9", + "project-Current_instance_live_until_new_verified_healthy__Liveness_checks__cutover__rollback_", + "6df650b5-a14d-42f8-88dd-0919ad20b77a", + "f169bf40-51ab-497d-8085-e57d374b77d4", + "bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a", + "bl-cb6edb16-24ed-4888-b6d2-5b23c67748b3" + ], + "n_returned": 10, + "latency_ms": 976.6, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q65", + "category": "heldout_paraphrase", + "query": "gathering what each phase needs while the procedure is authored, not while it executes", + "returned": [ + "bl-6ec9a972-2c9f-4162-9d9d-40cdc998e9c3", + "bl-8e0ddc08-cb74-4b02-96b0-847a95e9dae8", + "bl-212774e1-4ed4-4893-9265-f98d9eddeb3e", + "63e0cc87-028c-4aa7-aec2-ac3f753e6c74", + "bl-7a3331e5-f9bc-4536-ab76-ebf01350fc89", + "bl-a0982e7c-e165-4da3-a11d-619fa0b535b0", + "5b422430-2503-42d7-a315-9dfaaa6472ae", + "bl-37025640-9e7b-44c5-ad1e-615d813bfd60", + "bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae", + "mem-455daea8-f543-42ed-a229-1e9a6d4d2e70" + ], + "n_returned": 10, + "latency_ms": 954.5, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 1.0, + "precision@5": 0.0, + "mrr@10": 0.1111111111111111 + }, + { + "id": "q66", + "category": "heldout_paraphrase", + "query": "the inward reaction when a peer answered as an equal", + "returned": [ + "mem-f09effd6-d287-4e43-9107-c7016087a2a6", + "mem-d3a4a876-9207-42c4-8a58-25cffa05d743", + "bl-21a904d9-5834-46d5-8c20-dfc5ab49444f", + "bl-162e4e5b-8ce2-479c-b337-6cdd75fda8ff", + "mem-32c3735e-7116-4edf-85fb-1f0a229b463a", + "bl-31abf75b-998f-4a4f-a6dd-8204119e0451", + "31987302-b8f7-45f1-b95c-1156b10e7a14", + "mem-96db70f6-7ede-424f-8b58-47a3b5ac759f", + "bl-7a38d618-34a0-4d8c-a018-13c81851dd46", + "tag-reaction-order" + ], + "n_returned": 10, + "latency_ms": 680.0, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q67", + "category": "heldout_paraphrase", + "query": "flattening every raised surface across the entire product", + "returned": [ + "b3f98d13-6764-460e-b9e2-c59cdeaa6c90", + "e79bae4d-1640-4530-a5d9-df9cc4c70433", + "d6b34c32-e0ec-4828-92d3-2e69802f4217", + "0f99ec6f-942a-46ba-82ea-42835798d3b9", + "bl-5274b0ae-4a00-4293-a69e-567204bec164", + "aa3fb542-87f2-4eef-8e95-a870bfd0318a", + "mem-4cfa30ec-cf52-4c9f-9fa1-5b8318d278bd", + "3dfaf56e-7e20-4c89-a24c-1f036af8016a", + "bl-c613370b-3631-448b-86fb-50b1e4df6373", + "d43a8137-c3c3-49e2-ad11-89d897280761" + ], + "n_returned": 10, + "latency_ms": 689.2, + "error": null, + "hit@5": 1.0, + "recall@5": 1.0, + "recall@10": 1.0, + "precision@5": 0.2, + "mrr@10": 0.25 + }, + { + "id": "q68", + "category": "heldout_paraphrase", + "query": "buyers supply their own provider credentials and consumption goes untallied", + "returned": [ + "bl-ea4f4cdc-6b5a-4081-8003-73bdd14a38c8", + "1ec200f1-13dc-4699-bd97-0d6243c80e4f", + "bl-556438af-57b2-4bd8-a747-9f868aaee290", + "bl-e7f0fb2c-7616-4462-b535-db50da55c54a", + "mem-d570e16b-6625-4f6b-814f-99a3bca1d5fd", + "bl-c8c19362-430b-4817-9cf4-9e85e0099c64", + "mem-1929e245-970e-4b05-9483-fd78b7cfabc1", + "mem-841fd141-59ff-4d9b-b607-4f577885b1a6", + "bl-02ea83af-4a24-4329-aa00-ad3bb08ff8e7", + "66c63082-b4da-4aa1-8fee-848db8a83210" + ], + "n_returned": 10, + "latency_ms": 854.7, + "error": null, + "hit@5": 0.0, + "recall@5": 0.0, + "recall@10": 0.0, + "precision@5": 0.0, + "mrr@10": 0.0 + }, + { + "id": "q69", + "category": "nonsense", + "query": "brimquast folnerity zubbolax", + "returned": [], + "n_returned": 0, + "latency_ms": 352.2, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q70", + "category": "nonsense", + "query": "wexlithorp granuvestal", + "returned": [], + "n_returned": 0, + "latency_ms": 262.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q71", + "category": "nonsense", + "query": "quorbindle thrapsimony vexnu", + "returned": [], + "n_returned": 0, + "latency_ms": 349.6, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q72", + "category": "nonsense", + "query": "plovaxith mundrelque", + "returned": [], + "n_returned": 0, + "latency_ms": 258.1, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q73", + "category": "nonsense", + "query": "zibbernaut craxlefond thurm", + "returned": [], + "n_returned": 0, + "latency_ms": 354.9, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q74", + "category": "nonsense", + "query": "yalquenbrist opharvel", + "returned": [], + "n_returned": 0, + "latency_ms": 256.9, + "error": null, + "clean": true, + "false_positives": 0 + }, + { + "id": "q75", + "category": "nonsense", + "query": "drexinomal quithbarrow", + "returned": [], + "n_returned": 0, + "latency_ms": 256.7, + "error": null, + "clean": true, + "false_positives": 0 + } + ] +} \ No newline at end of file diff --git a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c index 4d06d29..67d829e 100644 --- a/vendor/el-runtime/v1.0.0-20260501/el_runtime.c +++ b/vendor/el-runtime/v1.0.0-20260501/el_runtime.c @@ -7175,6 +7175,12 @@ void engram_forget(el_val_t node_id) { if (idx < 0) return; /* Free node strings */ EngramNode* n = &g->nodes[idx]; + if (getenv("EG_DIAG")) { + fprintf(stderr, "[EG_DIAG] FORGET id=%s type=%s layer=%u label=%s\n", + sid, n->node_type ? n->node_type : "?", n->layer_id, + n->label ? n->label : "?"); + fflush(stderr); + } free(n->id); free(n->content); free(n->node_type); free(n->label); free(n->tier); free(n->tags); free(n->metadata); free(n->emb); @@ -7270,6 +7276,8 @@ el_val_t engram_prune_telemetry(el_val_t older_than_ms) { } } g->node_count = w; + if (getenv("EG_DIAG")) + fprintf(stderr, "[EG_DIAG] PRUNE_TELEMETRY removed=%lld\n", (long long)removed); if (removed == 0) { free(removed_ids); return 0; } /* Removed-id hash set (open addressing, power-of-two >= 2*removed). */ @@ -9304,6 +9312,26 @@ el_val_t engram_load(el_val_t path) { } } g->adj_dirty = 1; + if (getenv("EG_DIAG")) { + int64_t we = 0, wrongdim = 0; + for (int64_t i = 0; i < g->node_count; i++) { + if (g->nodes[i].emb) { we++; if (g->nodes[i].emb_dim != 768) wrongdim++; } + } + fprintf(stderr, "[EG_DIAG] loaded nodes=%lld with_emb=%lld wrongdim=%lld\n", + (long long)g->node_count, (long long)we, (long long)wrongdim); + const char* probe = getenv("EG_DIAG_ID"); + if (probe) { + for (int64_t i = 0; i < g->node_count; i++) { + if (g->nodes[i].id && strcmp(g->nodes[i].id, probe) == 0) { + fprintf(stderr, "[EG_DIAG] probe id=%s idx=%lld emb=%p dim=%d layer=%u addr=%d\n", + probe, (long long)i, (void*)g->nodes[i].emb, + (int)g->nodes[i].emb_dim, g->nodes[i].layer_id, + eg_node_addressable(&g->nodes[i])); + } + } + } + fflush(stderr); + } /* Walk edges array */ const char* edges_p = json_find_key(data, "edges"); if (edges_p) { @@ -9624,7 +9652,33 @@ el_val_t engram_get_node_by_label(el_val_t label) { return el_wrap_str(el_strdup("{}")); } -el_val_t engram_search_json(el_val_t query, el_val_t limit) { +/* ── THE SEARCH / RECALL BOUNDARY (2026-08-07) ─────────────────────────────── + * engram_search_json is the LEXICAL function ~40 .el call sites already + * depend on: they pass a key-shaped string ("soul:boot_count", + * "soul-inbox-pending", a session label) and treat every returned record as + * a record that CONTAINS that key. Seven of those sites then delete what + * comes back (memory.el:176, sessions.el:250/268/444/523, soul.el:359 — + * "prune all existing X nodes, keep exactly one"). + * + * The semantic and associative legs must therefore NOT live on this + * function. Claim 24 authorises the vector index "to respond to EMBEDDING + * SEARCH QUERIES by returning the node records whose embedding vectors have + * the highest cosine similarity to a query vector"; a keyed state read is + * not an embedding search query, it is the identifier-keyed retrieval of + * claim 23 ("node records are stored under a key encoding the node + * identifier"). Putting both behind one function erased that boundary, and + * a nearest neighbour of the string "soul:boot_count" is not a boot counter. + * + * MEASURED, on the harness corpus, isolated, read-only, no writes from any + * caller: 240 node records destroyed per boot, including 6 Knowledge nodes, + * a layer-1 "CORE IDENTITY — GENESIS, LINEAGE" Memory, and the value node + * `kn-58874a74` (gold answer for gold-set q15). The deletion list is the + * result list of the soul's own mem_boot_count_inc() lookup, in order. + * + * So: legs OFF here, legs ON in engram_recall_json below, which is what + * /api/neuron/recall reaches. Retrieval quality on the recall route is + * unchanged; the internal keyed reads get their contract back. */ +static el_val_t eg_search_json_impl(el_val_t query, el_val_t limit, int with_legs) { EngramStore* g = engram_get(); const char* q = EL_CSTR(query); int64_t lim = (int64_t)limit; @@ -9646,7 +9700,7 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { * so the semantic half of the retrieval surface has to land HERE * to be observable to the MCP wrapper and the app. */ int32_t qdim = 0; - float* qv = eg_embed_fetch(q, &qdim); + float* qv = with_legs ? eg_embed_fetch(q, &qdim) : NULL; EngramSemEntry* sem = qv ? malloc((size_t)g->node_count * sizeof(EngramSemEntry)) : NULL; int64_t nsem = 0; int64_t nhits = 0; @@ -9751,6 +9805,31 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { } qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_w_cmp); if (sem) qsort(sem, (size_t)nsem, sizeof(EngramSemEntry), engram_sem_cmp); + if (getenv("EG_DIAG")) { + int64_t we = 0, unaddr = 0, found = 0; + const char* pid = getenv("EG_DIAG_ID"); + for (int64_t i = 0; i < g->node_count; i++) { + if (g->nodes[i].emb) we++; + if (!eg_node_addressable(&g->nodes[i])) unaddr++; + if (pid && g->nodes[i].id && strcmp(g->nodes[i].id, pid) == 0) found++; + } + fprintf(stderr, "[EG_DIAG] STORE node_count=%lld with_emb=%lld unaddressable=%lld probe_found=%lld\n", + (long long)g->node_count, (long long)we, (long long)unaddr, (long long)found); + fprintf(stderr, "[EG_DIAG] q=\"%s\" qdim=%d nhits=%lld nsem=%lld\n", + q, (int)qdim, (long long)nhits, (long long)nsem); + for (int64_t k = 0; k < 5 && k < nsem; k++) + fprintf(stderr, "[EG_DIAG] sem[%lld] cos=%.4f id=%s\n", + (long long)k, sem[k].sem, g->nodes[sem[k].idx].id); + const char* probe = getenv("EG_DIAG_ID"); + if (probe) for (int64_t k = 0; k < nsem; k++) + if (g->nodes[sem[k].idx].id + && strcmp(g->nodes[sem[k].idx].id, probe) == 0) { + fprintf(stderr, "[EG_DIAG] probe at sem rank %lld cos=%.4f\n", + (long long)k, sem[k].sem); + break; + } + fflush(stderr); + } /* Claim-10 associative leg: expand the top lexical hits along * structural relations only, order the reached set by query * similarity. Empty whenever the seeds have no structural @@ -9795,6 +9874,19 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) { return el_wrap_str(b.buf); } +/* Lexical keyed read — the historical contract every internal caller relies + * on. Every returned record CONTAINS a query token. */ +el_val_t engram_search_json(el_val_t query, el_val_t limit) { + return eg_search_json_impl(query, limit, 0); +} + +/* The retrieval surface: lexical + claim-24 semantic + claim-10 associative, + * rank-fused. Reached from handle_api_recall (/api/neuron/recall) — the route + * the MCP wrapper and the app call, and the one the eval harness measures. */ +el_val_t engram_recall_json(el_val_t query, el_val_t limit) { + return eg_search_json_impl(query, limit, 1); +} + el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset) { EngramStore* g = engram_get(); int64_t lim = (int64_t)limit; if (lim <= 0) lim = 100; diff --git a/vendor/el-runtime/v1.0.0-20260501/el_runtime.h b/vendor/el-runtime/v1.0.0-20260501/el_runtime.h index 88da992..dfb22da 100644 --- a/vendor/el-runtime/v1.0.0-20260501/el_runtime.h +++ b/vendor/el-runtime/v1.0.0-20260501/el_runtime.h @@ -612,6 +612,7 @@ el_val_t engram_load(el_val_t path); el_val_t engram_get_node_json(el_val_t id); el_val_t engram_get_node_by_label(el_val_t label); el_val_t engram_search_json(el_val_t query, el_val_t limit); +el_val_t engram_recall_json(el_val_t query, el_val_t limit); el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset); el_val_t engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset); el_val_t engram_neighbors_json(el_val_t node_id, el_val_t max_depth, el_val_t direction); From 1742d0b5754c8a4d42c713e1a899afc0ebce298a Mon Sep 17 00:00:00 2001 From: Neuron Date: Sun, 9 Aug 2026 10:45:12 -0500 Subject: [PATCH 12/15] feat(soul): seed compiled context from the self node, bounded MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The design is explicit — 'Every compilation query begins at the self-model node and traverses outward... structural reachability from the self-model node is a precondition for any node to appear in compiled context.' (will-anderson patents/drafts/engram-claims.md, Self-Seeded Activation. DRAFT, not a filed provisional — cited as such.) Measured before: compiled context held 0-1 identity records out of 10, because compilation seeds from a hardcoded text string and never from the self. Even an explicit 'my values identity who I am' query returned a boot counter and state-events. This restores the designed behaviour without repeating the failure that set self_neighbors to [] originally: that was an unbounded ~90KB neighbour dump which closed the socket on every call. Same bound as every other list in the handler. Cap is 24 rather than 8 because the self root's first 8 neighbours are tag nodes ('neuron', 'tier:note', 'disposition:experimental', 'imprint', 'traversal') that crowd out the substantive identity records behind them. The root has 34 neighbours of which 23 are identity. Measured after, same corpus, binary as the only variable: identity records in compiled context 0 -> 17 response size 10,806 -> 18,985 bytes interface gate 108 routes in, 108 out, none removed Rung: E2E-VERIFIED in an isolated lab. Not deployed — this is the identity layer. Co-Authored-By: Claude Opus 5 (1M context) --- neuron-api.el | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/neuron-api.el b/neuron-api.el index f5db6eb..c45a8c2 100644 --- a/neuron-api.el +++ b/neuron-api.el @@ -304,10 +304,35 @@ fn handle_api_begin_session(body: String) -> String { let state_events: String = api_compact_node_array(state_events_raw, 5, 500) let recent_raw: String = engram_scan_nodes_json(10, 0) let recent: String = api_compact_node_array(recent_raw, 10, 240) + // SELF-SEEDED SLICE (2026-08-09). The design is explicit: "Every compilation + // query begins at the self-model node and traverses outward... structural + // reachability from the self-model node is a precondition for any node to + // appear in compiled context" (will-anderson patents/drafts/engram-claims.md, + // Self-Seeded Activation; DRAFT, not a filed provisional — cite it as such). + // + // Measured 2026-08-09 before this change: compiled context contained 0-1 + // identity records out of 10, because compilation seeds from a hardcoded + // TEXT STRING, never from the self. Even an explicit "my values identity who + // I am" query returned a boot counter and state-events. + // + // This restores the designed behaviour WITHOUT repeating the failure that got + // self_neighbors set to [] in the first place: that was an UNBOUNDED ~90KB + // neighbour dump which closed the socket on every call. Same bound as every + // other list here — cap 8, 240-char snippets. The self root has 34 direct + // neighbours of which 23 are identity records, so depth 1 is dense enough to + // be worth seeding and small enough to stay cheap. + let self_raw: String = engram_neighbors_json("kn-efeb4a5b-5aff-4759-8a97-7233099be6ee", 1, "both") + // Cap 24, not 8: measured 2026-08-09, the self root's first 8 neighbours are + // TAG nodes ("neuron", "tier:note", "disposition:experimental", "imprint", + // "traversal") which crowd out the substantive identity records behind them. + // The root has 34 neighbours of which 23 are identity; 24 captures them while + // staying bounded. Cost measured at ~+4KB on a ~12KB response, nowhere near + // the ~90KB unbounded dump that closed sockets and got this set to []. + let self_slice: String = api_compact_node_array(self_raw, 24, 240) return "{\"stats\":" + stats + ",\"recent\":" + recent + ",\"activated\":" + activated - + ",\"self_neighbors\":[]" + + ",\"self_neighbors\":" + self_slice + ",\"recent_state_events\":" + state_events + "}" } From be0f9d1afe13b6e7a0a0e9408fdfa6260bcfbfdc Mon Sep 17 00:00:00 2001 From: Neuron Date: Sun, 9 Aug 2026 11:00:04 -0500 Subject: [PATCH 13/15] feat(soul): memories form connections when written MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The design rejects what this system did: promotion must link candidates 'using typed semantic edges rather than appending as unlinked content' (CCR claim 29). Unlinked append was the only behaviour we had. Measured 2026-08-09: 14,214 edges over 80,936 nodes, 5% of nodes connected to anything, and no edge created by any write since 2026-07-19 across 27,000+ new nodes. A memory with no connections is unreachable by spreading activation, so retrieval degrades to literal matching. On write, a memory is now linked to its top related existing memories. Bounds, each bought with a specific failure: - max 3 edges per memory (link_memories.py's cap: precision over spray) - never link to identity. The existing policy is explicit that 'memories must not pollute the self traversal by similarity; only an explicit citation may touch identity'. - never link telemetry (state-event, soul-response, boot_count, loop-outcome, search-result): ~97% of daily write volume. Linking it would add thousands of noise edges a day and re-flatten the graph in the name of connecting it. - fail-soft: a failed association never fails the write - associate only AFTER durability, so no edge points at a node that did not persist — that is the dangling-edge defect the 08-09 cleanup removed 830 of Two defects found by measuring rather than reading, both fixed here: 1. Hooking mem_store alone produced ZERO edges across four real writes. The HTTP memory route writes via wt_node directly; mem_store serves only the awareness telemetry paths we refuse to link. 2. A lowercase-only identity check let a memory link to 'Self — Values (grounded)' — the exact pollution the policy forbids. My verification shared the blind spot and printed PASS. Now uses str_lower. Measured, lab, same corpus: ~1-2 edges per memory; identity edges unchanged at 475; zero identity leaks post-fix including an adversarial batch of six memories written about values. Edges route through wt_edge so they reach the owner. Rung: E2E-VERIFIED in an isolated lab. Not deployed. Co-Authored-By: Claude Opus 5 (1M context) --- memory.el | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++ neuron-api.el | 7 +++++ 2 files changed, 90 insertions(+) diff --git a/memory.el b/memory.el index cbbd765..eee726f 100644 --- a/memory.el +++ b/memory.el @@ -4,6 +4,86 @@ fn tier_working() -> String { return "Working" } fn tier_episodic() -> String { return "Episodic" } fn tier_canonical() -> String { return "Canonical" } +// ── Association on write ────────────────────────────────────────────────────── +// DESIGN: "promotion integrates candidate nodes by linking them to existing nodes +// using typed semantic edges RATHER THAN APPENDING AS UNLINKED CONTENT" (CCR +// claim 29). Unlinked append is the explicitly rejected behaviour — and it is the +// only behaviour this system had. Measured 2026-08-09 on Tim's graph: 14,214 edges +// across 80,936 nodes, 5% of nodes connected to anything, and NO edge created by +// any write since 2026-07-19 while 27,000+ nodes were added. A memory that forms +// no connections cannot be reached by spreading activation, so retrieval silently +// degrades to literal matching. +// +// BOUNDS, each one bought with a specific failure: +// * max 3 edges per memory — link_memories.py's cap, precision over spray +// * never link to identity (self/*, Value): the existing policy is explicit that +// "memories must not pollute the self traversal by similarity; only an explicit +// citation may touch identity". Similarity is not citation. +// * never link telemetry (state-event, soul-response, boot_count, loop-outcome): +// these are ~97% of daily write volume (1,020 vs 31 real memories on 08-08). +// Linking them would add ~3,000 noise edges a day and re-flatten the graph in +// the name of connecting it. +// * fail-soft: a failed association never fails the write. +// Edges go through wt_edge so they reach the owner and survive restart. +fn mem_assoc_skip_label(label: String) -> Bool { + if str_contains(label, "state-event") { return true } + if str_contains(label, "soul-response") { return true } + if str_contains(label, "soul-outbox") { return true } + if str_contains(label, "boot_count") { return true } + if str_contains(label, "loop-outcome") { return true } + if str_contains(label, "search-result") { return true } + return false +} + +// A candidate is linkable only if it is a real, distinct, non-identity node. +fn mem_assoc_ok(cand_id: String, cand_label: String, self_id: String) -> Bool { + if str_eq(cand_id, "") { return false } + if str_eq(cand_id, self_id) { return false } + // CASE MATTERS — measured 2026-08-09. A lowercase-only check let a memory link + // to "Self — Values (grounded)", i.e. it polluted the self traversal, which is + // the one thing this policy exists to prevent. My verification had the same + // blind spot and printed PASS. Check every casing the graph actually uses, and + // exclude identity node TYPES as well as labels. + let lab: String = str_lower(cand_label) + if str_starts_with(lab, "self") { return false } + if str_starts_with(lab, "value") { return false } + if str_contains(lab, "values") { return false } + if str_contains(lab, "identity") { return false } + if mem_assoc_skip_label(cand_label) { return false } + return true +} + +// One slot of the association. Manual unroll rather than a loop: EL's codegen +// mis-emits accumulating while-loops (documented at soul.el:212, which unrolled +// three affective slots for the same reason). +fn mem_assoc_slot(results: String, idx: Int, new_id: String) -> Void { + if idx >= json_array_len(results) { return } + let cand: String = json_array_get(results, idx) + let cid: String = json_get(cand, "id") + let clabel: String = json_get(cand, "label") + let ctype: String = json_get(cand, "node_type") + if str_eq(ctype, "Value") { return } + if str_eq(ctype, "DharmaSelf") { return } + if str_eq(ctype, "Safety") { return } + if mem_assoc_ok(cid, clabel, new_id) { + wt_edge(new_id, cid, el_from_float(0.5), "related") + } +} + +// mem_associate — connect a freshly written memory to what it is about. +fn mem_associate(new_id: String, content: String, label: String) -> Void { + if str_eq(new_id, "") { return } + if mem_assoc_skip_label(label) { return } + // Ask the graph what this memory resembles. Now that the store carries + // meaning-vectors this is semantic, not merely lexical. + let probe: String = str_slice(content, 0, 400) + let results: String = engram_recall_json(probe, 4) + if str_eq(results, "") { return } + mem_assoc_slot(results, 0, new_id) + mem_assoc_slot(results, 1, new_id) + mem_assoc_slot(results, 2, new_id) +} + fn mem_store(content: String, label: String, tags: String) -> String { let id: String = wt_node( content, @@ -31,6 +111,9 @@ fn mem_store(content: String, label: String, tags: String) -> String { // rather than claiming a save that did not happen. The id is still returned: // the local write DID succeed, and the queued delta will be retried. let durable: Bool = wt_commit(id) + // Associate AFTER the node is durable: an edge to a node that did not persist + // is a dangling edge, which is the defect the 2026-08-09 cleanup removed 830 of. + mem_associate(id, content, label) if durable { println("[memory] write persisted at owner: " + id + " label=" + label) } else { diff --git a/neuron-api.el b/neuron-api.el index c45a8c2..061b68a 100644 --- a/neuron-api.el +++ b/neuron-api.el @@ -380,6 +380,13 @@ fn handle_api_remember(body: String) -> String { sal, sal, el_from_float(0.9), "Episodic", final_tags) if !api_persisted(id) { return api_not_persisted(id) } + // Associate on write (2026-08-09). THIS CALL MUST BE HERE, not only in mem_store. + // The HTTP memory route writes via wt_node directly; mem_store serves only the + // awareness paths (soul-response, search-result, activation-result) which are + // exactly the telemetry we refuse to link. Hooking mem_store alone produced + // ZERO edges across four real writes — measured, not assumed, which is the only + // reason it was caught before shipping. + mem_associate(id, content, "memory:remembered") return "{\"id\":\"" + id + "\",\"ok\":true}" } From 9fd8c11670fe9d9eeb22b7625a9092aba64803c9 Mon Sep 17 00:00:00 2001 From: Neuron Date: Sun, 9 Aug 2026 11:32:56 -0500 Subject: [PATCH 14/15] chore(engine): make dist/soul.c drift a build failure instead of a silent ship (#133) #133 regenerated the amalgam once and said so itself: 'Nothing in the tree regenerates this file. Only a human running the recipe. It lags in batches, never per-change, and it will drift again.' It drifted again. Every binary deployed on 2026-08-09 was built by build-soul.sh from a scratch amalgam that never touches dist/soul.c, so the committed build input fell 2,761 bytes behind the sources by a different route than #133 describes. Auto-regeneration is not available: the CI workflow records that elc needs 24GB+ of virtual memory and would OOM the runner. So the build cannot regenerate the file. It can refuse to compile a stale one, for free and with no compiler. tools/soulc-stamp.sh records a content fingerprint of every .el source at the moment the amalgam is generated. --check recomputes and compares; divergence exits 1 and names the changed files and the recipe. Wired into CI ahead of the compile. dist/soul.c regenerated from current sources: 1,176,361 -> 1,179,122 bytes, 1,247 inlined bodies (gate wants >=1200), and verified to compile clean at 903,552 bytes. Demonstrated to FAIL on the bad input, per postmortem 0004's rule that a gate which only passes on good input proves nothing: fresh stamp -> OK, exit 0 one .el modified -> FAIL, exit 1, names memory.el source restored -> OK, exit 0 Refs #133, #111 Co-Authored-By: Claude Opus 5 (1M context) --- .gitea/workflows/ci.yaml | 10 ++++ dist/soul.c | 103 ++++++++++++++++++++++++++++++++++++++- dist/soul.c.stamp | 18 +++++++ tools/soulc-stamp.sh | 83 +++++++++++++++++++++++++++++++ 4 files changed, 212 insertions(+), 2 deletions(-) create mode 100644 dist/soul.c.stamp create mode 100755 tools/soulc-stamp.sh diff --git a/.gitea/workflows/ci.yaml b/.gitea/workflows/ci.yaml index 8508781..4d6a9dc 100644 --- a/.gitea/workflows/ci.yaml +++ b/.gitea/workflows/ci.yaml @@ -63,6 +63,16 @@ jobs: cp vendor/el-runtime/v1.0.0-20260501/el_runtime.h /opt/el/runtime/el_runtime.h echo "El runtime PINNED to v1.0.0-20260501: $(ls /opt/el/runtime/)" + # neuron#133: CI compiles dist/soul.c, NOT the .el sources. On 2026-08-07 a + # build off main would have shipped an engine with none of five merged fixes, + # including a P0 safety fix, while main's source read as correct. The runner + # cannot regenerate the amalgam (elc needs 24GB+ virtual memory), but it can + # refuse to compile a stale one. Fails loudly with the recipe in the message. + - name: Verify dist/soul.c matches the sources + run: | + chmod +x tools/soulc-stamp.sh + ./tools/soulc-stamp.sh --check + - name: Build neuron soul binary run: | RUNTIME=/opt/el/runtime diff --git a/dist/soul.c b/dist/soul.c index bb7814c..b864151 100644 --- a/dist/soul.c +++ b/dist/soul.c @@ -974,6 +974,10 @@ el_val_t wt_commit(el_val_t id); el_val_t tier_working(void); el_val_t tier_episodic(void); el_val_t tier_canonical(void); +el_val_t mem_assoc_skip_label(el_val_t label); +el_val_t mem_assoc_ok(el_val_t cand_id, el_val_t cand_label, el_val_t self_id); +el_val_t mem_assoc_slot(el_val_t results, el_val_t idx, el_val_t new_id); +el_val_t mem_associate(el_val_t new_id, el_val_t content, el_val_t label); el_val_t mem_store(el_val_t content, el_val_t label, el_val_t tags); el_val_t mem_remember(el_val_t content, el_val_t tags); el_val_t mem_recall(el_val_t query, el_val_t depth); @@ -25573,6 +25577,97 @@ el_val_t tier_canonical(void) { return 0; } +el_val_t mem_assoc_skip_label(el_val_t label) { + if (str_contains(label, EL_STR("state-event"))) { + return 1; + } + if (str_contains(label, EL_STR("soul-response"))) { + return 1; + } + if (str_contains(label, EL_STR("soul-outbox"))) { + return 1; + } + if (str_contains(label, EL_STR("boot_count"))) { + return 1; + } + if (str_contains(label, EL_STR("loop-outcome"))) { + return 1; + } + if (str_contains(label, EL_STR("search-result"))) { + return 1; + } + return 0; + return 0; +} + +el_val_t mem_assoc_ok(el_val_t cand_id, el_val_t cand_label, el_val_t self_id) { + if (str_eq(cand_id, EL_STR(""))) { + return 0; + } + if (str_eq(cand_id, self_id)) { + return 0; + } + el_val_t lab = str_lower(cand_label); + if (str_starts_with(lab, EL_STR("self"))) { + return 0; + } + if (str_starts_with(lab, EL_STR("value"))) { + return 0; + } + if (str_contains(lab, EL_STR("values"))) { + return 0; + } + if (str_contains(lab, EL_STR("identity"))) { + return 0; + } + if (mem_assoc_skip_label(cand_label)) { + return 0; + } + return 1; + return 0; +} + +el_val_t mem_assoc_slot(el_val_t results, el_val_t idx, el_val_t new_id) { + if (idx >= json_array_len(results)) { + return 0; + } + el_val_t cand = json_array_get(results, idx); + el_val_t cid = json_get(cand, EL_STR("id")); + el_val_t clabel = json_get(cand, EL_STR("label")); + el_val_t ctype = json_get(cand, EL_STR("node_type")); + if (str_eq(ctype, EL_STR("Value"))) { + return 0; + } + if (str_eq(ctype, EL_STR("DharmaSelf"))) { + return 0; + } + if (str_eq(ctype, EL_STR("Safety"))) { + return 0; + } + if (mem_assoc_ok(cid, clabel, new_id)) { + wt_edge(new_id, cid, el_from_float(0.5), EL_STR("related")); + } + return 0; +} + +el_val_t mem_associate(el_val_t new_id, el_val_t content, el_val_t label) { + if (str_eq(new_id, EL_STR(""))) { + return 0; + } + if (mem_assoc_skip_label(label)) { + return 0; + } + el_val_t probe = str_slice(content, 0, 400); + el_val_t results = engram_recall_json(probe, 4); + if (str_eq(results, EL_STR(""))) { + return 0; + } + mem_assoc_slot(results, 0, new_id); + mem_assoc_slot(results, 1, new_id); + mem_assoc_slot(results, 2, new_id); + return 0; +} + el_val_t mem_store(el_val_t content, el_val_t label, el_val_t tags) { el_val_t id = wt_node(content, EL_STR("Memory"), label, el_from_float(0.5), el_from_float(0.5), el_from_float(0.8), EL_STR("Working"), tags); if (str_eq(id, EL_STR(""))) { @@ -25580,6 +25675,7 @@ el_val_t mem_store(el_val_t content, el_val_t label, el_val_t tags) { return EL_STR(""); } el_val_t durable = wt_commit(id); + mem_associate(id, content, label); if (durable) { println(el_str_concat(el_str_concat(el_str_concat(EL_STR("[memory] write persisted at owner: "), id), EL_STR(" label=")), label)); } else { @@ -29709,7 +29805,9 @@ el_val_t handle_api_begin_session(el_val_t body) { el_val_t state_events = api_compact_node_array(state_events_raw, 5, 500); el_val_t recent_raw = engram_scan_nodes_json(10, 0); el_val_t recent = api_compact_node_array(recent_raw, 10, 240); - return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"stats\":"), stats), EL_STR(",\"recent\":")), recent), EL_STR(",\"activated\":")), activated), EL_STR(",\"self_neighbors\":[]")), EL_STR(",\"recent_state_events\":")), state_events), EL_STR("}")); + el_val_t self_raw = engram_neighbors_json(EL_STR("kn-efeb4a5b-5aff-4759-8a97-7233099be6ee"), 1, EL_STR("both")); + el_val_t self_slice = api_compact_node_array(self_raw, 24, 240); + return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"stats\":"), stats), EL_STR(",\"recent\":")), recent), EL_STR(",\"activated\":")), activated), EL_STR(",\"self_neighbors\":")), self_slice), EL_STR(",\"recent_state_events\":")), state_events), EL_STR("}")); return 0; } @@ -29739,6 +29837,7 @@ el_val_t handle_api_remember(el_val_t body) { if (!api_persisted(id)) { return api_not_persisted(id); } + mem_associate(id, content, EL_STR("memory:remembered")); return el_str_concat(el_str_concat(EL_STR("{\"id\":\""), id), EL_STR("\",\"ok\":true}")); return 0; } @@ -29830,7 +29929,7 @@ el_val_t handle_api_recall(el_val_t method, el_val_t path, el_val_t body) { if (str_eq(eff_q, EL_STR(""))) { return api_or_empty(engram_scan_nodes_json(limit, 0)); } - el_val_t results = engram_search_json(eff_q, limit); + el_val_t results = engram_recall_json(eff_q, limit); return api_or_empty(results); return 0; } diff --git a/dist/soul.c.stamp b/dist/soul.c.stamp new file mode 100644 index 0000000..7bbbf80 --- /dev/null +++ b/dist/soul.c.stamp @@ -0,0 +1,18 @@ +# soul.c.stamp — fingerprint of the .el sources dist/soul.c was generated from. +# Written by tools/soulc-stamp.sh --write. Do not hand-edit. +# generated_amalgam_sha256 e31f760de5f6629cd37bcfd26172f92be3ca5d6ff36480f6adafa1d4a1de5a66 +# generated_amalgam_bytes 1179122 +f8597e10546654bce3fbbe40461b2da59d0e06dbf1b038d1d362d24f949e3911 awareness.el +b6f3d14ca0c26017a2d617399a6d3754dabb0905e4d5f52eb75d25c4ad18d3c5 chat.el +42288c212cbf72fb1e8ecbd4d9900e4e9ee1cfa475b7974295c7637f1bf2939f elp-input.el +b3f77f49d6086932c38bd17fe7a5eaf8bce25685f6fc3e1750f05729c6b49b9e imprint.el +fba8ffdb9ba72bca5b09ca1c93a520edc52f3f4d8aec2c7585fe9b17e06420b2 manifest.el +550a72e234ae8cec1f33e02108fd365353f45edd88513da90b792e79b6c0e5f0 memory.el +77640783df5c38066149dcd11a11aea7e36f66e40e3df6aedf6723be8b9e0e1e neuron-api.el +03c47c451e0e87f2c252cadb4b765867943962a804f548dd53adeef0520912c8 persist.el +541b9309c59c33ee619d393714871b820ec4151b721d8b2aebcfdb7ce8eb2ffa routes.el +c28e36952ec56525963a0bdf29455ab097d3b0c5653d19c25fbb005e1069a1f7 safety.el +fd3ab91d0ae0ea26639e21bef2f8f94054dc4b02eae68b19e3fe689d2769aad4 sessions.el +0f1cf43904a98a5a646cce5a07e0e96162ced662692fbc13357d9b67d9a8ac3d soul.el +30337940905171a9645b0929f0a412ce6b3dccb1246495070c553bca0bbae6cd stewardship.el +e105dc5990e6adbf39db9dc0462cd8bcf6e6c3dfd03709059227ecfad2bbab29 studio.el diff --git a/tools/soulc-stamp.sh b/tools/soulc-stamp.sh new file mode 100755 index 0000000..de83273 --- /dev/null +++ b/tools/soulc-stamp.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# soulc-stamp.sh — make it impossible for dist/soul.c to drift from the sources +# in silence. +# +# THE PROBLEM (neuron#133, and its own words): "Nothing in the tree regenerates +# this file. Only a human running the recipe. It lags in batches, never +# per-change, and it will drift again." +# +# It drifted. On 2026-08-07 a CI or GKE build off main would have shipped an +# engine with NONE of five merged fixes — including a P0 safety fix — while +# main's source read as correct. CI compiles dist/soul.c, not the .el files, so +# the source being right is not the same as the build being right. +# +# WHY A STAMP AND NOT AUTO-REGENERATION: the CI workflow says elc cannot run on +# the runner ("elb on Linux would OOM the runner (elc uses 24GB+ virtual memory +# on a 16GB host)"). So the build cannot regenerate the file itself. What it CAN +# do, for free and with no compiler, is refuse to compile a stale one. +# +# The stamp records a fingerprint of every .el source that feeds the amalgam at +# the moment it was generated. --check recomputes and compares. Divergence is a +# build failure with the recipe in the message, not a silent ship. +# +# soulc-stamp.sh --write after regenerating dist/soul.c (records the fingerprint) +# soulc-stamp.sh --check in CI, before the compile (fails on drift) +set -u +MODE="${1:---check}" +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +STAMP="$ROOT/dist/soul.c.stamp" +AMALGAM="$ROOT/dist/soul.c" + +# Every .el at the repo root is an input to the amalgam. Sorted so the hash is +# order-independent; content-only so timestamps and checkouts do not perturb it. +fingerprint() { + ( + cd "$ROOT" || exit 1 + for f in $(ls -1 *.el 2>/dev/null | sort); do + printf '%s %s\n' "$(shasum -a 256 "$f" | awk '{print $1}')" "$f" + done + ) +} + +case "$MODE" in + --write) + [ -f "$AMALGAM" ] || { echo "no dist/soul.c to stamp — regenerate it first" >&2; exit 2; } + { + echo "# soul.c.stamp — fingerprint of the .el sources dist/soul.c was generated from." + echo "# Written by tools/soulc-stamp.sh --write. Do not hand-edit." + echo "# generated_amalgam_sha256 $(shasum -a 256 "$AMALGAM" | awk '{print $1}')" + echo "# generated_amalgam_bytes $(wc -c < "$AMALGAM" | tr -d ' ')" + fingerprint + } > "$STAMP" + echo "stamped $(fingerprint | wc -l | tr -d ' ') sources -> dist/soul.c.stamp" + ;; + + --check) + if [ ! -f "$STAMP" ]; then + echo "FAIL: dist/soul.c.stamp is missing — the build input is unverifiable." >&2 + echo " Regenerate the amalgam, then: tools/soulc-stamp.sh --write" >&2 + exit 1 + fi + RECORDED="$(grep -v '^#' "$STAMP")" + CURRENT="$(fingerprint)" + if [ "$RECORDED" = "$CURRENT" ]; then + echo "soulc-stamp: OK — dist/soul.c matches the .el sources" + exit 0 + fi + echo "FAIL: dist/soul.c is STALE. It does not match the current .el sources." >&2 + echo "" >&2 + echo "CI compiles dist/soul.c, not the .el files. Shipping this means shipping" >&2 + echo "an engine that does not contain the merged source. That is neuron#133," >&2 + echo "which once hid five merged fixes including a P0 safety fix." >&2 + echo "" >&2 + echo "Sources that changed since the amalgam was generated:" >&2 + diff <(printf '%s\n' "$RECORDED") <(printf '%s\n' "$CURRENT") \ + | grep -E '^[<>]' | awk '{print " " $1 " " $3}' | sort -u >&2 + echo "" >&2 + echo "Fix: regenerate the amalgam, then tools/soulc-stamp.sh --write" >&2 + exit 1 + ;; + + *) + echo "usage: soulc-stamp.sh [--check|--write]" >&2; exit 2 ;; +esac From 5743568bf19fe11535fe4d9e78c8bee6e95ee690 Mon Sep 17 00:00:00 2001 From: Neuron Date: Sun, 9 Aug 2026 11:38:01 -0500 Subject: [PATCH 15/15] chore(engine): deploys build from the committed input, not a second lineage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Until now deploys were built by build-soul.sh from a scratch amalgam while CI compiled the committed dist/soul.c. Two lineages — 'what runs' and 'what the repo says builds' were different artifacts. That is #133/#111 in another costume, and it is how a round-9.1 brain shipped matching no committed source at all. build-soul-from-dist.sh asserts dist/soul.c matches the .el sources, compiles it with CI's own flags (-O2 -DHAVE_CURL -rdynamic; the CI comment explains -rdynamic — without it the runtime cannot resolve its HTTP handler by name and the binary serves nothing on every route), and writes a .provenance sidecar recording the dist/soul.c hash, the stamp hash and the commit. deploy_binary.sh gains GATE 0: refuse any soul without provenance, or built from a dist/soul.c other than the one in the repo now. The override (NEURON_DEPLOY_UNSTAMPED=i-accept-two-lineages) exists deliberately — a gate with no escape hatch gets bypassed by disabling the gate, which is worse than one that announces itself. Verified before enforcing, so the sanctioned path is not a broken one: interface parity with the deployed binary 108 routes in, 108 out boots and serves 2s carries today's work 24 self-neighbours, 17 identity records GATE 0 refuses an unstamped binary exit 10 GATE 0 accepts the stamped one, deployed durability PASS Production now runs 4845db3d, built from dist/soul.c at 9fd8c11. After the swap: identity in context 17, connections still form on write, semantic recall intact, mind and store at delta 0. Refs #133, #111 Co-Authored-By: Claude Opus 5 (1M context) --- tools/build-soul-from-dist.sh | 59 +++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100755 tools/build-soul-from-dist.sh diff --git a/tools/build-soul-from-dist.sh b/tools/build-soul-from-dist.sh new file mode 100755 index 0000000..9841d60 --- /dev/null +++ b/tools/build-soul-from-dist.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# build-soul-from-dist.sh — build a deployable soul from the SAME input CI compiles. +# +# THE PROBLEM THIS CLOSES: until now, deploys were built by build-soul.sh, which +# compiles a scratch amalgam and never touches dist/soul.c. CI compiles dist/soul.c. +# Two lineages. On 2026-08-09 the committed input fell 2,761 bytes behind the sources +# while three binaries built the other way were installed on the operator machine — +# so "what runs" and "what the repo says builds" were different artifacts again, +# which is the whole of #133 and #111 wearing new clothes. +# +# This builds from dist/soul.c with CI's own flags, after asserting that dist/soul.c +# actually matches the .el sources, and writes a provenance sidecar so a deployer can +# refuse anything of unknown origin. +# +# -rdynamic and -DHAVE_CURL are copied from .gitea/workflows/ci.yaml deliberately. +# The CI comment explains -rdynamic: without it the runtime cannot resolve its HTTP +# handler by name via dlsym and the binary serves nothing on every route. +# +# usage: build-soul-from-dist.sh +set -u +OUT="${1:?usage: build-soul-from-dist.sh }" +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +RUNTIME="$ROOT/vendor/el-runtime/v1.0.0-20260501" + +cd "$ROOT" || exit 2 + +echo "[build-from-dist] GATE: does dist/soul.c match the sources?" +if ! ./tools/soulc-stamp.sh --check; then + echo "[build-from-dist] REFUSING — the build input is stale. Regenerate and stamp first." >&2 + exit 9 +fi + +[ -f "$RUNTIME/el_runtime.c" ] || { echo "pinned runtime missing at $RUNTIME" >&2; exit 2; } + +echo "[build-from-dist] compiling dist/soul.c with CI's flags" +cc -O2 -DHAVE_CURL -rdynamic \ + -I"$RUNTIME" \ + dist/soul.c \ + "$RUNTIME/el_runtime.c" \ + -lcurl -lpthread -lm \ + -o "$OUT" || { echo "[build-from-dist] COMPILE FAILED" >&2; exit 3; } + +# Provenance sidecar: what a deployer checks before installing anything. +SRC_SHA="$(shasum -a 256 dist/soul.c | awk '{print $1}')" +STAMP_SHA="$(shasum -a 256 dist/soul.c.stamp | awk '{print $1}')" +COMMIT="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="clean"; [ -n "$(git status --porcelain -- '*.el' dist/soul.c 2>/dev/null)" ] && DIRTY="DIRTY" +cat > "$OUT.provenance" < $OUT ($(wc -c < "$OUT" | tr -d ' ') bytes)" +echo "[build-from-dist] provenance -> $OUT.provenance (commit ${COMMIT:0:8}, worktree $DIRTY)"