Neuron
cf41d12d22
Nothing else on the memory roadmap should be built until a change can be shown
to help. Right now we judge by feel, and the benchmark literature is full of
systems that felt better and measured worse. This is the missing gate.
WHAT IT MEASURES, AND WHY IT BOOTS A REAL SOUL
The subject is Will's designed retrieval — spreading activation over the
weighted directed graph, four-factor multiplicative scoring — not a proxy for
it. A Python re-implementation would measure my reading of the design, so the
harness compiles the actual soul.el amalgam from a git ref and asks it over
HTTP on /api/neuron/recall, exactly as the MCP wrapper and the app do.
BUILT ON WHAT WAS ALREADY HERE, NOT AROUND IT
docs/research/graphrag_eval/{collect,score}.py — per-query relevant-id
scoring and fixed-denominator precision@5 (kept verbatim: an empty result
should be punished like a page of junk).
docs/research-archive/p0-prototypes/eval_pinned_40q_20260715.py — the pinned
ground truth + --check winnability gate, so every run judges alike.
scripts/verify-soul-contract.sh — the isolation recipe, including the
non-obvious SOUL_ISE_URL pin without which an "isolated" soul silently
syncs the operator's live brain.
gen-soul-amalgam.sh + .gitea/workflows/ci.yaml — the build recipe and flags.
New here: ids rather than regexes as ground truth, an associative category
derived from real edges, a superseded category scored on ranking, a
machine-checked zero-lexical-overlap guarantee on paraphrases, paired
significance testing, and measurement of the real compiled soul rather than an
offline replica of one leg of it.
THE GOLD SET IS AUDITABLE, NOT VIBES
38 queries over the real 78,768-node corpus, each carrying a `derivation`
string, each re-validated by `build_gold_set.py --check`. exact_rare is mined
(document frequency 1). phrase is mined (verbatim scan; >25 matches rejected as
too diffuse). paraphrase is hand-selected then PROVEN to share zero content
words with its target — a leak fails the build, so the category cannot decay
into lexical matching. associative is derived from real hub edges with
lexically-reachable siblings dropped. nonsense is verified absent. superseded
pairs are kept only when both sides survive as distinct nodes.
HONEST ABOUT NOISE
Minimum detectable swing on 38 queries is 6: if every changed query moves the
same way, p = 2*0.5^n first clears 0.05 at n=6. Run-to-run drift is measured,
not assumed — activation is a stateful read, and it shows: main is fully
deterministic across 3 runs, the candidate drifts by 1 query. compare.py
reports "no measurable difference" for anything inside max(6, drift+1).
FIRST VERDICT — feat/recall-through-activation
hit@5 34.3% -> 22.9%, phrase 85.7% -> 28.6%, latency p50 2.81x. Five discordant
pairs, all five against the candidate, none for it; McNemar exact p = 0.0625,
so by the stated rule this is one query short of significant and is reported as
such rather than as a win for main. The latency regression is deterministic and
not in any noise band.
The benefit the branch was written for is absent: associative recall is 0/6 on
BOTH builds. Probed directly, the traversal returns the lexical seed at rank 8
and none of its 12 hub siblings. Two measured corpus facts explain it — only
4,060 of 78,768 nodes (5.2%) carry any edge, and no node has an embedding, so
the fourth factor of the four-factor product has nothing to compute from. The
mechanism runs; the corpus lacks the structure it needs.
SAFETY
Throwaway port, throwaway HOME, disposable per-run copy of the corpus; live
ports refused by name. Every soul started is killed AND confirmed dead by pid
probe, with the confirmation written into the results file; run_comparison.sh
sweeps for strays and exits non-zero if any survive. Nothing under ~/.neuron,
/Applications/Neuron*, or ~/neuron-dev-stack is read, written, or restarted.
Rung: E2E-VERIFIED — 6 full runs (3 per config) against the real compiled
binaries on the real corpus; numbers above are measured, not projected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
945 lines
22 KiB
JSON
945 lines
22 KiB
JSON
{
|
|
"label": "main-r1",
|
|
"soul_binary": "/private/tmp/claude-501/-Users-timlingo/82369039-a20e-4b5a-8a5e-28234a57b996/scratchpad/soul-main",
|
|
"soul_md5": "caf822425540114984bfcd713ad37162",
|
|
"corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json",
|
|
"corpus_nodes": 78768,
|
|
"corpus_edges": 14214,
|
|
"gold_set": "/Users/timlingo/Development/neuron-technologies/_wt-eval/tools/retrieval-eval/gold_set.json",
|
|
"limit": 10,
|
|
"port": 7893,
|
|
"wall_clock_s": 45.9,
|
|
"child_pid": 78554,
|
|
"child_confirmed_dead": true,
|
|
"aggregate": {
|
|
"n_queries": 38,
|
|
"n_scored": 35,
|
|
"hit@5": 0.34285714285714286,
|
|
"recall@5": 0.26947278911564626,
|
|
"recall@10": 0.3333333333333333,
|
|
"precision@5": 0.12000000000000001,
|
|
"mrr@10": 0.2943197278911564,
|
|
"nonsense_clean": "2/3",
|
|
"superseded_outranks": "1/3",
|
|
"latency_ms_p50": 1140.4,
|
|
"latency_ms_p95": 1584.1,
|
|
"latency_ms_max": 1627.6,
|
|
"errors": 0,
|
|
"by_category": {
|
|
"associative": {
|
|
"n": 6,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
"exact_rare": {
|
|
"n": 6,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"mrr@10": 1.0
|
|
},
|
|
"nonsense": {
|
|
"n": 3,
|
|
"clean": 2,
|
|
"avg_false_positives": 3.3333333333333335
|
|
},
|
|
"paraphrase": {
|
|
"n": 13,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
"phrase": {
|
|
"n": 7,
|
|
"hit@5": 0.8571428571428571,
|
|
"recall@5": 0.4902210884353741,
|
|
"recall@10": 0.6666666666666666,
|
|
"mrr@10": 0.5965986394557822
|
|
},
|
|
"superseded": {
|
|
"n": 3,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.3333333333333333,
|
|
"mrr@10": 0.041666666666666664,
|
|
"outranks": 1
|
|
}
|
|
}
|
|
},
|
|
"rows": [
|
|
{
|
|
"id": "q01",
|
|
"category": "exact_rare",
|
|
"query": "unjailbreakable",
|
|
"returned": [
|
|
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
|
],
|
|
"n_returned": 1,
|
|
"latency_ms": 240.2,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"precision@5": 0.2,
|
|
"mrr@10": 1.0
|
|
},
|
|
{
|
|
"id": "q02",
|
|
"category": "exact_rare",
|
|
"query": "engram-migrate",
|
|
"returned": [
|
|
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5"
|
|
],
|
|
"n_returned": 1,
|
|
"latency_ms": 271.3,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"precision@5": 0.2,
|
|
"mrr@10": 1.0
|
|
},
|
|
{
|
|
"id": "q03",
|
|
"category": "exact_rare",
|
|
"query": "cartabandonedevent",
|
|
"returned": [
|
|
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
|
],
|
|
"n_returned": 1,
|
|
"latency_ms": 232.0,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"precision@5": 0.2,
|
|
"mrr@10": 1.0
|
|
},
|
|
{
|
|
"id": "q04",
|
|
"category": "exact_rare",
|
|
"query": "pre-apprenticeship",
|
|
"returned": [
|
|
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
|
],
|
|
"n_returned": 1,
|
|
"latency_ms": 227.8,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"precision@5": 0.2,
|
|
"mrr@10": 1.0
|
|
},
|
|
{
|
|
"id": "q05",
|
|
"category": "exact_rare",
|
|
"query": "inferencenodemanager",
|
|
"returned": [
|
|
"mem-73969486-143f-4431-b5e6-6845d1cc9848"
|
|
],
|
|
"n_returned": 1,
|
|
"latency_ms": 259.1,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"precision@5": 0.2,
|
|
"mrr@10": 1.0
|
|
},
|
|
{
|
|
"id": "q06",
|
|
"category": "exact_rare",
|
|
"query": "clear-eyed",
|
|
"returned": [
|
|
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
|
],
|
|
"n_returned": 1,
|
|
"latency_ms": 237.4,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"precision@5": 0.2,
|
|
"mrr@10": 1.0
|
|
},
|
|
{
|
|
"id": "q07",
|
|
"category": "phrase",
|
|
"query": "patterns not returns",
|
|
"returned": [
|
|
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
|
"?ǚ?7??????",
|
|
"mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482",
|
|
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
|
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
|
"????7???Ջ3",
|
|
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
|
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
|
"art-d24fd6dd-2cda-4eed-92f3-67b535a0d71b",
|
|
"7?e?7???\f3?"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 543.7,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"precision@5": 0.2,
|
|
"mrr@10": 0.3333333333333333
|
|
},
|
|
{
|
|
"id": "q08",
|
|
"category": "phrase",
|
|
"query": "thirty moves",
|
|
"returned": [
|
|
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
|
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
|
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
|
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
|
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
|
"knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6",
|
|
"Z[?<S???H??",
|
|
"rQ??m?;?x?'",
|
|
"kn-f230b362-b201-4402-9833-4160c89ab3d4",
|
|
"kn-6061318f-046b-4935-907d-8eafdce14930"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 467.1,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 0.3125,
|
|
"recall@10": 0.5,
|
|
"precision@5": 1.0,
|
|
"mrr@10": 1.0
|
|
},
|
|
{
|
|
"id": "q09",
|
|
"category": "phrase",
|
|
"query": "Grandma Lucas",
|
|
"returned": [
|
|
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
|
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
|
"art-79042b8b-6192-440f-90b0-60708f7e6325"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 475.2,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 0.3333333333333333,
|
|
"recall@10": 0.5555555555555556,
|
|
"precision@5": 0.6,
|
|
"mrr@10": 1.0
|
|
},
|
|
{
|
|
"id": "q10",
|
|
"category": "phrase",
|
|
"query": "Directed Harmonic",
|
|
"returned": [
|
|
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
|
"ԍ????X????",
|
|
"bl-dcee1887-34c4-4ffa-9119-1e291685ba08",
|
|
"?ǚ?7??????",
|
|
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
|
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
|
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf",
|
|
"????7???Ջ3",
|
|
"??????X??2c",
|
|
"g?e?7???'c?"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 449.6,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.1111111111111111,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.14285714285714285
|
|
},
|
|
{
|
|
"id": "q11",
|
|
"category": "phrase",
|
|
"query": "Sarah Bishop",
|
|
"returned": [
|
|
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
|
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
|
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
|
"? ?}&?#??X\b",
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
|
"mem-a9a9ce95-0d64-46eb-9db8-ff81d78ade35",
|
|
"mem-b8ecd23e-77ce-42f7-984c-f51453fec16d"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 463.9,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 0.5,
|
|
"recall@10": 1.0,
|
|
"precision@5": 0.2,
|
|
"mrr@10": 0.2
|
|
},
|
|
{
|
|
"id": "q12",
|
|
"category": "phrase",
|
|
"query": "Directed Autonomous Runtime Modification",
|
|
"returned": [
|
|
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
|
"bl-145a0985-2382-400f-a7c5-c335c5e30a72",
|
|
"mem-82b93b21-a865-410f-9ec1-fc54121d9bb5",
|
|
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
|
"mem-e6327f52-2bda-4ce7-9471-2fffd1e172de",
|
|
"7?e?7???\f3?",
|
|
"?of?7???",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b"
|
|
],
|
|
"n_returned": 9,
|
|
"latency_ms": 821.5,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 0.2857142857142857,
|
|
"recall@10": 0.5,
|
|
"precision@5": 0.8,
|
|
"mrr@10": 1.0
|
|
},
|
|
{
|
|
"id": "q13",
|
|
"category": "phrase",
|
|
"query": "zero-knowledge encrypted backup",
|
|
"returned": [
|
|
"7774a16c-1027-4e3b-a21e-67f1f95a4acd",
|
|
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a",
|
|
"mem-dba009a2-d2ea-4f5a-b9e8-0f04bc9ab32f",
|
|
"mem-a3c97012-5fa3-4915-a839-2c75c72005e0",
|
|
"mem-7cd90611-88a3-423d-a38a-0db2812952fa",
|
|
"bl-07375bf9-a169-42cd-adb3-7d32b25982f0",
|
|
"bl-ec84b63d-b278-4944-8d7f-4aa7a51c0315",
|
|
"7?e?7???\f3?",
|
|
"?of?7???",
|
|
"? ?}&?#??X\b"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 688.0,
|
|
"error": null,
|
|
"hit@5": 1.0,
|
|
"recall@5": 1.0,
|
|
"recall@10": 1.0,
|
|
"precision@5": 0.2,
|
|
"mrr@10": 0.5
|
|
},
|
|
{
|
|
"id": "q14",
|
|
"category": "paraphrase",
|
|
"query": "the elderly relative who passed while he stayed away",
|
|
"returned": [
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
|
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
|
"????7???Ջ3",
|
|
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1559.5,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q15",
|
|
"category": "paraphrase",
|
|
"query": "a soldier sidelined by illness who refused to quit",
|
|
"returned": [
|
|
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
|
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
|
"art-2f29ad36-6ee6-4a0e-8d72-0eaf7d12d3a9",
|
|
"? ?}&?#??X\b",
|
|
"kn-a31e1001-342e-4deb-a2e6-6d02d1f22dee",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1627.6,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q16",
|
|
"category": "paraphrase",
|
|
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
|
"returned": [
|
|
"7?e?7???\f3?",
|
|
"? ?}&?#??X\b",
|
|
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
|
"?of?7???",
|
|
"????7???Ջ3",
|
|
"kn-d97920d0-1649-4223-9508-c0bb621e7fc0",
|
|
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
|
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
|
"kn-81c24d13-a73b-4767-819c-dafaacc1498e",
|
|
"? ?}&?#??X\b"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1317.2,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q17",
|
|
"category": "paraphrase",
|
|
"query": "a tight payload beats a bloated one",
|
|
"returned": [
|
|
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491",
|
|
"7?e?7???\f3?",
|
|
"? ?}&?#??X\b",
|
|
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"??o?'?B???k",
|
|
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
|
"? ?}&?#??X\b",
|
|
"?of?7???"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1055.2,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q18",
|
|
"category": "paraphrase",
|
|
"query": "if you are able and nobody is coming the job is yours",
|
|
"returned": [
|
|
"7?e?7???\f3?",
|
|
"?of?7???",
|
|
"imp-dce1da0f-8776-4a9e-972b-33411a7ca138",
|
|
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
|
"kn-83bb86c6-521d-416c-a86e-6e29c2d8f102",
|
|
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b"
|
|
],
|
|
"n_returned": 9,
|
|
"latency_ms": 1352.5,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q19",
|
|
"category": "paraphrase",
|
|
"query": "learning is the wealth creditors cannot seize",
|
|
"returned": [
|
|
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
|
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
|
"knw-e24d6339-5ff3-4bed-ba53-707ffd0dc70a",
|
|
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
|
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
|
"7?e?7???\f3?",
|
|
"??o?'?B???k",
|
|
"? ?}&?#??X\b",
|
|
"kn-0625e393-067c-4bba-8389-7e1b79265142",
|
|
"ע?RGk?\tH(?"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1213.0,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q20",
|
|
"category": "paraphrase",
|
|
"query": "reliability proven by track record not assertion",
|
|
"returned": [
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
|
"7?e?7???\f3?",
|
|
"?of?7???",
|
|
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef",
|
|
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1545.8,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q21",
|
|
"category": "paraphrase",
|
|
"query": "boundaries that enable instead of confine",
|
|
"returned": [
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"??????X??2c",
|
|
"ԍ????X????",
|
|
"dR????X?-?S",
|
|
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
|
"dz????Xƹ?i",
|
|
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1327.3,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q22",
|
|
"category": "paraphrase",
|
|
"query": "what shifts tells you where to cut a system apart",
|
|
"returned": [
|
|
"7?e?7???\f3?",
|
|
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
|
"?of?7???",
|
|
"kn-f8974b26-78a6-4aad-b893-19a73b20013d",
|
|
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
|
"dR????X?-?S",
|
|
"dz????Xƹ?i",
|
|
"ԍ????X????",
|
|
"??????X??2c",
|
|
"kn-d7c1e0fb-fa59-46d3-b4c9-a0d1d437a491"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1584.1,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q23",
|
|
"category": "paraphrase",
|
|
"query": "a mind that compounds instead of resetting each day",
|
|
"returned": [
|
|
"????7???Ջ3",
|
|
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
|
"?ǚ?7??????",
|
|
"7?e?7???\f3?",
|
|
"??f?7???",
|
|
"? ?}&?#??X\b",
|
|
"%???2??jH??",
|
|
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
|
"? ?}&?#??X\b",
|
|
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1432.6,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q24",
|
|
"category": "paraphrase",
|
|
"query": "loved for the unedited self and not the polished exterior",
|
|
"returned": [
|
|
"mem-bbb126a1-b297-42bb-86be-796871829c94",
|
|
"mem-45022957-2d78-48aa-a714-16d6eca52e0f",
|
|
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
|
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
|
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
|
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
|
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
|
"7?e?7???\f3?",
|
|
"? ?}&?#??X\b",
|
|
"'?T?a\"B~-?8"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1505.5,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q25",
|
|
"category": "paraphrase",
|
|
"query": "cheerfulness you arrive at instead of assuming",
|
|
"returned": [
|
|
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
|
"?ǚ?7??????",
|
|
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
|
"7?e?7???\f3?",
|
|
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
|
"??S?7???",
|
|
"? ?}&?#??X\b",
|
|
"??f?7???",
|
|
"??S?7???",
|
|
"knw-5578cb21-e899-4822-b7f4-0d96fa094e3d"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1201.1,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q26",
|
|
"category": "paraphrase",
|
|
"query": "a childhood offering no solid foundation to inherit",
|
|
"returned": [
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
|
"7?e?7???\f3?",
|
|
"kn-e8423822-eacf-4029-aa7b-10d4d28d621e",
|
|
"? ?}&?#??X\b",
|
|
"??o?'?B???k",
|
|
"?of?7???",
|
|
"? ?}&?#??X\b"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1303.3,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q27",
|
|
"category": "associative",
|
|
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
|
"returned": [
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
|
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
|
"?ǚ?7??????"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1416.3,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q28",
|
|
"category": "associative",
|
|
"query": "Marines hernia sepsis medical ward",
|
|
"returned": [
|
|
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
|
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
|
"bl-33ecccc2-e37f-43db-91b3-c2a86f08aaac",
|
|
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
|
"art-e0bdf5d8-d163-491f-b649-453fee8b721d",
|
|
"7?e?7???\f3?",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"kn-9397c74b-35f3-4428-b4b0-5123353bbcd1",
|
|
"?of?7???"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1090.3,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q29",
|
|
"category": "associative",
|
|
"query": "Sarah Bishop Dyer trailer performance",
|
|
"returned": [
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
|
"7?e?7???\f3?",
|
|
"?of?7???",
|
|
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
|
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
|
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1167.7,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q30",
|
|
"category": "associative",
|
|
"query": "Swarm Architecture containment lateral worker",
|
|
"returned": [
|
|
"art-ee615cdb-e599-423d-9a4d-977859390ed3",
|
|
"bl-9bde67c1-f0ba-4c3a-8fe5-de0deee0ce43",
|
|
"8cbb60c5-4999-4ec1-8682-2592aedc4249",
|
|
"bl-0fac287f-f4c0-4f15-bc4d-ff7f8a7af3ae",
|
|
"7?e?7???\f3?",
|
|
"kn-6f248a50-355b-47bb-aec8-e0e646a9b077",
|
|
"? ?}&?#??X\b",
|
|
"h??I?cB?Q??",
|
|
"?of?7???",
|
|
"? ?}&?#??X\b"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1140.4,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q31",
|
|
"category": "associative",
|
|
"query": "hope won inside the narrative preface",
|
|
"returned": [
|
|
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
|
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
|
"knw-9e74ee95-ba7d-49b1-9262-977eae9729d1",
|
|
"7?e?7???\f3?",
|
|
"rQ??m?;?x?'",
|
|
"?Q??m?;?u?'",
|
|
"R^??m?;��?'",
|
|
"?Q??m?;��`'",
|
|
"? ?}&?#??X\b"
|
|
],
|
|
"n_returned": 9,
|
|
"latency_ms": 1157.6,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q32",
|
|
"category": "associative",
|
|
"query": "man of the house six years old expectation",
|
|
"returned": [
|
|
"? ?}&?#??X\b",
|
|
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"kn-57b4c5e7-40c6-4c90-bf14-71841b0081d4",
|
|
"knw-35940684-abc4-42f0-b942-818f66b1f69a",
|
|
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
|
"art-2fabd873-d787-49cb-ad30-d4ed9fcff8ef"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1379.7,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0
|
|
},
|
|
{
|
|
"id": "q33",
|
|
"category": "nonsense",
|
|
"query": "zqxjvw plimforth grebulon",
|
|
"returned": [],
|
|
"n_returned": 0,
|
|
"latency_ms": 677.2,
|
|
"error": null,
|
|
"clean": true,
|
|
"false_positives": 0
|
|
},
|
|
{
|
|
"id": "q34",
|
|
"category": "nonsense",
|
|
"query": "flarnbistle quommetry",
|
|
"returned": [],
|
|
"n_returned": 0,
|
|
"latency_ms": 443.2,
|
|
"error": null,
|
|
"clean": true,
|
|
"false_positives": 0
|
|
},
|
|
{
|
|
"id": "q35",
|
|
"category": "nonsense",
|
|
"query": "xxqzzt vurblenacht throom",
|
|
"returned": [
|
|
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
|
"knw-920c891f-bb8c-48c4-9afc-018ef12dcdc4",
|
|
"art-79042b8b-6192-440f-90b0-60708f7e6325",
|
|
"art-ddfcd045-2c3b-4a1e-9966-fec5ce44e1dd",
|
|
"bl-4476e856-c567-4b49-8ff7-d7dca3e5715e",
|
|
"kn-66a21179-2adc-4b19-a109-880cf4674d7d",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 679.8,
|
|
"error": null,
|
|
"clean": false,
|
|
"false_positives": 10
|
|
},
|
|
{
|
|
"id": "q36",
|
|
"category": "superseded",
|
|
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
|
"returned": [
|
|
"7?e?7???\f3?",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"?of?7???",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"mem-80d7416b-20e9-48a0-b176-b215527e2f56",
|
|
"mem-f3b37427-b7d1-4f7e-b32c-0241a20ce8da",
|
|
"art-80ca3d31-84dc-4502-83f4-538372b9764f"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1245.4,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 1.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.125,
|
|
"outranks": true,
|
|
"rank_correct": 8,
|
|
"rank_stale": null
|
|
},
|
|
{
|
|
"id": "q37",
|
|
"category": "superseded",
|
|
"query": "how many provisional patents does Will actually have",
|
|
"returned": [
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"7?e?7???\f3?",
|
|
"?of?7???",
|
|
"?ǚ?7??????",
|
|
"[?MO5????G",
|
|
"art-ee615cdb-e599-423d-9a4d-977859390ed3"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 1618.4,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0,
|
|
"outranks": false,
|
|
"rank_correct": null,
|
|
"rank_stale": null
|
|
},
|
|
{
|
|
"id": "q38",
|
|
"category": "superseded",
|
|
"query": "is MCP still the live integration layer",
|
|
"returned": [
|
|
"kn-5584ef9c-7f9d-4d7c-a10a-4ee6bc5cf356",
|
|
"7?e?7???\f3?",
|
|
"? ?}&?#??X\b",
|
|
"%???2??jH??",
|
|
"???Ͼd??W\b?",
|
|
"??o?'?B???k",
|
|
"? ?}&?#??X\b",
|
|
"63307ac5-cf6b-46e0-8296-07503b461cfa",
|
|
"? ?}&?#??X\b",
|
|
"?of?7???"
|
|
],
|
|
"n_returned": 10,
|
|
"latency_ms": 956.7,
|
|
"error": null,
|
|
"hit@5": 0.0,
|
|
"recall@5": 0.0,
|
|
"recall@10": 0.0,
|
|
"precision@5": 0.0,
|
|
"mrr@10": 0.0,
|
|
"outranks": false,
|
|
"rank_correct": null,
|
|
"rank_stale": null
|
|
}
|
|
]
|
|
} |