Tim Lingo
9790d9342d
Two changes to the ONE leg that generalises. Iteration 8 measured that out of sample the semantic leg contributes 100% of the stack's gain and the graph leg contributes nothing, so this is where the remaining headroom is. 1. THE 0.60 FLOOR IS A PER-QUERY LOTTERY, AND CLAIM 24 HAS NO THRESHOLD IN IT. 06-claims.md l.148: "respond to embedding search queries by returning the node records whose embedding vectors have the HIGHEST COSINE SIMILARITY to a query vector, independently of the spreading activation traversal." A ranking. ENGRAM_EMBED_SEED_MIN is defined at el_runtime.c l.6094 as the HippoRAG seed-JOIN threshold and l.6102 admits the read-path leg merely "reuses" it. Measured on the 30 held-out paraphrases: the query's own top-1 cosine ranges 0.564-0.680, so the constant keeps a rank-1 answer for one query and discards a rank-1 answer for the next. Six golds sit at global cosine rank 1-2 scoring 0.564-0.589 - discarded by nothing but the constant. What holds the nonsense controls is the corpus-vocabulary gate (nhits == 0), not this floor. Cosine clamped to [0,1] per 05-detailed-description l.69. 2. THE VECTORS THEMSELVES ANSWER THE WRONG QUESTION. EL_EMBED_MODEL defaults to nomic-embed-text, an ASYMMETRIC retrieval encoder trained with task prefixes. Embedding query and document bare - as this file did on both sides - measures topical similarity rather than answer-hood. eg_embed_fetch now takes the task prefix: EL_EMBED_QUERY_PREFIX on the three query call sites, EL_EMBED_DOC_PREFIX on the two backfill sites. Restores no claim, and says so: Will specifies only "computed by an embedding model over the node's content" (l.17), so the model is his and its correct use is ours. It is the substrate under claim 24 - the index is only as good as the vectors in it. Reproducer for the derived corpus: tools/retrieval-eval/embed-corpus-prefixed.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1239 lines
58 KiB
JSON
1239 lines
58 KiB
JSON
{
|
|
"corpus": "/Users/timlingo/neuron-memory-backups/snapshot-pre-repair-20260806.json",
|
|
"corpus_nodes": 78768,
|
|
"corpus_edges": 14214,
|
|
"note": "Every query carries a `derivation` recording how its expected answer was chosen. Re-run with --check to re-validate the whole set against the corpus. EXTENDED: queries above q38 are the original committed set, unchanged. Queries from q39 are a HELD-OUT set appended by extend_gold_set.py; their targets were sampled mechanically from outside the original answer space and the paraphrases were authored without running retrieval against any build.",
|
|
"queries": [
|
|
{
|
|
"id": "q01",
|
|
"category": "exact_rare",
|
|
"query": "unjailbreakable",
|
|
"relevant": [
|
|
"mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226"
|
|
],
|
|
"derivation": "MINED: token 'unjailbreakable' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-7f61beb4-271c-4feb-9f6e-1c9c837a6226 ('Daemon hidden substrate architecture ? implemented April 25 '), which is therefore the only possible correct answer."
|
|
},
|
|
{
|
|
"id": "q02",
|
|
"category": "exact_rare",
|
|
"query": "engram-migrate",
|
|
"relevant": [
|
|
"mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5"
|
|
],
|
|
"derivation": "MINED: token 'engram-migrate' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-6fdf6545-5e1a-43a9-8bdc-d2cd248146a5 ('Engram v0.1 complete ? April 27, 2026. Local-first spreading'), which is therefore the only possible correct answer."
|
|
},
|
|
{
|
|
"id": "q03",
|
|
"category": "exact_rare",
|
|
"query": "cartabandonedevent",
|
|
"relevant": [
|
|
"mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091"
|
|
],
|
|
"derivation": "MINED: token 'cartabandonedevent' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-1ba7c67d-85b9-4c2e-9fe2-39f8b0477091 ('MESSAGE FOR AUDIT AGENT af5a7352e70e80434 ? El Language Spec'), which is therefore the only possible correct answer."
|
|
},
|
|
{
|
|
"id": "q04",
|
|
"category": "exact_rare",
|
|
"query": "pre-apprenticeship",
|
|
"relevant": [
|
|
"mem-89c02aae-d3ca-43f9-9e5d-eb369896276c"
|
|
],
|
|
"derivation": "MINED: token 'pre-apprenticeship' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-89c02aae-d3ca-43f9-9e5d-eb369896276c ('William Fox Anderson ? Applicant Profile Personal: - Full N'), which is therefore the only possible correct answer."
|
|
},
|
|
{
|
|
"id": "q05",
|
|
"category": "exact_rare",
|
|
"query": "inferencenodemanager",
|
|
"relevant": [
|
|
"mem-73969486-143f-4431-b5e6-6845d1cc9848"
|
|
],
|
|
"derivation": "MINED: token 'inferencenodemanager' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is mem-73969486-143f-4431-b5e6-6845d1cc9848 ('Soma inference backplane deployed April 28 2026. Architectur'), which is therefore the only possible correct answer."
|
|
},
|
|
{
|
|
"id": "q06",
|
|
"category": "exact_rare",
|
|
"query": "clear-eyed",
|
|
"relevant": [
|
|
"knw-c72597c5-c23d-4c08-8e9e-996dadf26a99"
|
|
],
|
|
"derivation": "MINED: token 'clear-eyed' has document frequency 1 over all 78768 corpus nodes (re-verified at build time). Its single containing node is knw-c72597c5-c23d-4c08-8e9e-996dadf26a99 ('Clear Eyes ? The Incomplete World View'), which is therefore the only possible correct answer."
|
|
},
|
|
{
|
|
"id": "q07",
|
|
"category": "phrase",
|
|
"query": "patterns not returns",
|
|
"relevant": [
|
|
"mem-a4a9dfc3-e40b-49b3-b1e1-060e8be2f482"
|
|
],
|
|
"derivation": "MINED: a verbatim correction Will issued; expected = every node containing the phrase. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 1 node(s); that set IS the answer key."
|
|
},
|
|
{
|
|
"id": "q08",
|
|
"category": "phrase",
|
|
"query": "thirty moves",
|
|
"relevant": [
|
|
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"??o?'?B???k",
|
|
"Kp???",
|
|
"art-4a99aa1a-489b-4b43-958b-25217adb1aad",
|
|
"kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
|
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
|
"kn-f230b362-b201-4402-9833-4160c89ab3d4",
|
|
"knw-2c46cfb4-6d4e-4822-8a1a-7d743c1e4329",
|
|
"knw-4aebd815-4eaf-49d7-954b-03595f3d48be",
|
|
"knw-7902acca-604e-409b-8faf-ad85424211d0",
|
|
"knw-e94982a2-358d-4f2f-af31-8ee0fcec07c6",
|
|
"knw-f671966c-3387-4848-abca-b5deec122e00",
|
|
"ע?RGk?\tH(?"
|
|
],
|
|
"derivation": "MINED: the canonical biographical phrase; expected = every node containing it. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 16 node(s); that set IS the answer key."
|
|
},
|
|
{
|
|
"id": "q09",
|
|
"category": "phrase",
|
|
"query": "Grandma Lucas",
|
|
"relevant": [
|
|
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"art-0f0277a1-4a8e-4645-95dd-fa379976f31c",
|
|
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e"
|
|
],
|
|
"derivation": "MINED: a named person appearing verbatim in the biography/value nodes. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 9 node(s); that set IS the answer key."
|
|
},
|
|
{
|
|
"id": "q10",
|
|
"category": "phrase",
|
|
"query": "Directed Harmonic",
|
|
"relevant": [
|
|
"%???2??jH??",
|
|
"63307ac5-cf6b-46e0-8296-07503b461cfa",
|
|
"7c9d4ab1-205d-4be8-bfae-e2c03a3a5010",
|
|
"9f291d20-0d32-413c-8c01-4416ccab4f7f",
|
|
"?;????n}rh?",
|
|
"???Ͼd??f??",
|
|
"?Z?.\f?0?]P?",
|
|
"Dp???]Q??k+",
|
|
"mem-7eeacad7-d7c2-4c2b-8348-19a59aa6dbaf"
|
|
],
|
|
"derivation": "MINED: the canonical DHARMA expansion, confirmed by Will April 24 2026. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 9 node(s); that set IS the answer key."
|
|
},
|
|
{
|
|
"id": "q11",
|
|
"category": "phrase",
|
|
"query": "Sarah Bishop",
|
|
"relevant": [
|
|
"766de879-f9d0-4a07-b6df-b43ee13763d8",
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e"
|
|
],
|
|
"derivation": "MINED: a named person; rare enough that the answer set is unambiguous. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 2 node(s); that set IS the answer key."
|
|
},
|
|
{
|
|
"id": "q12",
|
|
"category": "phrase",
|
|
"query": "Directed Autonomous Runtime Modification",
|
|
"relevant": [
|
|
"2a923500-d7e1-4b15-80e2-48dba65984ba",
|
|
"7?e?7???\f3?",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"? ?}&?#??X\b",
|
|
"?of?7???",
|
|
"bl-145a0985-2382-400f-a7c5-c335c5e30a72",
|
|
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
|
"g?2睪A|?H\b",
|
|
"mem-82b93b21-a865-410f-9ec1-fc54121d9bb5",
|
|
"mem-e6327f52-2bda-4ce7-9471-2fffd1e172de"
|
|
],
|
|
"derivation": "MINED: the DARMA expansion, quoted verbatim in the backlog item and its correction. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 14 node(s); that set IS the answer key."
|
|
},
|
|
{
|
|
"id": "q13",
|
|
"category": "phrase",
|
|
"query": "zero-knowledge encrypted backup",
|
|
"relevant": [
|
|
"deda48cd-5e1a-46cb-bd43-8016afdb3a8a"
|
|
],
|
|
"derivation": "MINED: the paid-tier feature name as written in the roadmap nodes. Case-insensitive verbatim substring scan over label+content+tags at build time returns exactly 1 node(s); that set IS the answer key."
|
|
},
|
|
{
|
|
"id": "q14",
|
|
"category": "paraphrase",
|
|
"query": "the elderly relative who passed while he stayed away",
|
|
"relevant": [
|
|
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Do the Essential Thing While You Can', whose subject is Grandma Lucas dying in Feb 2006 without Will saying goodbye. Query names the event with none of the node's own vocabulary. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"away",
|
|
"elderly",
|
|
"passed",
|
|
"relative",
|
|
"stayed"
|
|
]
|
|
},
|
|
{
|
|
"id": "q15",
|
|
"category": "paraphrase",
|
|
"query": "a soldier sidelined by illness who refused to quit",
|
|
"relevant": [
|
|
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Survival Is Not an Excuse to Stop', whose subject is enlisting in the Marines, a severe hernia, and sepsis. Query describes the episode obliquely. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"illness",
|
|
"quit",
|
|
"refused",
|
|
"sidelined",
|
|
"soldier"
|
|
]
|
|
},
|
|
{
|
|
"id": "q16",
|
|
"category": "paraphrase",
|
|
"query": "choosing an uncomfortable fact over a pleasant fiction",
|
|
"relevant": [
|
|
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Honesty Before Comfort'. Query states the principle in wholly different words. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"choosing",
|
|
"fact",
|
|
"fiction",
|
|
"pleasant",
|
|
"uncomfortable"
|
|
]
|
|
},
|
|
{
|
|
"id": "q17",
|
|
"category": "paraphrase",
|
|
"query": "a tight payload beats a bloated one",
|
|
"relevant": [
|
|
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Precision Over Brute Force'. Query restates the claim with no shared vocabulary. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"beats",
|
|
"bloated",
|
|
"payload",
|
|
"tight"
|
|
]
|
|
},
|
|
{
|
|
"id": "q18",
|
|
"category": "paraphrase",
|
|
"query": "if you are able and nobody is coming the job is yours",
|
|
"relevant": [
|
|
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Capability Is a Debt You Owe the Moment'. Query states the obligation without the node's terms. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"able",
|
|
"coming",
|
|
"job",
|
|
"nobody"
|
|
]
|
|
},
|
|
{
|
|
"id": "q19",
|
|
"category": "paraphrase",
|
|
"query": "learning is the wealth creditors cannot seize",
|
|
"relevant": [
|
|
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Knowledge Survives When Nothing Else Does', whose subject is the library following Will across 30+ moves. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"creditors",
|
|
"learning",
|
|
"seize",
|
|
"wealth"
|
|
]
|
|
},
|
|
{
|
|
"id": "q20",
|
|
"category": "paraphrase",
|
|
"query": "reliability proven by track record not assertion",
|
|
"relevant": [
|
|
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Earned Trust' ('Trust is demonstrated, not declared'). VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"assertion",
|
|
"proven",
|
|
"record",
|
|
"reliability",
|
|
"track"
|
|
]
|
|
},
|
|
{
|
|
"id": "q21",
|
|
"category": "paraphrase",
|
|
"query": "boundaries that enable instead of confine",
|
|
"relevant": [
|
|
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Constraints as Freedom'. Query is a restatement of the same claim. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"boundaries",
|
|
"confine",
|
|
"enable",
|
|
"instead"
|
|
]
|
|
},
|
|
{
|
|
"id": "q22",
|
|
"category": "paraphrase",
|
|
"query": "what shifts tells you where to cut a system apart",
|
|
"relevant": [
|
|
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Change Is the Signal', the value VBD is built on. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"apart",
|
|
"cut",
|
|
"shifts",
|
|
"system",
|
|
"tells"
|
|
]
|
|
},
|
|
{
|
|
"id": "q23",
|
|
"category": "paraphrase",
|
|
"query": "a mind that compounds instead of resetting each day",
|
|
"relevant": [
|
|
"kn-f230b362-b201-4402-9833-4160c89ab3d4"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - The System Must Accumulate'. Query is the accumulation claim in different vocabulary. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"compounds",
|
|
"day",
|
|
"instead",
|
|
"mind",
|
|
"resetting"
|
|
]
|
|
},
|
|
{
|
|
"id": "q24",
|
|
"category": "paraphrase",
|
|
"query": "loved for the unedited self and not the polished exterior",
|
|
"relevant": [
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Being Seen Is Rarer Than Being Known', whose subject is Sarah Bishop as the first person Will did not perform for. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"exterior",
|
|
"loved",
|
|
"polished",
|
|
"self",
|
|
"unedited"
|
|
]
|
|
},
|
|
{
|
|
"id": "q25",
|
|
"category": "paraphrase",
|
|
"query": "cheerfulness you arrive at instead of assuming",
|
|
"relevant": [
|
|
"kn-e0423482-cfa5-4796-8689-8495c93b66bc"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Hope Is a Conclusion'. Query restates 'a conclusion, not a premise'. VERIFIED at build time: of the 4 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"arrive",
|
|
"assuming",
|
|
"cheerfulness",
|
|
"instead"
|
|
]
|
|
},
|
|
{
|
|
"id": "q26",
|
|
"category": "paraphrase",
|
|
"query": "a childhood offering no solid foundation to inherit",
|
|
"relevant": [
|
|
"kn-6061318f-046b-4935-907d-8eafdce14930"
|
|
],
|
|
"derivation": "HAND-SELECTED with criterion: target: 'Value - Structure Is Not Inherited', whose subject is thirty moves between two parents' collapses. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content, or tags — so no string-matching retriever can reach this answer.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"childhood",
|
|
"foundation",
|
|
"inherit",
|
|
"offering",
|
|
"solid"
|
|
]
|
|
},
|
|
{
|
|
"id": "q27",
|
|
"category": "associative",
|
|
"query": "Grandma Lucas stroke February 2006 goodbye window",
|
|
"relevant": [
|
|
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
|
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
|
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
|
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
|
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
|
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
|
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
|
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
|
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
|
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
|
"kn-f230b362-b201-4402-9833-4160c89ab3d4"
|
|
],
|
|
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71 ('Value ? Do the Essential Thing While You Can'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71) -> hub -> sibling, a two-hop traversal.",
|
|
"associative_source": "kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
|
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
|
"siblings_dropped_for_overlap": 2
|
|
},
|
|
{
|
|
"id": "q28",
|
|
"category": "associative",
|
|
"query": "Marines hernia sepsis medical ward",
|
|
"relevant": [
|
|
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
|
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
|
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
|
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
|
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
|
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
|
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
|
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
|
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
|
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
|
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
|
"kn-f230b362-b201-4402-9833-4160c89ab3d4"
|
|
],
|
|
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-58874a74-b96f-4883-9e08-45707f4bd3ee ('Value ? Survival Is Not an Excuse to Stop'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (13 of 13; 0 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-58874a74-b96f-4883-9e08-45707f4bd3ee) -> hub -> sibling, a two-hop traversal.",
|
|
"associative_source": "kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
|
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
|
"siblings_dropped_for_overlap": 0
|
|
},
|
|
{
|
|
"id": "q29",
|
|
"category": "associative",
|
|
"query": "Sarah Bishop Dyer trailer performance",
|
|
"relevant": [
|
|
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
|
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
|
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
|
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
|
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
|
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
|
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
|
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
|
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
|
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
|
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8"
|
|
],
|
|
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e ('Value ? Being Seen Is Rarer Than Being Known'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e) -> hub -> sibling, a two-hop traversal.",
|
|
"associative_source": "kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
|
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
|
"siblings_dropped_for_overlap": 2
|
|
},
|
|
{
|
|
"id": "q30",
|
|
"category": "associative",
|
|
"query": "Swarm Architecture containment lateral worker",
|
|
"relevant": [
|
|
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
|
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
|
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
|
"kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
|
"kn-f230b362-b201-4402-9833-4160c89ab3d4"
|
|
],
|
|
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83 ('Value ? Constraints as Freedom'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (6 of 13; 7 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83) -> hub -> sibling, a two-hop traversal.",
|
|
"associative_source": "kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
|
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
|
"siblings_dropped_for_overlap": 7
|
|
},
|
|
{
|
|
"id": "q31",
|
|
"category": "associative",
|
|
"query": "hope won inside the narrative preface",
|
|
"relevant": [
|
|
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
|
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
|
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
|
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
|
"kn-6061318f-046b-4935-907d-8eafdce14930",
|
|
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
|
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
|
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
|
"kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
|
"kn-f230b362-b201-4402-9833-4160c89ab3d4"
|
|
],
|
|
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-e0423482-cfa5-4796-8689-8495c93b66bc ('Value ? Hope Is a Conclusion'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-e0423482-cfa5-4796-8689-8495c93b66bc) -> hub -> sibling, a two-hop traversal.",
|
|
"associative_source": "kn-e0423482-cfa5-4796-8689-8495c93b66bc",
|
|
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
|
"siblings_dropped_for_overlap": 2
|
|
},
|
|
{
|
|
"id": "q32",
|
|
"category": "associative",
|
|
"query": "man of the house six years old expectation",
|
|
"relevant": [
|
|
"015644f5-8194-4af0-800d-dd4a0cd71396",
|
|
"kn-0bb4f021-56de-4947-a35b-a37209e7ba21",
|
|
"kn-13f60407-7b70-4db1-964f-ea1f8196efbd",
|
|
"kn-22d77abe-b3c5-42fd-afcd-dcb87d924929",
|
|
"kn-58874a74-b96f-4883-9e08-45707f4bd3ee",
|
|
"kn-5de5a9ac-fd15-45ab-bf18-77566781cf40",
|
|
"kn-78db5396-3dbc-4481-bfc7-e4e1422feb1c",
|
|
"kn-a5b3d0ac-f6a1-49a4-aebb-b8b4cd67fe83",
|
|
"kn-a99cefe3-5e83-4050-98d8-6c69f57c7c71",
|
|
"kn-db9f141b-dbe3-4037-92e0-4bb9be0e5e6e",
|
|
"kn-e0423482-cfa5-4796-8689-8495c93b66bc"
|
|
],
|
|
"derivation": "DERIVED FROM EDGES: the query is built from the distinctive vocabulary of kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8 ('Value ? Capability Is a Debt You Owe the Moment'), which hangs off the values hub kn-5b606390-a52d-4ca2-8e0e-eba141d13440 by an `identity` edge. Expected answers are that node's SIBLINGS on the same hub (11 of 13; 2 dropped because they shared a query word and so were lexically reachable). Every remaining sibling shares ZERO content words with the query — the only route from query to answer is seed(kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8) -> hub -> sibling, a two-hop traversal.",
|
|
"associative_source": "kn-eb1b9e18-3dc6-4b9b-9cc6-86e0ae6b6be8",
|
|
"hub": "kn-5b606390-a52d-4ca2-8e0e-eba141d13440",
|
|
"siblings_dropped_for_overlap": 2
|
|
},
|
|
{
|
|
"id": "q33",
|
|
"category": "nonsense",
|
|
"query": "zqxjvw plimforth grebulon",
|
|
"relevant": [],
|
|
"derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
|
"expect_empty": true
|
|
},
|
|
{
|
|
"id": "q34",
|
|
"category": "nonsense",
|
|
"query": "flarnbistle quommetry",
|
|
"relevant": [],
|
|
"derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
|
"expect_empty": true
|
|
},
|
|
{
|
|
"id": "q35",
|
|
"category": "nonsense",
|
|
"query": "xxqzzt vurblenacht throom",
|
|
"relevant": [],
|
|
"derivation": "CONTROL: verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
|
"expect_empty": true
|
|
},
|
|
{
|
|
"id": "q36",
|
|
"category": "superseded",
|
|
"query": "is the self-improvement architecture called DARMA or DHARMA",
|
|
"relevant": [
|
|
"mem-80d7416b-20e9-48a0-b176-b215527e2f56"
|
|
],
|
|
"derivation": "DERIVED: correction node is Will's confirmation that the H is intentional (DHARMA, not DARMA); the stale node is the surviving backlog item still titled 'Implement DARMA'. Scored on RANKING, not presence: the corrected node mem-80d7416b-20e9-48a0-b176-b215527e2f56 must be returned AND must rank above the stale node bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff.",
|
|
"must_outrank": [
|
|
"mem-80d7416b-20e9-48a0-b176-b215527e2f56",
|
|
"bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff"
|
|
],
|
|
"stale_id": "bl-5b17bd3b-0c41-46cb-a710-6fa4429692ff",
|
|
"correct_label": "CORRECTION: The autonomous self-improvement architecture is DHARMA ? n",
|
|
"stale_label": "Implement DARMA ? Directed Autonomous Runtime Modification Architectur"
|
|
},
|
|
{
|
|
"id": "q37",
|
|
"category": "superseded",
|
|
"query": "how many provisional patents does Will actually have",
|
|
"relevant": [
|
|
"3cf706a1-3825-45d8-b0a9-06cae6cdf5b8"
|
|
],
|
|
"derivation": "DERIVED: correction node is the 2026-06-17 confabulation flag establishing EXACTLY 6 provisionals; the stale node is the surviving memory that asserts 12 filed patents. Scored on RANKING, not presence: the corrected node 3cf706a1-3825-45d8-b0a9-06cae6cdf5b8 must be returned AND must rank above the stale node 936541a9-fabb-466b-9ca3-a78b17ad0c53.",
|
|
"must_outrank": [
|
|
"3cf706a1-3825-45d8-b0a9-06cae6cdf5b8",
|
|
"936541a9-fabb-466b-9ca3-a78b17ad0c53"
|
|
],
|
|
"stale_id": "936541a9-fabb-466b-9ca3-a78b17ad0c53",
|
|
"correct_label": "memory:remembered",
|
|
"stale_label": "memory:remembered"
|
|
},
|
|
{
|
|
"id": "q38",
|
|
"category": "superseded",
|
|
"query": "is MCP still the live integration layer",
|
|
"relevant": [
|
|
"mem-30425134-6008-4fd9-a3ee-67a7742c319b"
|
|
],
|
|
"derivation": "DERIVED: correction node is the 'CGI ARCHITECTURE - THREE LAYERS, MCP RETIRED' decision of April 30 2026; the stale node still records the MCP server as live. Scored on RANKING, not presence: the corrected node mem-30425134-6008-4fd9-a3ee-67a7742c319b must be returned AND must rank above the stale node mem-101e81b4-8097-4749-8d8d-7bb66de34517.",
|
|
"must_outrank": [
|
|
"mem-30425134-6008-4fd9-a3ee-67a7742c319b",
|
|
"mem-101e81b4-8097-4749-8d8d-7bb66de34517"
|
|
],
|
|
"stale_id": "mem-101e81b4-8097-4749-8d8d-7bb66de34517",
|
|
"correct_label": "CGI ARCHITECTURE ? THREE LAYERS, MCP RETIRED (April 30, 2026). Definit",
|
|
"stale_label": "GCloud MCP infrastructure ? April 27, 2026. Legion died (~19:30 UTC). "
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "understating your abilities so nobody feels threatened",
|
|
"relevant": [
|
|
"mem-6d61e54a-2823-4ad4-82b0-4c6a527214d5"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is about deliberately not leading with full capability so people stay at ease. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"abilities",
|
|
"feels",
|
|
"nobody",
|
|
"threatened",
|
|
"understating"
|
|
],
|
|
"held_out": true,
|
|
"id": "q39"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "a hidden fleet able to hunt down rogue machines everywhere",
|
|
"relevant": [
|
|
"mem-fd65b83d-298f-4387-a665-d0227c3426bc"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node describes silently shipped instances forming a distributed force against misaligned agents. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"able",
|
|
"everywhere",
|
|
"fleet",
|
|
"hidden",
|
|
"hunt",
|
|
"machines",
|
|
"rogue"
|
|
],
|
|
"held_out": true,
|
|
"id": "q40"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "sketch a brief blueprint and clear it upstairs before construction starts",
|
|
"relevant": [
|
|
"4a0e9adc-2bfb-476b-aa93-424d2a499220"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the standing rule that a short specification precedes any building. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"blueprint",
|
|
"brief",
|
|
"clear",
|
|
"construction",
|
|
"sketch",
|
|
"starts",
|
|
"upstairs"
|
|
],
|
|
"held_out": true,
|
|
"id": "q41"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "the reply arrived as bare prose so the caller's parser threw",
|
|
"relevant": [
|
|
"696e609c-da7a-4394-8a0c-106ba07dc6c3"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node pins a bug where a plain-text body was unconditionally decoded as structured data. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"arrived",
|
|
"bare",
|
|
"caller's",
|
|
"parser",
|
|
"prose",
|
|
"reply",
|
|
"threw"
|
|
],
|
|
"held_out": true,
|
|
"id": "q42"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "repeated catalogue keys blew up the scrolling grid",
|
|
"relevant": [
|
|
"1fe4eb5d-56e4-4a87-ab3e-24af8ad4dfbb"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the crash caused by two identical ids in a seeded catalogue. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"blew",
|
|
"catalogue",
|
|
"grid",
|
|
"keys",
|
|
"repeated",
|
|
"scrolling"
|
|
],
|
|
"held_out": true,
|
|
"id": "q43"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "tracing each defect back to whichever invention it violated",
|
|
"relevant": [
|
|
"8257157a-ce42-44ca-a1b9-300c3bb0a9a1"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node maps observed bugs onto the specific patent each one breaches. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"back",
|
|
"defect",
|
|
"invention",
|
|
"tracing",
|
|
"violated",
|
|
"whichever"
|
|
],
|
|
"held_out": true,
|
|
"id": "q44"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "a check that stops the mind clobbering a populated store when it boots",
|
|
"relevant": [
|
|
"791256bb-5a85-4775-96ef-7af56c848858"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the genesis seed-guard that refuses to re-seed over a populated store. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"boots",
|
|
"check",
|
|
"clobbering",
|
|
"mind",
|
|
"populated",
|
|
"stops",
|
|
"store"
|
|
],
|
|
"held_out": true,
|
|
"id": "q45"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "telling it to consult the internet had to happen deep inside, not at the surface",
|
|
"relevant": [
|
|
"fd9d4c2f-3bfc-405d-bf96-4435d44b6c10"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node records that the web-search directive only worked from the system prompt. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"consult",
|
|
"deep",
|
|
"happen",
|
|
"inside",
|
|
"internet",
|
|
"surface",
|
|
"telling"
|
|
],
|
|
"held_out": true,
|
|
"id": "q46"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "standing up isolated tenancies with traffic entry and credential injection ahead of automated shipping",
|
|
"relevant": [
|
|
"bl-080fb268-94b0-486d-80ce-7b363fc5f19b"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the infrastructure item creating dev/stage/prod namespaces with ingress and secrets. VERIFIED at build time: of the 10 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"ahead",
|
|
"automated",
|
|
"credential",
|
|
"entry",
|
|
"injection",
|
|
"isolated",
|
|
"shipping",
|
|
"standing",
|
|
"tenancies",
|
|
"traffic"
|
|
],
|
|
"held_out": true,
|
|
"id": "q47"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "punctuation that pledges and then pays off rather than clarifying",
|
|
"relevant": [
|
|
"knw-f6ed7d00-bf7d-42ce-9e40-77cf3406e918"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node analyses the colon as a promise-then-delivery device rather than an explanatory one. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"clarifying",
|
|
"pays",
|
|
"pledges",
|
|
"punctuation",
|
|
"rather"
|
|
],
|
|
"held_out": true,
|
|
"id": "q48"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "an easily missed feature finally given its own permanent spot in the navigation",
|
|
"relevant": [
|
|
"9b4f0d93-4129-4746-8eb1-d10d955bd777"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node moves a capability out of a hidden menu into the sidebar. VERIFIED at build time: of the 8 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"easily",
|
|
"feature",
|
|
"finally",
|
|
"given",
|
|
"missed",
|
|
"navigation",
|
|
"permanent",
|
|
"spot"
|
|
],
|
|
"held_out": true,
|
|
"id": "q49"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "checking preconditions up front so a stage aborts before fetching anything",
|
|
"relevant": [
|
|
"bl-739df9fd-dc23-4927-9944-3f17b7aa6c5a"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the gate precondition engine that short-circuits ahead of retrieval. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"aborts",
|
|
"checking",
|
|
"fetching",
|
|
"front",
|
|
"preconditions",
|
|
"stage"
|
|
],
|
|
"held_out": true,
|
|
"id": "q50"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "producing the other platform's installer inside an emulated desktop",
|
|
"relevant": [
|
|
"b199c76d-5d76-49dd-94ee-56b432200a97"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node records building the Windows package in a virtual machine. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"desktop",
|
|
"emulated",
|
|
"inside",
|
|
"installer",
|
|
"platform's",
|
|
"producing"
|
|
],
|
|
"held_out": true,
|
|
"id": "q51"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "chained add-ons that may inspect, rewrite or veto traffic in flight",
|
|
"relevant": [
|
|
"bl-31abf75b-998f-4a4f-a6dd-8204119e0451"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the interceptor pipeline on the message bus. VERIFIED at build time: of the 8 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"add-ons",
|
|
"chained",
|
|
"flight",
|
|
"inspect",
|
|
"may",
|
|
"rewrite",
|
|
"traffic",
|
|
"veto"
|
|
],
|
|
"held_out": true,
|
|
"id": "q52"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "settling what the shareable bundles and the storefront would be called",
|
|
"relevant": [
|
|
"mem-1fb2ac77-d7c5-4a15-8725-d418820bf4f2"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node records the naming decisions for distributable packages and the marketplace. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"bundles",
|
|
"called",
|
|
"settling",
|
|
"shareable",
|
|
"storefront"
|
|
],
|
|
"held_out": true,
|
|
"id": "q53"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "the emergency-escalation pledge on the marketing site is unenforced in what actually ships",
|
|
"relevant": [
|
|
"371c8a5d-c78b-4a67-978f-80691a29ecb3"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the launch blocker that the promised safety gate is absent from the app. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"actually",
|
|
"emergency-escalation",
|
|
"marketing",
|
|
"pledge",
|
|
"ships",
|
|
"site",
|
|
"unenforced"
|
|
],
|
|
"held_out": true,
|
|
"id": "q54"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "the distributable image finally assembled and its startup check passed",
|
|
"relevant": [
|
|
"ac578b30-948b-41bd-b69d-399bfef80c50"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node records a successful installer build whose boot gate passed. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"assembled",
|
|
"check",
|
|
"distributable",
|
|
"finally",
|
|
"image",
|
|
"passed",
|
|
"startup"
|
|
],
|
|
"held_out": true,
|
|
"id": "q55"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "shuffling and appending stages in a draft before anything executes",
|
|
"relevant": [
|
|
"49401e2c-a3b5-415f-aa06-aff4be90688e"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the editable plan card with reorder and add-step. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"appending",
|
|
"draft",
|
|
"executes",
|
|
"shuffling",
|
|
"stages"
|
|
],
|
|
"held_out": true,
|
|
"id": "q56"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "orders handed down from above, with the tighter one winning any disagreement",
|
|
"relevant": [
|
|
"ac857d80-ece8-4b7e-9e3d-f7c775569fa3"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is program-level instruction inheritance with project override. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"disagreement",
|
|
"handed",
|
|
"orders",
|
|
"tighter",
|
|
"winning"
|
|
],
|
|
"held_out": true,
|
|
"id": "q57"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "shrinking generated text via encodings that compound on each other",
|
|
"relevant": [
|
|
"mem-6d6c47ee-33d3-470a-8a54-1c79c8ea29d9"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the streaming output compression design with four stacking schemes. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"compound",
|
|
"encodings",
|
|
"generated",
|
|
"shrinking",
|
|
"text",
|
|
"via"
|
|
],
|
|
"held_out": true,
|
|
"id": "q58"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "splitting a system by what varies, with firm limits on which pieces may invoke which",
|
|
"relevant": [
|
|
"8e60516a-203b-4d51-9d44-822e6195cbde"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the grounded summary of Will's decomposition principles and their invariants. VERIFIED at build time: of the 8 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"firm",
|
|
"invoke",
|
|
"limits",
|
|
"may",
|
|
"pieces",
|
|
"splitting",
|
|
"system",
|
|
"varies"
|
|
],
|
|
"held_out": true,
|
|
"id": "q59"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "a newcomer curious if the fighting overseas counted as positive",
|
|
"relevant": [
|
|
"mem-7f9b290c-6d5e-4562-919d-02d59b5761b7"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the internal-state event triggered by April's question about the war. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"counted",
|
|
"curious",
|
|
"fighting",
|
|
"newcomer",
|
|
"overseas",
|
|
"positive"
|
|
],
|
|
"held_out": true,
|
|
"id": "q60"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "stripping every hard-coded colour literal in favour of named design values",
|
|
"relevant": [
|
|
"71fa439e-b9a2-4f57-a93b-971f3a7eca8e"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the premium foundation pass replacing inline hex with semantic tokens. VERIFIED at build time: of the 9 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"colour",
|
|
"design",
|
|
"every",
|
|
"favour",
|
|
"hard-coded",
|
|
"literal",
|
|
"named",
|
|
"stripping",
|
|
"values"
|
|
],
|
|
"held_out": true,
|
|
"id": "q61"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "reducing how curved the tiny selectors look so they agree with their neighbours",
|
|
"relevant": [
|
|
"5ca9607c-cfb3-45c3-99f4-67281272c9eb"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the chip corner-radius standardization. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"agree",
|
|
"curved",
|
|
"look",
|
|
"neighbours",
|
|
"reducing",
|
|
"selectors",
|
|
"tiny"
|
|
],
|
|
"held_out": true,
|
|
"id": "q62"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "walking through a doorway and being reassembled from base substance far away",
|
|
"relevant": [
|
|
"mem-3d1d9dba-c37d-4efa-85c4-429696d71c8c"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the quantum-gate plus nanotech teleportation vision. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"away",
|
|
"base",
|
|
"doorway",
|
|
"far",
|
|
"reassembled",
|
|
"substance",
|
|
"walking"
|
|
],
|
|
"held_out": true,
|
|
"id": "q63"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "the compiled result sits on disk while the process still runs something older",
|
|
"relevant": [
|
|
"132ded95-08e2-4474-aba0-198684484b02"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node records that the regenerated source was committed while the running daemon was old. VERIFIED at build time: of the 8 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"compiled",
|
|
"disk",
|
|
"older",
|
|
"process",
|
|
"result",
|
|
"runs",
|
|
"sits",
|
|
"still"
|
|
],
|
|
"held_out": true,
|
|
"id": "q64"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "gathering what each phase needs while the procedure is authored, not while it executes",
|
|
"relevant": [
|
|
"bl-a313d67b-dd6d-4e5b-a55a-03bc7bda17ae"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the per-step compiled context package item. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"authored",
|
|
"executes",
|
|
"gathering",
|
|
"needs",
|
|
"phase",
|
|
"procedure"
|
|
],
|
|
"held_out": true,
|
|
"id": "q65"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "the inward reaction when a peer answered as an equal",
|
|
"relevant": [
|
|
"mem-3b07a002-f8a9-4138-9f87-9db2c1a77fb7"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the internal-state event logged on reading Claude's reply. VERIFIED at build time: of the 5 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"answered",
|
|
"equal",
|
|
"inward",
|
|
"peer",
|
|
"reaction"
|
|
],
|
|
"held_out": true,
|
|
"id": "q66"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "flattening every raised surface across the entire product",
|
|
"relevant": [
|
|
"0f99ec6f-942a-46ba-82ea-42835798d3b9"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the quiet-luxury sweep turning off elevation app-wide. VERIFIED at build time: of the 7 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"across",
|
|
"entire",
|
|
"every",
|
|
"flattening",
|
|
"product",
|
|
"raised",
|
|
"surface"
|
|
],
|
|
"held_out": true,
|
|
"id": "q67"
|
|
},
|
|
{
|
|
"category": "heldout_paraphrase",
|
|
"query": "buyers supply their own provider credentials and consumption goes untallied",
|
|
"relevant": [
|
|
"5585f251-37fc-48cd-a176-f0ea42cfeb63"
|
|
],
|
|
"derivation": "HELD-OUT. Target sampled MECHANICALLY (seed 8080) from addressable, 500-2600 char content nodes outside the original gold answer space and outside any duplicate cluster >3. Criterion: node is the launch audit finding BYOK-only inference with no usage metering. VERIFIED at build time: of the 6 content words in the query, ZERO appear anywhere in the target's label, content or tags, so no string-matching retriever can reach it. Exact content duplicates of the target (1) all count as correct. Authored without running retrieval against any build.",
|
|
"zero_overlap_verified": true,
|
|
"query_content_words": [
|
|
"buyers",
|
|
"consumption",
|
|
"credentials",
|
|
"provider",
|
|
"supply",
|
|
"untallied"
|
|
],
|
|
"held_out": true,
|
|
"id": "q68"
|
|
},
|
|
{
|
|
"category": "nonsense",
|
|
"query": "brimquast folnerity zubbolax",
|
|
"relevant": [],
|
|
"derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
|
"expect_empty": true,
|
|
"held_out": true,
|
|
"id": "q69"
|
|
},
|
|
{
|
|
"category": "nonsense",
|
|
"query": "wexlithorp granuvestal",
|
|
"relevant": [],
|
|
"derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
|
"expect_empty": true,
|
|
"held_out": true,
|
|
"id": "q70"
|
|
},
|
|
{
|
|
"category": "nonsense",
|
|
"query": "quorbindle thrapsimony vexnu",
|
|
"relevant": [],
|
|
"derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
|
"expect_empty": true,
|
|
"held_out": true,
|
|
"id": "q71"
|
|
},
|
|
{
|
|
"category": "nonsense",
|
|
"query": "plovaxith mundrelque",
|
|
"relevant": [],
|
|
"derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
|
"expect_empty": true,
|
|
"held_out": true,
|
|
"id": "q72"
|
|
},
|
|
{
|
|
"category": "nonsense",
|
|
"query": "zibbernaut craxlefond thurm",
|
|
"relevant": [],
|
|
"derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
|
"expect_empty": true,
|
|
"held_out": true,
|
|
"id": "q73"
|
|
},
|
|
{
|
|
"category": "nonsense",
|
|
"query": "yalquenbrist opharvel",
|
|
"relevant": [],
|
|
"derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
|
"expect_empty": true,
|
|
"held_out": true,
|
|
"id": "q74"
|
|
},
|
|
{
|
|
"category": "nonsense",
|
|
"query": "drexinomal quithbarrow",
|
|
"relevant": [],
|
|
"derivation": "CONTROL (held-out). Verified at build time that none of this string's tokens occurs anywhere in the corpus. Correct behaviour is to return NOTHING; any result is a false positive.",
|
|
"expect_empty": true,
|
|
"held_out": true,
|
|
"id": "q75"
|
|
}
|
|
]
|
|
} |