Changing a shared function turned retrieval into deletion — 234 nodes destroyed per boot, dormant in prod only because embeddings are empty #146

Open
opened 2026-08-08 00:34:14 +00:00 by tim.lingo · 0 comments
Member

This is the most important thing the ten-iteration retrieval run produced, and it is a process defect as much as a code one.

What happened

The run added a semantic (embedding) leg to engram_search_json and measured it through /api/neuron/recall, the one route it cared about. That function has ~40 other call sites. Seven of them delete based on what it returns.

Result: 234 node records destroyed on every boot — including identity records, a core-identity node, and the gold answers to 8 of the run's own test queries. It stood for nine iterations before anyone noticed, because the metric it damaged was not the metric being watched: accuracy moved by 3 queries while the damage was 78 times larger.

It is fixed on the stack branch (#141) — records destroyed per boot back to 1, the pre-existing boot-counter node the code intends to replace.

Why it is still open after being fixed

It has been dormant in production only by accident. The live graph has no embeddings, so the new leg returns nothing there and the deleting callers delete nothing.

The moment embeddings are populated in production — the step every iteration recommended — every restart begins shredding real memories. Any embedding rollout must be gated on the #141 fix landing first. That ordering is the point of this issue.

The class, which is what actually needs a gate

Neither review nor the type checker nor the test suite could catch this, because the change was correct at the site it was made. The defect was entirely in the unexamined callers.

Proposed rule, cheap to enforce: before changing the behaviour of a shared function, enumerate every caller, and specifically every caller that writes or deletes based on the return value. A build check that flags "a function with N callers changed its return semantics" is the mechanical version.

This is the same family as inventory rows G-39 (no check rejects new zero-reference code) and G-41 (the contract check asserts presence only). Filed so the gate gets built rather than the lesson being carried in someone's head.

**This is the most important thing the ten-iteration retrieval run produced, and it is a process defect as much as a code one.** ## What happened The run added a semantic (embedding) leg to `engram_search_json` and measured it through `/api/neuron/recall`, the one route it cared about. That function has **~40 other call sites**. Seven of them **delete** based on what it returns. Result: **234 node records destroyed on every boot** — including identity records, a core-identity node, and the gold answers to 8 of the run's own test queries. It stood for nine iterations before anyone noticed, because the metric it damaged was not the metric being watched: accuracy moved by 3 queries while the damage was 78 times larger. It is **fixed** on the stack branch (#141) — records destroyed per boot back to 1, the pre-existing boot-counter node the code intends to replace. ## Why it is still open after being fixed **It has been dormant in production only by accident.** The live graph has no embeddings, so the new leg returns nothing there and the deleting callers delete nothing. **The moment embeddings are populated in production — the step every iteration recommended — every restart begins shredding real memories.** Any embedding rollout must be gated on the #141 fix landing first. That ordering is the point of this issue. ## The class, which is what actually needs a gate Neither review nor the type checker nor the test suite could catch this, because the change was correct at the site it was made. The defect was entirely in the *unexamined* callers. Proposed rule, cheap to enforce: **before changing the behaviour of a shared function, enumerate every caller, and specifically every caller that writes or deletes based on the return value.** A build check that flags "a function with N callers changed its return semantics" is the mechanical version. This is the same family as inventory rows G-39 (no check rejects new zero-reference code) and G-41 (the contract check asserts presence only). Filed so the gate gets built rather than the lesson being carried in someone's head.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/neuron#146