Tim Lingo
9c39084e60
engram_search_json ranked its lexical leg by raw distinct-token coverage with salience as tiebreak: a token in 30,000 nodes counted the same as a token in 1, and a 1.3 MB record matched nearly every query token by surface area alone. Score it BM25-shaped instead - Lucene-form IDF and length normalisation over the corpus mean - with per-token document frequency accumulated in the SAME corpus pass that finds the hits (no extra scan, no extra round-trip). Also refuse to return records whose identifier is not printable ASCII. This corpus carries 1,032 such records (453 by the printable test) from a save-side corruption; they occupy 125 of 303 returned slots on main. Claims 12, 23 and 27 all key on the node identifier, so such a record is unfetchable by any caller. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>