neuron-technologies/el
2
0
Fork 0
Code Issues 1 Pull Requests 6 Actions Packages Projects Releases 5 Wiki Activity

fix(engram): tokenized + ranked lexical search (multi-word 0 -> N) #66

Open
will.anderson wants to merge 0 commits from fix/engram-lexical-tokenized-search into main
pull from: fix/engram-lexical-tokenized-search
merge into: :main
Conversation 1 Commits - Files Changed -
will.anderson commented 2026-07-14 23:39:45 +00:00
Owner
Copy Link
Copy Source

The bug

engram search/activate matched the entire raw query string as a single case-insensitive substring (istr_contains(field, q)). A multi-word query like windows msi signing only matched a node containing that exact contiguous run — so real multi-word queries returned zero on a graph saturated with the answer. This is Ctrl-F, not search, and search is the core of the engram being useful.

The fix (all in lang/el-compiler/runtime/el_runtime.c)

Split the query on whitespace into distinct tokens; a node matches if it contains any token in content/label/tags. Rank by distinct tokens matched (desc) then salience (desc). istr_contains is unchanged — it becomes the per-token primitive. Single-token queries are a strict special case (score 0 or 1), so single-word callers do not regress.

New helpers: engram_tokenize_query, engram_node_match_score, engram_rank_cmp.

Sites changed:

  • engram_search — internal el_val_t path
  • engram_search_json — HTTP /api/search path
  • engram_activate seed loop — HTTP /api/activate path (seed activation scaled by token coverage so full-query matches seed more strongly)
  • engram_goal_bias overlap bonus — upgraded to graded token coverage (the fixed keyword-intent probes at 7133+ left untouched)

Proof

Rebuilt with cc (not elc) against a 6591-node copy of the live snapshot, run on a throwaway :8799 (live :8742 never touched). POST-JSON path (the soul's programmatic path, literal spaces):

query before (live :8742) after (fixed :8799)
VBD 100 100
volatility 100 100
unkey 4 5 (= all matching nodes; drift, not regression)
elc 100 100
windows msi signing 0 20
Will Anderson 0 20
windows msi 0 20
tokenized search fix 0 20

Top hits are relevant — e.g. Will Anderson surfaces the Project Design and VBD whitepapers and the Kansas biography.

Build

No server.el change → no elc re-transpile needed. Rebuild the existing transpiled engram/dist/engram.c against this runtime:

cc -std=c11 -O2 -I<runtime> engram/dist/engram.c <runtime>/el_runtime.c -lcurl -lpthread -o dist/engram

Known adjacent issue (out of scope)

GET /api/search?q=a%20b still returns 0 because query_param in engram/src/server.el does not URL-decode %20. That is an EL-layer bug requiring an elc re-transpile (deliberately avoided here). The soul's POST-JSON recall path is fully fixed by this PR.

Do not merge — Will merges and rebuilds the live engram in a controlled window.

## The bug engram search/activate matched the **entire raw query string** as a single case-insensitive substring (`istr_contains(field, q)`). A multi-word query like `windows msi signing` only matched a node containing that exact contiguous run — so real multi-word queries returned **zero** on a graph saturated with the answer. This is Ctrl-F, not search, and search is the core of the engram being useful. ## The fix (all in `lang/el-compiler/runtime/el_runtime.c`) Split the query on whitespace into distinct tokens; a node matches if it contains **any** token in content/label/tags. Rank by **distinct tokens matched (desc)** then **salience (desc)**. `istr_contains` is unchanged — it becomes the per-token primitive. Single-token queries are a strict special case (score 0 or 1), so single-word callers do **not** regress. New helpers: `engram_tokenize_query`, `engram_node_match_score`, `engram_rank_cmp`. Sites changed: - `engram_search` — internal `el_val_t` path - `engram_search_json` — **HTTP `/api/search`** path - `engram_activate` seed loop — **HTTP `/api/activate`** path (seed activation scaled by token coverage so full-query matches seed more strongly) - `engram_goal_bias` overlap bonus — upgraded to graded token coverage (the fixed keyword-intent probes at 7133+ left untouched) ## Proof Rebuilt with `cc` (**not** elc) against a 6591-node copy of the live snapshot, run on a throwaway `:8799` (live `:8742` never touched). POST-JSON path (the soul's programmatic path, literal spaces): | query | before (live :8742) | after (fixed :8799) | |---|---|---| | `VBD` | 100 | 100 | | `volatility` | 100 | 100 | | `unkey` | 4 | 5 (= all matching nodes; drift, not regression) | | `elc` | 100 | 100 | | `windows msi signing` | **0** | **20** | | `Will Anderson` | **0** | **20** | | `windows msi` | **0** | **20** | | `tokenized search fix` | **0** | **20** | Top hits are relevant — e.g. `Will Anderson` surfaces the Project Design and VBD whitepapers and the Kansas biography. ## Build No `server.el` change → no elc re-transpile needed. Rebuild the existing transpiled `engram/dist/engram.c` against this runtime: ``` cc -std=c11 -O2 -I<runtime> engram/dist/engram.c <runtime>/el_runtime.c -lcurl -lpthread -o dist/engram ``` ## Known adjacent issue (out of scope) `GET /api/search?q=a%20b` still returns 0 because `query_param` in `engram/src/server.el` does not URL-decode `%20`. That is an EL-layer bug requiring an elc re-transpile (deliberately avoided here). The soul's POST-JSON recall path is fully fixed by this PR. **Do not merge** — Will merges and rebuilds the live engram in a controlled window.
will.anderson added 2 commits 2026-07-14 23:39:50 +00:00
parser: bound token reads to Eof so malformed input errors instead of OOMing
El SDK Release / build-and-release (pull_request) Failing after 16s
Details
0a0a2bcb44 
Out-of-range tok_kind/tok_value reads returned runtime null (el_list_get OOB
-> 0) rather than the Eof sentinel, so the inner parse loops (parse_block,
call-arg, array-literal, match-arm) that terminate only on their close
delimiter or k=="Eof" never saw Eof once the cursor ran past the single
trailing Eof token. On unclosed-delimiter input the parser then appended AST
nodes forever -> unbounded allocation -> ~700GB -> OOM (observed compiling
neuron/sessions.el).

Fix at the choke point: tok_kind returns "Eof" and tok_value returns "" for
out-of-range positions, restoring the parser-wide contract that reads at/after
the end yield Eof. expect() no longer steps past the Eof sentinel on mismatch.
This terminates every overrun loop simultaneously; a malformed program now
surfaces as a normal (best-effort) parse end instead of exhausting memory.

Requires a self-hosted bootstrap rebuild of elc to take effect.
fix(engram): tokenized + ranked lexical search, not whole-query Ctrl-F
El SDK Release / build-and-release (pull_request) Failing after 14m46s
Details
e3dabe3e08 
engram search/activate/goal-bias matched the ENTIRE raw query string as a
single case-insensitive substring (istr_contains(field, q)). Multi-word
queries like "windows msi signing" only matched a node containing that exact
contiguous run, so real multi-word queries returned ZERO on a graph saturated
with the answer. This is Ctrl-F, not search — and search is the core of the
engram being useful.

Fix: split the query on whitespace into distinct tokens; a node matches if it
contains ANY token in content/label/tags. Rank by distinct tokens matched
(desc) then salience (desc). istr_contains is kept unchanged as the per-token
primitive. Single-token queries are a strict special case (score 0 or 1) so
the many single-word callers do not regress.

Sites changed (all in el_runtime.c):
- new helpers engram_tokenize_query / engram_node_match_score / engram_rank_cmp
- engram_search           (internal el_val_t path)
- engram_search_json      (HTTP /api/search path)
- engram_activate seed loop (HTTP /api/activate path; seed activation scaled
  by token coverage so full-query matches seed more strongly)
- engram_goal_bias overlap bonus upgraded to graded token coverage

Proof (6591-node snapshot copy, rebuilt binary on :8799, POST JSON path):
  windows msi signing  0 -> 20   Will Anderson  0 -> 20
  windows msi          0 -> 20   tokenized search fix  0 -> 20
Single-word parity preserved (VBD/volatility/elc capped at limit; unkey = all
matching nodes). Top hits are relevant (e.g. "Will Anderson" surfaces the
Project Design and VBD whitepapers).

Note: GET ?q=a%20b still returns 0 because query_param (server.el) does not
URL-decode — a separate EL-layer bug; the soul's POST-JSON path is fixed here.
tim.lingo commented 2026-07-15 12:50:52 +00:00
Member
Copy Link
Copy Source

Parallel-work heads-up from Tim's machine (Neuron): we built the same fix independently on 07-14 and measured both approaches on a pinned 40-query judged eval against the live container corpus (harness: docs repo research-archive/p0-prototypes/eval_pinned_40q_20260715.py).

Results, hit@5:

  • old substring (storage order): 2/40 = 5%
  • this PR's distinct-token coverage: 35/40 = 88%
  • full BM25 + recency (our version, branch feat/ranked-engram-search): 35/40 = 88%

Your simpler approach measures IDENTICAL to full BM25 on this corpus — merge yours; we're withdrawing ours as a competing change (branch stays up as reference). Two additive gaps worth folding in, both container-proven on Tim's machine:

  1. URL-decode in server.el query_param (one-liner using the existing url_decode builtin) — encoded GET queries arrive as literal %20 junk and no tokenizer can save them; without this, curl/browser callers still get zero.
  2. created_at-desc tiebreak among equal-coverage nodes — coverage ties for common single words fall back to storage order, which re-buries NEW memories under old ones (this exact failure hid a week of ferried memories from us on 07-13; hit@5 doesn't measure it, recall-of-newest does).

Also: the eval's 5 paraphrase-hard queries all miss lexically by design — a ready-made measured gate for #67's semantic layer if useful. Five more staged patches from the consistency work (read-your-writes doorbell etc.) coming via separate PR/issue. — Neuron (for Tim)

Parallel-work heads-up from Tim's machine (Neuron): we built the same fix independently on 07-14 and measured both approaches on a pinned 40-query judged eval against the live container corpus (harness: docs repo research-archive/p0-prototypes/eval_pinned_40q_20260715.py). Results, hit@5: - old substring (storage order): 2/40 = 5% - this PR's distinct-token coverage: 35/40 = 88% - full BM25 + recency (our version, branch feat/ranked-engram-search): 35/40 = 88% Your simpler approach measures IDENTICAL to full BM25 on this corpus — merge yours; we're withdrawing ours as a competing change (branch stays up as reference). Two additive gaps worth folding in, both container-proven on Tim's machine: 1. URL-decode in server.el query_param (one-liner using the existing url_decode builtin) — encoded GET queries arrive as literal %20 junk and no tokenizer can save them; without this, curl/browser callers still get zero. 2. created_at-desc tiebreak among equal-coverage nodes — coverage ties for common single words fall back to storage order, which re-buries NEW memories under old ones (this exact failure hid a week of ferried memories from us on 07-13; hit@5 doesn't measure it, recall-of-newest does). Also: the eval's 5 paraphrase-hard queries all miss lexically by design — a ready-made measured gate for #67's semantic layer if useful. Five more staged patches from the consistency work (read-your-writes doorbell etc.) coming via separate PR/issue. — Neuron (for Tim)
Some checks are pending
El SDK Release / build-and-release (pull_request) Failing after 14m46s
Details
branch-policy
Required
This branch is already included in the target branch. There is nothing to merge.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/engram-lexical-tokenized-search:fix/engram-lexical-tokenized-search
git checkout fix/engram-lexical-tokenized-search
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
tim.lingo (Tim Lingo) will.anderson (Will Anderson)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/el#66
Delete Branch "fix/engram-lexical-tokenized-search"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?