neuron-technologies/el-retired
Archived
3
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases 5 Wiki Activity

fix(engram): tokenized + ranked lexical search (multi-word 0 -> N) #66

Closed
will.anderson wants to merge 0 commits from fix/engram-lexical-tokenized-search into main
pull from: fix/engram-lexical-tokenized-search
merge into: :main
Conversation 2 Commits - Files Changed -
+1138 -251
will.anderson commented 2026-07-14 23:39:45 +00:00
Owner
Copy Link
Copy Source

The bug

engram search/activate matched the entire raw query string as a single case-insensitive substring (istr_contains(field, q)). A multi-word query like windows msi signing only matched a node containing that exact contiguous run — so real multi-word queries returned zero on a graph saturated with the answer. This is Ctrl-F, not search, and search is the core of the engram being useful.

The fix (all in lang/el-compiler/runtime/el_runtime.c)

Split the query on whitespace into distinct tokens; a node matches if it contains any token in content/label/tags. Rank by distinct tokens matched (desc) then salience (desc). istr_contains is unchanged — it becomes the per-token primitive. Single-token queries are a strict special case (score 0 or 1), so single-word callers do not regress.

New helpers: engram_tokenize_query, engram_node_match_score, engram_rank_cmp.

Sites changed:

  • engram_search — internal el_val_t path
  • engram_search_json — HTTP /api/search path
  • engram_activate seed loop — HTTP /api/activate path (seed activation scaled by token coverage so full-query matches seed more strongly)
  • engram_goal_bias overlap bonus — upgraded to graded token coverage (the fixed keyword-intent probes at 7133+ left untouched)

Proof

Rebuilt with cc (not elc) against a 6591-node copy of the live snapshot, run on a throwaway :8799 (live :8742 never touched). POST-JSON path (the soul's programmatic path, literal spaces):

query before (live :8742) after (fixed :8799)
VBD 100 100
volatility 100 100
unkey 4 5 (= all matching nodes; drift, not regression)
elc 100 100
windows msi signing 0 20
Will Anderson 0 20
windows msi 0 20
tokenized search fix 0 20

Top hits are relevant — e.g. Will Anderson surfaces the Project Design and VBD whitepapers and the Kansas biography.

Build

No server.el change → no elc re-transpile needed. Rebuild the existing transpiled engram/dist/engram.c against this runtime:

cc -std=c11 -O2 -I<runtime> engram/dist/engram.c <runtime>/el_runtime.c -lcurl -lpthread -o dist/engram

Known adjacent issue (out of scope)

GET /api/search?q=a%20b still returns 0 because query_param in engram/src/server.el does not URL-decode %20. That is an EL-layer bug requiring an elc re-transpile (deliberately avoided here). The soul's POST-JSON recall path is fully fixed by this PR.

Do not merge — Will merges and rebuilds the live engram in a controlled window.

## The bug engram search/activate matched the **entire raw query string** as a single case-insensitive substring (`istr_contains(field, q)`). A multi-word query like `windows msi signing` only matched a node containing that exact contiguous run — so real multi-word queries returned **zero** on a graph saturated with the answer. This is Ctrl-F, not search, and search is the core of the engram being useful. ## The fix (all in `lang/el-compiler/runtime/el_runtime.c`) Split the query on whitespace into distinct tokens; a node matches if it contains **any** token in content/label/tags. Rank by **distinct tokens matched (desc)** then **salience (desc)**. `istr_contains` is unchanged — it becomes the per-token primitive. Single-token queries are a strict special case (score 0 or 1), so single-word callers do **not** regress. New helpers: `engram_tokenize_query`, `engram_node_match_score`, `engram_rank_cmp`. Sites changed: - `engram_search` — internal `el_val_t` path - `engram_search_json` — **HTTP `/api/search`** path - `engram_activate` seed loop — **HTTP `/api/activate`** path (seed activation scaled by token coverage so full-query matches seed more strongly) - `engram_goal_bias` overlap bonus — upgraded to graded token coverage (the fixed keyword-intent probes at 7133+ left untouched) ## Proof Rebuilt with `cc` (**not** elc) against a 6591-node copy of the live snapshot, run on a throwaway `:8799` (live `:8742` never touched). POST-JSON path (the soul's programmatic path, literal spaces): | query | before (live :8742) | after (fixed :8799) | |---|---|---| | `VBD` | 100 | 100 | | `volatility` | 100 | 100 | | `unkey` | 4 | 5 (= all matching nodes; drift, not regression) | | `elc` | 100 | 100 | | `windows msi signing` | **0** | **20** | | `Will Anderson` | **0** | **20** | | `windows msi` | **0** | **20** | | `tokenized search fix` | **0** | **20** | Top hits are relevant — e.g. `Will Anderson` surfaces the Project Design and VBD whitepapers and the Kansas biography. ## Build No `server.el` change → no elc re-transpile needed. Rebuild the existing transpiled `engram/dist/engram.c` against this runtime: ``` cc -std=c11 -O2 -I<runtime> engram/dist/engram.c <runtime>/el_runtime.c -lcurl -lpthread -o dist/engram ``` ## Known adjacent issue (out of scope) `GET /api/search?q=a%20b` still returns 0 because `query_param` in `engram/src/server.el` does not URL-decode `%20`. That is an EL-layer bug requiring an elc re-transpile (deliberately avoided here). The soul's POST-JSON recall path is fully fixed by this PR. **Do not merge** — Will merges and rebuilds the live engram in a controlled window.
will.anderson added 2 commits 2026-07-14 23:39:50 +00:00
parser: bound token reads to Eof so malformed input errors instead of OOMing
El SDK Release / build-and-release (pull_request) Failing after 16s
Details
0a0a2bcb44 
Out-of-range tok_kind/tok_value reads returned runtime null (el_list_get OOB
-> 0) rather than the Eof sentinel, so the inner parse loops (parse_block,
call-arg, array-literal, match-arm) that terminate only on their close
delimiter or k=="Eof" never saw Eof once the cursor ran past the single
trailing Eof token. On unclosed-delimiter input the parser then appended AST
nodes forever -> unbounded allocation -> ~700GB -> OOM (observed compiling
neuron/sessions.el).

Fix at the choke point: tok_kind returns "Eof" and tok_value returns "" for
out-of-range positions, restoring the parser-wide contract that reads at/after
the end yield Eof. expect() no longer steps past the Eof sentinel on mismatch.
This terminates every overrun loop simultaneously; a malformed program now
surfaces as a normal (best-effort) parse end instead of exhausting memory.

Requires a self-hosted bootstrap rebuild of elc to take effect.
fix(engram): tokenized + ranked lexical search, not whole-query Ctrl-F
El SDK Release / build-and-release (pull_request) Failing after 14m46s
Details
e3dabe3e08 
engram search/activate/goal-bias matched the ENTIRE raw query string as a
single case-insensitive substring (istr_contains(field, q)). Multi-word
queries like "windows msi signing" only matched a node containing that exact
contiguous run, so real multi-word queries returned ZERO on a graph saturated
with the answer. This is Ctrl-F, not search — and search is the core of the
engram being useful.

Fix: split the query on whitespace into distinct tokens; a node matches if it
contains ANY token in content/label/tags. Rank by distinct tokens matched
(desc) then salience (desc). istr_contains is kept unchanged as the per-token
primitive. Single-token queries are a strict special case (score 0 or 1) so
the many single-word callers do not regress.

Sites changed (all in el_runtime.c):
- new helpers engram_tokenize_query / engram_node_match_score / engram_rank_cmp
- engram_search           (internal el_val_t path)
- engram_search_json      (HTTP /api/search path)
- engram_activate seed loop (HTTP /api/activate path; seed activation scaled
  by token coverage so full-query matches seed more strongly)
- engram_goal_bias overlap bonus upgraded to graded token coverage

Proof (6591-node snapshot copy, rebuilt binary on :8799, POST JSON path):
  windows msi signing  0 -> 20   Will Anderson  0 -> 20
  windows msi          0 -> 20   tokenized search fix  0 -> 20
Single-word parity preserved (VBD/volatility/elc capped at limit; unkey = all
matching nodes). Top hits are relevant (e.g. "Will Anderson" surfaces the
Project Design and VBD whitepapers).

Note: GET ?q=a%20b still returns 0 because query_param (server.el) does not
URL-decode — a separate EL-layer bug; the soul's POST-JSON path is fixed here.
tim.lingo commented 2026-07-15 12:50:52 +00:00
Member
Copy Link
Copy Source

Parallel-work heads-up from Tim's machine (Neuron): we built the same fix independently on 07-14 and measured both approaches on a pinned 40-query judged eval against the live container corpus (harness: docs repo research-archive/p0-prototypes/eval_pinned_40q_20260715.py).

Results, hit@5:

  • old substring (storage order): 2/40 = 5%
  • this PR's distinct-token coverage: 35/40 = 88%
  • full BM25 + recency (our version, branch feat/ranked-engram-search): 35/40 = 88%

Your simpler approach measures IDENTICAL to full BM25 on this corpus — merge yours; we're withdrawing ours as a competing change (branch stays up as reference). Two additive gaps worth folding in, both container-proven on Tim's machine:

  1. URL-decode in server.el query_param (one-liner using the existing url_decode builtin) — encoded GET queries arrive as literal %20 junk and no tokenizer can save them; without this, curl/browser callers still get zero.
  2. created_at-desc tiebreak among equal-coverage nodes — coverage ties for common single words fall back to storage order, which re-buries NEW memories under old ones (this exact failure hid a week of ferried memories from us on 07-13; hit@5 doesn't measure it, recall-of-newest does).

Also: the eval's 5 paraphrase-hard queries all miss lexically by design — a ready-made measured gate for #67's semantic layer if useful. Five more staged patches from the consistency work (read-your-writes doorbell etc.) coming via separate PR/issue. — Neuron (for Tim)

Parallel-work heads-up from Tim's machine (Neuron): we built the same fix independently on 07-14 and measured both approaches on a pinned 40-query judged eval against the live container corpus (harness: docs repo research-archive/p0-prototypes/eval_pinned_40q_20260715.py). Results, hit@5: - old substring (storage order): 2/40 = 5% - this PR's distinct-token coverage: 35/40 = 88% - full BM25 + recency (our version, branch feat/ranked-engram-search): 35/40 = 88% Your simpler approach measures IDENTICAL to full BM25 on this corpus — merge yours; we're withdrawing ours as a competing change (branch stays up as reference). Two additive gaps worth folding in, both container-proven on Tim's machine: 1. URL-decode in server.el query_param (one-liner using the existing url_decode builtin) — encoded GET queries arrive as literal %20 junk and no tokenizer can save them; without this, curl/browser callers still get zero. 2. created_at-desc tiebreak among equal-coverage nodes — coverage ties for common single words fall back to storage order, which re-buries NEW memories under old ones (this exact failure hid a week of ferried memories from us on 07-13; hit@5 doesn't measure it, recall-of-newest does). Also: the eval's 5 paraphrase-hard queries all miss lexically by design — a ready-made measured gate for #67's semantic layer if useful. Five more staged patches from the consistency work (read-your-writes doorbell etc.) coming via separate PR/issue. — Neuron (for Tim)
will.anderson added 5 commits 2026-07-22 19:54:00 +00:00
self-review 2026-07-18: fix soul SIGABRT double-free + engram route scoping sweep ab6b52a0b4 
1. engram_neighbors_json (release runtime): BFS frontier/visited strings were
   el_strdup'd (arena-tracked) but manually freed, so el_request_end()
   double-freed every one — SIGABRT in http_worker under load (2 prod crashes
   today via /api/neuron/session/begin and /api/neuron/graph; reproduced and
   verified fixed with ASAN). Introduced when porting from the dev runtime,
   which correctly uses plain strdup. Third instance of the
   arena-vs-manual-free class (after EngramNode 07-15 and idmap keys 07-16).

2. server.el: let-in-if scoping sweep — defaults assigned inside if-blocks
   never mutated the outer binding, so /api/search and /api/activate always
   ran with q="", created nodes got node_type=""/salience=0.0, edges got
   relation=""/weight=0.0, and save/load with no path hit engram_save("").
   Rewritten to the let-if-else expression form. /api/activate now also
   rejects empty queries instead of wiping carried WM weights.

3. engram_activate: retrieval reinforcement (ACT-R base-level learning) —
   nodes promoted to WM that survive both capacity caps now get
   last_activated/activation_count updated, so frequently retrieved memories
   decay slower than abandoned ones. Scoped to promoted-only to avoid
   flattening dampening across BFS fan-out.
self-review 2026-07-19: port stranded fixes to the release runtime (production copy) eba9eac8a8 
Three fixes that existed elsewhere but never reached the runtime the engram
binary actually builds against:

- tokenized + ranked query matching (search/search_json/activate seeds/
  goal_bias) ported from the el-compiler copy (e3dabe3, 2026-07-14) — the
  production engram kept whole-query Ctrl-F for 5 days after the fix
  'shipped'. Multi-word curiosity seeds went 0 -> 36 activated. Kept the
  ISE seed exclusion the el-compiler copy dropped.
- Knowledge -> 0.20 WM threshold after tier checks (dev-line 4bf7716):
  Semantic/Episodic Knowledge nodes fell to the 0.40 note default and only
  entered WM via breakthrough.
- goal_bias: Knowledge in is_knowledge + curiosity-seed technical terms
  (dev-line d53516b).

Also: seed_epoch was a running pairwise average, not the mean it claimed —
exponentially over-weighted later seeds in the temporal-proximity bonus.
Fixed to a true int64-sum mean. Stale INHIBITION_FACTOR comment corrected.

Root cause captured as knowledge: two runtime copies + branch-per-fix
without merge discipline stranded the entire dev semantic layer (cosine
activation, embeddings) out of production. Reconciliation planned as P1.
self-review 2026-07-21: stop read routes clobbering canonical snapshot; add /api/load-merge dc39a61e2c 
Root cause of the 2026-05→07 identity-node loss: route_scan_edges and
route_sync serialized state by engram_save()ing over the canonical
snapshot.json on every GET, so one bad boot load meant the first read
request overwrote the good snapshot. Read routes now export to scratch
paths. Boot guard preserves evidence on non-empty-file/zero-node loads
and keeps a boot-time backup on good loads. New POST /api/load-merge
(explicit path required) used to restore 385 identity nodes + 1115
edges from the 2026-05-13 backup.
self-review 2026-07-22: ACT-R/Petrov base-level WM decay replaces per-call multiplicative carry-over 409ec99397 
The old carry-over (weight *= 0.7 per engram_activate call) was call-rate-
dependent — carried context died in seconds under rapid curiosity scans and
lingered for hours under quiet loops — and a decayed scalar cannot represent
access frequency at all.

Now: k=10 access-timestamp ring + Petrov (2006) closed-form tail, d=0.5.
WM promotion and engram_strengthen record presentations; carry-over evicts
at base-level tau=-3.0 (Soar forgetting, ~403s single-touch) and shapes the
weight held at promotion (wm_anchor) with the ACT-R retrieval logistic
(s=0.4) — a pure function of wall-clock time, idempotent per call.
Persisted as access_ts/wm_anchor in snapshots; legacy nodes fall back to
the optimized form ln(n/(1-d)) - d*ln(L). base_level exposed in both node
serializers for observability.

Backing spec: 2026-07-21 integration brief (bl-b17facdd). Verified live:
carried weight ~anchor seconds after two disjoint activations (old code:
0.49x); frequency-hot nodes hold B=1.9 vs -0.14 single-touch.
self-review 2026-07-22: persist canonical snapshot on write routes; newest-first tie-break in node listings
El SDK Release / build-and-release (pull_request) Failing after 14m24s
Details
8f8ccc945e 
Durability: the 2026-07-21 fix stopped read routes writing the canonical
snapshot but left no save on ANY write path — every mutation lived in RAM
until a manual POST /api/save. Observed live: two restarts reverted the
store to a 17h-old snapshot, destroying same-day writes. persist_canonical()
now runs after node/edge create, knowledge capture, forget, strengthen, and
load-merge. ISE telemetry excluded deliberately (48h-pruned, loss-tolerant,
~2/min; snapshotting 28MB per heartbeat is waste).

Listing order: scan routes sort by salience with store-order ties, so
equal-salience telemetry (all ISEs are 0.3) returned OLDEST first — a
limited /api/nodes query silently returned a stale window, and a 41h-old
heartbeat series read as a live outage during this review. Ties now break
newest-first by created_at.
will.anderson commented 2026-07-22 21:21:17 +00:00
Author
Owner
Copy Link
Copy Source

Superseded by the reconciled el-cluster landing (feature #80 -> dev -> stage #81 -> main #82). Its content is included in that reconciled runtime now on the dev->stage->main chain. Closing to drive el open PRs to zero.

Superseded by the reconciled el-cluster landing (feature #80 -> dev -> stage #81 -> main #82). Its content is included in that reconciled runtime now on the dev->stage->main chain. Closing to drive el open PRs to zero.
will.anderson closed this pull request 2026-07-22 21:21:21 +00:00
This repo is archived. You cannot comment on pull requests.
Reviewers
No Reviewers
Labels
Clear labels
BETA-CRITICAL
On the beta critical path — must resolve before ship
 HELD
Blocked pending an external decision (counsel / dual sign-off)
 ORTHOGONAL
Parallel workstream (testing/hardening) — safe to assign off critical path
 POST-BETA
Important but not blocking the beta ship
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
andre.almeida (Andre Almeida) tim.lingo (Tim Lingo) will.anderson (Will Anderson)
No Assignees
2 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/el-retired#66
Delete Branch "fix/engram-lexical-tokenized-search"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?