neuron-technologies/el-retired
Archived
3
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases 5 Wiki Activity

feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical) #67

Closed
will.anderson wants to merge 0 commits from feat/engram-semantic-search into hotfix/elc-fixes
pull from: feat/engram-semantic-search
merge into: :hotfix/elc-fixes
Conversation 1 Commits - Files Changed -
+314 -23
will.anderson commented 2026-07-14 23:49:13 +00:00
Owner
Copy Link
Copy Source

Semantic search layer — nomic-embed-text (the upgrade beyond lexical)

Lexical istr_contains can only match text that literally appears. It cannot
surface a node whose words never appear in the query but whose meaning
matches. This PR adds an optional dense-vector layer: node content and the
query are embedded via Ollama (nomic-embed-text), ranked by cosine
similarity, and unioned with lexical hits.

What changed (el_runtime.c)

A self-contained semantic block (embed client, FNV-1a-keyed in-memory vector
cache, cosine, one-shot probe) plus wiring into all three query entry
points — the routing reality is that server.el calls the _json builtins,
so fixing only the internal twin would never reach :8742:

  • engram_search_json (HTTP /api/search): collect lexical ∪ semantic
    candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
    rank, emit top-N. Stable sort ⇒ byte-identical to old output when semantic
    is off.
  • engram_search (internal el_val_t twin): lexical ∪ semantic union.
  • engram_activate seed loop (HTTP /api/activate): a node seeds if it
    lexically matches or clears the cosine threshold; pure-semantic seeds
    enter scaled by cosine so a paraphrase spreads activation without
    overpowering exact lexical seeds.

Degradable by design

The whole layer is gated on HAVE_CURL and a runtime probe. If curl is
compiled out, Ollama is unreachable, or ENGRAM_SEMANTIC=0, every entry point
yields zero semantic signal and callers fall back byte-for-byte to the
pre-existing lexical search. Verified against a dead endpoint and with the
flag off (see evidence).

Where vectors live

In-process cache keyed by node id, invalidated by an FNV-1a hash of content
(edited content re-embeds). The query is embedded once per call; the graph
is not re-embedded per query. nomic task prefixes
(search_query: / search_document:) are applied — empirically they separate
a paraphrase (0.72) cleanly from distractors (<0.48).

Config

ENGRAM_SEMANTIC (0 disables), ENGRAM_EMBED_URL
(def http://localhost:11434/api/embeddings), ENGRAM_EMBED_MODEL
(def nomic-embed-text), ENGRAM_SEMANTIC_MIN (cosine threshold, def 0.6).

Build steps gain -DHAVE_CURL (the engram artifact linked -lcurl but never
defined the guard, so the HTTP client + this layer were compiled out).

Evidence — rebuilt with cc -DHAVE_CURL, run on :8797 against a throwaway

7-node snapshot (never touched live :8742/:7770/~/.neuron):

Test Query Result
Semantic paraphrase how are employee wages and salaries paid returns sem-payroll ("Twice a month the organization transfers monetary compensation to its workforce…" — zero shared words)
Lexical intact Will Anderson sem-will
Lexical intact windows msi signing sem-lexctl
Activate semantic seed paraphrase, depth 2 sem-payroll seeded at hops=0, act=0.5009 (= salience 0.7 × cosine 0.716), spreads to sem-will
Degrade ENGRAM_SEMANTIC=0 paraphrase 0 (pure lexical) — lexical queries still return
Degrade Ollama down paraphrase 0, server healthy, lexical still returns

Compiles clean with -Wall -Wextra both with and without -DHAVE_CURL.

⚠️ Merge-order dependency — LEXICAL FIRST

This branch was cut from hotfix/elc-fixes before the lexical tokenize+rank
fix (fix/engram-lexical-tokenized-search, commit e3dabe3). Both branches
edit the same three match sites in el_runtime.c. Merge the lexical PR
first, then rebase this one. On rebase, the lexical predicate inside each
site (currently whole-query istr_contains) should adopt the tokenized matcher
— my changes wrap that predicate in lex || sem, so reconciliation is
mechanical. My semantic block, the ranking scaffold, and the -DHAVE_CURL
build flag are additive and independent of the lexical work.

Do not merge without the lexical PR landing first.

Documented as not-yet-done (partial scope, per brief)

  • Cache persistence across process restarts (sidecar next to snapshot) — a
    cold process still pays O(N) embeds the first time each node is scanned. In
    memory the cache already prevents per-query re-embedding.
  • Node-vec lookup is O(N) linear (fine at current graph sizes; a hash map
    is the next optimization). First semantic query on a large live graph will
    be slow until warm — a boot-time warmup or persisted cache addresses this.
## Semantic search layer — nomic-embed-text (the upgrade beyond lexical) Lexical `istr_contains` can only match text that literally appears. It cannot surface a node whose **words never appear in the query** but whose **meaning matches**. This PR adds an optional dense-vector layer: node content and the query are embedded via Ollama (`nomic-embed-text`), ranked by cosine similarity, and **unioned with lexical hits**. ### What changed (`el_runtime.c`) A self-contained semantic block (embed client, FNV-1a-keyed in-memory vector cache, cosine, one-shot probe) plus wiring into **all three** query entry points — the routing reality is that `server.el` calls the `_json` builtins, so fixing only the internal twin would never reach `:8742`: - **`engram_search_json`** (HTTP `/api/search`): collect lexical ∪ semantic candidates, score (lexical base `1.0 + cosine`; pure-semantic `= cosine`), rank, emit top-N. Stable sort ⇒ byte-identical to old output when semantic is off. - **`engram_search`** (internal `el_val_t` twin): lexical ∪ semantic union. - **`engram_activate` seed loop** (HTTP `/api/activate`): a node seeds if it lexically matches **or** clears the cosine threshold; pure-semantic seeds enter scaled by cosine so a paraphrase spreads activation without overpowering exact lexical seeds. ### Degradable by design The whole layer is gated on `HAVE_CURL` **and** a runtime probe. If curl is compiled out, Ollama is unreachable, or `ENGRAM_SEMANTIC=0`, every entry point yields zero semantic signal and callers fall back **byte-for-byte** to the pre-existing lexical search. Verified against a dead endpoint and with the flag off (see evidence). ### Where vectors live In-process cache keyed by node id, invalidated by an FNV-1a hash of content (edited content re-embeds). The **query is embedded once per call**; the graph is **not** re-embedded per query. nomic task prefixes (`search_query:` / `search_document:`) are applied — empirically they separate a paraphrase (0.72) cleanly from distractors (<0.48). ### Config `ENGRAM_SEMANTIC` (`0` disables), `ENGRAM_EMBED_URL` (def `http://localhost:11434/api/embeddings`), `ENGRAM_EMBED_MODEL` (def `nomic-embed-text`), `ENGRAM_SEMANTIC_MIN` (cosine threshold, def `0.6`). Build steps gain `-DHAVE_CURL` (the engram artifact linked `-lcurl` but never defined the guard, so the HTTP client + this layer were compiled out). ### Evidence — rebuilt with `cc -DHAVE_CURL`, run on `:8797` against a throwaway 7-node snapshot (never touched live `:8742`/`:7770`/`~/.neuron`): | Test | Query | Result | |------|-------|--------| | **Semantic paraphrase** | `how are employee wages and salaries paid` | returns `sem-payroll` (*"Twice a month the organization transfers monetary compensation to its workforce…"* — **zero shared words**) | | Lexical intact | `Will Anderson` | `sem-will` | | Lexical intact | `windows msi signing` | `sem-lexctl` | | **Activate semantic seed** | paraphrase, depth 2 | `sem-payroll` seeded at hops=0, `act=0.5009` (= salience 0.7 × cosine 0.716), spreads to `sem-will` | | Degrade `ENGRAM_SEMANTIC=0` | paraphrase | `0` (pure lexical) — lexical queries still return | | Degrade Ollama down | paraphrase | `0`, server healthy, lexical still returns | Compiles clean with `-Wall -Wextra` both **with and without** `-DHAVE_CURL`. --- ### ⚠️ Merge-order dependency — LEXICAL FIRST This branch was cut from `hotfix/elc-fixes` **before** the lexical tokenize+rank fix (`fix/engram-lexical-tokenized-search`, commit `e3dabe3`). Both branches edit the **same three match sites** in `el_runtime.c`. **Merge the lexical PR first**, then rebase this one. On rebase, the lexical predicate inside each site (currently whole-query `istr_contains`) should adopt the tokenized matcher — my changes wrap that predicate in `lex || sem`, so reconciliation is mechanical. My semantic block, the ranking scaffold, and the `-DHAVE_CURL` build flag are additive and independent of the lexical work. **Do not merge** without the lexical PR landing first. ### Documented as not-yet-done (partial scope, per brief) - **Cache persistence** across process restarts (sidecar next to snapshot) — a cold process still pays O(N) embeds the first time each node is scanned. In memory the cache already prevents per-query re-embedding. - **Node-vec lookup is O(N) linear** (fine at current graph sizes; a hash map is the next optimization). First semantic query on a large live graph will be slow until warm — a boot-time warmup or persisted cache addresses this.
will.anderson added 1 commit 2026-07-14 23:49:19 +00:00
feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical) b4967af13e 
Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.

Wired into all three query entry points in el_runtime.c:
  - engram_search_json (HTTP /api/search): collect lexical ∪ semantic
    candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
    rank, emit top-N. Stable sort preserves old order when semantic is off.
  - engram_search (internal el_val twin): lexical ∪ semantic union.
  - engram_activate seed loop (HTTP /api/activate): a node seeds if it
    lexically matches OR clears the cosine threshold; pure-semantic seeds
    enter scaled by cosine so paraphrase spreads without overpowering.

Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.

Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.

Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).
will.anderson commented 2026-07-22 21:21:29 +00:00
Author
Owner
Copy Link
Copy Source

Superseded by the reconciled el-cluster landing (feature #80 -> dev -> stage #81 -> main #82). Its content is included in that reconciled runtime now on the dev->stage->main chain. Closing to drive el open PRs to zero.

Superseded by the reconciled el-cluster landing (feature #80 -> dev -> stage #81 -> main #82). Its content is included in that reconciled runtime now on the dev->stage->main chain. Closing to drive el open PRs to zero.
will.anderson closed this pull request 2026-07-22 21:21:35 +00:00
This repo is archived. You cannot comment on pull requests.
Reviewers
No Reviewers
Labels
Clear labels
BETA-CRITICAL
On the beta critical path — must resolve before ship
 HELD
Blocked pending an external decision (counsel / dual sign-off)
 ORTHOGONAL
Parallel workstream (testing/hardening) — safe to assign off critical path
 POST-BETA
Important but not blocking the beta ship
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
andre.almeida (Andre Almeida) tim.lingo (Tim Lingo) will.anderson (Will Anderson)
No Assignees
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: neuron-technologies/el-retired#67
Delete Branch "feat/engram-semantic-search"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?