feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical) #67
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: neuron-technologies/el#67
Reference in New Issue
Block a user
Delete Branch "feat/engram-semantic-search"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Semantic search layer — nomic-embed-text (the upgrade beyond lexical)
Lexical
istr_containscan only match text that literally appears. It cannotsurface a node whose words never appear in the query but whose meaning
matches. This PR adds an optional dense-vector layer: node content and the
query are embedded via Ollama (
nomic-embed-text), ranked by cosinesimilarity, and unioned with lexical hits.
What changed (
el_runtime.c)A self-contained semantic block (embed client, FNV-1a-keyed in-memory vector
cache, cosine, one-shot probe) plus wiring into all three query entry
points — the routing reality is that
server.elcalls the_jsonbuiltins,so fixing only the internal twin would never reach
:8742:engram_search_json(HTTP/api/search): collect lexical ∪ semanticcandidates, score (lexical base
1.0 + cosine; pure-semantic= cosine),rank, emit top-N. Stable sort ⇒ byte-identical to old output when semantic
is off.
engram_search(internalel_val_ttwin): lexical ∪ semantic union.engram_activateseed loop (HTTP/api/activate): a node seeds if itlexically matches or clears the cosine threshold; pure-semantic seeds
enter scaled by cosine so a paraphrase spreads activation without
overpowering exact lexical seeds.
Degradable by design
The whole layer is gated on
HAVE_CURLand a runtime probe. If curl iscompiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry pointyields zero semantic signal and callers fall back byte-for-byte to the
pre-existing lexical search. Verified against a dead endpoint and with the
flag off (see evidence).
Where vectors live
In-process cache keyed by node id, invalidated by an FNV-1a hash of content
(edited content re-embeds). The query is embedded once per call; the graph
is not re-embedded per query. nomic task prefixes
(
search_query:/search_document:) are applied — empirically they separatea paraphrase (0.72) cleanly from distractors (<0.48).
Config
ENGRAM_SEMANTIC(0disables),ENGRAM_EMBED_URL(def
http://localhost:11434/api/embeddings),ENGRAM_EMBED_MODEL(def
nomic-embed-text),ENGRAM_SEMANTIC_MIN(cosine threshold, def0.6).Build steps gain
-DHAVE_CURL(the engram artifact linked-lcurlbut neverdefined the guard, so the HTTP client + this layer were compiled out).
Evidence — rebuilt with
cc -DHAVE_CURL, run on:8797against a throwaway7-node snapshot (never touched live
:8742/:7770/~/.neuron):how are employee wages and salaries paidsem-payroll("Twice a month the organization transfers monetary compensation to its workforce…" — zero shared words)Will Andersonsem-willwindows msi signingsem-lexctlsem-payrollseeded at hops=0,act=0.5009(= salience 0.7 × cosine 0.716), spreads tosem-willENGRAM_SEMANTIC=00(pure lexical) — lexical queries still return0, server healthy, lexical still returnsCompiles clean with
-Wall -Wextraboth with and without-DHAVE_CURL.⚠️ Merge-order dependency — LEXICAL FIRST
This branch was cut from
hotfix/elc-fixesbefore the lexical tokenize+rankfix (
fix/engram-lexical-tokenized-search, commite3dabe3). Both branchesedit the same three match sites in
el_runtime.c. Merge the lexical PRfirst, then rebase this one. On rebase, the lexical predicate inside each
site (currently whole-query
istr_contains) should adopt the tokenized matcher— my changes wrap that predicate in
lex || sem, so reconciliation ismechanical. My semantic block, the ranking scaffold, and the
-DHAVE_CURLbuild flag are additive and independent of the lexical work.
Do not merge without the lexical PR landing first.
Documented as not-yet-done (partial scope, per brief)
cold process still pays O(N) embeds the first time each node is scanned. In
memory the cache already prevents per-query re-embedding.
is the next optimization). First semantic query on a large live graph will
be slow until warm — a boot-time warmup or persisted cache addresses this.
Lexical istr_contains alone can't surface a node whose words don't appear in the query. This adds an optional dense-vector layer: node content and the query are embedded through Ollama (nomic-embed-text), and nodes are ranked by cosine similarity unioned with lexical hits, so a paraphrase query reaches the right node. Wired into all three query entry points in el_runtime.c: - engram_search_json (HTTP /api/search): collect lexical ∪ semantic candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine), rank, emit top-N. Stable sort preserves old order when semantic is off. - engram_search (internal el_val twin): lexical ∪ semantic union. - engram_activate seed loop (HTTP /api/activate): a node seeds if it lexically matches OR clears the cosine threshold; pure-semantic seeds enter scaled by cosine so paraphrase spreads without overpowering. Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot runtime probe. If curl is compiled out, Ollama is unreachable, or ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers fall back byte-for-byte to the pre-existing lexical search. Node embeddings are cached in process memory keyed by node id with an FNV-1a content hash for invalidation; the query is embedded once per call — so the graph is not re-embedded on every query. nomic task prefixes (search_query:/search_document:) are applied for retrieval separation. Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in (-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL, ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.