feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical) #67

Open

will.anderson wants to merge 1 commits from feat/engram-semantic-search into hotfix/elc-fixes

Author	SHA1	Message	Date
will.anderson	b4967af13e	feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical) Lexical istr_contains alone can't surface a node whose words don't appear in the query. This adds an optional dense-vector layer: node content and the query are embedded through Ollama (nomic-embed-text), and nodes are ranked by cosine similarity unioned with lexical hits, so a paraphrase query reaches the right node. Wired into all three query entry points in el_runtime.c: - engram_search_json (HTTP /api/search): collect lexical ∪ semantic candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine), rank, emit top-N. Stable sort preserves old order when semantic is off. - engram_search (internal el_val twin): lexical ∪ semantic union. - engram_activate seed loop (HTTP /api/activate): a node seeds if it lexically matches OR clears the cosine threshold; pure-semantic seeds enter scaled by cosine so paraphrase spreads without overpowering. Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot runtime probe. If curl is compiled out, Ollama is unreachable, or ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers fall back byte-for-byte to the pre-existing lexical search. Node embeddings are cached in process memory keyed by node id with an FNV-1a content hash for invalidation; the query is embedded once per call — so the graph is not re-embedded on every query. nomic task prefixes (search_query:/search_document:) are applied for retrieval separation. Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in (-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL, ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).	2026-07-14 18:48:16 -05:00

Author

SHA1

Message

Date

will.anderson

b4967af13e

feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical)

Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.

Wired into all three query entry points in el_runtime.c:
  - engram_search_json (HTTP /api/search): collect lexical ∪ semantic
    candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
    rank, emit top-N. Stable sort preserves old order when semantic is off.
  - engram_search (internal el_val twin): lexical ∪ semantic union.
  - engram_activate seed loop (HTTP /api/activate): a node seeds if it
    lexically matches OR clears the cosine threshold; pure-semantic seeds
    enter scaled by cosine so paraphrase spreads without overpowering.

Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.

Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.

Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).

2026-07-14 18:48:16 -05:00

feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical) #67

1 Commits