will.anderson
b4967af13e
Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.
Wired into all three query entry points in el_runtime.c:
- engram_search_json (HTTP /api/search): collect lexical ∪ semantic
candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
rank, emit top-N. Stable sort preserves old order when semantic is off.
- engram_search (internal el_val twin): lexical ∪ semantic union.
- engram_activate seed loop (HTTP /api/activate): a node seeds if it
lexically matches OR clears the cosine threshold; pure-semantic seeds
enter scaled by cosine so paraphrase spreads without overpowering.
Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.
Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.
Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).
122 lines
4.4 KiB
YAML
122 lines
4.4 KiB
YAML
name: Engram CI — dev
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- dev
|
|
pull_request:
|
|
branches:
|
|
- dev
|
|
|
|
jobs:
|
|
build-and-test:
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Install build dependencies
|
|
run: |
|
|
apt-get update -qq
|
|
apt-get install -y gcc libcurl4-openssl-dev
|
|
|
|
# Install El SDK from dev artifact registry
|
|
- name: Install El SDK from dev registry
|
|
env:
|
|
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
|
|
run: |
|
|
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
|
|
apt-get install -y -qq apt-transport-https ca-certificates gnupg curl
|
|
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
|
|
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
|
|
apt-get update -qq && apt-get install -y google-cloud-cli
|
|
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
|
|
gcloud config set project neuron-785695
|
|
|
|
mkdir -p /usr/local/bin /usr/local/lib/el
|
|
|
|
# Download the latest-versioned elc from dev registry
|
|
# Fall back to the Gitea latest release if no dev artifact exists yet
|
|
LATEST_VERSION=$(gcloud artifacts versions list \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el/elc \
|
|
--format="value(name)" 2>/dev/null | sort | tail -1 | sed 's|.*/||' || true)
|
|
|
|
if [ -n "${LATEST_VERSION}" ]; then
|
|
echo "Downloading elc ${LATEST_VERSION} from foundation-dev..."
|
|
gcloud artifacts generic download \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el/elc \
|
|
--version="${LATEST_VERSION}" \
|
|
--destination=/usr/local/bin/elc
|
|
chmod +x /usr/local/bin/elc
|
|
else
|
|
echo "No dev artifact found, falling back to Gitea latest release..."
|
|
RELEASE_BASE="https://git.neuralplatform.ai/neuron-technologies/el/releases/download/latest"
|
|
curl -fsSL "${RELEASE_BASE}/elc" -o /usr/local/bin/elc
|
|
chmod +x /usr/local/bin/elc
|
|
fi
|
|
|
|
# Always get runtime from Gitea latest release
|
|
RELEASE_BASE="https://git.neuralplatform.ai/neuron-technologies/el/releases/download/latest"
|
|
curl -fsSL "${RELEASE_BASE}/el_runtime.c" -o /usr/local/lib/el/el_runtime.c
|
|
curl -fsSL "${RELEASE_BASE}/el_runtime.h" -o /usr/local/lib/el/el_runtime.h
|
|
|
|
echo "El SDK installed:"
|
|
elc --version || true
|
|
rm -f /tmp/gcp-key.json
|
|
|
|
# Compile server.el → dist/engram.c
|
|
- name: Compile server.el
|
|
run: |
|
|
mkdir -p dist
|
|
elc src/server.el > dist/engram.c
|
|
echo "Compiled src/server.el → dist/engram.c"
|
|
|
|
# Link to produce the engram binary
|
|
- name: Link engram binary
|
|
run: |
|
|
cc -std=c11 -O2 -DHAVE_CURL \
|
|
-I /usr/local/lib/el \
|
|
-o dist/engram \
|
|
dist/engram.c \
|
|
/usr/local/lib/el/el_runtime.c \
|
|
-lcurl -lpthread
|
|
echo "Linked dist/engram"
|
|
ls -lh dist/engram
|
|
|
|
# Run tests if spec directory contains runnable tests
|
|
- name: Run tests
|
|
run: |
|
|
if [ -f spec/run.sh ]; then
|
|
bash spec/run.sh
|
|
else
|
|
echo "No spec/run.sh found — skipping test run"
|
|
fi
|
|
|
|
# Publish artifact to GCP Artifact Registry (dev)
|
|
- name: Publish engram to Artifact Registry (dev)
|
|
env:
|
|
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
|
|
run: |
|
|
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
|
|
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
|
|
gcloud config set project neuron-785695
|
|
|
|
VERSION="${GITEA_SHA:0:8}"
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=engram/engram \
|
|
--version="${VERSION}" \
|
|
--source=dist/engram
|
|
|
|
echo "Published engram version=${VERSION} to foundation-dev/engram/engram"
|
|
rm -f /tmp/gcp-key.json
|