release: promote stage -> main (tokenized search, get_node_by_label, epm fix, win portability) #74

Merged

will.anderson merged 16 commits from stage into main 2026-07-15 18:24:41 +00:00

Conversation 0 Commits 16 Files Changed 14

+516 -34

will.anderson commented 2026-07-15 18:19:50 +00:00

Owner

Copy Link

Copy Source

Promote stage (b97ce74d) to main for production SDK publish. Payload verified in foundation-stage today (el-runtime-c/-h/-elc b97ce74d @ 13:08). sdk-release push will publish el-runtime-c/-h to foundation-prod, which neuron's soul build consumes. Authorized production artifact release (Will approved).

Promote stage (b97ce74d) to main for production SDK publish. Payload verified in foundation-stage today (el-runtime-c/-h/-elc b97ce74d @ 13:08). sdk-release push will publish el-runtime-c/-h to foundation-prod, which neuron's soul build consumes. Authorized production artifact release (Will approved).

will.anderson added 16 commits 2026-07-15 18:19:51 +00:00

Merge pull request 'fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only' (#60) from fix/windows-rusage-guard into stage ... 5c41c66a0f

fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only

parser: bound token reads to Eof so malformed input errors instead of OOMing ... 0a0a2bcb44

Out-of-range tok_kind/tok_value reads returned runtime null (el_list_get OOB
-> 0) rather than the Eof sentinel, so the inner parse loops (parse_block,
call-arg, array-literal, match-arm) that terminate only on their close
delimiter or k=="Eof" never saw Eof once the cursor ran past the single
trailing Eof token. On unclosed-delimiter input the parser then appended AST
nodes forever -> unbounded allocation -> ~700GB -> OOM (observed compiling
neuron/sessions.el).

Fix at the choke point: tok_kind returns "Eof" and tok_value returns "" for
out-of-range positions, restoring the parser-wide contract that reads at/after
the end yield Eof. expect() no longer steps past the Eof sentinel on mismatch.
This terminates every overrun loop simultaneously; a malformed program now
surfaces as a normal (best-effort) parse end instead of exhausting memory.

Requires a self-hosted bootstrap rebuild of elc to take effect.

fix(engram): tokenized + ranked lexical search, not whole-query Ctrl-F ... e3dabe3e08

engram search/activate/goal-bias matched the ENTIRE raw query string as a
single case-insensitive substring (istr_contains(field, q)). Multi-word
queries like "windows msi signing" only matched a node containing that exact
contiguous run, so real multi-word queries returned ZERO on a graph saturated
with the answer. This is Ctrl-F, not search — and search is the core of the
engram being useful.

Fix: split the query on whitespace into distinct tokens; a node matches if it
contains ANY token in content/label/tags. Rank by distinct tokens matched
(desc) then salience (desc). istr_contains is kept unchanged as the per-token
primitive. Single-token queries are a strict special case (score 0 or 1) so
the many single-word callers do not regress.

Sites changed (all in el_runtime.c):
- new helpers engram_tokenize_query / engram_node_match_score / engram_rank_cmp
- engram_search           (internal el_val_t path)
- engram_search_json      (HTTP /api/search path)
- engram_activate seed loop (HTTP /api/activate path; seed activation scaled
  by token coverage so full-query matches seed more strongly)
- engram_goal_bias overlap bonus upgraded to graded token coverage

Proof (6591-node snapshot copy, rebuilt binary on :8799, POST JSON path):
  windows msi signing  0 -> 20   Will Anderson  0 -> 20
  windows msi          0 -> 20   tokenized search fix  0 -> 20
Single-word parity preserved (VBD/volatility/elc capped at limit; unkey = all
matching nodes). Top hits are relevant (e.g. "Will Anderson" surfaces the
Project Design and VBD whitepapers).

Note: GET ?q=a%20b still returns 0 because query_param (server.el) does not
URL-decode — a separate EL-layer bug; the soul's POST-JSON path is fixed here.

feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical) ... b4967af13e

Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.

Wired into all three query entry points in el_runtime.c:
  - engram_search_json (HTTP /api/search): collect lexical ∪ semantic
    candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
    rank, emit top-N. Stable sort preserves old order when semantic is off.
  - engram_search (internal el_val twin): lexical ∪ semantic union.
  - engram_activate seed loop (HTTP /api/activate): a node seeds if it
    lexically matches OR clears the cosine threshold; pure-semantic seeds
    enter scaled by cosine so paraphrase spreads without overpowering.

Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.

Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.

Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).

Add engram_get_node_by_label runtime native to unblock soul link ... 6b9d9e6c4a

chat.el calls the runtime native engram_get_node_by_label to fetch
well-known nodes (conv:history, session:summary) by stable label rather
than by ID — immune to vector-index drift across restarts. The current
runtime never defined it, so the regenerated dist/soul.c fails to link.

Backport the function verbatim (idiom-adapted to jb_finish) from release
runtime v1.0.0-20260501 and register it as an EL builtin exactly like its
siblings: runtime definition + prototype, __-prefixed seed wrapper +
prototype, and codegen arity entry. No search-site code is touched.

runtime: make Windows soul reproducible from a clean el checkout ... 4773dd0aa2

Two el_runtime portability defects only ever lived in staged local copies
used to hand-build neuron-ui PR #136's curl-enabled Windows neuron.exe.
gcc 15 promotes both to hard errors, so a clean el checkout cannot rebuild
that soul. Upstream the minimal fixes so the build is reproducible:

- http_serve_async: cast setsockopt optval to (const char*). Win32/mingw
  setsockopt wants const char*, not int*; the cast is a no-op on POSIX and
  matches the four already-cast sites elsewhere in this file.
- engram_save persist path: map fsync -> _commit in the _WIN32-only
  el_platform_win.h (io.h already included). Windows has no fsync(); the
  POSIX path is untouched.

Merge remote-tracking branch 'origin/hotfix/elc-fixes' into hotfix/stage-elc-engram-integration 9d565ca080

Merge remote-tracking branch 'origin/hotfix/runtime-engram-get-node-by-label' into hotfix/stage-elc-engram-integration 0481bea44d

Merge remote-tracking branch 'origin/hotfix/win-runtime-portability' into hotfix/stage-elc-engram-integration 40287c4cfc

Merge remote-tracking branch 'origin/fix/engram-lexical-tokenized-search' into hotfix/stage-elc-engram-integration e8f0b5a9de

Merge remote-tracking branch 'origin/feat/engram-semantic-search' into hotfix/stage-elc-engram-integration ... 1e49560f1f

# Conflicts:
#	lang/el-compiler/runtime/el_runtime.c

epm: declare cross-module callees as extern fn so strict compilers accept generated C ... 8ce8656de2

epm's sibling modules (registry/install/update) call functions defined in other
modules and in the El runtime (config, read_installed, registry_find,
manifest_deps, manifest_name, registry_latest_version, registry_token,
install_vessel, installed_version) without importing them, so elc emits no C
prototype for those calls. gcc<=13 treated the resulting implicit declarations
as warnings; gcc>=14 and clang reject them as hard errors, which is why the
"Build epm" CI step fails and blocks the whole dev/stage pipeline.

Add `extern fn` forward declarations -- El's own separate-compilation mechanism
-- for each cross-module callee at the top of registry/install/update. This
gives elc the correct C prototype in every generated translation unit, so the
calls compile cleanly and still resolve at link time. Simply suppressing
-Wimplicit-function-declaration would be unsafe: an implicit int return
truncates the 64-bit pointer returns of config/registry_find into a latent
crash, so declaring the true signatures is the correct fix. Localized to epm;
touches neither elc nor the runtime.

Merge pull request 'integrate: stack PRs #65–#69 (elc OOM guard, tokenized+semantic engram search, get_node_by_label, win portability) for green CI' (#71) from hotfix/stage-elc-engram-integration into dev 581a351fb1

ci(dev): make SDK publish fail loudly, decouple ci-base rebuild ... 4696fd6833

The dev push build went green-then-red while nothing published: the
Publish step had no set -e, so an auth/upload failure exited 0 (silent
no-publish), while the ci-base rebuild (set -euo pipefail + Docker on the
host-mode runner) hard-failed the job. Add set -euo pipefail + an empty-key
guard + active-account echo to the Publish step so failures surface with a
retrievable log, and mark the ci-base cache rebuild continue-on-error so
the fragile Docker step can never block the actual SDK artifact publish.

Merge pull request 'ci(dev): make SDK publish fail loudly, decouple ci-base rebuild' (#72) from hotfix/ci-dev-publish-hardening into dev 155a449c4e

Merge pull request 'release: promote dev -> stage (tokenized search, get_node_by_label, epm fix)' (#73) from dev into stage b97ce74d1f

will.anderson merged commit 208e36c899 into main 2026-07-15 18:24:41 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

tim.lingo (Tim Lingo) will.anderson (Will Anderson)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: neuron-technologies/el#74