Compare commits

..

21 Commits

Author SHA1 Message Date
will.anderson 43636aed99 runtime: pair fs_read length hint with its buffer in BOTH runtimes — kill response truncation for good
El SDK Release / build-and-release (pull_request) Failing after 7s
The binary-safe fs_read length (_tl_fs_read_len) was consumed by the HTTP
response path for ANY body, even when a handler wrapped a smaller file into a
larger reply. Content-Length then lied AND the send stopped short: the
safety-contact (988) routes returned 178 of 208/218 bytes, cut mid-'set_at' —
unparseable JSON. The desktop app read that as failure. On Windows the shipped
brain is an OLD build without even the per-handler workaround, so EVERY reply
truncated: the app can't read confirmations and refuses the new user.

Durable fix: pair the length hint with the exact buffer pointer it describes
(_tl_fs_read_buf). Apply the raw byte count ONLY when the response IS that
buffer (binary file serving stays correct); every wrapped/enveloped/derived
body is measured with strlen. Reset both at request start and in fs_read /
json_get_raw. This also closes the stale-hint heap over-read (a length larger
than a later body would read past it out the socket) that a plain max() leaves
open — so this class of bug dies on every platform, not just where a handler
happened to be patched.

Applied identically to the mainline runtime (lang/el-compiler/runtime) AND the
frozen release runtime (lang/releases/v1.0.0-20260501) the desktop souls
compile against — the release copy still carried the raw leak, which is why the
Windows brain kept truncating. Same proven approach as PR #78 (Tim Lingo),
extended to cover the release runtime and rebased onto current main.

Both runtimes: gcc -fsyntax-only clean.
2026-07-22 15:04:25 -05:00
will.anderson 2baa0b9a41 Merge pull request 'release: promote stage -> main (ci publish hardening for sdk-release)' (#77) from stage into main
El SDK Release / build-and-release (push) Successful in 7m55s
2026-07-15 21:21:28 +00:00
will.anderson 6a8b2461cd Merge pull request 'release: promote dev -> stage (ci publish hardening for stage/main)' (#76) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 8m19s
El SDK Release / build-and-release (pull_request) Failing after 13m1s
2026-07-15 21:16:11 +00:00
will.anderson bcb356fe69 Merge pull request 'ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly' (#75) from hotfix/ci-stage-main-publish-hardening into dev
El SDK CI - stage / build-and-test (pull_request) Successful in 4m27s
El SDK CI - dev / build-and-test (push) Failing after 14m3s
2026-07-15 21:15:27 +00:00
will.anderson dd7827059a ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly
El SDK CI - dev / build-and-test (pull_request) Failing after 14m30s
Mirror the PR #72 fix (applied to ci-dev.yaml) onto ci-stage.yaml and
sdk-release.yaml. The stage and prod release jobs reported FAILURE even
when the el-runtime-c/-h publish SUCCEEDED, because the ancillary ci-base
Docker rebuild (a CI-cache optimization on the fragile host-mode GCE
runner) reddened the whole job.

- Rebuild ci-base step: continue-on-error: true — never blocks/reddens
  the job; the SDK publish is the deliverable.
- Publish step: set -euo pipefail + empty-key guard + active-account echo
  so a real publish failure still fails loud and is diagnosable.
2026-07-15 16:14:50 -05:00
will.anderson 208e36c899 Merge pull request 'release: promote stage -> main (tokenized search, get_node_by_label, epm fix, win portability)' (#74) from stage into main
El SDK Release / build-and-release (push) Successful in 8m23s
2026-07-15 18:24:39 +00:00
will.anderson b97ce74d1f Merge pull request 'release: promote dev -> stage (tokenized search, get_node_by_label, epm fix)' (#73) from dev into stage
El SDK CI - stage / build-and-test (push) Failing after 8m45s
El SDK Release / build-and-release (pull_request) Successful in 4m1s
2026-07-15 17:20:11 +00:00
will.anderson 155a449c4e Merge pull request 'ci(dev): make SDK publish fail loudly, decouple ci-base rebuild' (#72) from hotfix/ci-dev-publish-hardening into dev
El SDK CI - dev / build-and-test (push) Successful in 8m56s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m10s
2026-07-15 16:34:14 +00:00
will.anderson 4696fd6833 ci(dev): make SDK publish fail loudly, decouple ci-base rebuild
El SDK CI - dev / build-and-test (pull_request) Successful in 8m51s
The dev push build went green-then-red while nothing published: the
Publish step had no set -e, so an auth/upload failure exited 0 (silent
no-publish), while the ci-base rebuild (set -euo pipefail + Docker on the
host-mode runner) hard-failed the job. Add set -euo pipefail + an empty-key
guard + active-account echo to the Publish step so failures surface with a
retrievable log, and mark the ci-base cache rebuild continue-on-error so
the fragile Docker step can never block the actual SDK artifact publish.
2026-07-15 11:33:37 -05:00
will.anderson 581a351fb1 Merge pull request 'integrate: stack PRs #65–#69 (elc OOM guard, tokenized+semantic engram search, get_node_by_label, win portability) for green CI' (#71) from hotfix/stage-elc-engram-integration into dev
El SDK CI - dev / build-and-test (push) Failing after 14m31s
2026-07-15 15:49:57 +00:00
will.anderson 8ce8656de2 epm: declare cross-module callees as extern fn so strict compilers accept generated C
El SDK CI - dev / build-and-test (pull_request) Successful in 7m33s
epm's sibling modules (registry/install/update) call functions defined in other
modules and in the El runtime (config, read_installed, registry_find,
manifest_deps, manifest_name, registry_latest_version, registry_token,
install_vessel, installed_version) without importing them, so elc emits no C
prototype for those calls. gcc<=13 treated the resulting implicit declarations
as warnings; gcc>=14 and clang reject them as hard errors, which is why the
"Build epm" CI step fails and blocks the whole dev/stage pipeline.

Add `extern fn` forward declarations -- El's own separate-compilation mechanism
-- for each cross-module callee at the top of registry/install/update. This
gives elc the correct C prototype in every generated translation unit, so the
calls compile cleanly and still resolve at link time. Simply suppressing
-Wimplicit-function-declaration would be unsafe: an implicit int return
truncates the 64-bit pointer returns of config/registry_find into a latent
crash, so declaring the true signatures is the correct fix. Localized to epm;
touches neither elc nor the runtime.
2026-07-15 10:14:43 -05:00
will.anderson 1e49560f1f Merge remote-tracking branch 'origin/feat/engram-semantic-search' into hotfix/stage-elc-engram-integration
El SDK CI - dev / build-and-test (pull_request) Failing after 14m39s
# Conflicts:
#	lang/el-compiler/runtime/el_runtime.c
2026-07-15 09:33:05 -05:00
will.anderson e8f0b5a9de Merge remote-tracking branch 'origin/fix/engram-lexical-tokenized-search' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 40287c4cfc Merge remote-tracking branch 'origin/hotfix/win-runtime-portability' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 0481bea44d Merge remote-tracking branch 'origin/hotfix/runtime-engram-get-node-by-label' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 9d565ca080 Merge remote-tracking branch 'origin/hotfix/elc-fixes' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 4773dd0aa2 runtime: make Windows soul reproducible from a clean el checkout
El SDK Release / build-and-release (pull_request) Failing after 16s
Two el_runtime portability defects only ever lived in staged local copies
used to hand-build neuron-ui PR #136's curl-enabled Windows neuron.exe.
gcc 15 promotes both to hard errors, so a clean el checkout cannot rebuild
that soul. Upstream the minimal fixes so the build is reproducible:

- http_serve_async: cast setsockopt optval to (const char*). Win32/mingw
  setsockopt wants const char*, not int*; the cast is a no-op on POSIX and
  matches the four already-cast sites elsewhere in this file.
- engram_save persist path: map fsync -> _commit in the _WIN32-only
  el_platform_win.h (io.h already included). Windows has no fsync(); the
  POSIX path is untouched.
2026-07-15 04:24:08 -05:00
will.anderson 6b9d9e6c4a Add engram_get_node_by_label runtime native to unblock soul link
El SDK Release / build-and-release (pull_request) Failing after 22s
chat.el calls the runtime native engram_get_node_by_label to fetch
well-known nodes (conv:history, session:summary) by stable label rather
than by ID — immune to vector-index drift across restarts. The current
runtime never defined it, so the regenerated dist/soul.c fails to link.

Backport the function verbatim (idiom-adapted to jb_finish) from release
runtime v1.0.0-20260501 and register it as an EL builtin exactly like its
siblings: runtime definition + prototype, __-prefixed seed wrapper +
prototype, and codegen arity entry. No search-site code is touched.
2026-07-15 04:07:33 -05:00
will.anderson b4967af13e feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical)
Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.

Wired into all three query entry points in el_runtime.c:
  - engram_search_json (HTTP /api/search): collect lexical ∪ semantic
    candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
    rank, emit top-N. Stable sort preserves old order when semantic is off.
  - engram_search (internal el_val twin): lexical ∪ semantic union.
  - engram_activate seed loop (HTTP /api/activate): a node seeds if it
    lexically matches OR clears the cosine threshold; pure-semantic seeds
    enter scaled by cosine so paraphrase spreads without overpowering.

Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.

Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.

Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).
2026-07-14 18:48:16 -05:00
will.anderson 2b2a1246e7 Merge pull request 'runtime: fix the memory-leak + write-corruption pair in el_runtime.c' (#64) from hotfix/el-runtime-leak-and-persist into main
El SDK Release / build-and-release (push) Failing after 10m52s
2026-07-13 21:23:31 +00:00
will.anderson 5c41c66a0f Merge pull request 'fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only' (#60) from fix/windows-rusage-guard into stage
El SDK CI - stage / build-and-test (push) Failing after 13m21s
fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only
2026-06-25 16:48:13 +00:00
19 changed files with 765 additions and 1202 deletions
+15
View File
@@ -214,9 +214,18 @@ jobs:
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: | run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695 gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}" VERSION="${GITHUB_SHA:0:8}"
@@ -268,6 +277,12 @@ jobs:
# Patches ci-base:dev in-place: pulls the existing image (which has all # Patches ci-base:dev in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly # system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental. # built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push' if: github.event_name == 'push'
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+15
View File
@@ -212,12 +212,21 @@ jobs:
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: | run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695 gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}" VERSION="${GITHUB_SHA:0:8}"
@@ -253,6 +262,12 @@ jobs:
# Patches ci-base:stage in-place: pulls the existing image (which has all # Patches ci-base:stage in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly # system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental. # built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push' if: github.event_name == 'push'
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+15
View File
@@ -288,12 +288,21 @@ jobs:
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: | run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695 gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}" VERSION="${GITHUB_SHA:0:8}"
@@ -345,6 +354,12 @@ jobs:
# Patches ci-base:latest in-place: pulls the existing image (which has all # Patches ci-base:latest in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly # system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental. # built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push' if: github.event_name == 'push'
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+1 -1
View File
@@ -81,7 +81,7 @@ jobs:
# Link to produce the engram binary # Link to produce the engram binary
- name: Link engram binary - name: Link engram binary
run: | run: |
cc -std=c11 -O2 \ cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \ -I /usr/local/lib/el \
-o dist/engram \ -o dist/engram \
dist/engram.c \ dist/engram.c \
+1 -1
View File
@@ -88,7 +88,7 @@ jobs:
# Link to produce the engram binary # Link to produce the engram binary
- name: Link engram binary - name: Link engram binary
run: | run: |
cc -std=c11 -O2 \ cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \ -I /usr/local/lib/el \
-o dist/engram \ -o dist/engram \
dist/engram.c \ dist/engram.c \
+1 -1
View File
@@ -62,7 +62,7 @@ jobs:
# Link to produce the engram binary # Link to produce the engram binary
- name: Link engram binary - name: Link engram binary
run: | run: |
cc -std=c11 -O2 \ cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \ -I /usr/local/lib/el \
-o dist/engram \ -o dist/engram \
dist/engram.c \ dist/engram.c \
BIN
View File
Binary file not shown.
+95 -142
View File
@@ -10,7 +10,6 @@ el_val_t query_param(el_val_t path, el_val_t key);
el_val_t query_int(el_val_t path, el_val_t key, el_val_t default_val); el_val_t query_int(el_val_t path, el_val_t key, el_val_t default_val);
el_val_t extract_id(el_val_t path, el_val_t prefix); el_val_t extract_id(el_val_t path, el_val_t prefix);
el_val_t route_stats(el_val_t method, el_val_t path, el_val_t body); el_val_t route_stats(el_val_t method, el_val_t path, el_val_t body);
el_val_t persist_canonical(void);
el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body); el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_get_node(el_val_t method, el_val_t path, el_val_t body); el_val_t route_get_node(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_scan_nodes(el_val_t method, el_val_t path, el_val_t body); el_val_t route_scan_nodes(el_val_t method, el_val_t path, el_val_t body);
@@ -21,23 +20,18 @@ el_val_t route_create_edge(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neighbors(el_val_t method, el_val_t path, el_val_t body); el_val_t route_neighbors(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_strengthen(el_val_t method, el_val_t path, el_val_t body); el_val_t route_strengthen(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_forget(el_val_t method, el_val_t path, el_val_t body); el_val_t route_forget(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_create_ise(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_save(el_val_t method, el_val_t path, el_val_t body); el_val_t route_save(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_load(el_val_t method, el_val_t path, el_val_t body); el_val_t route_load(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_health(el_val_t method, el_val_t path, el_val_t body); el_val_t route_health(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_load_merge(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_emit_ise(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_capture_knowledge(el_val_t method, el_val_t path, el_val_t body);
el_val_t check_auth_ok(el_val_t method, el_val_t body); el_val_t check_auth_ok(el_val_t method, el_val_t body);
el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body); el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body);
el_val_t bind_raw;
el_val_t bind_str; el_val_t bind_str;
el_val_t port; el_val_t port;
el_val_t data_dir_raw;
el_val_t data_dir; el_val_t data_dir;
el_val_t snapshot_path; el_val_t snapshot_path;
el_val_t boot_snap;
el_val_t parse_port(el_val_t bind) { el_val_t parse_port(el_val_t bind) {
el_val_t colon = str_index_of(bind, EL_STR(":")); el_val_t colon = str_index_of(bind, EL_STR(":"));
@@ -116,22 +110,17 @@ el_val_t route_stats(el_val_t method, el_val_t path, el_val_t body) {
return 0; return 0;
} }
el_val_t persist_canonical(void) {
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_1 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_1 = (EL_STR("/tmp/engram")); } else { _if_result_1 = (dir_raw); } _if_result_1; });
engram_save(el_str_concat(dir, EL_STR("/snapshot.json")));
return 1;
return 0;
}
el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body) {
el_val_t content = json_get_string(body, EL_STR("content")); el_val_t content = json_get_string(body, EL_STR("content"));
el_val_t nt_raw = json_get_string(body, EL_STR("node_type")); el_val_t node_type = json_get_string(body, EL_STR("node_type"));
el_val_t node_type = ({ el_val_t _if_result_2 = 0; if (str_eq(nt_raw, EL_STR(""))) { _if_result_2 = (EL_STR("Memory")); } else { _if_result_2 = (nt_raw); } _if_result_2; }); if (str_eq(node_type, EL_STR(""))) {
el_val_t sal_raw = json_get_float(body, EL_STR("salience")); node_type = EL_STR("Memory");
el_val_t salience = ({ el_val_t _if_result_3 = 0; if ((sal_raw == el_from_float(0.0))) { _if_result_3 = (el_from_float(0.5)); } else { _if_result_3 = (sal_raw); } _if_result_3; }); }
el_val_t salience = json_get_float(body, EL_STR("salience"));
if (salience == el_from_float(0.0)) {
salience = el_from_float(0.5);
}
el_val_t id = engram_node(content, node_type, salience); el_val_t id = engram_node(content, node_type, salience);
el_val_t saved = persist_canonical();
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"id\":\""), id), EL_STR("\",\"content\":\"")), content), EL_STR("\",\"node_type\":\"")), node_type), EL_STR("\"}")); return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"id\":\""), id), EL_STR("\",\"content\":\"")), content), EL_STR("\",\"node_type\":\"")), node_type), EL_STR("\"}"));
return 0; return 0;
} }
@@ -157,9 +146,11 @@ el_val_t route_scan_nodes(el_val_t method, el_val_t path, el_val_t body) {
} }
el_val_t route_scan_edges(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_scan_edges(el_val_t method, el_val_t path, el_val_t body) {
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR")); el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_4 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_4 = (EL_STR("/tmp/engram")); } else { _if_result_4 = (dir_raw); } _if_result_4; }); if (str_eq(dir, EL_STR(""))) {
el_val_t snap_path = el_str_concat(dir, EL_STR("/.scan-export.json")); dir = EL_STR("/tmp/engram");
}
el_val_t snap_path = el_str_concat(dir, EL_STR("/snapshot.json"));
engram_save(snap_path); engram_save(snap_path);
el_val_t snap = fs_read(snap_path); el_val_t snap = fs_read(snap_path);
if (str_eq(snap, EL_STR(""))) { if (str_eq(snap, EL_STR(""))) {
@@ -174,22 +165,36 @@ el_val_t route_scan_edges(el_val_t method, el_val_t path, el_val_t body) {
} }
el_val_t route_search(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_search(el_val_t method, el_val_t path, el_val_t body) {
el_val_t q = ({ el_val_t _if_result_5 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_5 = (query_param(path, EL_STR("q"))); } else { _if_result_5 = (json_get_string(body, EL_STR("query"))); } _if_result_5; }); el_val_t q = EL_STR("");
el_val_t lim_url = query_int(path, EL_STR("limit"), 0); if (str_eq(method, EL_STR("GET"))) {
el_val_t lim_body = json_get_int(body, EL_STR("limit")); q = query_param(path, EL_STR("q"));
el_val_t lim_either = ({ el_val_t _if_result_6 = 0; if ((lim_url > 0)) { _if_result_6 = (lim_url); } else { _if_result_6 = (lim_body); } _if_result_6; }); } else {
el_val_t limit = ({ el_val_t _if_result_7 = 0; if ((lim_either > 0)) { _if_result_7 = (lim_either); } else { _if_result_7 = (20); } _if_result_7; }); q = json_get_string(body, EL_STR("query"));
}
el_val_t limit = query_int(path, EL_STR("limit"), 20);
if (limit == 0) {
limit = json_get_int(body, EL_STR("limit"));
}
if (limit == 0) {
limit = 20;
}
return engram_search_json(q, limit); return engram_search_json(q, limit);
return 0; return 0;
} }
el_val_t route_activate(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_activate(el_val_t method, el_val_t path, el_val_t body) {
el_val_t q = ({ el_val_t _if_result_8 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_8 = (query_param(path, EL_STR("q"))); } else { _if_result_8 = (json_get_string(body, EL_STR("query"))); } _if_result_8; }); el_val_t q = EL_STR("");
if (str_eq(q, EL_STR(""))) { el_val_t depth = 3;
return err_json(EL_STR("missing query")); if (str_eq(method, EL_STR("GET"))) {
q = query_param(path, EL_STR("q"));
depth = query_int(path, EL_STR("depth"), 3);
} else {
q = json_get_string(body, EL_STR("query"));
el_val_t bd = json_get_int(body, EL_STR("depth"));
if (bd > 0) {
depth = bd;
}
} }
el_val_t d_raw = ({ el_val_t _if_result_9 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_9 = (query_int(path, EL_STR("depth"), 3)); } else { _if_result_9 = (json_get_int(body, EL_STR("depth"))); } _if_result_9; });
el_val_t depth = ({ el_val_t _if_result_10 = 0; if ((d_raw > 0)) { _if_result_10 = (d_raw); } else { _if_result_10 = (3); } _if_result_10; });
return el_str_concat(el_str_concat(EL_STR("{\"results\":"), engram_activate_json(q, depth)), EL_STR("}")); return el_str_concat(el_str_concat(EL_STR("{\"results\":"), engram_activate_json(q, depth)), EL_STR("}"));
return 0; return 0;
} }
@@ -197,12 +202,15 @@ el_val_t route_activate(el_val_t method, el_val_t path, el_val_t body) {
el_val_t route_create_edge(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_create_edge(el_val_t method, el_val_t path, el_val_t body) {
el_val_t from_id = json_get_string(body, EL_STR("from_id")); el_val_t from_id = json_get_string(body, EL_STR("from_id"));
el_val_t to_id = json_get_string(body, EL_STR("to_id")); el_val_t to_id = json_get_string(body, EL_STR("to_id"));
el_val_t rel_raw = json_get_string(body, EL_STR("relation")); el_val_t relation = json_get_string(body, EL_STR("relation"));
el_val_t relation = ({ el_val_t _if_result_11 = 0; if (str_eq(rel_raw, EL_STR(""))) { _if_result_11 = (EL_STR("associates")); } else { _if_result_11 = (rel_raw); } _if_result_11; }); if (str_eq(relation, EL_STR(""))) {
el_val_t w_raw = json_get_float(body, EL_STR("weight")); relation = EL_STR("associates");
el_val_t weight = ({ el_val_t _if_result_12 = 0; if ((w_raw == el_from_float(0.0))) { _if_result_12 = (el_from_float(0.5)); } else { _if_result_12 = (w_raw); } _if_result_12; }); }
el_val_t weight = json_get_float(body, EL_STR("weight"));
if (weight == el_from_float(0.0)) {
weight = el_from_float(0.5);
}
engram_connect(from_id, to_id, weight, relation); engram_connect(from_id, to_id, weight, relation);
el_val_t saved = persist_canonical();
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"from_id\":\""), from_id), EL_STR("\",\"to_id\":\"")), to_id), EL_STR("\",\"relation\":\"")), relation), EL_STR("\"}")); return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"from_id\":\""), from_id), EL_STR("\",\"to_id\":\"")), to_id), EL_STR("\",\"relation\":\"")), relation), EL_STR("\"}"));
return 0; return 0;
} }
@@ -223,7 +231,6 @@ el_val_t route_strengthen(el_val_t method, el_val_t path, el_val_t body) {
return err_json(EL_STR("missing node_id")); return err_json(EL_STR("missing node_id"));
} }
engram_strengthen(id); engram_strengthen(id);
el_val_t saved = persist_canonical();
return ok_json(); return ok_json();
return 0; return 0;
} }
@@ -234,40 +241,29 @@ el_val_t route_forget(el_val_t method, el_val_t path, el_val_t body) {
return err_json(EL_STR("missing id")); return err_json(EL_STR("missing id"));
} }
engram_forget(id); engram_forget(id);
el_val_t saved = persist_canonical();
return ok_json(); return ok_json();
return 0; return 0;
} }
el_val_t route_save(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_create_ise(el_val_t method, el_val_t path, el_val_t body) {
el_val_t p_raw = json_get_string(body, EL_STR("path")); el_val_t content = json_get_string(body, EL_STR("content"));
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR")); if (str_eq(content, EL_STR(""))) {
el_val_t dir = ({ el_val_t _if_result_13 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_13 = (EL_STR("/tmp/engram")); } else { _if_result_13 = (dir_raw); } _if_result_13; }); return err_json(EL_STR("missing content"));
el_val_t p = ({ el_val_t _if_result_14 = 0; if (str_eq(p_raw, EL_STR(""))) { _if_result_14 = (el_str_concat(dir, EL_STR("/snapshot.json"))); } else { _if_result_14 = (p_raw); } _if_result_14; }); }
engram_save(p); el_val_t sal = el_from_float(0.3);
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"path\":\""), p), EL_STR("\"}")); el_val_t imp = el_from_float(0.3);
return 0; el_val_t conf = el_from_float(0.8);
} el_val_t id = engram_node_full(content, EL_STR("InternalStateEvent"), EL_STR("state-event"), sal, imp, conf, EL_STR("Episodic"), EL_STR("[\"internal-state\",\"InternalStateEvent\"]"));
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\"}"));
el_val_t route_load(el_val_t method, el_val_t path, el_val_t body) {
el_val_t p_raw = json_get_string(body, EL_STR("path"));
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_15 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_15 = (EL_STR("/tmp/engram")); } else { _if_result_15 = (dir_raw); } _if_result_15; });
el_val_t p = ({ el_val_t _if_result_16 = 0; if (str_eq(p_raw, EL_STR(""))) { _if_result_16 = (el_str_concat(dir, EL_STR("/snapshot.json"))); } else { _if_result_16 = (p_raw); } _if_result_16; });
engram_load(p);
return ok_json();
return 0;
}
el_val_t route_health(el_val_t method, el_val_t path, el_val_t body) {
return EL_STR("{\"status\":\"ok\",\"engine\":\"engram-runtime-native\"}");
return 0; return 0;
} }
el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body) {
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR")); el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_17 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_17 = (EL_STR("/tmp/engram")); } else { _if_result_17 = (dir_raw); } _if_result_17; }); if (str_eq(dir, EL_STR(""))) {
el_val_t snap_path = el_str_concat(dir, EL_STR("/.sync-export.json")); dir = EL_STR("/tmp/engram");
}
el_val_t snap_path = el_str_concat(dir, EL_STR("/sync-export.json"));
engram_save(snap_path); engram_save(snap_path);
el_val_t snap = fs_read(snap_path); el_val_t snap = fs_read(snap_path);
if (str_eq(snap, EL_STR(""))) { if (str_eq(snap, EL_STR(""))) {
@@ -277,68 +273,36 @@ el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body) {
return 0; return 0;
} }
el_val_t route_load_merge(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_save(el_val_t method, el_val_t path, el_val_t body) {
el_val_t p = json_get_string(body, EL_STR("path")); el_val_t p = json_get_string(body, EL_STR("path"));
if (str_eq(p, EL_STR(""))) { if (str_eq(p, EL_STR(""))) {
return err_json(EL_STR("path is required")); el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
p = el_str_concat(dir, EL_STR("/snapshot.json"));
} }
if (str_eq(fs_read(p), EL_STR(""))) { engram_save(p);
return err_json(EL_STR("file missing or empty")); return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"path\":\""), p), EL_STR("\"}"));
}
el_val_t before_n = engram_node_count();
el_val_t before_e = engram_edge_count();
engram_load_merge(p);
el_val_t added_n = (engram_node_count() - before_n);
el_val_t added_e = (engram_edge_count() - before_e);
el_val_t saved = persist_canonical();
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"nodes_added\":"), int_to_str(added_n)), EL_STR(",\"edges_added\":")), int_to_str(added_e)), EL_STR(",\"node_count\":")), int_to_str(engram_node_count())), EL_STR("}"));
return 0; return 0;
} }
el_val_t route_emit_ise(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_load(el_val_t method, el_val_t path, el_val_t body) {
el_val_t content = json_get_string(body, EL_STR("content")); el_val_t p = json_get_string(body, EL_STR("path"));
if (str_eq(content, EL_STR(""))) { if (str_eq(p, EL_STR(""))) {
return err_json(EL_STR("missing content")); el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
p = el_str_concat(dir, EL_STR("/snapshot.json"));
} }
el_val_t sal = el_from_float(0.3); engram_load(p);
el_val_t imp = el_from_float(0.3); return ok_json();
el_val_t conf = el_from_float(0.8);
el_val_t id = engram_node_full(content, EL_STR("InternalStateEvent"), EL_STR("state-event"), sal, imp, conf, EL_STR("Episodic"), EL_STR("[\"internal-state\",\"InternalStateEvent\"]"));
el_val_t ret_raw = env(EL_STR("ENGRAM_ISE_RETENTION_MS"));
el_val_t ret_ms = ({ el_val_t _if_result_18 = 0; if (str_eq(ret_raw, EL_STR(""))) { _if_result_18 = (172800000); } else { _if_result_18 = (str_to_int(ret_raw)); } _if_result_18; });
el_val_t pruned = engram_prune_telemetry(ret_ms);
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\",\"pruned\":")), int_to_str(pruned)), EL_STR("}"));
return 0; return 0;
} }
el_val_t route_capture_knowledge(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_health(el_val_t method, el_val_t path, el_val_t body) {
el_val_t content = json_get_string(body, EL_STR("content")); return EL_STR("{\"status\":\"ok\",\"engine\":\"engram-runtime-native\"}");
if (str_eq(content, EL_STR(""))) {
return err_json(EL_STR("missing content"));
}
el_val_t title = json_get_string(body, EL_STR("title"));
el_val_t label = ({ el_val_t _if_result_19 = 0; if (str_eq(title, EL_STR(""))) { _if_result_19 = (str_slice(content, 0, 60)); } else { _if_result_19 = (title); } _if_result_19; });
el_val_t category_raw = json_get_string(body, EL_STR("category"));
el_val_t category = ({ el_val_t _if_result_20 = 0; if (str_eq(category_raw, EL_STR(""))) { _if_result_20 = (EL_STR("other")); } else { _if_result_20 = (category_raw); } _if_result_20; });
el_val_t ktier_raw = json_get_string(body, EL_STR("tier"));
el_val_t ktier = ({ el_val_t _if_result_21 = 0; if (str_eq(ktier_raw, EL_STR(""))) { _if_result_21 = (EL_STR("note")); } else { _if_result_21 = (ktier_raw); } _if_result_21; });
el_val_t project = json_get_string(body, EL_STR("project"));
el_val_t tags_raw = json_get_raw(body, EL_STR("tags"));
el_val_t tags_base = ({ el_val_t _if_result_22 = 0; if (str_eq(tags_raw, EL_STR(""))) { _if_result_22 = (EL_STR("[]")); } else { _if_result_22 = (tags_raw); } _if_result_22; });
el_val_t base_len = str_len(tags_base);
el_val_t head = str_slice(tags_base, 0, (base_len - 1));
el_val_t sep = ({ el_val_t _if_result_23 = 0; if (str_eq(head, EL_STR("["))) { _if_result_23 = (EL_STR("")); } else { _if_result_23 = (EL_STR(",")); } _if_result_23; });
el_val_t safe_cat = str_replace(category, EL_STR("\""), EL_STR("'"));
el_val_t safe_tier = str_replace(ktier, EL_STR("\""), EL_STR("'"));
el_val_t safe_proj = str_replace(project, EL_STR("\""), EL_STR("'"));
el_val_t proj_tag = ({ el_val_t _if_result_24 = 0; if (str_eq(safe_proj, EL_STR(""))) { _if_result_24 = (EL_STR("")); } else { _if_result_24 = (el_str_concat(el_str_concat(EL_STR(",\"project:"), safe_proj), EL_STR("\""))); } _if_result_24; });
el_val_t tags = el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(head, sep), EL_STR("\"category:")), safe_cat), EL_STR("\",\"tier:")), safe_tier), EL_STR("\"")), proj_tag), EL_STR("]"));
el_val_t sal = el_from_float(0.5);
el_val_t imp = el_from_float(0.5);
el_val_t conf = el_from_float(0.9);
el_val_t id = engram_node_full(content, EL_STR("Knowledge"), label, sal, imp, conf, EL_STR("Semantic"), tags);
el_val_t saved = persist_canonical();
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\"}"));
return 0; return 0;
} }
@@ -365,15 +329,12 @@ el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body) {
return route_health(method, path, body); return route_health(method, path, body);
} }
} }
if (str_eq(method, EL_STR("POST")) && str_eq(clean, EL_STR("/api/neuron/state-events"))) { if (str_eq(method, EL_STR("POST")) && str_starts_with(clean, EL_STR("/api/neuron/state-events"))) {
return route_emit_ise(method, path, body); return route_create_ise(method, path, body);
} }
if (!check_auth_ok(method, body)) { if (!check_auth_ok(method, body)) {
return err_json(EL_STR("unauthorized")); return err_json(EL_STR("unauthorized"));
} }
if (str_eq(method, EL_STR("POST")) && str_eq(clean, EL_STR("/api/neuron/knowledge/capture"))) {
return route_capture_knowledge(method, path, body);
}
if (str_eq(method, EL_STR("GET")) && (str_eq(clean, EL_STR("/api/stats")) || str_eq(clean, EL_STR("/stats")))) { if (str_eq(method, EL_STR("GET")) && (str_eq(clean, EL_STR("/api/stats")) || str_eq(clean, EL_STR("/stats")))) {
return route_stats(method, path, body); return route_stats(method, path, body);
} }
@@ -413,40 +374,32 @@ el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body) {
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/strengthen")) || str_eq(clean, EL_STR("/strengthen")))) { if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/strengthen")) || str_eq(clean, EL_STR("/strengthen")))) {
return route_strengthen(method, path, body); return route_strengthen(method, path, body);
} }
if (str_eq(method, EL_STR("GET")) && (str_eq(clean, EL_STR("/api/sync")) || str_eq(clean, EL_STR("/sync")))) {
return route_sync(method, path, body);
}
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/save")) || str_eq(clean, EL_STR("/save")))) { if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/save")) || str_eq(clean, EL_STR("/save")))) {
return route_save(method, path, body); return route_save(method, path, body);
} }
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/load")) || str_eq(clean, EL_STR("/load")))) { if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/load")) || str_eq(clean, EL_STR("/load")))) {
return route_load(method, path, body); return route_load(method, path, body);
} }
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/load-merge")) || str_eq(clean, EL_STR("/load-merge")))) {
return route_load_merge(method, path, body);
}
if (str_eq(method, EL_STR("GET")) && str_eq(clean, EL_STR("/api/sync"))) {
return route_sync(method, path, body);
}
return el_str_concat(el_str_concat(EL_STR("{\"error\":\"not found\",\"path\":\""), clean), EL_STR("\"}")); return el_str_concat(el_str_concat(EL_STR("{\"error\":\"not found\",\"path\":\""), clean), EL_STR("\"}"));
return 0; return 0;
} }
int main(int _argc, char** _argv) { int main(int _argc, char** _argv) {
el_runtime_init_args(_argc, _argv); el_runtime_init_args(_argc, _argv);
bind_raw = env(EL_STR("ENGRAM_BIND")); bind_str = env(EL_STR("ENGRAM_BIND"));
bind_str = ({ el_val_t _if_result_25 = 0; if (str_eq(bind_raw, EL_STR(""))) { _if_result_25 = (EL_STR(":8742")); } else { _if_result_25 = (bind_raw); } _if_result_25; }); if (str_eq(bind_str, EL_STR(""))) {
bind_str = EL_STR(":8742");
}
port = parse_port(bind_str); port = parse_port(bind_str);
data_dir_raw = env(EL_STR("ENGRAM_DATA_DIR")); data_dir = env(EL_STR("ENGRAM_DATA_DIR"));
data_dir = ({ el_val_t _if_result_26 = 0; if (str_eq(data_dir_raw, EL_STR(""))) { _if_result_26 = (EL_STR("/tmp/engram")); } else { _if_result_26 = (data_dir_raw); } _if_result_26; }); if (str_eq(data_dir, EL_STR(""))) {
data_dir = EL_STR("/tmp/engram");
}
snapshot_path = el_str_concat(data_dir, EL_STR("/snapshot.json")); snapshot_path = el_str_concat(data_dir, EL_STR("/snapshot.json"));
engram_load(snapshot_path); engram_load(snapshot_path);
boot_snap = fs_read(snapshot_path);
if (!str_eq(boot_snap, EL_STR(""))) {
if (engram_node_count() == 0) {
println(EL_STR("[engram] WARNING: snapshot.json is non-empty but load produced 0 nodes \xe2\x80\x94 preserving copy at snapshot.failed-load.json"));
fs_write(el_str_concat(data_dir, EL_STR("/snapshot.failed-load.json")), boot_snap);
} else {
fs_write(el_str_concat(data_dir, EL_STR("/snapshot.boot-backup.json")), boot_snap);
}
}
println(EL_STR("[engram] runtime-native graph engine")); println(EL_STR("[engram] runtime-native graph engine"));
println(el_str_concat(EL_STR("[engram] data_dir="), data_dir)); println(el_str_concat(EL_STR("[engram] data_dir="), data_dir));
println(el_str_concat(EL_STR("[engram] node_count="), int_to_str(engram_node_count()))); println(el_str_concat(EL_STR("[engram] node_count="), int_to_str(engram_node_count())));
+54 -190
View File
@@ -76,43 +76,13 @@ fn route_stats(method: String, path: String, body: String) -> String {
engram_stats_json() engram_stats_json()
} }
// (2026-07-18 self-review) Scoping sweep: `let` inside an if-block creates an
// inner scope only it does NOT mutate the outer binding (documented with
// evidence in awareness.el, 2026-05-25). Every default/reassignment below used
// that broken pattern, so defaults never applied: nodes were created with
// node_type="" and salience=0.0, /api/search and /api/activate ALWAYS ran with
// q="" regardless of input, edges defaulted to relation=""/weight=0.0, and
// save/load with no "path" hit engram_save(""). Rewritten to the
// `let x = if cond { a } else { b }` expression form (the pattern the newer
// routes route_emit_ise/route_capture_knowledge already use correctly).
// persist_canonical save the canonical snapshot after a durable write.
//
// WHY (2026-07-22 self-review): the 2026-07-21 fix correctly stopped READ
// routes from writing the canonical snapshot.json but nothing was left
// that saved it on WRITE. Every mutation (node create, edge create,
// knowledge capture, forget, merge) lived only in RAM until someone POSTed
// /api/save manually; a process restart silently discarded everything since
// the last manual save. Observed live: two engram restarts during the
// 2026-07-22 review reverted the store to a ~17h-old snapshot, destroying
// same-day writes. Reads must never write the canonical; writes must always
// persist it. ISE telemetry is deliberately excluded (48h-pruned, loss-
// tolerant, ~2/min snapshotting the whole store per heartbeat is waste;
// any durable write that follows persists the pruning too).
fn persist_canonical() -> Int {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw }
engram_save(dir + "/snapshot.json")
return 1
}
fn route_create_node(method: String, path: String, body: String) -> String { fn route_create_node(method: String, path: String, body: String) -> String {
let content: String = json_get_string(body, "content") let content: String = json_get_string(body, "content")
let nt_raw: String = json_get_string(body, "node_type") let node_type: String = json_get_string(body, "node_type")
let node_type: String = if str_eq(nt_raw, "") { "Memory" } else { nt_raw } if str_eq(node_type, "") { let node_type = "Memory" }
let sal_raw: Float = json_get_float(body, "salience") let salience: Float = json_get_float(body, "salience")
let salience: Float = if sal_raw == 0.0 { 0.5 } else { sal_raw } if salience == 0.0 { let salience = 0.5 }
let id: String = engram_node(content, node_type, salience) let id: String = engram_node(content, node_type, salience)
let saved: Int = persist_canonical()
"{\"id\":\"" + id + "\",\"content\":\"" + content + "\",\"node_type\":\"" + node_type + "\"}" "{\"id\":\"" + id + "\",\"content\":\"" + content + "\",\"node_type\":\"" + node_type + "\"}"
} }
@@ -133,14 +103,13 @@ fn route_scan_nodes(method: String, path: String, body: String) -> String {
} }
// route_scan_edges bulk export of all edges as a JSON array. Implemented // route_scan_edges bulk export of all edges as a JSON array. Implemented
// via engram_save fs_read of a SCRATCH export path. (2026-07-21 self-review: // via engram_save fs_read of the canonical on-disk snapshot, which the
// previously this saved over the canonical snapshot.json on every GET if the // runtime keeps in lockstep with the in-memory graph. Live against the
// process ever booted with a partial/empty store, the first read request // running graph, not a stale export.
// clobbered the good snapshot. Read routes must never write the canonical path.)
fn route_scan_edges(method: String, path: String, body: String) -> String { fn route_scan_edges(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR") let dir: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw } if str_eq(dir, "") { let dir = "/tmp/engram" }
let snap_path: String = dir + "/.scan-export.json" let snap_path: String = dir + "/snapshot.json"
engram_save(snap_path) engram_save(snap_path)
let snap: String = fs_read(snap_path) let snap: String = fs_read(snap_path)
if str_eq(snap, "") { return "[]" } if str_eq(snap, "") { return "[]" }
@@ -153,34 +122,40 @@ fn route_scan_edges(method: String, path: String, body: String) -> String {
} }
fn route_search(method: String, path: String, body: String) -> String { fn route_search(method: String, path: String, body: String) -> String {
let q: String = if str_eq(method, "GET") { query_param(path, "q") } else { json_get_string(body, "query") } let q: String = ""
let lim_url: Int = query_int(path, "limit", 0) if str_eq(method, "GET") {
let lim_body: Int = json_get_int(body, "limit") let q = query_param(path, "q")
let lim_either: Int = if lim_url > 0 { lim_url } else { lim_body } } else {
let limit: Int = if lim_either > 0 { lim_either } else { 20 } let q = json_get_string(body, "query")
}
let limit: Int = query_int(path, "limit", 20)
if limit == 0 { let limit = json_get_int(body, "limit") }
if limit == 0 { let limit = 20 }
return engram_search_json(q, limit) return engram_search_json(q, limit)
} }
fn route_activate(method: String, path: String, body: String) -> String { fn route_activate(method: String, path: String, body: String) -> String {
let q: String = if str_eq(method, "GET") { query_param(path, "q") } else { json_get_string(body, "query") } let q: String = ""
// Guard: engram_activate with an empty query matches zero seeds, which let depth: Int = 3
// zeroes ALL carried working-memory weights (documented in awareness.el if str_eq(method, "GET") {
// perceive()). Never let an empty activation through to wipe WM. let q = query_param(path, "q")
if str_eq(q, "") { return err_json("missing query") } let depth = query_int(path, "depth", 3)
let d_raw: Int = if str_eq(method, "GET") { query_int(path, "depth", 3) } else { json_get_int(body, "depth") } } else {
let depth: Int = if d_raw > 0 { d_raw } else { 3 } let q = json_get_string(body, "query")
let bd: Int = json_get_int(body, "depth")
if bd > 0 { let depth = bd }
}
return "{\"results\":" + engram_activate_json(q, depth) + "}" return "{\"results\":" + engram_activate_json(q, depth) + "}"
} }
fn route_create_edge(method: String, path: String, body: String) -> String { fn route_create_edge(method: String, path: String, body: String) -> String {
let from_id: String = json_get_string(body, "from_id") let from_id: String = json_get_string(body, "from_id")
let to_id: String = json_get_string(body, "to_id") let to_id: String = json_get_string(body, "to_id")
let rel_raw: String = json_get_string(body, "relation") let relation: String = json_get_string(body, "relation")
let relation: String = if str_eq(rel_raw, "") { "associates" } else { rel_raw } if str_eq(relation, "") { let relation = "associates" }
let w_raw: Float = json_get_float(body, "weight") let weight: Float = json_get_float(body, "weight")
let weight: Float = if w_raw == 0.0 { 0.5 } else { w_raw } if weight == 0.0 { let weight = 0.5 }
engram_connect(from_id, to_id, weight, relation) engram_connect(from_id, to_id, weight, relation)
let saved: Int = persist_canonical()
"{\"ok\":true,\"from_id\":\"" + from_id + "\",\"to_id\":\"" + to_id + "\",\"relation\":\"" + relation + "\"}" "{\"ok\":true,\"from_id\":\"" + from_id + "\",\"to_id\":\"" + to_id + "\",\"relation\":\"" + relation + "\"}"
} }
@@ -195,7 +170,6 @@ fn route_strengthen(method: String, path: String, body: String) -> String {
let id: String = json_get_string(body, "node_id") let id: String = json_get_string(body, "node_id")
if str_eq(id, "") { return err_json("missing node_id") } if str_eq(id, "") { return err_json("missing node_id") }
engram_strengthen(id) engram_strengthen(id)
let saved: Int = persist_canonical()
ok_json() ok_json()
} }
@@ -203,24 +177,27 @@ fn route_forget(method: String, path: String, body: String) -> String {
let id: String = extract_id(path, "/api/nodes/") let id: String = extract_id(path, "/api/nodes/")
if str_eq(id, "") { return err_json("missing id") } if str_eq(id, "") { return err_json("missing id") }
engram_forget(id) engram_forget(id)
let saved: Int = persist_canonical()
ok_json() ok_json()
} }
fn route_save(method: String, path: String, body: String) -> String { fn route_save(method: String, path: String, body: String) -> String {
let p_raw: String = json_get_string(body, "path") let p: String = json_get_string(body, "path")
let dir_raw: String = env("ENGRAM_DATA_DIR") if str_eq(p, "") {
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw } let dir: String = env("ENGRAM_DATA_DIR")
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw } if str_eq(dir, "") { let dir = "/tmp/engram" }
let p = dir + "/snapshot.json"
}
engram_save(p) engram_save(p)
"{\"ok\":true,\"path\":\"" + p + "\"}" "{\"ok\":true,\"path\":\"" + p + "\"}"
} }
fn route_load(method: String, path: String, body: String) -> String { fn route_load(method: String, path: String, body: String) -> String {
let p_raw: String = json_get_string(body, "path") let p: String = json_get_string(body, "path")
let dir_raw: String = env("ENGRAM_DATA_DIR") if str_eq(p, "") {
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw } let dir: String = env("ENGRAM_DATA_DIR")
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw } if str_eq(dir, "") { let dir = "/tmp/engram" }
let p = dir + "/snapshot.json"
}
engram_load(p) engram_load(p)
ok_json() ok_json()
} }
@@ -242,36 +219,15 @@ fn route_health(method: String, path: String, body: String) -> String {
// (it skips nodes already present by ID). Auth-exempt: same-host internal call. // (it skips nodes already present by ID). Auth-exempt: same-host internal call.
// (2026-06-27 self-review: added this route to fix silent 10-min sync failures) // (2026-06-27 self-review: added this route to fix silent 10-min sync failures)
fn route_sync(method: String, path: String, body: String) -> String { fn route_sync(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR") let dir: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw } if str_eq(dir, "") { let dir = "/tmp/engram" }
// 2026-07-21 self-review: export to a scratch path, never the canonical let snap_path: String = dir + "/snapshot.json"
// snapshot.json read routes must not be able to clobber the good snapshot.
let snap_path: String = dir + "/.sync-export.json"
engram_save(snap_path) engram_save(snap_path)
let snap: String = fs_read(snap_path) let snap: String = fs_read(snap_path)
if str_eq(snap, "") { return "{\"nodes\":[],\"edges\":[]}" } if str_eq(snap, "") { return "{\"nodes\":[],\"edges\":[]}" }
return snap return snap
} }
// route_load_merge POST /api/load-merge {"path": "..."} merge a snapshot
// file into the live store WITHOUT resetting it (engram_load_merge skips nodes
// already present by id). Added 2026-07-21 self-review to restore the 244 kn-
// identity Knowledge nodes lost from the snapshot lineage between 05-13 and
// 07-13. Requires an explicit path: refuses to run without one so it can never
// be triggered accidentally against a default.
fn route_load_merge(method: String, path: String, body: String) -> String {
let p: String = json_get_string(body, "path")
if str_eq(p, "") { return err_json("path is required") }
if str_eq(fs_read(p), "") { return err_json("file missing or empty") }
let before_n: Int = engram_node_count()
let before_e: Int = engram_edge_count()
engram_load_merge(p)
let added_n: Int = engram_node_count() - before_n
let added_e: Int = engram_edge_count() - before_e
let saved: Int = persist_canonical()
"{\"ok\":true,\"nodes_added\":" + int_to_str(added_n) + ",\"edges_added\":" + int_to_str(added_e) + ",\"node_count\":" + int_to_str(engram_node_count()) + "}"
}
// route_emit_ise write an InternalStateEvent node from the soul daemon. // route_emit_ise write an InternalStateEvent node from the soul daemon.
// //
// Endpoint: POST /api/neuron/state-events // Endpoint: POST /api/neuron/state-events
@@ -285,20 +241,10 @@ fn route_load_merge(method: String, path: String, body: String) -> String {
// //
// Salience/importance set to match engram_node_full ISE defaults used by the // Salience/importance set to match engram_node_full ISE defaults used by the
// in-process fallback path in awareness.el (salience=0.3, importance=0.3, // in-process fallback path in awareness.el (salience=0.3, importance=0.3,
// confidence=0.8, tier=Episodic). // confidence=0.8, tier=Episodic). High temporal_decay_rate (1.617) ISEs
// are inherently transient; they should decay faster than structural knowledge.
// (2026-06-26 self-review: added this route after discovering ise_post was // (2026-06-26 self-review: added this route after discovering ise_post was
// silently failing the soul posts here but the endpoint didn't exist.) // silently failing the soul posts here but the endpoint didn't exist.)
//
// Retention (2026-07-16 self-review): an earlier comment here claimed ISEs
// got temporal_decay_rate=1.617 that was never implemented (engram_node_full
// hardcodes 0.0), and per-node decay only dampens activation anyway; it never
// removes nodes. By 2026-07-16 ISEs were 75% of the store (10,175 of 13,522
// nodes, ~4,300/day, unbounded). ISEs are already WM-excluded in
// engram_activate, so the fix is retention, not decay: every insert calls
// engram_prune_telemetry(), a single O(nodes+edges) compaction pass that
// removes ISEs older than ENGRAM_ISE_RETENTION_MS (default 48h), protecting
// "session-start" labels and self_review events as durable history. At
// ~3 ISEs/min this bounds telemetry at ~8.6k nodes instead of growing forever.
fn route_emit_ise(method: String, path: String, body: String) -> String { fn route_emit_ise(method: String, path: String, body: String) -> String {
let content: String = json_get_string(body, "content") let content: String = json_get_string(body, "content")
if str_eq(content, "") { return err_json("missing content") } if str_eq(content, "") { return err_json("missing content") }
@@ -310,64 +256,6 @@ fn route_emit_ise(method: String, path: String, body: String) -> String {
sal, imp, conf, sal, imp, conf,
"Episodic", "[\"internal-state\",\"InternalStateEvent\"]" "Episodic", "[\"internal-state\",\"InternalStateEvent\"]"
) )
let ret_raw: String = env("ENGRAM_ISE_RETENTION_MS")
let ret_ms: Int = if str_eq(ret_raw, "") { 172800000 } else { str_to_int(ret_raw) }
let pruned: Int = engram_prune_telemetry(ret_ms)
"{\"ok\":true,\"id\":\"" + id + "\",\"pruned\":" + int_to_str(pruned) + "}"
}
// Knowledge capture
//
// route_capture_knowledge direct Knowledge-node capture over HTTP.
//
// Endpoint: POST /api/neuron/knowledge/capture (auth required: "_auth" in body)
// Body: {"content": "...", "title": "...", "category": "...",
// "tier": "note|lesson|canonical", "tags": [...], "project": "...",
// "_auth": "<key>"}
//
// WHY (2026-07-15 self-review): the world-ingestor integrator was designed
// against this endpoint (its MCP-unavailable fallback), but the route never
// existed every direct push 404'd, and because the auth gate ran before
// routing, the failure surfaced as {"error":"unauthorized"} and was
// misdiagnosed for two weeks while world knowledge silently dropped.
// POST /api/nodes was no substitute: it discards label/tags/tier, which
// makes captured knowledge invisible to tag-scoped search and curiosity.
//
// The incoming knowledge tier (note/lesson/canonical) is preserved as a
// "tier:<x>" tag rather than mapped onto Engram's cognitive tiers Knowledge
// nodes land in Semantic (stable reference), and the epistemic tier stays
// queryable without inventing a lossy mapping.
fn route_capture_knowledge(method: String, path: String, body: String) -> String {
let content: String = json_get_string(body, "content")
if str_eq(content, "") { return err_json("missing content") }
let title: String = json_get_string(body, "title")
let label: String = if str_eq(title, "") { str_slice(content, 0, 60) } else { title }
let category_raw: String = json_get_string(body, "category")
let category: String = if str_eq(category_raw, "") { "other" } else { category_raw }
let ktier_raw: String = json_get_string(body, "tier")
let ktier: String = if str_eq(ktier_raw, "") { "note" } else { ktier_raw }
let project: String = json_get_string(body, "project")
let tags_raw: String = json_get_raw(body, "tags")
let tags_base: String = if str_eq(tags_raw, "") { "[]" } else { tags_raw }
// Merge category/tier/project markers into the tag array. Search matches
// against the tags string, so these make captures findable by facet.
let base_len: Int = str_len(tags_base)
let head: String = str_slice(tags_base, 0, base_len - 1)
let sep: String = if str_eq(head, "[") { "" } else { "," }
let safe_cat: String = str_replace(category, "\"", "'")
let safe_tier: String = str_replace(ktier, "\"", "'")
let safe_proj: String = str_replace(project, "\"", "'")
let proj_tag: String = if str_eq(safe_proj, "") { "" } else { ",\"project:" + safe_proj + "\"" }
let tags: String = head + sep + "\"category:" + safe_cat + "\",\"tier:" + safe_tier + "\"" + proj_tag + "]"
let sal: Float = 0.5
let imp: Float = 0.5
let conf: Float = 0.9
let id: String = engram_node_full(
content, "Knowledge", label,
sal, imp, conf,
"Semantic", tags
)
let saved: Int = persist_canonical()
"{\"ok\":true,\"id\":\"" + id + "\"}" "{\"ok\":true,\"id\":\"" + id + "\"}"
} }
@@ -407,12 +295,6 @@ fn handle_request(method: String, path: String, body: String) -> String {
return err_json("unauthorized") return err_json("unauthorized")
} }
// Knowledge capture (auth enforced above; the world-ingestor integrator
// and any headless session without MCP push knowledge through this)
if str_eq(method, "POST") && str_eq(clean, "/api/neuron/knowledge/capture") {
return route_capture_knowledge(method, path, body)
}
// Stats // Stats
if str_eq(method, "GET") && (str_eq(clean, "/api/stats") || str_eq(clean, "/stats")) { if str_eq(method, "GET") && (str_eq(clean, "/api/stats") || str_eq(clean, "/stats")) {
return route_stats(method, path, body) return route_stats(method, path, body)
@@ -469,9 +351,6 @@ fn handle_request(method: String, path: String, body: String) -> String {
if str_eq(method, "POST") && (str_eq(clean, "/api/load") || str_eq(clean, "/load")) { if str_eq(method, "POST") && (str_eq(clean, "/api/load") || str_eq(clean, "/load")) {
return route_load(method, path, body) return route_load(method, path, body)
} }
if str_eq(method, "POST") && (str_eq(clean, "/api/load-merge") || str_eq(clean, "/load-merge")) {
return route_load_merge(method, path, body)
}
// Sync soul daemon periodic pull of non-ISE knowledge into in-process graph // Sync soul daemon periodic pull of non-ISE knowledge into in-process graph
if str_eq(method, "GET") && str_eq(clean, "/api/sync") { if str_eq(method, "GET") && str_eq(clean, "/api/sync") {
@@ -483,31 +362,16 @@ fn handle_request(method: String, path: String, body: String) -> String {
// Entry // Entry
let bind_raw: String = env("ENGRAM_BIND") let bind_str: String = env("ENGRAM_BIND")
let bind_str: String = if str_eq(bind_raw, "") { ":8742" } else { bind_raw } if str_eq(bind_str, "") { let bind_str = ":8742" }
let port: Int = parse_port(bind_str) let port: Int = parse_port(bind_str)
// On startup, try to load any existing snapshot (best effort). // On startup, try to load any existing snapshot (best effort).
let data_dir_raw: String = env("ENGRAM_DATA_DIR") let data_dir: String = env("ENGRAM_DATA_DIR")
let data_dir: String = if str_eq(data_dir_raw, "") { "/tmp/engram" } else { data_dir_raw } if str_eq(data_dir, "") { let data_dir = "/tmp/engram" }
let snapshot_path: String = data_dir + "/snapshot.json" let snapshot_path: String = data_dir + "/snapshot.json"
engram_load(snapshot_path) engram_load(snapshot_path)
// 2026-07-21 self-review boot guard: if the snapshot file has content but the
// load produced 0 nodes, something is wrong (corrupt file / parse failure).
// Preserve the evidence and warn loudly and since read routes no longer write
// the canonical path, a bad boot can no longer clobber the good snapshot.
let boot_snap: String = fs_read(snapshot_path)
if !str_eq(boot_snap, "") {
if engram_node_count() == 0 {
println("[engram] WARNING: snapshot.json is non-empty but load produced 0 nodes — preserving copy at snapshot.failed-load.json")
fs_write(data_dir + "/snapshot.failed-load.json", boot_snap)
} else {
// Good load: keep a boot-time backup of the snapshot as loaded.
fs_write(data_dir + "/snapshot.boot-backup.json", boot_snap)
}
}
println("[engram] runtime-native graph engine") println("[engram] runtime-native graph engine")
println("[engram] data_dir=" + data_dir) println("[engram] data_dir=" + data_dir)
println("[engram] node_count=" + int_to_str(engram_node_count())) println("[engram] node_count=" + int_to_str(engram_node_count()))
+10
View File
@@ -17,6 +17,16 @@
// 4. Append dep to order after all its transitive deps // 4. Append dep to order after all its transitive deps
// 5. Deduplicate: skip already-ordered vessels // 5. Deduplicate: skip already-ordered vessels
// Cross-module forward declarations
// Defined in sibling epm modules; resolved at link time. The `extern fn` decls
// give elc the C prototypes so generated install.c compiles cleanly under strict
// compilers (gcc>=14 / clang) that reject implicit function declarations.
extern fn manifest_name(src: String) -> String // manifest.el
extern fn manifest_deps(src: String) -> String // manifest.el
extern fn registry_token() -> String // registry.el
extern fn registry_find(name: String, version: String) -> String // registry.el
extern fn registry_latest_version(name: String) -> String // registry.el
// Install paths // Install paths
// packages_dir returns the root directory for installed vessels. // packages_dir returns the root directory for installed vessels.
+9
View File
@@ -14,6 +14,15 @@
// EPM_REGISTRY_ORG org name that hosts vessel repos (default: neuron-technologies) // EPM_REGISTRY_ORG org name that hosts vessel repos (default: neuron-technologies)
// EPM_TOKEN Gitea personal access token (required for publish) // EPM_TOKEN Gitea personal access token (required for publish)
// Cross-module forward declarations
// These symbols are defined in sibling epm modules or the El runtime and are
// resolved at link time. The `extern fn` decls give elc the C prototype so the
// generated registry.c compiles cleanly under strict compilers (gcc>=14 / clang)
// that reject implicit function declarations. Signature arity must match the
// definition; return/param types are informational (all lower to el_val_t).
extern fn config(key: String) -> String // El runtime builtin
extern fn read_installed() -> String // install.el
// Config helpers // Config helpers
// registry_api_url returns the Gitea API base URL with no trailing slash. // registry_api_url returns the Gitea API base URL with no trailing slash.
+9
View File
@@ -6,6 +6,15 @@
// Depends on: registry.el (registry_latest_version, registry_find), // Depends on: registry.el (registry_latest_version, registry_find),
// install.el (read_installed, install_vessel, installed_version) // install.el (read_installed, install_vessel, installed_version)
// Cross-module forward declarations
// Defined in sibling epm modules; resolved at link time. The `extern fn` decls
// give elc the C prototypes so generated update.c compiles cleanly under strict
// compilers (gcc>=14 / clang) that reject implicit function declarations.
extern fn read_installed() -> String // install.el
extern fn installed_version(name: String) -> String // install.el
extern fn install_vessel(name: String, version: String) -> Bool // install.el
extern fn registry_latest_version(name: String) -> String // registry.el
// Semver helpers // Semver helpers
// semver_part extracts the Nth dot-separated component from a semver string. // semver_part extracts the Nth dot-separated component from a semver string.
@@ -75,6 +75,7 @@ static inline void* el_win_dlsym(void* handle, const char* name) {
#include <direct.h> /* _mkdir */ #include <direct.h> /* _mkdir */
#define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */ #define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */
#define timegm _mkgmtime /* UTC tm → time_t */ #define timegm _mkgmtime /* UTC tm → time_t */
#define fsync(fd) _commit(fd) /* no fsync() on Windows; _commit() (<io.h>) is the equiv */
/* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */ /* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */
static inline int setenv(const char* name, const char* value, int overwrite) { static inline int setenv(const char* name, const char* value, int overwrite) {
+387 -51
View File
@@ -82,8 +82,14 @@ static _Thread_local ElArena _tl_arena = {NULL, 0, 0};
static _Thread_local int _tl_arena_active = 0; static _Thread_local int _tl_arena_active = 0;
/* Binary-safe fs_read length — set by fs_read, consumed by http_send_response. /* Binary-safe fs_read length — set by fs_read, consumed by http_send_response.
* Allows serving PNGs and other binary files without strlen truncation. */ * Allows serving PNGs and other binary files without strlen truncation.
static _Thread_local size_t _tl_fs_read_len = 0; * PAIRED with the buffer pointer it describes: the length may only be applied
* to the exact buffer fs_read returned. Without the pairing, any handler that
* fs_read a file and then WRAPPED it into a larger response had that response
* truncated to the file's length (Content-Length lied AND the send stopped
* short) the safety-contact onboarding trap, 2026-07-17. */
static _Thread_local size_t _tl_fs_read_len = 0;
static _Thread_local const char* _tl_fs_read_buf = NULL;
static void el_arena_track(char* p) { static void el_arena_track(char* p) {
if (!_tl_arena_active || !p) return; if (!_tl_arena_active || !p) return;
@@ -101,6 +107,8 @@ static void el_arena_track(char* p) {
void el_request_start(void) { void el_request_start(void) {
_tl_arena.count = 0; _tl_arena.count = 0;
_tl_arena_active = 1; _tl_arena_active = 1;
_tl_fs_read_len = 0; /* never let a previous request's file length */
_tl_fs_read_buf = NULL; /* leak into this response's byte accounting */
} }
/* Called by http_worker after the El handler returns and the response is sent. /* Called by http_worker after the El handler returns and the response is sent.
@@ -1484,11 +1492,14 @@ static void http_send_response(int fd, const char* body) {
} }
const char* eff_body = is_envelope ? env_body : body; const char* eff_body = is_envelope ? env_body : body;
/* Use the real byte count from fs_read if available (handles binary files /* Use the real byte count from fs_read ONLY when this body IS the exact
* with embedded null bytes PNG, WOFF2, etc.). Fall back to strlen for * buffer fs_read returned (binary files with embedded null bytes PNG,
* normal text/JSON responses where _tl_fs_read_len is 0. */ * WOFF2, etc.). Any other body wrapped, enveloped, or derived must be
size_t blen = (_tl_fs_read_len > 0) ? _tl_fs_read_len : strlen(eff_body); * measured with strlen, or it is truncated/over-read to the file's size. */
size_t blen = (_tl_fs_read_len > 0 && eff_body == _tl_fs_read_buf)
? _tl_fs_read_len : strlen(eff_body);
_tl_fs_read_len = 0; /* consume — one-shot per response */ _tl_fs_read_len = 0; /* consume — one-shot per response */
_tl_fs_read_buf = NULL;
int head_only = _tl_http_head_only; int head_only = _tl_http_head_only;
JsonBuf hdrs; jb_init(&hdrs); JsonBuf hdrs; jb_init(&hdrs);
@@ -1568,11 +1579,22 @@ static void* http_worker(void* arg) {
const char* rs = EL_CSTR(r); const char* rs = EL_CSTR(r);
/* Copy response out BEFORE arena teardown. /* Copy response out BEFORE arena teardown.
* For binary files, _tl_fs_read_len holds the real byte count * For binary files, _tl_fs_read_len holds the real byte count
* use memcpy instead of strdup so null bytes are preserved. */ * use memcpy instead of strdup so null bytes are preserved.
size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0); * The stored length applies ONLY when the response IS the exact
* fs_read buffer; a wrapped/derived response must use strlen or
* it gets truncated (or over-read) to the file's length. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1); response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; } if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; } else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
} else { } else {
response = el_strdup_persist("el-runtime: no http handler registered"); response = el_strdup_persist("el-runtime: no http handler registered");
} }
@@ -1822,10 +1844,20 @@ static void* http_worker_v2(void* arg) {
el_val_t hmap = http_build_headers_map(hdr_block ? hdr_block : ""); el_val_t hmap = http_build_headers_map(hdr_block ? hdr_block : "");
el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), hmap, EL_STR(body)); el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), hmap, EL_STR(body));
const char* rs = EL_CSTR(r); const char* rs = EL_CSTR(r);
size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0); /* Same pairing rule as the v1 worker: the fs_read length is only
* trustworthy for the exact buffer fs_read returned. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1); response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; } if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; } else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
el_release(hmap); el_release(hmap);
} else { } else {
response = el_strdup_persist( response = el_strdup_persist(
@@ -1963,8 +1995,9 @@ void http_serve_async(el_val_t port, el_val_t handler) {
int sock = socket(AF_INET6, SOCK_STREAM, 0); int sock = socket(AF_INET6, SOCK_STREAM, 0);
if (sock < 0) { perror("socket"); return; } if (sock < 0) { perror("socket"); return; }
int yes = 1; int no = 0; int yes = 1; int no = 0;
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes)); /* Win32/mingw setsockopt takes optval as (const char*); the cast is portable on POSIX too. */
setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, &no, sizeof(no)); setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (const char*)&yes, sizeof(yes));
setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, (const char*)&no, sizeof(no));
struct sockaddr_in6 addr; struct sockaddr_in6 addr;
memset(&addr, 0, sizeof(addr)); memset(&addr, 0, sizeof(addr));
addr.sin6_family = AF_INET6; addr.sin6_family = AF_INET6;
@@ -2023,6 +2056,7 @@ el_val_t http_response(el_val_t status, el_val_t headers_json, el_val_t body) {
el_val_t fs_read(el_val_t pathv) { el_val_t fs_read(el_val_t pathv) {
const char* path = EL_CSTR(pathv); const char* path = EL_CSTR(pathv);
_tl_fs_read_len = 0; _tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!path) return el_wrap_str(el_strdup("")); if (!path) return el_wrap_str(el_strdup(""));
FILE* f = fopen(path, "rb"); FILE* f = fopen(path, "rb");
if (!f) return el_wrap_str(el_strdup("")); if (!f) return el_wrap_str(el_strdup(""));
@@ -2034,6 +2068,7 @@ el_val_t fs_read(el_val_t pathv) {
size_t got = fread(buf, 1, (size_t)sz, f); size_t got = fread(buf, 1, (size_t)sz, f);
buf[got] = '\0'; buf[got] = '\0';
_tl_fs_read_len = got; /* store real byte count for binary-safe send */ _tl_fs_read_len = got; /* store real byte count for binary-safe send */
_tl_fs_read_buf = buf; /* ...valid ONLY for this exact buffer */
fclose(f); fclose(f);
return el_wrap_str(buf); return el_wrap_str(buf);
} }
@@ -3576,8 +3611,10 @@ el_val_t json_get_raw(el_val_t json_str, el_val_t key) {
const char* k = EL_CSTR(key); const char* k = EL_CSTR(key);
const char* p = json_find_key(json, k); const char* p = json_find_key(json, k);
/* Clear fs_read binary-length hint — result is a fresh null-terminated /* Clear fs_read binary-length hint — result is a fresh null-terminated
* string, not the raw file bytes, so Content-Length must use strlen. */ * string, not the raw file bytes, so Content-Length must use strlen.
* (Kept although the pointer pairing now makes this redundant.) */
_tl_fs_read_len = 0; _tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!p) return el_wrap_str(el_strdup("")); if (!p) return el_wrap_str(el_strdup(""));
const char* end = json_skip_value(p); const char* end = json_skip_value(p);
size_t n = (size_t)(end - p); size_t n = (size_t)(end - p);
@@ -6895,6 +6932,243 @@ static int engram_rank_cmp(const void* a, const void* b) {
return 0; return 0;
} }
/* ══════════════════════════════════════════════════════════════════════════
* SEMANTIC SEARCH LAYER nomic-embed-text via Ollama /api/embeddings
*
* Augments the lexical (istr_contains) matcher with dense-vector retrieval.
* Node content and the query are embedded through a local Ollama server;
* nodes are ranked by cosine similarity and UNIONED with lexical hits. This
* lets a paraphrase query surface a node whose words never appear in it.
*
* DEGRADABLE BY DESIGN. The whole layer is gated on HAVE_CURL plus a one-shot
* runtime probe of the embedding endpoint. If curl is not compiled in, or
* Ollama is unreachable, or ENGRAM_SEMANTIC=0, every entry point returns
* "no semantic signal" and callers fall back to pure lexical behaviour
* byte-for-byte the pre-existing search.
*
* CACHE. Node embeddings are computed lazily on first use and cached in
* process memory keyed by node id, with an FNV-1a content hash for
* invalidation (edited content re-embeds). The query is embedded once per
* search call. This is what "avoid re-embedding the whole graph every query"
* buys us: a warm cache serves cosine from RAM. (A cold process still pays
* O(N) embed calls the first time each node is scanned persisting the cache
* to a snapshot sidecar is the documented next step, not done here.)
*
* nomic task prefixes ("search_query:" / "search_document:") are applied
* because nomic-embed-text is trained with them; they materially improve
* retrieval separation (empirically: paraphrase 0.72 vs distractors <0.48).
*
* ENV:
* ENGRAM_SEMANTIC "0" disables; unset/other = auto-probe
* ENGRAM_EMBED_URL default http://localhost:11434/api/embeddings
* ENGRAM_EMBED_MODEL default nomic-embed-text
* ENGRAM_SEMANTIC_MIN cosine threshold for a pure-semantic match (def 0.6)
* */
static double engram_semantic_min(void) {
static double v = -1.0;
if (v >= 0.0) return v;
const char* s = getenv("ENGRAM_SEMANTIC_MIN");
double d = 0.6;
if (s && *s) { char* e = NULL; double t = strtod(s, &e);
if (e != s && t >= 0.0 && t <= 1.0) d = t; }
v = d; return v;
}
#ifdef HAVE_CURL
typedef struct { char* id; uint64_t hash; float* vec; int dim; } EngramEmbEntry;
static EngramEmbEntry* g_emb_items = NULL;
static int64_t g_emb_count = 0, g_emb_cap = 0;
static int g_emb_state = 0; /* 0=unprobed, 1=available, -1=disabled */
static uint64_t engram_fnv1a(const char* s) {
uint64_t h = 1469598103934665603ULL;
if (s) for (const unsigned char* p = (const unsigned char*)s; *p; p++) {
h ^= *p; h *= 1099511628211ULL;
}
return h;
}
/* Parse "embedding":[f,f,...] from an Ollama response. malloc'd vec, or NULL. */
static float* engram_parse_embedding(const char* json, int* out_dim) {
if (!json) return NULL;
const char* p = strstr(json, "\"embedding\"");
if (!p) return NULL;
p = strchr(p, '[');
if (!p) return NULL;
p++;
int cap = 1024, n = 0;
float* v = malloc((size_t)cap * sizeof(float));
if (!v) return NULL;
while (*p && *p != ']') {
while (*p == ' ' || *p == '\t' || *p == '\n' || *p == '\r' || *p == ',') p++;
if (*p == ']' || !*p) break;
char* e = NULL;
double d = strtod(p, &e);
if (e == p) break;
if (n >= cap) { cap *= 2; float* nv = realloc(v, (size_t)cap * sizeof(float));
if (!nv) { free(v); return NULL; } v = nv; }
v[n++] = (float)d;
p = e;
}
if (n == 0) { free(v); return NULL; }
*out_dim = n;
return v;
}
/* JSON-escape src into a malloc'd buffer (no surrounding quotes). */
static char* engram_json_escape(const char* src) {
if (!src) src = "";
size_t n = strlen(src);
char* out = malloc(n * 2 + 1);
if (!out) return NULL;
size_t j = 0;
for (size_t i = 0; i < n; i++) {
unsigned char c = (unsigned char)src[i];
if (c == '"') { out[j++] = '\\'; out[j++] = '"'; }
else if (c == '\\') { out[j++] = '\\'; out[j++] = '\\'; }
else if (c == '\n') { out[j++] = '\\'; out[j++] = 'n'; }
else if (c == '\r') { out[j++] = '\\'; out[j++] = 'r'; }
else if (c == '\t') { out[j++] = '\\'; out[j++] = 't'; }
else if (c < 0x20) { /* drop other control bytes */ }
else { out[j++] = (char)c; }
}
out[j] = '\0';
return out;
}
/* Embed `prefix+text` via Ollama. Returns malloc'd vec (caller frees), or NULL. */
static float* engram_embed_raw(const char* prefix, const char* text, int* out_dim) {
if (!text) return NULL;
const char* url = getenv("ENGRAM_EMBED_URL");
if (!url || !*url) url = "http://localhost:11434/api/embeddings";
const char* model = getenv("ENGRAM_EMBED_MODEL");
if (!model || !*model) model = "nomic-embed-text";
/* Bound content length to keep latency/memory sane on huge nodes. */
char* trunc = NULL;
size_t maxlen = 8192;
if (strlen(text) > maxlen) {
trunc = malloc(maxlen + 1);
if (trunc) { memcpy(trunc, text, maxlen); trunc[maxlen] = '\0'; text = trunc; }
}
char* esc_prefix = engram_json_escape(prefix ? prefix : "");
char* esc = engram_json_escape(text);
free(trunc);
if (!esc || !esc_prefix) { free(esc); free(esc_prefix); return NULL; }
size_t blen = strlen(esc) + strlen(esc_prefix) + strlen(model) + 64;
char* body = malloc(blen);
if (!body) { free(esc); free(esc_prefix); return NULL; }
snprintf(body, blen, "{\"model\":\"%s\",\"prompt\":\"%s%s\"}", model, esc_prefix, esc);
free(esc); free(esc_prefix);
CURL* c = curl_easy_init();
if (!c) { free(body); return NULL; }
HttpBuf rb; httpbuf_init(&rb);
struct curl_slist* h = curl_slist_append(NULL, "Content-Type: application/json");
char errbuf[CURL_ERROR_SIZE]; errbuf[0] = '\0';
curl_easy_setopt(c, CURLOPT_URL, url);
curl_easy_setopt(c, CURLOPT_WRITEFUNCTION, http_write_cb);
curl_easy_setopt(c, CURLOPT_WRITEDATA, &rb);
curl_easy_setopt(c, CURLOPT_POST, 1L);
curl_easy_setopt(c, CURLOPT_POSTFIELDS, body);
curl_easy_setopt(c, CURLOPT_POSTFIELDSIZE, (long)strlen(body));
curl_easy_setopt(c, CURLOPT_HTTPHEADER, h);
curl_easy_setopt(c, CURLOPT_TIMEOUT_MS, el_http_timeout_ms());
curl_easy_setopt(c, CURLOPT_NOSIGNAL, 1L);
curl_easy_setopt(c, CURLOPT_ERRORBUFFER, errbuf);
CURLcode rc = curl_easy_perform(c);
curl_slist_free_all(h);
curl_easy_cleanup(c);
free(body);
if (rc != CURLE_OK) { free(rb.data); return NULL; }
float* v = engram_parse_embedding(rb.data, out_dim);
free(rb.data);
return v;
}
/* One-shot probe: is semantic search available? Caches the verdict. */
static int engram_semantic_enabled(void) {
if (g_emb_state != 0) return g_emb_state == 1;
const char* s = getenv("ENGRAM_SEMANTIC");
if (s && strcmp(s, "0") == 0) { g_emb_state = -1; return 0; }
int dim = 0;
float* v = engram_embed_raw("search_query: ", "probe", &dim);
if (v && dim > 0) { free(v); g_emb_state = 1; return 1; }
free(v);
g_emb_state = -1; return 0;
}
/* Embed the query. Returns malloc'd vec (caller frees), or NULL if semantic off. */
static float* engram_embed_query(const char* q, int* dim) {
if (!engram_semantic_enabled()) return NULL;
if (!q || !*q) return NULL;
return engram_embed_raw("search_query: ", q, dim);
}
/* Cached node embedding. Returns a pointer OWNED BY THE CACHE — do not free. */
static const float* engram_node_vec(EngramNode* n, int* out_dim) {
if (!n || !n->id) return NULL;
uint64_t h = engram_fnv1a(n->content);
for (int64_t i = 0; i < g_emb_count; i++) {
if (g_emb_items[i].id && strcmp(g_emb_items[i].id, n->id) == 0) {
if (g_emb_items[i].hash == h && g_emb_items[i].vec) {
*out_dim = g_emb_items[i].dim; return g_emb_items[i].vec;
}
/* content changed → re-embed in place */
int dim = 0;
float* v = engram_embed_raw("search_document: ", n->content ? n->content : "", &dim);
if (!v) return NULL;
free(g_emb_items[i].vec);
g_emb_items[i].vec = v; g_emb_items[i].dim = dim; g_emb_items[i].hash = h;
*out_dim = dim; return v;
}
}
int dim = 0;
float* v = engram_embed_raw("search_document: ", n->content ? n->content : "", &dim);
if (!v) return NULL;
if (g_emb_count >= g_emb_cap) {
int64_t nc = g_emb_cap ? g_emb_cap * 2 : 256;
EngramEmbEntry* ni = realloc(g_emb_items, (size_t)nc * sizeof(EngramEmbEntry));
if (!ni) { free(v); return NULL; }
g_emb_items = ni; g_emb_cap = nc;
}
g_emb_items[g_emb_count].id = strdup(n->id);
g_emb_items[g_emb_count].hash = h;
g_emb_items[g_emb_count].vec = v;
g_emb_items[g_emb_count].dim = dim;
g_emb_count++;
*out_dim = dim; return v;
}
static double engram_cosine(const float* a, const float* b, int dim) {
double dot = 0, na = 0, nb = 0;
for (int i = 0; i < dim; i++) { dot += (double)a[i] * b[i];
na += (double)a[i] * a[i];
nb += (double)b[i] * b[i]; }
if (na <= 0 || nb <= 0) return 0.0;
return dot / (sqrt(na) * sqrt(nb));
}
/* Cosine of node n against the query vector; 0 if unavailable / dim mismatch. */
static double engram_node_cosine(EngramNode* n, const float* qvec, int qdim) {
if (!qvec || qdim <= 0) return 0.0;
int ndim = 0;
const float* nv = engram_node_vec(n, &ndim);
if (!nv || ndim != qdim) return 0.0;
return engram_cosine(qvec, nv, qdim);
}
#else /* !HAVE_CURL — semantic layer compiled out; callers stay pure-lexical.
* Only the two boundary functions the always-compiled search/activate
* code calls are stubbed; the query embed always yields NULL so every
* cosine is 0 and every caller collapses to lexical-only. */
static float* engram_embed_query(const char* q, int* dim) { (void)q; (void)dim; return NULL; }
static double engram_node_cosine(EngramNode* n, const float* qvec, int qdim) {
(void)n; (void)qvec; (void)qdim; return 0.0;
}
#endif /* HAVE_CURL */
el_val_t engram_search(el_val_t query, el_val_t limit) { el_val_t engram_search(el_val_t query, el_val_t limit) {
EngramStore* g = engram_get(); EngramStore* g = engram_get();
const char* q = EL_CSTR(query); const char* q = EL_CSTR(query);
@@ -6905,8 +7179,15 @@ el_val_t engram_search(el_val_t query, el_val_t limit) {
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN]; char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS); int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
if (ntok == 0) return lst; if (ntok == 0) return lst;
/* Semantic augmentation: embed the query once; a node is a hit if it covers
* >=1 query token (tokenized-lexical, #66) OR its cosine clears the
* threshold (#67). qvec is NULL (cosine 0) when semantic is unavailable
* pure tokenized-lexical, byte-identical to the lexical-only behaviour. */
int qdim = 0;
float* qvec = engram_embed_query(q, &qdim);
double sem_min = engram_semantic_min();
EngramRankEntry* hits = malloc((size_t)g->node_count * sizeof(EngramRankEntry)); EngramRankEntry* hits = malloc((size_t)g->node_count * sizeof(EngramRankEntry));
if (!hits) return lst; if (!hits) { free(qvec); return lst; }
int64_t nhits = 0; int64_t nhits = 0;
for (int64_t i = 0; i < g->node_count; i++) { for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i]; EngramNode* n = &g->nodes[i];
@@ -6916,20 +7197,24 @@ el_val_t engram_search(el_val_t query, el_val_t limit) {
* + engram_compile_layered_json that's the legitimate path. */ * + engram_compile_layered_json that's the legitimate path. */
if (engram_layer_is_transparent(n->layer_id)) continue; if (engram_layer_is_transparent(n->layer_id)) continue;
int sc = engram_node_match_score(n, toks, ntok); int sc = engram_node_match_score(n, toks, ntok);
if (sc > 0) { double sem = qvec ? engram_node_cosine(n, qvec, qdim) : 0.0;
if (sc > 0 || sem >= sem_min) {
hits[nhits].idx = i; hits[nhits].idx = i;
hits[nhits].score = sc; hits[nhits].score = sc;
hits[nhits].salience = n->salience; hits[nhits].salience = n->salience;
nhits++; nhits++;
} }
} }
/* Rank by distinct tokens matched (desc) then salience (desc), then cap. */ /* Rank by distinct tokens matched (desc) then salience (desc), then cap.
* Pure-semantic hits (token score 0) sort after every lexical hit a
* lexical semantic union with lexical precedence. */
qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp); qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp);
int64_t end = nhits < lim ? nhits : lim; int64_t end = nhits < lim ? nhits : lim;
for (int64_t k = 0; k < end; k++) { for (int64_t k = 0; k < end; k++) {
lst = el_list_append(lst, engram_node_to_map(&g->nodes[hits[k].idx])); lst = el_list_append(lst, engram_node_to_map(&g->nodes[hits[k].idx]));
} }
free(hits); free(hits);
free(qvec);
return lst; return lst;
} }
@@ -7279,21 +7564,31 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
if (!seeds) { if (!seeds) {
free(best_bg); free(best_hops); free(reached); return out; free(best_bg); free(best_hops); free(reached); return out;
} }
/* Tokenize once: a node seeds if it matches ANY query token, and its seed /* Tokenized + semantic seeding: a node seeds if it covers >=1 query token
* activation is scaled by token coverage (fraction of distinct query * (tokenized-lexical, #66) OR its cosine clears the threshold (#67). A
* tokens it contains) so a node matching all words seeds more strongly * lexical seed's activation is scaled by token coverage (fraction of
* than one matching a single word. Single-word queries coverage 1.0, * distinct query tokens covered) so a node matching all words seeds more
* identical to the prior whole-query behavior. */ * strongly than one matching a single word; single-word queries coverage
* 1.0. A pure-semantic seed (no token match) is instead down-weighted by
* its cosine so paraphrase matches spread without overpowering exact seeds.
* q_vec is NULL (cosine 0) when semantic is unavailable the seed set is
* exactly the tokenized-lexical one. q_vec is freed right after this loop
* so the many downstream early-returns need no cleanup change. */
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN]; char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS); int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
int q_dim = 0;
float* q_vec = engram_embed_query(q, &q_dim);
double q_sem_min = engram_semantic_min();
for (int64_t i = 0; i < g->node_count; i++) { for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i]; EngramNode* n = &g->nodes[i];
int sc = engram_node_match_score(n, toks, ntok); int sc = engram_node_match_score(n, toks, ntok);
if (sc > 0) { double sem = q_vec ? engram_node_cosine(n, q_vec, q_dim) : 0.0;
if (sc > 0 || sem >= q_sem_min) {
double tdecay = engram_temporal_decay(n, now_ms); double tdecay = engram_temporal_decay(n, now_ms);
double dampen = engram_activation_dampen(n); double dampen = engram_activation_dampen(n);
double cover = ntok > 0 ? (double)sc / (double)ntok : 1.0; double act = n->salience * tdecay * dampen;
double act = n->salience * tdecay * dampen * cover; if (sc > 0) act *= (ntok > 0 ? (double)sc / (double)ntok : 1.0);
else act *= sem; /* pure-semantic seed: down-weight by cosine */
seeds[seed_count].idx = i; seeds[seed_count].idx = i;
seeds[seed_count].act = act; seeds[seed_count].act = act;
seeds[seed_count].created_at = n->created_at; seeds[seed_count].created_at = n->created_at;
@@ -7303,6 +7598,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
reached[i] = 1; reached[i] = 1;
} }
} }
free(q_vec);
/* Compute mean seed created_at for temporal proximity bonus. */ /* Compute mean seed created_at for temporal proximity bonus. */
int64_t seed_epoch = 0; int64_t seed_epoch = 0;
if (seed_count > 0) { if (seed_count > 0) {
@@ -7854,6 +8150,35 @@ el_val_t engram_get_node_json(el_val_t id) {
return el_wrap_str(jb_finish(&b)); return el_wrap_str(jb_finish(&b));
} }
/* engram_get_node_by_label — find the first node whose label field exactly
* matches the given string. Returns the node as a JSON object string, or "{}"
* if no match is found.
*
* Used by chat.el to retrieve well-known nodes (e.g. "conv:history",
* "session:summary") by their stable label rather than by ID, which is immune
* to vector index drift across restarts.
*
* Exact match (strcmp, not istr_contains) because labels like "conv:history"
* must not collide with nodes whose content happens to contain that substring.
*
* Backported verbatim (idiom-adapted to jb_finish) from release runtime
* v1.0.0-20260501 to unblock the soul regen link: chat.el references this
* native but the current runtime lacked its definition. */
el_val_t engram_get_node_by_label(el_val_t label) {
const char* lbl = EL_CSTR(label);
if (!lbl || !*lbl) return el_wrap_str(el_strdup("{}"));
EngramStore* g = engram_get();
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
if (n->label && strcmp(n->label, lbl) == 0) {
JsonBuf b; jb_init(&b);
engram_emit_node_json(&b, n);
return el_wrap_str(jb_finish(&b));
}
}
return el_wrap_str(el_strdup("{}"));
}
el_val_t engram_search_json(el_val_t query, el_val_t limit) { el_val_t engram_search_json(el_val_t query, el_val_t limit) {
EngramStore* g = engram_get(); EngramStore* g = engram_get();
const char* q = EL_CSTR(query); const char* q = EL_CSTR(query);
@@ -7861,39 +8186,50 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) {
if (lim <= 0) lim = 100; if (lim <= 0) lim = 100;
JsonBuf b; jb_init(&b); JsonBuf b; jb_init(&b);
jb_putc(&b, '['); jb_putc(&b, '[');
int first = 1; if (q && *q && g->node_count > 0) {
if (q && *q) { /* Collect candidates from the UNION of tokenized-lexical and semantic
* matches, score each, rank by score, emit the top `lim`. A node is a
* candidate if it covers >=1 query token (tokenized-lexical, #66) OR its
* query cosine clears the threshold (#67). Lexical score is the distinct
* token count (>=1), so any lexical hit outranks a pure-semantic hit
* (cosine < 1); pure-semantic hits are scored by cosine alone. When
* semantic is unavailable qvec is NULL, sem is 0, only tokenized-lexical
* hits are collected, and the stable insertion sort preserves order. */
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN]; char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS); int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
if (ntok > 0) { int qdim = 0;
EngramRankEntry* hits = float* qvec = engram_embed_query(q, &qdim);
malloc((size_t)g->node_count * sizeof(EngramRankEntry)); double sem_min = engram_semantic_min();
if (hits) { typedef struct { int64_t idx; double score; } Cand;
int64_t nhits = 0; Cand* cand = malloc((size_t)g->node_count * sizeof(Cand));
for (int64_t i = 0; i < g->node_count; i++) { if (cand) {
EngramNode* n = &g->nodes[i]; int64_t nc = 0;
/* Filter transparent layers — same as engram_search. */ for (int64_t i = 0; i < g->node_count; i++) {
if (engram_layer_is_transparent(n->layer_id)) continue; EngramNode* n = &g->nodes[i];
int sc = engram_node_match_score(n, toks, ntok); if (engram_layer_is_transparent(n->layer_id)) continue;
if (sc > 0) { int sc = engram_node_match_score(n, toks, ntok);
hits[nhits].idx = i; double sem = qvec ? engram_node_cosine(n, qvec, qdim) : 0.0;
hits[nhits].score = sc; if (sc > 0 || sem >= sem_min) {
hits[nhits].salience = n->salience; cand[nc].idx = i;
nhits++; cand[nc].score = (double)sc + sem;
} nc++;
} }
/* Rank by distinct tokens matched (desc) then salience (desc). */
qsort(hits, (size_t)nhits, sizeof(EngramRankEntry),
engram_rank_cmp);
int64_t end = nhits < lim ? nhits : lim;
for (int64_t k = 0; k < end; k++) {
if (!first) jb_putc(&b, ',');
engram_emit_node_json(&b, &g->nodes[hits[k].idx]);
first = 0;
}
free(hits);
} }
/* Insertion sort by score desc; stable for equal scores. */
for (int64_t i = 1; i < nc; i++) {
Cand k = cand[i]; int64_t j = i - 1;
while (j >= 0 && cand[j].score < k.score) { cand[j + 1] = cand[j]; j--; }
cand[j + 1] = k;
}
int first = 1;
for (int64_t i = 0; i < nc && i < lim; i++) {
if (!first) jb_putc(&b, ',');
engram_emit_node_json(&b, &g->nodes[cand[i].idx]);
first = 0;
}
free(cand);
} }
free(qvec);
} }
jb_putc(&b, ']'); jb_putc(&b, ']');
return el_wrap_str(jb_finish(&b)); return el_wrap_str(jb_finish(&b));
+1
View File
@@ -632,6 +632,7 @@ el_val_t engram_load(el_val_t path);
* can pass results straight through without round-tripping ElList/ElMap * can pass results straight through without round-tripping ElList/ElMap
* through json_stringify. */ * through json_stringify. */
el_val_t engram_get_node_json(el_val_t id); el_val_t engram_get_node_json(el_val_t id);
el_val_t engram_get_node_by_label(el_val_t label);
el_val_t engram_search_json(el_val_t query, el_val_t limit); el_val_t engram_search_json(el_val_t query, el_val_t limit);
el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset); el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset);
el_val_t engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset); el_val_t engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset);
+1
View File
@@ -1072,6 +1072,7 @@ el_val_t __engram_save(el_val_t path) { return engram_save
el_val_t __engram_load(el_val_t path) { return engram_load(path); } el_val_t __engram_load(el_val_t path) { return engram_load(path); }
el_val_t __engram_get_node_json(el_val_t id) { return engram_get_node_json(id); } el_val_t __engram_get_node_json(el_val_t id) { return engram_get_node_json(id); }
el_val_t __engram_get_node_by_label(el_val_t label) { return engram_get_node_by_label(label); }
el_val_t __engram_search_json(el_val_t query, el_val_t limit) { el_val_t __engram_search_json(el_val_t query, el_val_t limit) {
return engram_search_json(query, limit); return engram_search_json(query, limit);
+1
View File
@@ -226,6 +226,7 @@ el_val_t __engram_activate(el_val_t query, el_val_t depth);
el_val_t __engram_save(el_val_t path); el_val_t __engram_save(el_val_t path);
el_val_t __engram_load(el_val_t path); el_val_t __engram_load(el_val_t path);
el_val_t __engram_get_node_json(el_val_t id); el_val_t __engram_get_node_json(el_val_t id);
el_val_t __engram_get_node_by_label(el_val_t label);
el_val_t __engram_search_json(el_val_t query, el_val_t limit); el_val_t __engram_search_json(el_val_t query, el_val_t limit);
el_val_t __engram_scan_nodes_json(el_val_t limit, el_val_t offset); el_val_t __engram_scan_nodes_json(el_val_t limit, el_val_t offset);
el_val_t __engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset); el_val_t __engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset);
+1
View File
@@ -2670,6 +2670,7 @@ fn builtin_arity(name: String) -> Int {
if str_eq(name, "engram_save") { return 1 } if str_eq(name, "engram_save") { return 1 }
if str_eq(name, "engram_load") { return 1 } if str_eq(name, "engram_load") { return 1 }
if str_eq(name, "engram_get_node_json") { return 1 } if str_eq(name, "engram_get_node_json") { return 1 }
if str_eq(name, "engram_get_node_by_label") { return 1 }
if str_eq(name, "engram_search_json") { return 2 } if str_eq(name, "engram_search_json") { return 2 }
if str_eq(name, "engram_scan_nodes_json") { return 2 } if str_eq(name, "engram_scan_nodes_json") { return 2 }
if str_eq(name, "engram_neighbors_json") { return 3 } if str_eq(name, "engram_neighbors_json") { return 3 }
File diff suppressed because it is too large Load Diff