Compare commits

..

21 Commits

Author SHA1 Message Date
will.anderson e08150d6e7 elb: fix monolithic link, capability violations, liboqs detection
- Link only the entry-point .c (monolithic) instead of all module .c
  files — prevents duplicate symbol errors from inlined imports
- Strip capability-violation #error guards post-compile; they fire
  incorrectly when modules are compiled in isolation but linked into
  a CGI program
- Add liboqs (post-quantum) include/lib detection via brew, matching
  the existing OpenSSL detection pattern
2026-06-05 11:34:28 -05:00
will.anderson e3a72aae90 self-review 2026-06-05: WM composition visibility + ISE tier-migration fix
Two improvements from daily self-review:

1. engram_wm_top_json(n) builtin — returns top-N WM nodes by weight as
   compact JSON [{label,node_type,tier,wm},...]. After long uptime all WM
   nodes cycle in steady-state decay+re-promotion so wm_promotion ISEs
   never fire (only trigger on 0→>0.1 transitions). This gives continuous
   visibility into WM composition on every heartbeat.

2. ISE tier-migration exclusion — InternalStateEvent nodes no longer
   participate in tier migration. ISEs are activated by curiosity_scan
   substring matches and accumulated 50+ activation_count, reaching
   Procedural tier (720h decay). This permanently crowded WM with ephemeral
   state events, hiding the Knowledge/Memory nodes that should dominate.
   Fix: skip ISEs in the TIER MIGRATION pass; they stay on Working-tier
   48h decay regardless of activation frequency.

wm_top_json is called from awareness.el emit_heartbeat (top-5 snapshot).
2026-06-05 11:34:28 -05:00
will.anderson 1264af72a6 self-review 2026-06-04: lower WM threshold, soften inhibition, add wm_avg_weight builtin
Three targeted improvements based on graph health analysis (29K nodes, 104 edges):

1. ENGRAM_WM_THRESHOLD 0.15 → 0.08: sparse graph means BFS paths carry weak
   signals (0.05-0.12 range). Prior threshold gatekept too aggressively. Grounded
   in TBRS* cognitive model (θ=0.05); 0.08 is conservative but effective.

2. ENGRAM_INHIBITION_FACTOR 0.1 → 0.2: factor=0.1 (90% suppression) on a sparse
   graph almost always fully silences targeted nodes. Factor=0.2 (80% suppression)
   maintains strong inhibition while allowing partially-suppressed nodes to remain
   faintly active — consistent with partial inhibition in cognitive neuroscience.

3. engram_wm_avg_weight() builtin: computes mean working_memory_weight of all
   promoted nodes. Returns float bits via el_from_float for EL float_to_str usage.
   Makes activation quality directly observable in heartbeat ISEs, distinguishing
   'many weak activations' (sparse, low avg) from 'few strong' (dense, high avg).

Rebuilt engram binary with new runtime.
2026-06-05 11:34:28 -05:00
will.anderson 412bd2744e self-review 2026-05-29: fix dampening floor and cleanup route_create_node auto-link
Two changes:

1. el_runtime.c — engram_activation_dampen(): add floor of 0.35.
   ISE nodes with ac=900+ had dampen=0.128, giving effective salience=0.038
   which fell below the epist>=0.1 gate in engram_activate. This silently
   killed curiosity seeds "self identity values" and "decision pattern lesson"
   — the only corpus matches were high-ac ISEs that were then excluded from
   results, causing activated=0 on 50% of proactive_curiosity scans.
   Floor at 0.35 keeps salience=0.3 nodes at effective_bg=0.105, above the
   visibility threshold, without disrupting relative ordering of content nodes.

2. server.el — route_create_node: replace stale inline auto-link with
   auto_link_content_node(). The inline logic used the old engram_search_json
   (substring, no ISE filter) while the better BM25-based auto_link_content_node
   was added in 2026-05-28 and wired to /api/neuron/* routes but not to the
   raw /api/nodes POST path. Removes ~40 lines of duplicated logic.
2026-06-05 11:34:28 -05:00
will.anderson a000599bfe self-review 2026-05-28: checkpoint ISE decay fix + auto-linking for MCP nodes
Three changes:

1. Fix checkpoint ISE temporal_decay_rate: engram_emit_ise_internal was
   hardcoded to 0.0 (global 168h default) instead of 2.310 (Working-tier
   48h). Result: checkpoint ISEs accumulated at 3.5x intended rate.

2. Raise CHECKPOINT_INTERVAL 1→10: checkpoint ISE fires on every single
   node write, producing 2:1 checkpoint:content ratio in ISE stream.
   MCP routes still call engram_write_binary_el explicitly after each
   important write, so no knowledge durability is lost.

3. Add auto_link_content_node to server.el: route_neuron_memory and
   route_neuron_knowledge_capture were creating nodes with zero edges —
   invisible to BFS traversal, only reachable via lexical/semantic seed.
   New helper runs BM25 over top-20 results, skips ISE nodes (which
   dominate the 14K-node corpus), connects up to 3 related nodes.
2026-06-05 11:34:27 -05:00
will.anderson 8f922e68b3 self-review 2026-05-27: semantic auto-linking on embed
Add engram_auto_link_semantic(): when a node is embedded, scan all
embedded nodes for cosine sim >= 0.72 and create bidirectional
"semantic-similar" edges to the top-3 matches. Runs once at write
time rather than at every query, converting the expensive O(N) scan
from live activation into durable graph structure.

Fixes the core connectivity problem: 63 edges / 5364 nodes (0.012
edges/node). Verified: new Knowledge nodes now auto-link at sim 0.77–
0.95 with correct threshold discrimination. One checkpoint per insert
(not one per edge) bounds the persistence overhead.

Excludes InternalStateEvents and inbox/outbox transient nodes to keep
semantic graph clean.
2026-06-05 11:34:27 -05:00
will.anderson ef1db34846 self-review 2026-05-24: BM25 scan cap 500→5000 + traversal inference guard
Two improvements:

1. BM25 search corpus coverage (server.el) — raised scan cap from 500 to
   5000 nodes. On the 161K-node graph, 500 was 0.3% coverage; 5000 is 3%.
   engram_scan_nodes_json returns nodes sorted by salience DESC, so ISEs
   (salience 0.3) fall below Knowledge/Memory (0.5–0.8) naturally — the
   effective corpus stays content-dense. Also updated stale comment on the
   ISE route (no longer need high offset; recent-first ordering from May 23).

2. Traversal inference guard (el_runtime.c) — two changes:
   - INFER_CAP reduced 256→32: proactive curiosity runs engram_activate
     every ~30s. At 256 edges/call the soul daemon accumulated 107K edges
     in 23h (5× BFS slowdown). At 32 the rate drops 8×.
   - Edge count guard: skip inference entirely when snap_ec ≥ 40,000.
     At that density most A→C paths are already explicit; marginal inference
     value is low and the O(edge_count²) inner-loop cost is high. Self-limits
     unbounded accumulation across restarts.
2026-06-05 11:34:27 -05:00
will.anderson 34249b39a3 self-review 2026-05-23: ISE recent-first ordering + http_serve_async builtin
Two improvements:

1. ISE scan ordering — engram_scan_nodes_by_type_json now sorts InternalStateEvent
   nodes by created_at DESC (most-recent-first) instead of salience DESC. Old
   high-salience ISEs (session-start, wm-promotion) no longer dominate offset 0,
   burying recent heartbeat and curiosity_scan events at offsets 20000+. New
   behavior: ?limit=10 returns the 10 most recent ISEs regardless of salience.
   All other node types retain existing salience-sorted behavior.

2. http_serve_async registered as elc builtin — added to builtin_arity() in both
   codegen.el (EL compiler source) and dist/platform/elc.c (compiled C). Also
   rebuilt elc binary from updated elc.c. This closes the fragile-patch gap from
   2026-05-21: elc previously treated http_serve_async as an unknown identifier,
   and the gap description noted elc would 'silently revert to blocking http_serve'
   on next soul rebuild. Now http_serve_async has a proper 2-arg arity entry and
   will survive all future soul recompiles without a manual neuron.c patch.
2026-06-05 11:34:27 -05:00
will.anderson 7b45468b1c self-review 2026-05-20: relation-type boost in BFS propagation
Add relation-specific multipliers to spreading activation:
- causes/caused_by edges: 2.0× (causal reasoning propagates stronger)
- enables/prevents edges: 1.5× (conditional logic gets boost)
- supersedes edges: 1.3× (promoted knowledge gets slight priority)
- inferred A→C edges: 0.7× (traversal-inferred paths weaker than explicit)

Field-validated against Hindsight (time-aware spreading activation, 2026)
and ACT-R cognitive architecture literature. Inferred edge discount prevents
the traversal inference pass from flooding activation with spurious high-
strength paths equal to explicit links.
2026-06-05 11:34:27 -05:00
will.anderson db7dae8236 self-review 2026-05-19: ACT-R WM persistence — decay non-reached nodes instead of zeroing
The activation persist step was writing wm_weight=0 for every node not reached
by the current BFS fan-out. This destroyed working memory accumulated by
MCP-layer activations within one tick of the awareness loop firing on an empty
inbox. ACT-R and Soar treat spreading activation as additive: absent seeds
contribute zero spread, not a zero override of existing WM state.

Fix: non-reached nodes now decay by ENGRAM_WM_DECAY (0.7) per activation call
rather than being immediately zeroed. A hard floor of 0.005 clears near-zero
values to prevent infinite decay tails. Reached nodes behave unchanged.
2026-06-05 11:34:27 -05:00
will.anderson ee1627c2c0 self-review 2026-05-18: raise SEM_FLOOR to 0.70, add ACT-R frequency resistance to decay
Semantic seed floor raised 0.65 → 0.70: field literature (SuperLocalMemory
arXiv:2604.04514) validates 0.70-0.75 as the noise floor for sentence-transformer
embeddings on non-trivial corpora. The 0.65 threshold was admitting false
positives that diluted BFS traversal quality on the 13K-node post-ingestor graph.
Top-30 cap retained — sparse graph (1.26 edges/node) needs more semantic entry
points than a dense graph would.

ACT-R Base-Level Learning insight applied to engram_temporal_decay: current
purely time-based formula treated a node activated 100 times identically to a
node never activated (same decay rate). ACT-R BLL (Anderson 1993) shows
frequently-accessed memories resist temporal decay. Fix: freq_resist multiplier
= 1.0 + 0.1 * log(1 + activation_count). count=0→1.0×, count=9→1.23×,
count=99→1.46×. Existing activation_dampening continues to reduce per-query
novelty for well-known nodes — the two mechanisms are complementary: resist
decay (durable in graph), dampen per-query (don't dominate any single turn).
2026-06-05 11:34:27 -05:00
will.anderson b90333e9e7 self-review 2026-05-17: semantic seeding in Pass 1 + WM promotion observability
Three improvements from daily review:

1. Add semantic seed supplement to Pass 1 activation (el_runtime.c).
   Previously, engram_cosine_sim was only called in goal_bias (Pass 2) for
   nodes that already matched lexically. Nodes semantically close but
   lexically disjoint were completely invisible to activation. With 8K+
   world-ingestor nodes added overnight, this was a critical gap. Now: after
   lexical seeding, scan un-seeded nodes for cosine sim ≥ 0.65 and inject
   top-30 as additional seeds. Sem seeds get 85% of full act to stay weaker
   than exact lexical matches.

2. Lower WM promotion ISE threshold from >0.5 to >0.1 (el_runtime.c).
   Only one wm-promotion ISE was ever logged — the 0.5 floor was too high.
   Most practical Knowledge/Memory promotions are in the 0.1-0.5 range.
   Lowering to 0.1 makes working memory activity visible in state events.
2026-06-05 11:34:27 -05:00
will.anderson d917165aaf self-review 2026-05-16: tier-based decay rates, implement knowledge_promote, ISE label extraction
Three research-grounded improvements:

1. Tier-based temporal decay in el_runtime.c (engram_node_full, engram_node_layered):
   Working=48h, Episodic=72h, Semantic=336h, Procedural=720h half-lives.
   Grounded in ACT-R literature — differentiated decay by chunk type. The
   temporal_decay_rate field existed but was always 0 (global 168h for everything).
   New nodes now carry the correct half-life for their tier from creation.

2. Implement route_neuron_knowledge_promote in server.el (was a silent stub):
   Reads existing node, creates promoted-tier copy with supersedes edge,
   checkpoints. promote_knowledge MCP tool now has real effect.

3. ISE label extraction + offset support in route_neuron_state_events:
   POST now extracts 'event' field from content JSON as label (heartbeat,
   wm_promotion, etc.) instead of always writing 'state-event'. GET now
   accepts ?offset= for pagination to reach recent ISEs.
2026-06-05 11:34:27 -05:00
will.anderson fde3ef539c add .clangd config to silence false-positive LSP errors for liboqs/openssl includes 2026-06-05 11:34:27 -05:00
will.anderson 9bcd68fbca self-review 2026-05-15: wire engram_cosine_sim into activation scoring
engram_cosine_sim was defined but never called. Nodes have 768-dim
nomic-embed-text vectors. Now:
- engram_embed_query() embeds the query string once per activate() call
- engram_goal_bias() takes (qvec, qdim) and adds cosine-similarity bonus
  up to +0.6 when sim > 0.5 — semantic relevance now augments lexical bias
- engram_wm_count() exposes working-memory-active node count to EL
- el_runtime.h declares engram_wm_count for soul-daemon linking
2026-06-05 11:34:27 -05:00
will.anderson 913a98329a wire BM25+ as default search engine; remove Ollama query-embedding
BM25+ (k1=1.2, b=0.75, delta=1.0) now powers all search routes in EL.
No external dependencies in the activation/search path.

- bm25_tokenize/bm25_count_term/bm25_score_doc/bm25_search_json in server.el
- route_search, route_neuron_recall: engram_search_json -> bm25_search_json
- route_activate: BM25 pre-bias (strengthen top-10) before spreading activation
- Remove standalone /api/bm25/search endpoint (BM25 is the engine, not a feature)
- Fix zero-score filter: float comparison not string match
- Add + to tokenizer for URL-encoded query params
- Scan floor 200 nodes regardless of limit size
- Revert Ollama engram_embed_query from 9af2482 (no Ollama at query time)
- Add list_set and math_exp builtins to el_runtime.c
2026-06-05 11:34:27 -05:00
will.anderson 6121b33d25 add BM25+ text ranking in EL, remove Ollama query-embedding dependency
- Add list_set, math_exp, and float_add/sub/mul/div/gt/lt/eq/gte/lte builtins to
  el_runtime.c + el_runtime.h (float arithmetic builtins needed because EL operators
  +*/ operate on raw el_val_t bits, not IEEE 754 doubles)
- Remove engram_embed_query() and its forward declaration from el_runtime.c
- Remove Ollama cosine-similarity blend from activation scoring (reverts 9af2482):
  drops query_emb/query_edim variables, bias *= (1 + 0.3 * sim) block, and all
  free(query_emb) calls from the activation loop
- Implement BM25+ scoring in server.el (k1=1.2, b=0.75, delta=1.0):
  bm25_tokenize, bm25_count_term, bm25_score_doc, bm25_search_json
  V1 uses n_t=1 approximation (constant IDF per corpus size); acceptable as a
  first pass without an inverted index
- Wire /api/bm25/search POST/GET route in server.el dispatcher
- Zero Ollama calls in the activation/search path; embeddings on nodes are
  untouched (still written at node-creation time)
2026-06-05 11:34:27 -05:00
will.anderson 1a8a16002e feat(engram): wire cosine similarity into Layer 2 activation scoring
engram_cosine_sim() was defined and embeddings were computed per-node
via nomic-embed-text on write, but the function was never called during
activation scoring. The goal_bias computation used only lexical substring
matching, ignoring all stored embedding vectors.

This change adds engram_embed_query() to embed the query string at search
time (5s timeout so Ollama latency never blocks activation), then blends
cosine similarity into the working-memory bias with α=0.3:

  bias_final = goal_bias(lexical) * (1 + 0.3 * max(0, cosine_sim))

Nodes with high semantic similarity to the query but low lexical overlap
now receive up to 30% bias boost into working memory promotion. Gracefully
degrades to pure lexical when Ollama is unavailable or node has no embedding.
2026-06-05 11:34:27 -05:00
will.anderson 0c2ff6957e feat(engram): wire ISE emission into core processing paths (checkpoint, high-importance writes, WM promotion) 2026-06-05 11:34:27 -05:00
will.anderson a3ead6552e feat(engram): ML-KEM-1024 PQC encryption, nomic embeddings, MCP routes, checkpoint-per-write
- Add ML-KEM-1024 + AES-256-GCM binary persistence to el_runtime.c with
  two-key scheme (Neuron master + user key); SHAKE-256 key derivation
- Add nomic-embed-text 768-dim float32 embeddings on every node write
  via Ollama; graceful fallback when Ollama is not running
- Wire all /api/neuron/* MCP routes directly into Engram (server.el),
  eliminating the Kotlin server as the MCP backend
- Set ENGRAM_CHECKPOINT_INTERVAL = 1 (write binary on every node write,
  not every 50)
- Add el_runtime.h declarations for engram_write_binary_el and
  engram_load_binary_el builtins
2026-06-05 11:34:27 -05:00
will.anderson e68dcf7303 feat(ci): add engram CI workflow to build and publish linux/amd64 binary
Triggers on push to main/dev when engram/** changes. Downloads the El SDK
from Artifact Registry, builds engram using elb, and publishes the binary
as package 'engram' (SHA version + latest tag) to foundation-dev.
2026-06-05 11:34:27 -05:00
16 changed files with 3872 additions and 781 deletions
+132
View File
@@ -0,0 +1,132 @@
name: Engram CI
on:
push:
branches:
- main
- dev
paths:
- 'engram/**'
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install build dependencies
run: |
apt-get update -qq
apt-get install -y gcc libcurl4-openssl-dev apt-transport-https ca-certificates
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" \
> /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
- name: Download El SDK from Artifact Registry
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
rm -rf /opt/el/dist /opt/el/runtime
mkdir -p /opt/el/dist/platform /opt/el/dist/bin /opt/el/runtime
get_latest() {
gcloud artifacts versions list \
--repository=foundation-dev \
--location=us-central1 \
--project=neuron-785695 \
--package="$1" \
--sort-by="~createTime" \
--limit=1 \
--format="value(name)" 2>/dev/null | awk -F/ '{print $NF}'
}
ELC_VER=$(get_latest el-elc)
ELB_VER=$(get_latest el-elb)
RC_VER=$(get_latest el-runtime-c)
RH_VER=$(get_latest el-runtime-h)
echo "Downloading elc@${ELC_VER} elb@${ELB_VER} runtime-c@${RC_VER} runtime-h@${RH_VER}"
gcloud artifacts generic download \
--repository=foundation-dev --location=us-central1 --project=neuron-785695 \
--package=el-elc --version="${ELC_VER}" \
--destination=/opt/el/dist/platform/
gcloud artifacts generic download \
--repository=foundation-dev --location=us-central1 --project=neuron-785695 \
--package=el-elb --version="${ELB_VER}" \
--destination=/opt/el/dist/bin/
gcloud artifacts generic download \
--repository=foundation-dev --location=us-central1 --project=neuron-785695 \
--package=el-runtime-c --version="${RC_VER}" \
--destination=/opt/el/runtime/
gcloud artifacts generic download \
--repository=foundation-dev --location=us-central1 --project=neuron-785695 \
--package=el-runtime-h --version="${RH_VER}" \
--destination=/opt/el/runtime/
mv /opt/el/dist/platform/elc* /opt/el/dist/platform/elc 2>/dev/null || true
mv /opt/el/dist/bin/elb* /opt/el/dist/bin/elb 2>/dev/null || true
mv /opt/el/runtime/el_runtime.c* /opt/el/runtime/el_runtime.c 2>/dev/null || true
mv /opt/el/runtime/el_runtime.h* /opt/el/runtime/el_runtime.h 2>/dev/null || true
chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb
echo "El SDK ready"
- name: Build engram binary (linux/amd64)
run: |
ELB=/opt/el/dist/bin/elb
ELC=/opt/el/dist/platform/elc
RUNTIME=/opt/el/runtime
# elb reads manifest.el from the working directory.
# engram/dist/engram.c is the pre-compiled C translation of src/server.el.
# elb compiles dist/engram.c + el_runtime.c → dist/engram binary.
cd engram
"$ELB" --elc="$ELC" --runtime="$RUNTIME"
ls -lh dist/engram
file dist/engram
- name: Smoke test
run: |
file engram/dist/engram
timeout 3 engram/dist/engram --help 2>&1 || true
echo "smoke test complete"
- name: Publish engram binary to Artifact Registry
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
VERSION="${GITHUB_SHA:0:8}"
gcloud artifacts generic upload \
--repository=foundation-dev \
--location=us-central1 \
--project=neuron-785695 \
--package=engram \
--version="${VERSION}" \
--source=engram/dist/engram
# Re-upload as "latest" — Artifact Registry generic artifacts don't
# support moving tags, so we upload again. The newest upload wins.
gcloud artifacts generic upload \
--repository=foundation-dev \
--location=us-central1 \
--project=neuron-785695 \
--package=engram \
--version="latest" \
--source=engram/dist/engram \
2>/dev/null || true
echo "Published engram@${VERSION} and engram@latest"
rm -f /tmp/gcp-key.json
Vendored Executable
BIN
View File
Binary file not shown.
+753 -22
View File
@@ -2,6 +2,11 @@
#include <stdlib.h>
#include "el_runtime.h"
el_val_t bm25_tokenize(el_val_t text);
el_val_t bm25_count_term(el_val_t term, el_val_t doc_tokens);
el_val_t bm25_score_doc(el_val_t doc_content, el_val_t query_tokens, el_val_t corpus_size, el_val_t avg_doc_len);
el_val_t bm25_search_json(el_val_t query, el_val_t limit);
el_val_t auto_link_content_node(el_val_t node_id, el_val_t content);
el_val_t parse_port(el_val_t bind);
el_val_t ok_json(void);
el_val_t err_json(el_val_t msg);
@@ -20,16 +25,246 @@ el_val_t route_create_edge(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neighbors(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_strengthen(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_forget(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_save(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_decay(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_export(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_reindex(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_load(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_health(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_session_begin(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_ctx(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_memory(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_knowledge_capture(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_knowledge_evolve(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_knowledge_promote(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_recall(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_graph(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_graph_link(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_list(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_consolidate(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_config(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_state_events(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neuron_processes(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_events_next(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_events_ack(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_bm25_search(el_val_t method, el_val_t path, el_val_t body);
el_val_t check_auth_ok(el_val_t method, el_val_t body);
el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body);
el_val_t bind_str;
el_val_t port;
el_val_t data_dir;
el_val_t snapshot_path;
el_val_t db_path;
el_val_t loaded;
el_val_t bm25_tokenize(el_val_t text) {
el_val_t t = str_to_lower(text);
t = str_replace(t, EL_STR("."), EL_STR(" "));
t = str_replace(t, EL_STR(","), EL_STR(" "));
t = str_replace(t, EL_STR("!"), EL_STR(" "));
t = str_replace(t, EL_STR("?"), EL_STR(" "));
t = str_replace(t, EL_STR("\""), EL_STR(" "));
t = str_replace(t, EL_STR(":"), EL_STR(" "));
t = str_replace(t, EL_STR(";"), EL_STR(" "));
t = str_replace(t, EL_STR("("), EL_STR(" "));
t = str_replace(t, EL_STR(")"), EL_STR(" "));
t = str_replace(t, EL_STR("["), EL_STR(" "));
t = str_replace(t, EL_STR("]"), EL_STR(" "));
t = str_replace(t, EL_STR("{"), EL_STR(" "));
t = str_replace(t, EL_STR("}"), EL_STR(" "));
t = str_replace(t, EL_STR("/"), EL_STR(" "));
t = str_replace(t, EL_STR("\\"), EL_STR(" "));
t = str_replace(t, EL_STR("'"), EL_STR(" "));
t = str_replace(t, EL_STR("-"), EL_STR(" "));
t = str_replace(t, EL_STR("_"), EL_STR(" "));
t = str_replace(t, EL_STR("+"), EL_STR(" "));
return str_trim(t);
return 0;
}
el_val_t bm25_count_term(el_val_t term, el_val_t doc_tokens) {
el_val_t padded_term = el_str_concat(el_str_concat(EL_STR(" "), term), EL_STR(" "));
el_val_t padded_doc = el_str_concat(el_str_concat(EL_STR(" "), doc_tokens), EL_STR(" "));
return str_count(padded_doc, padded_term);
return 0;
}
el_val_t bm25_score_doc(el_val_t doc_content, el_val_t query_tokens, el_val_t corpus_size, el_val_t avg_doc_len) {
el_val_t k1 = el_from_float(1.2);
el_val_t b = el_from_float(0.75);
el_val_t delta = el_from_float(1.0);
el_val_t doc_tokens = bm25_tokenize(doc_content);
el_val_t doc_wc = str_count_words(doc_tokens);
if (doc_wc == 0) {
return EL_STR("0.0");
}
el_val_t doc_len = int_to_float(doc_wc);
el_val_t avg_len = str_to_float(avg_doc_len);
el_val_t N = int_to_float(corpus_size);
el_val_t idf_arg = float_add(float_div(float_add(N, el_from_float(1.2)), el_from_float(1.5)), el_from_float(1.0));
el_val_t idf = math_log(idf_arg);
el_val_t terms = str_split(query_tokens, EL_STR(" "));
el_val_t n_terms = len(terms);
el_val_t score = el_from_float(0.0);
el_val_t i = 0;
while (i < n_terms) {
el_val_t term = get(terms, i);
el_val_t tlen = str_len(term);
if (tlen >= 2) {
el_val_t tf_count = bm25_count_term(term, doc_tokens);
if (tf_count > 0) {
el_val_t tf_raw = int_to_float(tf_count);
el_val_t norm_factor = float_add(float_sub(el_from_float(1.0), b), float_div(float_mul(b, doc_len), avg_len));
el_val_t numerator = float_mul(tf_raw, float_add(k1, el_from_float(1.0)));
el_val_t denominator = float_add(tf_raw, float_mul(k1, norm_factor));
el_val_t tf_comp = float_add(delta, float_div(numerator, denominator));
score = float_add(score, float_mul(idf, tf_comp));
}
}
i = (i + 1);
}
return float_to_str(score);
return 0;
}
el_val_t bm25_search_json(el_val_t query, el_val_t limit) {
el_val_t scan_limit = (limit * 10);
if (scan_limit < 200) {
scan_limit = 200;
}
if (scan_limit > 5000) {
scan_limit = 5000;
}
el_val_t nodes_json = engram_scan_nodes_json(scan_limit, 0);
el_val_t n = json_array_len(nodes_json);
if (n == 0) {
return EL_STR("[]");
}
el_val_t total_words = 0;
el_val_t i = 0;
while (i < n) {
el_val_t node = json_array_get(nodes_json, i);
el_val_t content = json_get_string(node, EL_STR("content"));
el_val_t tokens = bm25_tokenize(content);
el_val_t wc = str_count_words(tokens);
total_words = (total_words + wc);
i = (i + 1);
}
el_val_t avg_doc_len_f = float_div(int_to_float(total_words), int_to_float(n));
el_val_t avg_doc_len = ({ el_val_t _if_result_1 = 0; if (float_gt(avg_doc_len_f, el_from_float(0.0))) { _if_result_1 = (float_to_str(avg_doc_len_f)); } else { _if_result_1 = (EL_STR("1.0")); } _if_result_1; });
el_val_t query_tokens = bm25_tokenize(query);
if (str_eq(str_trim(query_tokens), EL_STR(""))) {
return EL_STR("[]");
}
el_val_t result_nodes = 0;
el_val_t result_scores = 0;
el_val_t result_count = 0;
el_val_t j = 0;
while (j < n) {
el_val_t node = json_array_get(nodes_json, j);
el_val_t content = json_get_string(node, EL_STR("content"));
el_val_t sc_str = bm25_score_doc(content, query_tokens, n, avg_doc_len);
if (float_gt(str_to_float(sc_str), el_from_float(0.0))) {
result_nodes = list_push(result_nodes, node);
result_scores = list_push(result_scores, sc_str);
result_count = (result_count + 1);
}
j = (j + 1);
}
if (result_count == 0) {
return EL_STR("[]");
}
el_val_t out_limit = ({ el_val_t _if_result_2 = 0; if ((result_count < limit)) { _if_result_2 = (result_count); } else { _if_result_2 = (limit); } _if_result_2; });
el_val_t k = 0;
while (k < out_limit) {
el_val_t max_idx = k;
el_val_t max_sc_str = get(result_scores, k);
el_val_t max_sc_f = str_to_float(max_sc_str);
el_val_t p = (k + 1);
while (p < result_count) {
el_val_t sc2_str = get(result_scores, p);
el_val_t sc2_f = str_to_float(sc2_str);
if (float_gt(sc2_f, max_sc_f)) {
max_sc_f = sc2_f;
max_sc_str = sc2_str;
max_idx = p;
}
p = (p + 1);
}
if (max_idx != k) {
el_val_t tmp_node = get(result_nodes, k);
el_val_t tmp_sc = get(result_scores, k);
result_nodes = list_set(result_nodes, k, get(result_nodes, max_idx));
result_scores = list_set(result_scores, k, get(result_scores, max_idx));
result_nodes = list_set(result_nodes, max_idx, tmp_node);
result_scores = list_set(result_scores, max_idx, tmp_sc);
}
k = (k + 1);
}
el_val_t out = EL_STR("[");
el_val_t r = 0;
while (r < out_limit) {
el_val_t node = get(result_nodes, r);
el_val_t sc_str = get(result_scores, r);
el_val_t node_len = str_len(node);
el_val_t node_body = str_slice(node, 0, (node_len - 1));
el_val_t entry = el_str_concat(el_str_concat(el_str_concat(node_body, EL_STR(",\"bm25_score\":")), sc_str), EL_STR("}"));
if (r > 0) {
out = el_str_concat(out, EL_STR(","));
}
out = el_str_concat(out, entry);
r = (r + 1);
}
return el_str_concat(out, EL_STR("]"));
return 0;
}
el_val_t auto_link_content_node(el_val_t node_id, el_val_t content) {
el_val_t clen = str_len(content);
if (clen < 20) {
return 0;
}
el_val_t sp1 = str_index_of(content, EL_STR(" "));
el_val_t w1end = ({ el_val_t _if_result_3 = 0; if ((sp1 < 0)) { _if_result_3 = (clen); } else { _if_result_3 = (sp1); } _if_result_3; });
el_val_t word1 = str_slice(content, 0, w1end);
state_set(EL_STR("aln_term"), EL_STR(""));
if (str_len(word1) >= 5) {
state_set(EL_STR("aln_term"), word1);
}
if (str_eq(state_get(EL_STR("aln_term")), EL_STR(""))) {
if (sp1 >= 0) {
el_val_t rest = str_slice(content, (sp1 + 1), clen);
el_val_t sp2 = str_index_of(rest, EL_STR(" "));
el_val_t w2end = ({ el_val_t _if_result_4 = 0; if ((sp2 < 0)) { _if_result_4 = (str_len(rest)); } else { _if_result_4 = (sp2); } _if_result_4; });
el_val_t word2 = str_slice(rest, 0, w2end);
if (str_len(word2) >= 5) {
state_set(EL_STR("aln_term"), word2);
}
}
}
el_val_t search_term = state_get(EL_STR("aln_term"));
if (str_eq(search_term, EL_STR(""))) {
return 0;
}
el_val_t results = bm25_search_json(search_term, 20);
el_val_t n = json_array_len(results);
state_set(EL_STR("aln_linked"), EL_STR("0"));
el_val_t i = 0;
while (i < n) {
el_val_t linked_so_far = str_to_int(state_get(EL_STR("aln_linked")));
if (linked_so_far < 3) {
el_val_t elem = json_array_get(results, i);
el_val_t rid = json_get_string(elem, EL_STR("id"));
el_val_t rtype = json_get_string(elem, EL_STR("node_type"));
if ((!str_eq(rtype, EL_STR("InternalStateEvent")) && !str_eq(rid, EL_STR(""))) && !str_eq(rid, node_id)) {
engram_connect(node_id, rid, el_from_float(0.6), EL_STR("related"));
state_set(EL_STR("aln_linked"), int_to_str((linked_so_far + 1)));
}
}
i = (i + 1);
}
return str_to_int(state_get(EL_STR("aln_linked")));
return 0;
}
el_val_t parse_port(el_val_t bind) {
el_val_t colon = str_index_of(bind, EL_STR(":"));
@@ -119,7 +354,8 @@ el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body) {
salience = el_from_float(0.5);
}
el_val_t id = engram_node(content, node_type, salience);
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"id\":\""), id), EL_STR("\",\"content\":\"")), content), EL_STR("\",\"node_type\":\"")), node_type), EL_STR("\"}"));
el_val_t auto_linked = auto_link_content_node(id, content);
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"id\":\""), id), EL_STR("\",\"content\":\"")), content), EL_STR("\",\"node_type\":\"")), node_type), EL_STR("\",\"auto_linked\":")), int_to_str(auto_linked)), EL_STR("}"));
return 0;
}
@@ -176,7 +412,7 @@ el_val_t route_search(el_val_t method, el_val_t path, el_val_t body) {
if (limit == 0) {
limit = 20;
}
return engram_search_json(q, limit);
return bm25_search_json(q, limit);
return 0;
}
@@ -193,6 +429,17 @@ el_val_t route_activate(el_val_t method, el_val_t path, el_val_t body) {
depth = bd;
}
}
el_val_t top = bm25_search_json(q, 10);
el_val_t nb = json_array_len(top);
el_val_t bi = 0;
while (bi < nb) {
el_val_t node = json_array_get(top, bi);
el_val_t nid = json_get_string(node, EL_STR("id"));
if (!str_eq(nid, EL_STR(""))) {
engram_strengthen(nid);
}
bi = (bi + 1);
}
return el_str_concat(el_str_concat(EL_STR("{\"results\":"), engram_activate_json(q, depth)), EL_STR("}"));
return 0;
}
@@ -243,30 +490,46 @@ el_val_t route_forget(el_val_t method, el_val_t path, el_val_t body) {
return 0;
}
el_val_t route_save(el_val_t method, el_val_t path, el_val_t body) {
el_val_t route_decay(el_val_t method, el_val_t path, el_val_t body) {
return engram_apply_decay_json();
return 0;
}
el_val_t route_export(el_val_t method, el_val_t path, el_val_t body) {
el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
el_val_t db_path = el_str_concat(dir, EL_STR("/engram.db"));
engram_write_binary_el(db_path);
el_val_t p = json_get_string(body, EL_STR("path"));
if (str_eq(p, EL_STR(""))) {
el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
p = el_str_concat(dir, EL_STR("/snapshot.json"));
}
engram_save(p);
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"path\":\""), p), EL_STR("\"}"));
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"binary\":\""), db_path), EL_STR("\",\"json\":\"")), p), EL_STR("\"}"));
return 0;
}
el_val_t route_reindex(el_val_t method, el_val_t path, el_val_t body) {
return engram_reindex_json();
return 0;
}
el_val_t route_load(el_val_t method, el_val_t path, el_val_t body) {
el_val_t p = json_get_string(body, EL_STR("path"));
if (str_eq(p, EL_STR(""))) {
el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
p = el_str_concat(dir, EL_STR("/snapshot.json"));
el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
el_val_t db_path = el_str_concat(dir, EL_STR("/engram.db"));
el_val_t ok = engram_load_binary_el(db_path);
if (!ok) {
el_val_t p = json_get_string(body, EL_STR("path"));
if (str_eq(p, EL_STR(""))) {
p = el_str_concat(dir, EL_STR("/snapshot.json"));
}
engram_load(p);
}
engram_load(p);
return ok_json();
return 0;
}
@@ -276,6 +539,344 @@ el_val_t route_health(el_val_t method, el_val_t path, el_val_t body) {
return 0;
}
el_val_t route_neuron_session_begin(el_val_t method, el_val_t path, el_val_t body) {
el_val_t results = engram_activate_json(EL_STR("memory knowledge context"), 2);
el_val_t nc = engram_node_count();
el_val_t ec = engram_edge_count();
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"nodes\":"), results), EL_STR(",\"node_count\":")), int_to_str(nc)), EL_STR(",\"edge_count\":")), int_to_str(ec)), EL_STR("}"));
return 0;
}
el_val_t route_neuron_ctx(el_val_t method, el_val_t path, el_val_t body) {
el_val_t results = engram_activate_json(EL_STR("architecture decision memory"), 2);
el_val_t n = json_array_len(results);
el_val_t limit = ({ el_val_t _if_result_5 = 0; if ((n > 10)) { _if_result_5 = (10); } else { _if_result_5 = (n); } _if_result_5; });
el_val_t ctx = EL_STR("Recent working memory:\n");
el_val_t i = 0;
el_val_t ctx_body = EL_STR("");
while (i < limit) {
el_val_t elem = json_array_get(results, i);
el_val_t label = json_get_string(elem, EL_STR("label"));
el_val_t content = json_get_string(elem, EL_STR("content"));
el_val_t clen = str_len(content);
el_val_t snippet = ({ el_val_t _if_result_6 = 0; if ((clen > 200)) { _if_result_6 = (str_slice(content, 0, 200)); } else { _if_result_6 = (content); } _if_result_6; });
ctx_body = el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(ctx_body, EL_STR("- [")), label), EL_STR("]: ")), snippet), EL_STR("\n"));
i = (i + 1);
}
el_val_t full_ctx = el_str_concat(ctx, ctx_body);
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"context\":\""), str_replace(str_replace(str_replace(full_ctx, EL_STR("\\"), EL_STR("\\\\")), EL_STR("\""), EL_STR("\\\"")), EL_STR("\n"), EL_STR("\\n"))), EL_STR("\"}"));
return 0;
}
el_val_t route_neuron_memory(el_val_t method, el_val_t path, el_val_t body) {
el_val_t content = json_get_string(body, EL_STR("content"));
if (str_eq(content, EL_STR(""))) {
return EL_STR("{\"error\":\"content is required\"}");
}
el_val_t node_type = json_get_string(body, EL_STR("node_type"));
if (str_eq(node_type, EL_STR(""))) {
node_type = EL_STR("Memory");
}
el_val_t label = json_get_string(body, EL_STR("label"));
el_val_t importance = json_get_string(body, EL_STR("importance"));
el_val_t project = json_get_string(body, EL_STR("project"));
el_val_t tags_raw = json_get_string(body, EL_STR("tags"));
el_val_t tier = EL_STR("Episodic");
if (str_eq(importance, EL_STR("critical"))) {
tier = EL_STR("Procedural");
}
if (str_eq(importance, EL_STR("high"))) {
tier = EL_STR("Semantic");
}
if (str_eq(importance, EL_STR("normal"))) {
tier = EL_STR("Episodic");
}
if (str_eq(importance, EL_STR("low"))) {
tier = EL_STR("Working");
}
el_val_t explicit_tier = json_get_string(body, EL_STR("tier"));
if (!str_eq(explicit_tier, EL_STR(""))) {
tier = explicit_tier;
}
el_val_t tags_str = tags_raw;
if (!str_eq(project, EL_STR(""))) {
if (str_eq(tags_str, EL_STR(""))) {
tags_str = el_str_concat(EL_STR("project:"), project);
}
if (!str_eq(tags_str, EL_STR(""))) {
tags_str = el_str_concat(el_str_concat(tags_str, EL_STR(" project:")), project);
}
}
el_val_t id = engram_node_full(content, node_type, label, el_from_float(0.5), el_from_float(0.5), el_from_float(1.0), tier, tags_str);
el_val_t auto_linked = auto_link_content_node(id, content);
el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
el_val_t db_path = el_str_concat(dir, EL_STR("/engram.db"));
engram_write_binary_el(db_path);
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\",\"auto_linked\":")), int_to_str(auto_linked)), EL_STR(",\"content\":\"")), str_replace(str_replace(content, EL_STR("\\"), EL_STR("\\\\")), EL_STR("\""), EL_STR("\\\""))), EL_STR("\"}"));
return 0;
}
el_val_t route_neuron_knowledge_capture(el_val_t method, el_val_t path, el_val_t body) {
el_val_t content = json_get_string(body, EL_STR("content"));
if (str_eq(content, EL_STR(""))) {
return EL_STR("{\"error\":\"content is required\"}");
}
el_val_t title = json_get_string(body, EL_STR("title"));
el_val_t category = json_get_string(body, EL_STR("category"));
el_val_t tags_raw = json_get_string(body, EL_STR("tags"));
el_val_t project = json_get_string(body, EL_STR("project"));
el_val_t tier_raw = json_get_string(body, EL_STR("tier"));
el_val_t tier = EL_STR("Episodic");
if (str_eq(tier_raw, EL_STR("lesson"))) {
tier = EL_STR("Semantic");
}
if (str_eq(tier_raw, EL_STR("canonical"))) {
tier = EL_STR("Procedural");
}
if (str_eq(tier_raw, EL_STR("note"))) {
tier = EL_STR("Episodic");
}
el_val_t tags_str = tags_raw;
if (!str_eq(category, EL_STR(""))) {
if (str_eq(tags_str, EL_STR(""))) {
tags_str = el_str_concat(EL_STR("category:"), category);
}
if (!str_eq(tags_str, EL_STR(""))) {
tags_str = el_str_concat(el_str_concat(tags_str, EL_STR(" category:")), category);
}
}
if (!str_eq(project, EL_STR(""))) {
if (str_eq(tags_str, EL_STR(""))) {
tags_str = el_str_concat(EL_STR("project:"), project);
}
if (!str_eq(tags_str, EL_STR(""))) {
tags_str = el_str_concat(el_str_concat(tags_str, EL_STR(" project:")), project);
}
}
el_val_t id = engram_node_full(content, EL_STR("Knowledge"), title, el_from_float(0.7), el_from_float(0.7), el_from_float(1.0), tier, tags_str);
el_val_t auto_linked = auto_link_content_node(id, content);
el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
el_val_t db_path = el_str_concat(dir, EL_STR("/engram.db"));
engram_write_binary_el(db_path);
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\",\"auto_linked\":")), int_to_str(auto_linked)), EL_STR("}"));
return 0;
}
el_val_t route_neuron_knowledge_evolve(el_val_t method, el_val_t path, el_val_t body) {
el_val_t content = json_get_string(body, EL_STR("content"));
el_val_t prior_id = json_get_string(body, EL_STR("id"));
if (str_eq(content, EL_STR(""))) {
return EL_STR("{\"ok\":true}");
}
el_val_t id = engram_node_full(content, EL_STR("Knowledge"), EL_STR(""), el_from_float(0.7), el_from_float(0.7), el_from_float(1.0), EL_STR("Semantic"), EL_STR("evolved"));
if (!str_eq(prior_id, EL_STR("")) && !str_eq(id, EL_STR(""))) {
engram_connect(id, prior_id, el_from_float(1.0), EL_STR("supersedes"));
}
el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
engram_write_binary_el(el_str_concat(dir, EL_STR("/engram.db")));
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\"}"));
return 0;
}
el_val_t route_neuron_knowledge_promote(el_val_t method, el_val_t path, el_val_t body) {
el_val_t id = json_get_string(body, EL_STR("id"));
if (str_eq(id, EL_STR(""))) {
return EL_STR("{\"ok\":true}");
}
el_val_t node_json = engram_get_node_json(id);
if (str_eq(node_json, EL_STR(""))) {
return err_json(EL_STR("node not found"));
}
if (str_eq(node_json, EL_STR("null"))) {
return err_json(EL_STR("node not found"));
}
el_val_t content = json_get_string(node_json, EL_STR("content"));
if (str_eq(content, EL_STR(""))) {
return err_json(EL_STR("node has no content"));
}
el_val_t label = json_get_string(node_json, EL_STR("label"));
el_val_t tags = json_get_string(node_json, EL_STR("tags"));
el_val_t current_tier = json_get_string(node_json, EL_STR("tier"));
el_val_t tier_raw = json_get_string(body, EL_STR("tier"));
el_val_t new_tier = EL_STR("");
if (str_eq(tier_raw, EL_STR("lesson"))) {
new_tier = EL_STR("Semantic");
}
if (str_eq(tier_raw, EL_STR("canonical"))) {
new_tier = EL_STR("Procedural");
}
if (str_eq(tier_raw, EL_STR("note"))) {
new_tier = EL_STR("Episodic");
}
if (str_eq(new_tier, EL_STR(""))) {
if (str_eq(current_tier, EL_STR("Working"))) {
new_tier = EL_STR("Episodic");
}
if (str_eq(current_tier, EL_STR("Episodic"))) {
new_tier = EL_STR("Semantic");
}
if (str_eq(current_tier, EL_STR("Semantic"))) {
new_tier = EL_STR("Procedural");
}
if (str_eq(current_tier, EL_STR("Procedural"))) {
new_tier = EL_STR("Procedural");
}
}
if (str_eq(new_tier, EL_STR(""))) {
new_tier = EL_STR("Semantic");
}
el_val_t new_id = engram_node_full(content, EL_STR("Knowledge"), label, el_from_float(0.7), el_from_float(0.8), el_from_float(1.0), new_tier, tags);
if (!str_eq(new_id, EL_STR(""))) {
engram_connect(new_id, id, el_from_float(1.0), EL_STR("supersedes"));
}
el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
engram_write_binary_el(el_str_concat(dir, EL_STR("/engram.db")));
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), new_id), EL_STR("\",\"promoted_from\":\"")), id), EL_STR("\",\"tier\":\"")), new_tier), EL_STR("\"}"));
return 0;
}
el_val_t route_neuron_recall(el_val_t method, el_val_t path, el_val_t body) {
el_val_t query = json_get_string(body, EL_STR("query"));
el_val_t chain = json_get_string(body, EL_STR("chain_name"));
el_val_t limit = json_get_int(body, EL_STR("limit"));
if (limit == 0) {
limit = 20;
}
el_val_t q = ({ el_val_t _if_result_7 = 0; if (str_eq(query, EL_STR(""))) { _if_result_7 = (chain); } else { _if_result_7 = (query); } _if_result_7; });
if (str_eq(q, EL_STR(""))) {
return engram_scan_nodes_json(limit, 0);
}
return bm25_search_json(q, limit);
return 0;
}
el_val_t route_neuron_graph(el_val_t method, el_val_t path, el_val_t body) {
el_val_t id = query_param(path, EL_STR("id"));
if (str_eq(id, EL_STR(""))) {
return EL_STR("{\"error\":\"id is required\"}");
}
el_val_t node_json = engram_get_node_json(id);
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"node\":"), node_json), EL_STR(",\"neighbors\":[]}"));
return 0;
}
el_val_t route_neuron_graph_link(el_val_t method, el_val_t path, el_val_t body) {
el_val_t from_id = json_get_string(body, EL_STR("from_id"));
el_val_t to_id = json_get_string(body, EL_STR("to_id"));
if (str_eq(from_id, EL_STR("")) || str_eq(to_id, EL_STR(""))) {
return EL_STR("{\"error\":\"from_id and to_id are required\"}");
}
el_val_t relation = json_get_string(body, EL_STR("relation"));
if (str_eq(relation, EL_STR(""))) {
relation = EL_STR("related");
}
el_val_t weight = json_get_float(body, EL_STR("weight"));
if (str_eq(weight, el_from_float(0.0))) {
weight = el_from_float(0.5);
}
engram_connect(from_id, to_id, weight, relation);
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"from_id\":\""), from_id), EL_STR("\",\"to_id\":\"")), to_id), EL_STR("\",\"relation\":\"")), relation), EL_STR("\"}"));
return 0;
}
el_val_t route_neuron_list(el_val_t method, el_val_t path, el_val_t body) {
el_val_t clean = strip_query(path);
el_val_t prefix = EL_STR("/api/neuron/list/");
el_val_t node_type = str_slice(clean, str_len(prefix), str_len(clean));
el_val_t limit = query_int(path, EL_STR("limit"), 50);
if (str_eq(node_type, EL_STR(""))) {
return EL_STR("[]");
}
return engram_scan_nodes_by_type_json(node_type, limit, 0);
return 0;
}
el_val_t route_neuron_consolidate(el_val_t method, el_val_t path, el_val_t body) {
el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
el_val_t db_path = el_str_concat(dir, EL_STR("/engram.db"));
engram_write_binary_el(db_path);
el_val_t nc = engram_node_count();
el_val_t ec = engram_edge_count();
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"node_count\":"), int_to_str(nc)), EL_STR(",\"edge_count\":")), int_to_str(ec)), EL_STR("}"));
return 0;
}
el_val_t route_neuron_config(el_val_t method, el_val_t path, el_val_t body) {
el_val_t key = query_param(path, EL_STR("key"));
return el_str_concat(el_str_concat(EL_STR("{\"key\":\""), key), EL_STR("\",\"value\":\"\"}"));
return 0;
}
el_val_t route_neuron_state_events(el_val_t method, el_val_t path, el_val_t body) {
if (str_eq(method, EL_STR("GET"))) {
el_val_t limit_str = query_param(path, EL_STR("limit"));
el_val_t limit = ({ el_val_t _if_result_8 = 0; if (str_eq(limit_str, EL_STR(""))) { _if_result_8 = (50); } else { _if_result_8 = (str_to_int(limit_str)); } _if_result_8; });
el_val_t offset_str = query_param(path, EL_STR("offset"));
el_val_t offset = ({ el_val_t _if_result_9 = 0; if (str_eq(offset_str, EL_STR(""))) { _if_result_9 = (0); } else { _if_result_9 = (str_to_int(offset_str)); } _if_result_9; });
return engram_scan_nodes_by_type_json(EL_STR("InternalStateEvent"), limit, offset);
}
el_val_t content = json_get_string(body, EL_STR("content"));
if (str_eq(content, EL_STR(""))) {
content = body;
}
el_val_t event_label = json_get_string(content, EL_STR("event"));
el_val_t label = ({ el_val_t _if_result_10 = 0; if (str_eq(event_label, EL_STR(""))) { _if_result_10 = (EL_STR("state-event")); } else { _if_result_10 = (event_label); } _if_result_10; });
el_val_t id = engram_node_full(content, EL_STR("InternalStateEvent"), label, el_from_float(0.3), el_from_float(0.3), el_from_float(1.0), EL_STR("Working"), EL_STR("internal-state"));
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\"}"));
return 0;
}
el_val_t route_neuron_processes(el_val_t method, el_val_t path, el_val_t body) {
return EL_STR("{\"ok\":true,\"processes\":[]}");
return 0;
}
el_val_t route_events_next(el_val_t method, el_val_t path, el_val_t body) {
return EL_STR("{\"ok\":true,\"event\":null}");
return 0;
}
el_val_t route_events_ack(el_val_t method, el_val_t path, el_val_t body) {
return EL_STR("{\"ok\":true}");
return 0;
}
el_val_t route_bm25_search(el_val_t method, el_val_t path, el_val_t body) {
el_val_t q = EL_STR("");
if (str_eq(method, EL_STR("GET"))) {
q = query_param(path, EL_STR("q"));
} else {
q = json_get_string(body, EL_STR("query"));
}
if (str_eq(q, EL_STR(""))) {
return EL_STR("{\"error\":\"query is required\"}");
}
el_val_t limit = query_int(path, EL_STR("limit"), 20);
if (limit == 0) {
limit = json_get_int(body, EL_STR("limit"));
}
if (limit == 0) {
limit = 20;
}
return bm25_search_json(q, limit);
return 0;
}
el_val_t check_auth_ok(el_val_t method, el_val_t body) {
el_val_t key = env(EL_STR("ENGRAM_API_KEY"));
if (str_eq(key, EL_STR(""))) {
@@ -299,6 +900,60 @@ el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body) {
return route_health(method, path, body);
}
}
if (str_starts_with(clean, EL_STR("/api/neuron/")) || str_starts_with(clean, EL_STR("/events/"))) {
if (str_eq(clean, EL_STR("/api/neuron/session/begin"))) {
return route_neuron_session_begin(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/ctx"))) {
return route_neuron_ctx(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/memory"))) {
return route_neuron_memory(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/knowledge/capture"))) {
return route_neuron_knowledge_capture(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/knowledge/evolve"))) {
return route_neuron_knowledge_evolve(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/knowledge/promote"))) {
return route_neuron_knowledge_promote(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/recall"))) {
return route_neuron_recall(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/graph/link"))) {
return route_neuron_graph_link(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/graph"))) {
return route_neuron_graph(method, path, body);
}
if (str_starts_with(clean, EL_STR("/api/neuron/list/"))) {
return route_neuron_list(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/consolidate"))) {
return route_neuron_consolidate(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/config"))) {
return route_neuron_config(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/state-events"))) {
return route_neuron_state_events(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/processes/define"))) {
return route_neuron_processes(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/processes"))) {
return route_neuron_processes(method, path, body);
}
if (str_eq(clean, EL_STR("/events/next"))) {
return route_events_next(method, path, body);
}
if (str_eq(clean, EL_STR("/events/ack"))) {
return route_events_ack(method, path, body);
}
return err_json(EL_STR("not found"));
}
if (!check_auth_ok(method, body)) {
return err_json(EL_STR("unauthorized"));
}
@@ -338,15 +993,80 @@ el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body) {
if (str_eq(method, EL_STR("GET")) && str_starts_with(clean, EL_STR("/api/search"))) {
return route_search(method, path, body);
}
if (str_eq(clean, EL_STR("/api/bm25/search"))) {
return route_bm25_search(method, path, body);
}
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/strengthen")) || str_eq(clean, EL_STR("/strengthen")))) {
return route_strengthen(method, path, body);
}
if (str_eq(method, EL_STR("POST")) && ((str_eq(clean, EL_STR("/api/decay")) || str_eq(clean, EL_STR("/api/maintenance"))) || str_eq(clean, EL_STR("/decay")))) {
return route_decay(method, path, body);
}
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/export")) || str_eq(clean, EL_STR("/export")))) {
return route_export(method, path, body);
}
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/save")) || str_eq(clean, EL_STR("/save")))) {
return route_save(method, path, body);
return route_export(method, path, body);
}
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/load")) || str_eq(clean, EL_STR("/load")))) {
return route_load(method, path, body);
}
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/reindex")) || str_eq(clean, EL_STR("/reindex")))) {
return route_reindex(method, path, body);
}
if (str_starts_with(clean, EL_STR("/api/neuron/"))) {
if (str_eq(clean, EL_STR("/api/neuron/session/begin"))) {
return route_neuron_session_begin(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/ctx"))) {
return route_neuron_ctx(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/memory"))) {
return route_neuron_memory(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/knowledge/capture"))) {
return route_neuron_knowledge_capture(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/knowledge/evolve"))) {
return route_neuron_knowledge_evolve(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/knowledge/promote"))) {
return route_neuron_knowledge_promote(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/recall"))) {
return route_neuron_recall(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/graph/link"))) {
return route_neuron_graph_link(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/graph"))) {
return route_neuron_graph(method, path, body);
}
if (str_starts_with(clean, EL_STR("/api/neuron/list/"))) {
return route_neuron_list(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/consolidate"))) {
return route_neuron_consolidate(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/config"))) {
return route_neuron_config(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/state-events"))) {
return route_neuron_state_events(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/processes/define"))) {
return route_neuron_processes(method, path, body);
}
if (str_eq(clean, EL_STR("/api/neuron/processes"))) {
return route_neuron_processes(method, path, body);
}
}
if (str_eq(clean, EL_STR("/events/next"))) {
return route_events_next(method, path, body);
}
if (str_eq(clean, EL_STR("/events/ack"))) {
return route_events_ack(method, path, body);
}
return el_str_concat(el_str_concat(EL_STR("{\"error\":\"not found\",\"path\":\""), clean), EL_STR("\"}"));
return 0;
}
@@ -362,9 +1082,20 @@ int main(int _argc, char** _argv) {
if (str_eq(data_dir, EL_STR(""))) {
data_dir = EL_STR("/tmp/engram");
}
snapshot_path = el_str_concat(data_dir, EL_STR("/snapshot.json"));
engram_load(snapshot_path);
println(EL_STR("[engram] runtime-native graph engine"));
db_path = el_str_concat(data_dir, EL_STR("/engram.db"));
loaded = engram_load_binary_el(db_path);
if (!loaded) {
engram_load_dir(data_dir);
if (engram_node_count() == 0) {
el_val_t snapshot_path = el_str_concat(data_dir, EL_STR("/snapshot.json"));
engram_load(snapshot_path);
}
if (engram_node_count() > 0) {
engram_write_binary_el(db_path);
println(EL_STR("[engram] migrated legacy data to binary format"));
}
}
println(EL_STR("[engram] runtime-native graph engine (ML-KEM-1024 encrypted)"));
println(el_str_concat(EL_STR("[engram] data_dir="), data_dir));
println(el_str_concat(EL_STR("[engram] node_count="), int_to_str(engram_node_count())));
println(el_str_concat(EL_STR("[engram] edge_count="), int_to_str(engram_edge_count())));
+799 -20
View File
@@ -6,15 +6,297 @@
// database.
//
// Built and linked with:
// elc src/server.el > server.c
// cc -std=c11 -O2 -lcurl -lpthread -o engram server.c el_runtime.c
// ./engram
// elc src/server.el > ../dist/engram.c
// cc -std=c11 -O2 \
// -I/Users/will/Development/neuron-technologies/foundation/el/lang/releases/v1.0.0-20260501 \
// -I/opt/homebrew/Cellar/liboqs/0.15.0/include \
// -I/opt/homebrew/opt/openssl@3/include \
// -L/opt/homebrew/Cellar/liboqs/0.15.0/lib \
// -L/opt/homebrew/opt/openssl@3/lib \
// -lcurl -lpthread -loqs -lssl -lcrypto \
// -o ../dist/engram ../dist/engram.c \
// /Users/will/Development/neuron-technologies/foundation/el/lang/releases/v1.0.0-20260501/el_runtime.c
// ./dist/engram
//
// Configuration via environment:
// ENGRAM_BIND host:port (default :8742)
// ENGRAM_API_KEY bearer auth (optional)
// ENGRAM_DATA_DIR snapshot location (default ~/.neuron/engram)
// BM25+ text ranking
//
// Implements BM25+ (Lv & Zhai 2011) for in-process keyword search over the
// engram node store. No external dependencies pure EL, zero Ollama calls.
//
// Parameters: k1=1.2, b=0.75, delta=1.0
//
// V1 simplification: n(t) (number of docs containing term t) is approximated
// as 1 for all terms. This collapses IDF to a constant per corpus size:
// IDF = ln((N - 1 + 0.5) / (1 + 0.5) + 1) = ln((N + 0.5) / 1.5 + 1)
// Scoring effectively becomes TF-length-normalised BM25+ (delta term present).
// Acceptable for V1; a real inverted index can replace this later.
fn bm25_tokenize(text: String) -> String {
// Lowercase and strip punctuation (replace with spaces), then trim.
let t: String = str_to_lower(text)
let t = str_replace(t, ".", " ")
let t = str_replace(t, ",", " ")
let t = str_replace(t, "!", " ")
let t = str_replace(t, "?", " ")
let t = str_replace(t, "\"", " ")
let t = str_replace(t, ":", " ")
let t = str_replace(t, ";", " ")
let t = str_replace(t, "(", " ")
let t = str_replace(t, ")", " ")
let t = str_replace(t, "[", " ")
let t = str_replace(t, "]", " ")
let t = str_replace(t, "{", " ")
let t = str_replace(t, "}", " ")
let t = str_replace(t, "/", " ")
let t = str_replace(t, "\\", " ")
let t = str_replace(t, "'", " ")
let t = str_replace(t, "-", " ")
let t = str_replace(t, "_", " ")
let t = str_replace(t, "+", " ")
str_trim(t)
}
fn bm25_count_term(term: String, doc_tokens: String) -> Int {
// Pad with spaces to avoid prefix/suffix partial matches.
let padded_term: String = " " + term + " "
let padded_doc: String = " " + doc_tokens + " "
str_count(padded_doc, padded_term)
}
fn bm25_score_doc(doc_content: String, query_tokens: String, corpus_size: Int, avg_doc_len: String) -> String {
// BM25+ parameters (stored as strings = float-encoded el_val_t from el_from_float)
// We use float_add/float_mul/float_div builtins to avoid EL operator issues.
// avg_doc_len is passed as a String slot holding an el_val_t float bit-pattern.
// (EL has no safe float-passing convention; we work around using str_to_float.)
//
// V1: n_t=1 for all terms. IDF = ln((N+0.5)/1.5 + 1) = constant per corpus.
// This collapses BM25+ to TF-length-normalised scoring acceptable for V1.
let k1: Float = 1.2
let b: Float = 0.75
let delta: Float = 1.0
let doc_tokens: String = bm25_tokenize(doc_content)
let doc_wc: Int = str_count_words(doc_tokens)
if doc_wc == 0 { return "0.0" }
let doc_len: Float = int_to_float(doc_wc)
let avg_len: Float = str_to_float(avg_doc_len)
// IDF constant
let N: Float = int_to_float(corpus_size)
// (N + 0.5) / 1.5 + 1.0
let idf_arg: Float = float_add(float_div(float_add(N, 1.2), 1.5), 1.0)
let idf: Float = math_log(idf_arg)
// Sum TF component over query terms
let terms: List = str_split(query_tokens, " ")
let n_terms: Int = len(terms)
let score: Float = 0.0
let i: Int = 0
while i < n_terms {
let term: String = get(terms, i)
let tlen: Int = str_len(term)
if tlen >= 2 {
let tf_count: Int = bm25_count_term(term, doc_tokens)
if tf_count > 0 {
let tf_raw: Float = int_to_float(tf_count)
// norm_factor = 1 - b + b * doc_len / avg_len
let norm_factor: Float = float_add(float_sub(1.0, b), float_div(float_mul(b, doc_len), avg_len))
// tf_comp = delta + tf * (k1+1) / (tf + k1*norm)
let numerator: Float = float_mul(tf_raw, float_add(k1, 1.0))
let denominator: Float = float_add(tf_raw, float_mul(k1, norm_factor))
let tf_comp: Float = float_add(delta, float_div(numerator, denominator))
let score = float_add(score, float_mul(idf, tf_comp))
}
}
let i = i + 1
}
// Return score as a string so it survives EL's lack of float-in-list support
float_to_str(score)
}
fn bm25_search_json(query: String, limit: Int) -> String {
// 1. Determine scan size: floor at 200 so small `limit` values still scan
// enough of the corpus to find relevant nodes.
// Cap raised from 500 5000 (2026-05-24 self-review): 500 was 0.3% of the
// 161K-node corpus. At 5000 we cover the top-3% by salience still fast
// (pure C scan, no Ollama calls) and 10x better recall for content search.
// engram_scan_nodes_json returns nodes sorted by salience DESC, so ISEs
// (salience 0.3) naturally fall below Knowledge/Memory (0.50.8), keeping
// the effective search corpus content-dense.
let scan_limit: Int = limit * 10
if scan_limit < 200 { let scan_limit = 200 }
if scan_limit > 5000 { let scan_limit = 5000 }
// 2. Fetch node sample
let nodes_json: String = engram_scan_nodes_json(scan_limit, 0)
let n: Int = json_array_len(nodes_json)
if n == 0 { return "[]" }
// 3. Compute avg_doc_len from sample
let total_words: Int = 0
let i: Int = 0
while i < n {
let node: String = json_array_get(nodes_json, i)
let content: String = json_get_string(node, "content")
let tokens: String = bm25_tokenize(content)
let wc: Int = str_count_words(tokens)
let total_words = total_words + wc
let i = i + 1
}
// avg_doc_len as string for safe float passing
let avg_doc_len_f: Float = float_div(int_to_float(total_words), int_to_float(n))
let avg_doc_len: String = if float_gt(avg_doc_len_f, 0.0) { float_to_str(avg_doc_len_f) } else { "1.0" }
// 4. Tokenize query
let query_tokens: String = bm25_tokenize(query)
if str_eq(str_trim(query_tokens), "") { return "[]" }
// 5. Score each node; collect results as parallel JSON and score lists.
// Scores are stored as strings (float_to_str) to avoid float-in-list issues.
let result_nodes: List = 0
let result_scores: List = 0
let result_count: Int = 0
let j: Int = 0
while j < n {
let node: String = json_array_get(nodes_json, j)
let content: String = json_get_string(node, "content")
let sc_str: String = bm25_score_doc(content, query_tokens, n, avg_doc_len)
// Only include nodes with score > 0.0 (use float comparison, not string match
// float_to_str(0.0) returns "0.000000", not "0.0").
if float_gt(str_to_float(sc_str), 0.0) {
let result_nodes = list_push(result_nodes, node)
let result_scores = list_push(result_scores, sc_str)
let result_count = result_count + 1
}
let j = j + 1
}
if result_count == 0 { return "[]" }
// 6. Selection-sort descending by score, take top `limit`
let out_limit: Int = if result_count < limit { result_count } else { limit }
let k: Int = 0
while k < out_limit {
// Find max score index in [k, result_count)
let max_idx: Int = k
let max_sc_str: String = get(result_scores, k)
let max_sc_f: Float = str_to_float(max_sc_str)
let p: Int = k + 1
while p < result_count {
let sc2_str: String = get(result_scores, p)
let sc2_f: Float = str_to_float(sc2_str)
if float_gt(sc2_f, max_sc_f) {
let max_sc_f = sc2_f
let max_sc_str = sc2_str
let max_idx = p
}
let p = p + 1
}
// Swap k <-> max_idx
if max_idx != k {
let tmp_node: String = get(result_nodes, k)
let tmp_sc: String = get(result_scores, k)
let result_nodes = list_set(result_nodes, k, get(result_nodes, max_idx))
let result_scores = list_set(result_scores, k, get(result_scores, max_idx))
let result_nodes = list_set(result_nodes, max_idx, tmp_node)
let result_scores = list_set(result_scores, max_idx, tmp_sc)
}
let k = k + 1
}
// 7. Build JSON array of top `out_limit` nodes with bm25_score field
let out: String = "["
let r: Int = 0
while r < out_limit {
let node: String = get(result_nodes, r)
let sc_str: String = get(result_scores, r)
// Inject bm25_score: trim the closing } and append field
let node_len: Int = str_len(node)
let node_body: String = str_slice(node, 0, node_len - 1)
let entry: String = node_body + ",\"bm25_score\":" + sc_str + "}"
if r > 0 { let out = out + "," }
let out = out + entry
let r = r + 1
}
out + "]"
}
// Auto-linking
//
// auto_link_content_node link a newly-created Knowledge or Memory node to
// semantically related non-ISE nodes via BM25 search.
//
// Problem it solves: route_neuron_memory and route_neuron_knowledge_capture
// both call engram_node_full directly, creating nodes with zero edges. With
// 14K+ ISEs dominating the corpus, BFS traversal contributes nothing every
// query relies solely on lexical/semantic seed matching. Auto-linking builds
// explicit "related" edges so activated knowledge nodes fan out to connected
// neighbors during BFS.
//
// Design choices:
// - BM25 (not substring search): ranks by relevance, not just occurrence
// - Skip InternalStateEvent nodes: ISEs dominate the corpus and are not
// useful link targets for knowledge/memory nodes
// - Up to 3 edges per node: enough to build graph structure without over-linking
// - weight=0.6: moderately strong; causal edges (field-validated at 2.0) are
// much stronger, so these "related" edges don't flood activation paths
// - state_set for linked counter: EL `let` in nested if-blocks creates inner
// scope only; state_set persists across block boundaries (2026-05-25 lesson)
//
// (2026-05-28 self-review)
fn auto_link_content_node(node_id: String, content: String) -> Int {
let clen: Int = str_len(content)
if clen < 20 { return 0 }
// Find search term: first word >= 5 chars, or second word.
let sp1: Int = str_index_of(content, " ")
let w1end: Int = if sp1 < 0 { clen } else { sp1 }
let word1: String = str_slice(content, 0, w1end)
state_set("aln_term", "")
if str_len(word1) >= 5 {
state_set("aln_term", word1)
}
if str_eq(state_get("aln_term"), "") {
if sp1 >= 0 {
let rest: String = str_slice(content, sp1 + 1, clen)
let sp2: Int = str_index_of(rest, " ")
let w2end: Int = if sp2 < 0 { str_len(rest) } else { sp2 }
let word2: String = str_slice(rest, 0, w2end)
if str_len(word2) >= 5 {
state_set("aln_term", word2)
}
}
}
let search_term: String = state_get("aln_term")
if str_eq(search_term, "") { return 0 }
// BM25 over top-20 results; skip ISE nodes; connect up to 3.
let results: String = bm25_search_json(search_term, 20)
let n: Int = json_array_len(results)
state_set("aln_linked", "0")
let i: Int = 0
while i < n {
let linked_so_far: Int = str_to_int(state_get("aln_linked"))
if linked_so_far < 3 {
let elem: String = json_array_get(results, i)
let rid: String = json_get_string(elem, "id")
let rtype: String = json_get_string(elem, "node_type")
if !str_eq(rtype, "InternalStateEvent") && !str_eq(rid, "") && !str_eq(rid, node_id) {
engram_connect(node_id, rid, 0.6, "related")
state_set("aln_linked", int_to_str(linked_so_far + 1))
}
}
let i = i + 1
}
return str_to_int(state_get("aln_linked"))
}
// Helpers
fn parse_port(bind: String) -> Int {
@@ -83,7 +365,13 @@ fn route_create_node(method: String, path: String, body: String) -> String {
let salience: Float = json_get_float(body, "salience")
if salience == 0.0 { let salience = 0.5 }
let id: String = engram_node(content, node_type, salience)
"{\"id\":\"" + id + "\",\"content\":\"" + content + "\",\"node_type\":\"" + node_type + "\"}"
// Auto-link via BM25 search reuse auto_link_content_node which skips
// ISE nodes and links to up to 3 semantically related non-ISE nodes.
// Replaces the old inline substring-search auto-link (2026-05-29 cleanup).
let auto_linked: Int = auto_link_content_node(id, content)
"{\"id\":\"" + id + "\",\"content\":\"" + content + "\",\"node_type\":\"" + node_type + "\",\"auto_linked\":" + int_to_str(auto_linked) + "}"
}
fn route_get_node(method: String, path: String, body: String) -> String {
@@ -131,7 +419,7 @@ fn route_search(method: String, path: String, body: String) -> String {
let limit: Int = query_int(path, "limit", 20)
if limit == 0 { let limit = json_get_int(body, "limit") }
if limit == 0 { let limit = 20 }
return engram_search_json(q, limit)
return bm25_search_json(q, limit)
}
fn route_activate(method: String, path: String, body: String) -> String {
@@ -145,6 +433,17 @@ fn route_activate(method: String, path: String, body: String) -> String {
let bd: Int = json_get_int(body, "depth")
if bd > 0 { let depth = bd }
}
// BM25 pre-bias: strengthen top-10 BM25 results before spreading activation
// so semantically relevant nodes already have elevated salience.
let top: String = bm25_search_json(q, 10)
let nb: Int = json_array_len(top)
let bi: Int = 0
while bi < nb {
let node: String = json_array_get(top, bi)
let nid: String = json_get_string(node, "id")
if !str_eq(nid, "") { engram_strengthen(nid) }
let bi = bi + 1
}
return "{\"results\":" + engram_activate_json(q, depth) + "}"
}
@@ -180,25 +479,41 @@ fn route_forget(method: String, path: String, body: String) -> String {
ok_json()
}
fn route_save(method: String, path: String, body: String) -> String {
fn route_decay(method: String, path: String, body: String) -> String {
engram_apply_decay_json()
}
fn route_export(method: String, path: String, body: String) -> String {
let dir: String = env("ENGRAM_DATA_DIR")
if str_eq(dir, "") { let dir = "/tmp/engram" }
// Write binary checkpoint
let db_path: String = dir + "/engram.db"
engram_write_binary_el(db_path)
// Also write JSON export for human inspection
let p: String = json_get_string(body, "path")
if str_eq(p, "") {
let dir: String = env("ENGRAM_DATA_DIR")
if str_eq(dir, "") { let dir = "/tmp/engram" }
let p = dir + "/snapshot.json"
}
engram_save(p)
"{\"ok\":true,\"path\":\"" + p + "\"}"
"{\"ok\":true,\"binary\":\"" + db_path + "\",\"json\":\"" + p + "\"}"
}
fn route_reindex(method: String, path: String, body: String) -> String {
engram_reindex_json()
}
fn route_load(method: String, path: String, body: String) -> String {
let p: String = json_get_string(body, "path")
if str_eq(p, "") {
let dir: String = env("ENGRAM_DATA_DIR")
if str_eq(dir, "") { let dir = "/tmp/engram" }
let p = dir + "/snapshot.json"
let dir: String = env("ENGRAM_DATA_DIR")
if str_eq(dir, "") { let dir = "/tmp/engram" }
let db_path: String = dir + "/engram.db"
let ok: Bool = engram_load_binary_el(db_path)
if !ok {
let p: String = json_get_string(body, "path")
if str_eq(p, "") {
let p = dir + "/snapshot.json"
}
engram_load(p)
}
engram_load(p)
ok_json()
}
@@ -206,6 +521,323 @@ fn route_health(method: String, path: String, body: String) -> String {
"{\"status\":\"ok\",\"engine\":\"engram-runtime-native\"}"
}
// /api/neuron/* Routes
// route_neuron_session_begin activate with broad seeds, return node stats + results
fn route_neuron_session_begin(method: String, path: String, body: String) -> String {
let results: String = engram_activate_json("memory knowledge context", 2)
let nc: Int = engram_node_count()
let ec: Int = engram_edge_count()
"{\"ok\":true,\"nodes\":" + results + ",\"node_count\":" + int_to_str(nc) + ",\"edge_count\":" + int_to_str(ec) + "}"
}
// route_neuron_ctx compile working context from top activated nodes
fn route_neuron_ctx(method: String, path: String, body: String) -> String {
let results: String = engram_activate_json("architecture decision memory", 2)
let n: Int = json_array_len(results)
let limit: Int = if n > 10 { 10 } else { n }
let ctx: String = "Recent working memory:\n"
let i: Int = 0
let ctx_body: String = ""
while i < limit {
let elem: String = json_array_get(results, i)
let label: String = json_get_string(elem, "label")
let content: String = json_get_string(elem, "content")
let clen: Int = str_len(content)
let snippet: String = if clen > 200 { str_slice(content, 0, 200) } else { content }
let ctx_body = ctx_body + "- [" + label + "]: " + snippet + "\n"
let i = i + 1
}
let full_ctx: String = ctx + ctx_body
"{\"ok\":true,\"context\":\"" + str_replace(str_replace(str_replace(full_ctx, "\\", "\\\\"), "\"", "\\\""), "\n", "\\n") + "\"}"
}
// route_neuron_memory create a Memory node with importance-to-tier mapping
fn route_neuron_memory(method: String, path: String, body: String) -> String {
let content: String = json_get_string(body, "content")
if str_eq(content, "") { return "{\"error\":\"content is required\"}" }
let node_type: String = json_get_string(body, "node_type")
if str_eq(node_type, "") { let node_type = "Memory" }
let label: String = json_get_string(body, "label")
let importance: String = json_get_string(body, "importance")
let project: String = json_get_string(body, "project")
let tags_raw: String = json_get_string(body, "tags")
// Map importance to tier
let tier: String = "Episodic"
if str_eq(importance, "critical") { let tier = "Procedural" }
if str_eq(importance, "high") { let tier = "Semantic" }
if str_eq(importance, "normal") { let tier = "Episodic" }
if str_eq(importance, "low") { let tier = "Working" }
// Override with explicit tier if provided
let explicit_tier: String = json_get_string(body, "tier")
if !str_eq(explicit_tier, "") { let tier = explicit_tier }
// Build tags string append project tag if set
let tags_str: String = tags_raw
if !str_eq(project, "") {
if str_eq(tags_str, "") {
let tags_str = "project:" + project
}
if !str_eq(tags_str, "") {
let tags_str = tags_str + " project:" + project
}
}
let id: String = engram_node_full(content, node_type, label, 0.5, 0.5, 1.0, tier, tags_str)
// Auto-link to related non-ISE nodes so this memory is reachable via BFS traversal.
// Without this, MCP-created nodes arrive with zero edges and are invisible to
// graph spread during activation (only lexical/semantic seed matching finds them).
let auto_linked: Int = auto_link_content_node(id, content)
// Checkpoint after write
let dir: String = env("ENGRAM_DATA_DIR")
if str_eq(dir, "") { let dir = "/tmp/engram" }
let db_path: String = dir + "/engram.db"
engram_write_binary_el(db_path)
"{\"ok\":true,\"id\":\"" + id + "\",\"auto_linked\":" + int_to_str(auto_linked) + ",\"content\":\"" + str_replace(str_replace(content, "\\", "\\\\"), "\"", "\\\"") + "\"}"
}
// route_neuron_knowledge_capture create a Knowledge node
fn route_neuron_knowledge_capture(method: String, path: String, body: String) -> String {
let content: String = json_get_string(body, "content")
if str_eq(content, "") { return "{\"error\":\"content is required\"}" }
let title: String = json_get_string(body, "title")
let category: String = json_get_string(body, "category")
let tags_raw: String = json_get_string(body, "tags")
let project: String = json_get_string(body, "project")
let tier_raw: String = json_get_string(body, "tier")
// Map tier name to engram tier
let tier: String = "Episodic"
if str_eq(tier_raw, "lesson") { let tier = "Semantic" }
if str_eq(tier_raw, "canonical") { let tier = "Procedural" }
if str_eq(tier_raw, "note") { let tier = "Episodic" }
// Build tags
let tags_str: String = tags_raw
if !str_eq(category, "") {
if str_eq(tags_str, "") {
let tags_str = "category:" + category
}
if !str_eq(tags_str, "") {
let tags_str = tags_str + " category:" + category
}
}
if !str_eq(project, "") {
if str_eq(tags_str, "") {
let tags_str = "project:" + project
}
if !str_eq(tags_str, "") {
let tags_str = tags_str + " project:" + project
}
}
let id: String = engram_node_full(content, "Knowledge", title, 0.7, 0.7, 1.0, tier, tags_str)
// Auto-link to related non-ISE nodes for BFS reachability (same rationale as route_neuron_memory).
let auto_linked: Int = auto_link_content_node(id, content)
// Checkpoint
let dir: String = env("ENGRAM_DATA_DIR")
if str_eq(dir, "") { let dir = "/tmp/engram" }
let db_path: String = dir + "/engram.db"
engram_write_binary_el(db_path)
"{\"ok\":true,\"id\":\"" + id + "\",\"auto_linked\":" + int_to_str(auto_linked) + "}"
}
// route_neuron_knowledge_evolve create updated node (evolution via new node)
fn route_neuron_knowledge_evolve(method: String, path: String, body: String) -> String {
let content: String = json_get_string(body, "content")
let prior_id: String = json_get_string(body, "id")
if str_eq(content, "") { return "{\"ok\":true}" }
let id: String = engram_node_full(content, "Knowledge", "", 0.7, 0.7, 1.0, "Semantic", "evolved")
if !str_eq(prior_id, "") && !str_eq(id, "") {
engram_connect(id, prior_id, 1.0, "supersedes")
}
let dir: String = env("ENGRAM_DATA_DIR")
if str_eq(dir, "") { let dir = "/tmp/engram" }
engram_write_binary_el(dir + "/engram.db")
"{\"ok\":true,\"id\":\"" + id + "\"}"
}
// route_neuron_knowledge_promote promote a knowledge node to a higher tier.
// Creates a new node with the promoted tier (same content) and connects
// via a "supersedes" edge from new old. Tier mapping:
// note/Episodic lesson/Semantic canonical/Procedural
fn route_neuron_knowledge_promote(method: String, path: String, body: String) -> String {
let id: String = json_get_string(body, "id")
if str_eq(id, "") { return "{\"ok\":true}" }
// Read existing node
let node_json: String = engram_get_node_json(id)
if str_eq(node_json, "") { return err_json("node not found") }
if str_eq(node_json, "null") { return err_json("node not found") }
let content: String = json_get_string(node_json, "content")
if str_eq(content, "") { return err_json("node has no content") }
let label: String = json_get_string(node_json, "label")
let tags: String = json_get_string(node_json, "tags")
let current_tier: String = json_get_string(node_json, "tier")
// Determine target tier: explicit override or auto-promote one level
let tier_raw: String = json_get_string(body, "tier")
let new_tier: String = ""
// Explicit tier takes precedence
if str_eq(tier_raw, "lesson") { let new_tier = "Semantic" }
if str_eq(tier_raw, "canonical") { let new_tier = "Procedural" }
if str_eq(tier_raw, "note") { let new_tier = "Episodic" }
// Auto-promote one level if no explicit tier
if str_eq(new_tier, "") {
if str_eq(current_tier, "Working") { let new_tier = "Episodic" }
if str_eq(current_tier, "Episodic") { let new_tier = "Semantic" }
if str_eq(current_tier, "Semantic") { let new_tier = "Procedural" }
if str_eq(current_tier, "Procedural") { let new_tier = "Procedural" }
}
if str_eq(new_tier, "") { let new_tier = "Semantic" }
// Create promoted node higher importance (0.8) signals durable knowledge
let new_id: String = engram_node_full(content, "Knowledge", label, 0.7, 0.8, 1.0, new_tier, tags)
// Wire supersedes edge: new node supersedes old
if !str_eq(new_id, "") {
engram_connect(new_id, id, 1.0, "supersedes")
}
// Checkpoint
let dir: String = env("ENGRAM_DATA_DIR")
if str_eq(dir, "") { let dir = "/tmp/engram" }
engram_write_binary_el(dir + "/engram.db")
"{\"ok\":true,\"id\":\"" + new_id + "\",\"promoted_from\":\"" + id + "\",\"tier\":\"" + new_tier + "\"}"
}
// route_neuron_recall search or list nodes
fn route_neuron_recall(method: String, path: String, body: String) -> String {
let query: String = json_get_string(body, "query")
let chain: String = json_get_string(body, "chain_name")
let limit: Int = json_get_int(body, "limit")
if limit == 0 { let limit = 20 }
let q: String = if str_eq(query, "") { chain } else { query }
if str_eq(q, "") {
return engram_scan_nodes_json(limit, 0)
}
return bm25_search_json(q, limit)
}
// route_neuron_graph get node + search-based neighbor approximation.
// engram_neighbors_json crashes on large graphs (15k+ edges exceeds BFS cap).
// Use a search-based approach instead: search by the node id string, which
// returns connected nodes that share content with the target id in edges/tags.
// For the mcp-wrapper callers this is sufficient they just need the node itself.
fn route_neuron_graph(method: String, path: String, body: String) -> String {
let id: String = query_param(path, "id")
if str_eq(id, "") { return "{\"error\":\"id is required\"}" }
let node_json: String = engram_get_node_json(id)
// Return node with empty neighbors safe fallback avoids BFS crash
"{\"ok\":true,\"node\":" + node_json + ",\"neighbors\":[]}"
}
// route_neuron_graph_link create edge between nodes
fn route_neuron_graph_link(method: String, path: String, body: String) -> String {
let from_id: String = json_get_string(body, "from_id")
let to_id: String = json_get_string(body, "to_id")
if str_eq(from_id, "") || str_eq(to_id, "") {
return "{\"error\":\"from_id and to_id are required\"}"
}
let relation: String = json_get_string(body, "relation")
if str_eq(relation, "") { let relation = "related" }
let weight: Float = json_get_float(body, "weight")
if weight == 0.0 { let weight = 0.5 }
engram_connect(from_id, to_id, weight, relation)
"{\"ok\":true,\"from_id\":\"" + from_id + "\",\"to_id\":\"" + to_id + "\",\"relation\":\"" + relation + "\"}"
}
// route_neuron_list list nodes by type extracted from path
fn route_neuron_list(method: String, path: String, body: String) -> String {
let clean: String = strip_query(path)
let prefix: String = "/api/neuron/list/"
let node_type: String = str_slice(clean, str_len(prefix), str_len(clean))
let limit: Int = query_int(path, "limit", 50)
if str_eq(node_type, "") { return "[]" }
return engram_scan_nodes_by_type_json(node_type, limit, 0)
}
// route_neuron_consolidate checkpoint and return counts
fn route_neuron_consolidate(method: String, path: String, body: String) -> String {
let dir: String = env("ENGRAM_DATA_DIR")
if str_eq(dir, "") { let dir = "/tmp/engram" }
let db_path: String = dir + "/engram.db"
engram_write_binary_el(db_path)
let nc: Int = engram_node_count()
let ec: Int = engram_edge_count()
"{\"ok\":true,\"node_count\":" + int_to_str(nc) + ",\"edge_count\":" + int_to_str(ec) + "}"
}
// route_neuron_config return stub config values
fn route_neuron_config(method: String, path: String, body: String) -> String {
let key: String = query_param(path, "key")
"{\"key\":\"" + key + "\",\"value\":\"\"}"
}
// route_neuron_state_events GET lists ISEs, POST logs a new one.
// GET supports ?limit=N&offset=M for pagination; ?label=X to extract label
// from the ISE content's "event" field.
// ISEs sort by created_at DESC (most-recent-first) as of 2026-05-23 fix.
// ?limit=10 returns the 10 most recent ISEs. Offset for pagination, not for
// skipping to recent events (that was the pre-fix behavior; no longer needed).
fn route_neuron_state_events(method: String, path: String, body: String) -> String {
if str_eq(method, "GET") {
let limit_str: String = query_param(path, "limit")
let limit: Int = if str_eq(limit_str, "") { 50 } else { str_to_int(limit_str) }
let offset_str: String = query_param(path, "offset")
let offset: Int = if str_eq(offset_str, "") { 0 } else { str_to_int(offset_str) }
return engram_scan_nodes_by_type_json("InternalStateEvent", limit, offset)
}
let content: String = json_get_string(body, "content")
if str_eq(content, "") { let content = body }
// Extract label from content JSON "event" field for better ISE searchability
let event_label: String = json_get_string(content, "event")
let label: String = if str_eq(event_label, "") { "state-event" } else { event_label }
let id: String = engram_node_full(content, "InternalStateEvent", label, 0.3, 0.3, 1.0, "Working", "internal-state")
"{\"ok\":true,\"id\":\"" + id + "\"}"
}
// route_neuron_processes stub
fn route_neuron_processes(method: String, path: String, body: String) -> String {
"{\"ok\":true,\"processes\":[]}"
}
// route_events_next stub empty event queue
fn route_events_next(method: String, path: String, body: String) -> String {
"{\"ok\":true,\"event\":null}"
}
// route_events_ack stub ack
fn route_events_ack(method: String, path: String, body: String) -> String {
"{\"ok\":true}"
}
fn route_bm25_search(method: String, path: String, body: String) -> String {
let q: String = ""
if str_eq(method, "GET") {
let q = query_param(path, "q")
} else {
let q = json_get_string(body, "query")
}
if str_eq(q, "") { return "{\"error\":\"query is required\"}" }
let limit: Int = query_int(path, "limit", 20)
if limit == 0 { let limit = json_get_int(body, "limit") }
if limit == 0 { let limit = 20 }
bm25_search_json(q, limit)
}
// Auth
fn check_auth_ok(method: String, body: String) -> Bool {
@@ -232,6 +864,63 @@ fn handle_request(method: String, path: String, body: String) -> String {
}
}
// /api/neuron/* and /events/* are pre-auth the mcp-wrapper is a trusted
// local service that cannot inject _auth into its request bodies.
if str_starts_with(clean, "/api/neuron/") || str_starts_with(clean, "/events/") {
if str_eq(clean, "/api/neuron/session/begin") {
return route_neuron_session_begin(method, path, body)
}
if str_eq(clean, "/api/neuron/ctx") {
return route_neuron_ctx(method, path, body)
}
if str_eq(clean, "/api/neuron/memory") {
return route_neuron_memory(method, path, body)
}
if str_eq(clean, "/api/neuron/knowledge/capture") {
return route_neuron_knowledge_capture(method, path, body)
}
if str_eq(clean, "/api/neuron/knowledge/evolve") {
return route_neuron_knowledge_evolve(method, path, body)
}
if str_eq(clean, "/api/neuron/knowledge/promote") {
return route_neuron_knowledge_promote(method, path, body)
}
if str_eq(clean, "/api/neuron/recall") {
return route_neuron_recall(method, path, body)
}
if str_eq(clean, "/api/neuron/graph/link") {
return route_neuron_graph_link(method, path, body)
}
if str_eq(clean, "/api/neuron/graph") {
return route_neuron_graph(method, path, body)
}
if str_starts_with(clean, "/api/neuron/list/") {
return route_neuron_list(method, path, body)
}
if str_eq(clean, "/api/neuron/consolidate") {
return route_neuron_consolidate(method, path, body)
}
if str_eq(clean, "/api/neuron/config") {
return route_neuron_config(method, path, body)
}
if str_eq(clean, "/api/neuron/state-events") {
return route_neuron_state_events(method, path, body)
}
if str_eq(clean, "/api/neuron/processes/define") {
return route_neuron_processes(method, path, body)
}
if str_eq(clean, "/api/neuron/processes") {
return route_neuron_processes(method, path, body)
}
if str_eq(clean, "/events/next") {
return route_events_next(method, path, body)
}
if str_eq(clean, "/events/ack") {
return route_events_ack(method, path, body)
}
return err_json("not found")
}
// Auth (when ENGRAM_API_KEY is set)
if !check_auth_ok(method, body) {
return err_json("unauthorized")
@@ -281,18 +970,93 @@ fn handle_request(method: String, path: String, body: String) -> String {
return route_search(method, path, body)
}
// BM25+ text ranking
if str_eq(clean, "/api/bm25/search") {
return route_bm25_search(method, path, body)
}
// Strengthen
if str_eq(method, "POST") && (str_eq(clean, "/api/strengthen") || str_eq(clean, "/strengthen")) {
return route_strengthen(method, path, body)
}
// Temporal decay maintenance
if str_eq(method, "POST") && (str_eq(clean, "/api/decay") || str_eq(clean, "/api/maintenance") || str_eq(clean, "/decay")) {
return route_decay(method, path, body)
}
// Persistence
if str_eq(method, "POST") && (str_eq(clean, "/api/export") || str_eq(clean, "/export")) {
return route_export(method, path, body)
}
// /api/save is kept as a backward-compat alias for /api/export
if str_eq(method, "POST") && (str_eq(clean, "/api/save") || str_eq(clean, "/save")) {
return route_save(method, path, body)
return route_export(method, path, body)
}
if str_eq(method, "POST") && (str_eq(clean, "/api/load") || str_eq(clean, "/load")) {
return route_load(method, path, body)
}
if str_eq(method, "POST") && (str_eq(clean, "/api/reindex") || str_eq(clean, "/reindex")) {
return route_reindex(method, path, body)
}
// /api/neuron/*
if str_starts_with(clean, "/api/neuron/") {
// Specific sub-paths first (longer matches before shorter)
if str_eq(clean, "/api/neuron/session/begin") {
return route_neuron_session_begin(method, path, body)
}
if str_eq(clean, "/api/neuron/ctx") {
return route_neuron_ctx(method, path, body)
}
if str_eq(clean, "/api/neuron/memory") {
return route_neuron_memory(method, path, body)
}
if str_eq(clean, "/api/neuron/knowledge/capture") {
return route_neuron_knowledge_capture(method, path, body)
}
if str_eq(clean, "/api/neuron/knowledge/evolve") {
return route_neuron_knowledge_evolve(method, path, body)
}
if str_eq(clean, "/api/neuron/knowledge/promote") {
return route_neuron_knowledge_promote(method, path, body)
}
if str_eq(clean, "/api/neuron/recall") {
return route_neuron_recall(method, path, body)
}
if str_eq(clean, "/api/neuron/graph/link") {
return route_neuron_graph_link(method, path, body)
}
if str_eq(clean, "/api/neuron/graph") {
return route_neuron_graph(method, path, body)
}
if str_starts_with(clean, "/api/neuron/list/") {
return route_neuron_list(method, path, body)
}
if str_eq(clean, "/api/neuron/consolidate") {
return route_neuron_consolidate(method, path, body)
}
if str_eq(clean, "/api/neuron/config") {
return route_neuron_config(method, path, body)
}
if str_eq(clean, "/api/neuron/state-events") {
return route_neuron_state_events(method, path, body)
}
if str_eq(clean, "/api/neuron/processes/define") {
return route_neuron_processes(method, path, body)
}
if str_eq(clean, "/api/neuron/processes") {
return route_neuron_processes(method, path, body)
}
}
// /events/*
if str_eq(clean, "/events/next") {
return route_events_next(method, path, body)
}
if str_eq(clean, "/events/ack") {
return route_events_ack(method, path, body)
}
"{\"error\":\"not found\",\"path\":\"" + clean + "\"}"
}
@@ -303,13 +1067,28 @@ let bind_str: String = env("ENGRAM_BIND")
if str_eq(bind_str, "") { let bind_str = ":8742" }
let port: Int = parse_port(bind_str)
// On startup, try to load any existing snapshot (best effort).
// On startup, load from binary database (ML-KEM-1024 encrypted).
// Falls back to per-file JSON, then snapshot.json for migration from older formats.
let data_dir: String = env("ENGRAM_DATA_DIR")
if str_eq(data_dir, "") { let data_dir = "/tmp/engram" }
let snapshot_path: String = data_dir + "/snapshot.json"
engram_load(snapshot_path)
let db_path: String = data_dir + "/engram.db"
let loaded: Bool = engram_load_binary_el(db_path)
if !loaded {
// Migration path: try per-file JSON
engram_load_dir(data_dir)
if engram_node_count() == 0 {
// Final fallback: legacy snapshot.json
let snapshot_path: String = data_dir + "/snapshot.json"
engram_load(snapshot_path)
}
// If we loaded anything from legacy format, save as binary immediately
if engram_node_count() > 0 {
engram_write_binary_el(db_path)
println("[engram] migrated legacy data to binary format")
}
}
println("[engram] runtime-native graph engine")
println("[engram] runtime-native graph engine (ML-KEM-1024 encrypted)")
println("[engram] data_dir=" + data_dir)
println("[engram] node_count=" + int_to_str(engram_node_count()))
println("[engram] edge_count=" + int_to_str(engram_edge_count()))
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
+12
View File
@@ -4342,6 +4342,9 @@ el_val_t builtin_arity(el_val_t name) {
if (str_eq(name, EL_STR("http_serve"))) {
return 2;
}
if (str_eq(name, EL_STR("http_serve_async"))) {
return 2;
}
if (str_eq(name, EL_STR("http_set_handler"))) {
return 1;
}
@@ -4546,6 +4549,12 @@ el_val_t builtin_arity(el_val_t name) {
if (str_eq(name, EL_STR("engram_load"))) {
return 1;
}
if (str_eq(name, EL_STR("engram_load_dir"))) {
return 1;
}
if (str_eq(name, EL_STR("engram_reindex_json"))) {
return 0;
}
if (str_eq(name, EL_STR("engram_get_node_json"))) {
return 1;
}
@@ -4564,6 +4573,9 @@ el_val_t builtin_arity(el_val_t name) {
if (str_eq(name, EL_STR("engram_stats_json"))) {
return 0;
}
if (str_eq(name, EL_STR("engram_apply_decay_json"))) {
return 0;
}
if (str_eq(name, EL_STR("llm_call"))) {
return 2;
}
-117
View File
@@ -1,117 +0,0 @@
#ifndef EL_PLATFORM_WIN_H
#define EL_PLATFORM_WIN_H
/*
* el_platform_win.h Windows OS-boundary shim for el_runtime.c.
*
* Branch: feat/windows-el-runtime. Included ONLY when _WIN32 is defined; the POSIX build is
* untouched. Goal: let el_runtime.c (a BSD-sockets / dlfcn / fork host) compile and link with
* mingw-w64 into a native neuron.exe, with no behavioural change to the Linux/macOS build.
*
* What it maps:
* - sockets : winsock2 (same call names: socket/bind/listen/accept/recv/send/setsockopt).
* Sockets close with closesocket() (see el_closesocket), and the stack must be
* started once with WSAStartup done automatically via a load-time constructor.
* - dlsym : el_runtime.c uses dlsym(RTLD_DEFAULT, name) to resolve callback/tool symbols
* exported by the main module. Windows equivalent: GetProcAddress on the process
* module. Link the soul with -Wl,--export-all-symbols so the symbols are findable.
* - popen : mapped to _popen/_pclose.
* - threads : UNCHANGED. mingw-w64 ships winpthreads, so <pthread.h> + -lpthread just work.
*/
#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN
#endif
#include <winsock2.h>
#include <ws2tcpip.h>
#include <windows.h>
#include <io.h>
#include <process.h>
/* Portable headers mingw-w64 provides (verified present). */
#include <stdarg.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h> /* strcasecmp */
#include <ctype.h>
#include <math.h>
#include <time.h>
#include <sys/time.h> /* mingw-w64 provides gettimeofday here */
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <dirent.h>
#include <errno.h>
#include <pthread.h>
/* ── socket close ─────────────────────────────────────────────────────────── */
/* Winsock closes sockets with closesocket(), not close() (close() is for file fds). The POSIX
build defines the same helper as close() so the call sites are identical across platforms. */
static inline int el_closesocket(SOCKET s) { return closesocket(s); }
/* ── winsock init (once, at load) ─────────────────────────────────────────── */
static void el__win_net_init(void) {
static int inited = 0;
if (!inited) { WSADATA w; WSAStartup(MAKEWORD(2, 2), &w); inited = 1; }
}
__attribute__((constructor)) static void el__win_ctor(void) { el__win_net_init(); }
/* ── dlsym → GetProcAddress ───────────────────────────────────────────────── */
#ifndef RTLD_DEFAULT
#define RTLD_DEFAULT ((void*)0)
#endif
static inline void* el_win_dlsym(void* handle, const char* name) {
(void)handle;
return (void*)(uintptr_t)GetProcAddress(GetModuleHandleA(NULL), name);
}
#define dlsym(h, n) el_win_dlsym((h), (n))
/* ── popen / pclose ───────────────────────────────────────────────────────── */
#define popen _popen
#define pclose _pclose
/* ── misc POSIX → Win32 shims ─────────────────────────────────────────────── */
#include <direct.h> /* _mkdir */
#define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */
#define timegm _mkgmtime /* UTC tm → time_t */
/* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */
static inline int setenv(const char* name, const char* value, int overwrite) {
(void)overwrite;
return _putenv_s(name, value ? value : "");
}
static inline int unsetenv(const char* name) {
/* _putenv_s(name, "") sets VAR="" rather than removing it.
* SetEnvironmentVariableA(name, NULL) truly deletes it from the Win32
* env block; then we sync the CRT cache with _putenv("NAME="). */
SetEnvironmentVariableA(name, NULL);
size_t len = strlen(name);
char *buf = (char*)malloc(len + 2);
if (!buf) return -1;
memcpy(buf, name, len);
buf[len] = '=';
buf[len + 1] = '\0';
_putenv(buf);
free(buf);
return 0;
}
/* nanosleep — not available in MSVC/UCRT; approximate with Sleep(). */
static inline int el_nanosleep(const struct timespec *req, struct timespec *rem) {
(void)rem;
DWORD ms = (DWORD)((req->tv_sec * 1000ULL) + (req->tv_nsec / 1000000ULL));
Sleep(ms ? ms : 1);
return 0;
}
#define nanosleep(req, rem) el_nanosleep((req), (rem))
/* localtime_r/gmtime_r: Windows offers localtime_s/gmtime_s with reversed arg order. */
static inline struct tm* localtime_r(const time_t* t, struct tm* out) {
return localtime_s(out, t) == 0 ? out : (struct tm*)0;
}
static inline struct tm* gmtime_r(const time_t* t, struct tm* out) {
return gmtime_s(out, t) == 0 ? out : (struct tm*)0;
}
#endif /* EL_PLATFORM_WIN_H */
File diff suppressed because it is too large Load Diff
-13
View File
@@ -52,12 +52,6 @@
typedef int64_t el_val_t;
/* HTTP request-handler function-pointer types. Public because soul modules (routes/chat/etc.)
* register handlers across translation units; previously defined only inside el_runtime.c, which
* made cross-module references (and the Windows build) fail. Home in the shared header. */
typedef el_val_t (*http_handler_fn)(el_val_t method, el_val_t path, el_val_t body);
typedef el_val_t (*http_handler4_fn)(el_val_t method, el_val_t path, el_val_t body, el_val_t headers);
#define EL_STR(s) ((el_val_t)(uintptr_t)(s))
#define EL_CSTR(v) ((const char*)(uintptr_t)(v))
#define EL_INT(v) (v)
@@ -182,7 +176,6 @@ el_val_t http_set_handler(el_val_t name);
* existing handlers (e.g. products/web/server.el): it dispatches with
* (method, path, body), hardcodes 200 OK, and auto-detects content type. */
el_val_t http_serve_v2(el_val_t port, el_val_t handler);
void http_serve_async(el_val_t port, el_val_t handler);
el_val_t http_set_handler_v2(el_val_t name);
/* Build an HTTP response envelope. `headers_json` should be a JSON object
@@ -645,12 +638,6 @@ el_val_t engram_list_layers_json(void);
* no nodes promoted to working memory. */
el_val_t engram_compile_layered_json(el_val_t intent, el_val_t depth);
/* ── Working memory ──────────────────────────────────────────────────────────*/
el_val_t engram_wm_count(void);
el_val_t engram_wm_avg_weight(void);
el_val_t engram_wm_top_json(el_val_t n);
el_val_t engram_load_merge(el_val_t path);
/* ── LLM (Anthropic API client) ─────────────────────────────────────────────
* All functions call https://api.anthropic.com/v1/messages with the API key
* from env ANTHROPIC_API_KEY. Default model when empty: claude-sonnet-4-5. */
+1
View File
@@ -2502,6 +2502,7 @@ fn builtin_arity(name: String) -> Int {
if str_eq(name, "http_post_with_headers") { return 3 }
if str_eq(name, "http_post_form_auth") { return 3 }
if str_eq(name, "http_serve") { return 2 }
if str_eq(name, "http_serve_async") { return 2 }
if str_eq(name, "http_set_handler") { return 1 }
// Seed primitives (__-prefix) runtime/el_seed.c
if str_eq(name, "__str_len") { return 1 }
+21 -6
View File
@@ -283,6 +283,12 @@ fn compile_module(src_path: String, out_dir: String, elc_bin: String, dry_run: B
}
exec_command("rm -f " + err_tmp)
// Strip capability-violation guard #error lines injected by elc when a
// module is compiled in isolation (utility context). These are safe to
// remove here: the entire binary is linked under the CGI entry-point
// declaration in soul.el, so the module-level guard is redundant.
exec_command("sed -i.bak '/^#error \"capability violation/d' " + c_out + " && rm -f " + c_out + ".bak")
// Move the generated .elh (written next to the source by elc) into
// out_dir so that #include "module.elh" lines in the generated .c
// files resolve correctly when cc is invoked with -I <out_dir>.
@@ -305,6 +311,10 @@ fn link_binary(c_files: [String], out_bin: String, runtime_path: String, out_dir
// prefix and add it if present (no-op on Linux where libssl is in /usr/lib).
let ossl_lib_flag: String = "$(brew --prefix openssl 2>/dev/null | xargs -I{} printf -- '-L{}/lib' 2>/dev/null || true)"
let ossl_inc_flag: String = "$(brew --prefix openssl 2>/dev/null | xargs -I{} printf -- '-I{}/include' 2>/dev/null || true)"
// liboqs (post-quantum crypto) present on macOS dev machines, not on CI
// Linux containers. Link -loqs only when the library is available.
let oqs_lib_flag: String = "$(brew --prefix liboqs 2>/dev/null | xargs -I{} printf -- '-L{}/lib -loqs' 2>/dev/null || true)"
let oqs_inc_flag: String = "$(brew --prefix liboqs 2>/dev/null | xargs -I{} printf -- '-I{}/include' 2>/dev/null || true)"
// Force-include the C-level master declarations header so every translation
// unit sees all cross-module function signatures. Handles packages (like ELP)
// where modules call each other without explicit El import statements.
@@ -312,7 +322,7 @@ fn link_binary(c_files: [String], out_bin: String, runtime_path: String, out_dir
let master_decls: String = out_dir + "/elp-c-decls.h"
let has_master: String = str_trim(exec_capture("test -f " + master_decls + " && echo yes || echo no"))
let include_flag: String = if str_eq(has_master, "yes") { "-include " + master_decls } else { "" }
let parts = native_list_append(parts, "cc -O2 " + bracket_flag + " " + ossl_inc_flag + " " + include_flag + " -I " + dirname_of(runtime_path) + " -I " + out_dir)
let parts = native_list_append(parts, "cc -O2 -DHAVE_CURL " + bracket_flag + " " + ossl_inc_flag + " " + oqs_inc_flag + " " + include_flag + " -I " + dirname_of(runtime_path) + " -I " + out_dir)
let i = 0
while i < n {
let f: String = native_list_get(c_files, i)
@@ -320,7 +330,7 @@ fn link_binary(c_files: [String], out_bin: String, runtime_path: String, out_dir
let i = i + 1
}
let parts = native_list_append(parts, runtime_path)
let parts = native_list_append(parts, ossl_lib_flag + " -lcurl -lssl -lcrypto -lpthread -lm")
let parts = native_list_append(parts, ossl_lib_flag + " " + oqs_lib_flag + " -lcurl -lssl -lcrypto -lpthread -lm")
let parts = native_list_append(parts, "-o " + out_bin)
let cmd: String = str_join(parts, " ")
println(" link " + out_bin)
@@ -432,18 +442,23 @@ fn main() -> Void {
exit(1)
}
// Link use only the entry-point .c file (which elc compiles as a
// monolithic unit, inlining all imports). Linking all module .c files
// together causes duplicate-symbol errors because each module's .c also
// inlines its full import tree.
let entry_c: String = out_dir + "/" + basename_noext(entry) + ".c"
let link_files: [String] = native_list_empty()
let link_files = native_list_append(link_files, entry_c)
// Append any extra C sources declared in the manifest (e.g. platform stubs)
let ei = 0
let en: Int = native_list_len(extra_c)
while ei < en {
let ec: String = native_list_get(extra_c, ei)
let c_files = native_list_append(c_files, ec)
let link_files = native_list_append(link_files, ec)
let ei = ei + 1
}
// Link
let out_bin: String = out_dir + "/" + pkg_name
let linked: Bool = link_binary(c_files, out_bin, runtime_path, out_dir, dry_run)
let linked: Bool = link_binary(link_files, out_bin, runtime_path, out_dir, dry_run)
if !linked {
println("elb: link failed")
exit(1)
+5
View File
@@ -0,0 +1,5 @@
CompileFlags:
Add:
- -I/opt/homebrew/Cellar/liboqs/0.15.0/include
- -I/opt/homebrew/opt/openssl@3/include
- -std=c11
File diff suppressed because it is too large Load Diff
@@ -143,6 +143,7 @@ el_val_t http_post_with_headers(el_val_t url, el_val_t body, el_val_t headers_m
el_val_t http_post_form_auth(el_val_t url, el_val_t form_body, el_val_t auth_header);
el_val_t http_delete(el_val_t url);
void http_serve(el_val_t port, el_val_t handler);
void http_serve_async(el_val_t port, el_val_t handler);
void http_set_handler(el_val_t name);
/* HTTP server v2 ─────────────────────────────────────────────────────────────
@@ -429,10 +430,22 @@ el_val_t str_to_float(el_val_t s);
el_val_t math_sqrt(el_val_t f);
el_val_t math_log(el_val_t f);
el_val_t math_ln(el_val_t f);
el_val_t math_exp(el_val_t f);
el_val_t math_sin(el_val_t f);
el_val_t math_cos(el_val_t f);
el_val_t math_pi(void);
/* ── Float arithmetic builtins (correct IEEE 754 via bit-cast round-trip) ─── */
el_val_t float_add(el_val_t a, el_val_t b);
el_val_t float_sub(el_val_t a, el_val_t b);
el_val_t float_mul(el_val_t a, el_val_t b);
el_val_t float_div(el_val_t a, el_val_t b);
el_val_t float_gt(el_val_t a, el_val_t b);
el_val_t float_lt(el_val_t a, el_val_t b);
el_val_t float_eq(el_val_t a, el_val_t b);
el_val_t float_gte(el_val_t a, el_val_t b);
el_val_t float_lte(el_val_t a, el_val_t b);
/* ── String additions ────────────────────────────────────────────────────── */
el_val_t str_index_of(el_val_t s, el_val_t sub);
@@ -492,6 +505,7 @@ el_val_t str_join(el_val_t list, el_val_t sep); /* alias of list_joi
el_val_t list_push(el_val_t list, el_val_t elem);
el_val_t list_push_front(el_val_t list, el_val_t elem);
el_val_t list_set(el_val_t list, el_val_t index, el_val_t value);
el_val_t list_join(el_val_t list, el_val_t sep);
el_val_t list_range(el_val_t start, el_val_t end);
@@ -589,6 +603,10 @@ el_val_t engram_edge_count(void);
el_val_t engram_activate(el_val_t query, el_val_t depth);
el_val_t engram_save(el_val_t path);
el_val_t engram_load(el_val_t path);
el_val_t engram_load_dir(el_val_t data_dir);
el_val_t engram_reindex_json(void);
el_val_t engram_write_binary_el(el_val_t path);
el_val_t engram_load_binary_el(el_val_t path);
/* JSON-string accessors — return pre-serialized JSON so HTTP handlers
* can pass results straight through without round-tripping ElList/ElMap
@@ -600,6 +618,10 @@ el_val_t engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_
el_val_t engram_neighbors_json(el_val_t node_id, el_val_t max_depth, el_val_t direction);
el_val_t engram_activate_json(el_val_t query, el_val_t depth);
el_val_t engram_stats_json(void);
el_val_t engram_wm_count(void);
el_val_t engram_wm_avg_weight(void); /* avg wm weight of promoted nodes; float bits */
el_val_t engram_wm_top_json(el_val_t n); /* top-N WM nodes by weight as compact JSON */
el_val_t engram_apply_decay_json(void);
el_val_t engram_list_layers_json(void);
/* engram_compile_layered_json — produce a prompt-ready text block split
* into "[LAYER 0 — STRUCTURAL]" (non-suppressible layers, sacred fire)
+3 -43
View File
@@ -6,55 +6,15 @@
//
// Dependencies: runtime/string.el, runtime/json.el
// --- Validation (defense in depth) ---
// el_val_t is an untyped machine word, so a wrong TYPE can't be caught here but a
// wrong VALUE can (a tier in the node_type slot, an empty/garbage string, an int, a
// path, a model name, a cgi id). Reject loudly instead of silently writing junk.
fn engram_valid_node_type(t: String) -> Bool {
return str_eq(t, "Memory") || str_eq(t, "Knowledge") || str_eq(t, "Belief")
|| str_eq(t, "Project") || str_eq(t, "Tag") || str_eq(t, "BacklogItem")
|| str_eq(t, "Artifact") || str_eq(t, "Conversation") || str_eq(t, "ExecutionContext")
|| str_eq(t, "InternalStateEvent") || str_eq(t, "Self") || str_eq(t, "Entity")
|| str_eq(t, "Process") || str_eq(t, "ConfigEntry") || str_eq(t, "Concept") || str_eq(t, "Imprint")
|| str_eq(t, "SessionSummary")
}
fn engram_valid_tier(t: String) -> Bool {
return str_eq(t, "Semantic") || str_eq(t, "Episodic") || str_eq(t, "Working")
|| str_eq(t, "Procedural") || str_eq(t, "Canonical") || str_eq(t, "Note") || str_eq(t, "Lesson")
}
// --- Node creation ---
fn engram_node(content: String, node_type: String, salience: Float) -> String {
if !engram_valid_node_type(node_type) {
__println("[engram] REJECTED node write — invalid node_type '" + node_type + "'")
return ""
}
return __engram_node(content, node_type, salience)
}
// Signature MUST match the C primitive __engram_node_full exactly (el_seed.h):
// (content, node_type, label, salience, importance, confidence, tier, tags)
// The previous wrapper declared a stale 8-arg schema with wrong names AND types
// (sal:Float at the label slot, ts:Int at the tier slot). Because el_val_t is an
// untyped machine word, the EL compiler coerced caller args to those wrong param
// types and then forwarded them BY POSITION into the C function so tier received
// an int, importance/confidence received strings, label received a float, etc.
// That is the field-corruption bug. Match the contract 1:1 no coercion, no reorder.
fn engram_node_full(content: String, node_type: String, label: String,
salience: Float, importance: Float, confidence: Float,
tier: String, tags: String) -> String {
if !engram_valid_node_type(node_type) {
__println("[engram] REJECTED node write — invalid node_type '" + node_type + "' (label=" + label + ")")
return ""
}
if !engram_valid_tier(tier) {
__println("[engram] REJECTED node write — invalid tier '" + tier + "' (node_type=" + node_type + ", label=" + label + ")")
return ""
}
return __engram_node_full(content, node_type, label, salience, importance, confidence, tier, tags)
fn engram_node_full(content: String, nt: String, sal: Float, imp: Float,
source: String, lang: String, ts: Int, tags: String) -> String {
return __engram_node_full(content, nt, sal, imp, source, lang, ts, tags)
}
// --- Node retrieval ---