Compare commits

..

2 Commits

Author SHA1 Message Date
Neuron 40653d2aa1 fix(codegen): emit the declared cgi identity — it was searched for in a list that cannot contain it
El SDK Release / build-and-release (pull_request) Failing after 10m39s
A cgi block is a top-level declaration, so codegen_streaming classifies it via
is_top_level_decl and releases it. The identity emission then searched
toplevel_exec_stmts for that same block. Declarations are excluded from that list by
construction, so the search could never succeed. A probe printed what it actually
saw for a program whose first statement is a cgi block: [Let, Expr]. It emitted
nothing, silently, with no diagnostic on any channel.

The code documented its own assumption — 'Since cgi blocks are rare and small, they
end up in toplevel_exec_stmts' — and that assumption was false.

Capture the declared values before the release and emit from them. The search is
deleted rather than repaired, so the failure mode is removed rather than relocated.

Proven discriminating (old fails, new passes):
  minimal cgi program, old   -> 0 el_cgi_init
  minimal cgi program, fixed -> el_cgi_init with all four declared values
  neuron soul, fixed         -> principal present in the compiled binary (0 before),
                                boots in 2s, interface 110 routes in / 110 out

Consequence: a binary now carries its declared identity as a compiled constant,
which is what the identity protocol requires. Whether the runtime surfaces it to
state_get("soul_principal") is unverified and separate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 13:36:51 -05:00
Tim Lingo f76ccc0590 engram: ranked BM25+recency search replaces storage-order substring; URL-decode GET query params
Measured on the live container mind (pinned 40-query eval, judged): substring
2/40=5% hit@5 -> ranked 35/40=88%. Multi-word queries stop returning zero; new
memories stop losing to storage order (created_at tiebreak). Transparent-layer
identity filter preserved in both passes; jb_finish (#64) tail preserved.
query_param now url_decode()s values - %XX arrived literal before (pre-existing
GET defect, masked while multi-word substring returned nothing anyway).
E2E-verified in Tim's container deployment 2026-07-14/15; eval harness:
docs repo research-archive/p0-prototypes/eval_pinned_40q_20260715.py.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 07:48:09 -05:00
23 changed files with 451 additions and 2212 deletions
-15
View File
@@ -214,18 +214,9 @@ jobs:
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: | run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695 gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}" VERSION="${GITHUB_SHA:0:8}"
@@ -277,12 +268,6 @@ jobs:
# Patches ci-base:dev in-place: pulls the existing image (which has all # Patches ci-base:dev in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly # system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental. # built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push' if: github.event_name == 'push'
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
-15
View File
@@ -212,21 +212,12 @@ jobs:
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: | run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695 gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}" VERSION="${GITHUB_SHA:0:8}"
@@ -262,12 +253,6 @@ jobs:
# Patches ci-base:stage in-place: pulls the existing image (which has all # Patches ci-base:stage in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly # system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental. # built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push' if: github.event_name == 'push'
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
-15
View File
@@ -288,21 +288,12 @@ jobs:
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: | run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695 gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}" VERSION="${GITHUB_SHA:0:8}"
@@ -354,12 +345,6 @@ jobs:
# Patches ci-base:latest in-place: pulls the existing image (which has all # Patches ci-base:latest in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly # system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental. # built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push' if: github.event_name == 'push'
env: env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }} GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
-154
View File
@@ -1,154 +0,0 @@
# El
**A self-hosting, statically-typed language that compiles to C — built around a graph-native runtime instead of a database driver.**
El is the execution substrate for the Neuron agent runtime, the DHARMA network, and the Engram knowledge graph. This repository is the monorepo for the whole stack: the language itself, the graph memory engine it's built to talk to natively, and the tools (package manager, IDE, UI framework, diagramming) built on top of it.
---
## Why El exists
Every other language treats persistent, associative state as something you reach for through a driver — a SQL client, an ORM, a Redis library bolted on from outside. El inverts that: graph operations (`engram_*`) are runtime primitives, on the same footing as string or list operations. There is no separate database driver because the database is not separate.
El has four defining properties:
1. **Self-hosting compiler.** The compiler (`lexer.el`, `parser.el`, `codegen.el`, `compiler.el`) is written in El. It compiles El source to C, which `cc` compiles against a fixed runtime into a native binary. A Rust genesis compiler bootstrapped the first iteration; the self-hosted binary at `lang/dist/platform/elc` has been the canonical compiler ever since — every binary in `dist/platform/` was produced by an earlier version of itself compiling `el-compiler/src/`. The chain is auditable: source is the ground truth, not the binary. See [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) for the full recovery path if that binary is ever lost.
2. **C compilation target.** Every compiled program is plain C11. Every El value is `el_val_t` (`int64_t`); strings are heap pointers cast through it. Functions become C functions; top-level statements become `main()`.
3. **Graph-native runtime.** The runtime provides first-class graph operations over an in-process Engram store — no separate DB driver, no ORM.
4. **DHARMA-aware identity.** A `cgi` block declares a program's DHARMA identity at compile time. The runtime resolves identity before user code runs, so `dharma_*` calls have a stable principal and channel surface throughout.
---
## Architecture map
```
┌─────────────┐
│ lang │ El compiler + C runtime
│ (El itself) │ everything below is written in it,
└──────┬──────┘ or compiles down through it
┌─────────────┼─────────────┐
│ │ │
┌──────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
│ engram │ │ epm │ │ ide │
│ graph/mem │ │ package │ │ editor + │
│ substrate │ │ manager │ │ LSP │
└──────┬─────┘ └───────────┘ └───────────┘
┌───────┼────────────────┬─────────────────────┐
│ │ │ │
┌─────▼───┐ ┌─▼──────────┐ ┌──▼──────────┐ ┌─────▼──────┐
│ elp │ │ ql │ │ ui │ │ arbor │
│ NLG / │ │engram-el. │ |spreading- │ |arbor │
│ 31 langs│ │studio+tests│ |activation UI│ |diagram lang│
└─────────┘ └────────────┘ └─────────────┘ └────────────┘
```
`lang` is the foundation — the compiler and C runtime everything else builds on. `engram` is the graph-native memory/state engine that gives El its identity (property 3 above). Everything else is either a tool for working with El (`epm`, `ide`) or a system built on top of Engram's graph model (`elp`, `ql`, `ui`, `arbor`).
---
## Repository layout
### [lang/](lang/) — the El language
The compiler and runtime. Self-hosting: `elc-cli.el``compiler.el``lexer.el` / `parser.el` / `codegen.el` / `codegen-js.el`, textually inlined and compiled in one pass. Compiles to C11 and links against `el-compiler/runtime/el_seed.c`, a hand-maintained OS-boundary layer (libcurl HTTP, pthreads, filesystem, arena allocation) — everything else in the runtime is native El (`runtime/*.el`).
Two layers to know: **El programs** (`.el` files — where nearly all work belongs) and **the C seed** (`el_seed.c` — edit only for genuine OS-level access; never re-implement what El can already express).
Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented. In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), the real `engram_*` and `dharma_*` runtimes (currently stubs), and libcurl-backed `http_get`/`http_post`/`http_serve`. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language.
Key docs: [AGENTS.md](lang/AGENTS.md) (agent-facing orientation), [BOOTSTRAP.md](lang/BOOTSTRAP.md) (compiler recovery from scratch), [spec/language.md](lang/spec/language.md), [spec/codegen-js.md](lang/spec/codegen-js.md).
### [engram/](engram/) — graph intelligence substrate
**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. Rust core (`engram-core`, `engram-ffi`) exposed to El and other languages (Kotlin, TypeScript/WASM, Go bindings).
The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop (`strength = parent_strength × edge_weight × target_salience × cosine_sim`), gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works.
Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay**`importance × recency-decay × log(activation_count)`. Forgetting is adaptive pruning, not a bug: unreinforced memories stop competing for attention without being deleted.
Backed by `sled` (embedded, local-first, no daemon) with flat cosine scan for vector search — deliberately simple until scale demands an HNSW layer. Full API and design rationale in [engram/README.md](engram/README.md).
### [elp/](elp/) — Engram Language Protocol
Bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*``grammar``realizer``semantics``elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages.
### [epm/](epm/) — El Package Manager
Manages **vessels** (El's package unit): publish, install, resolve dependencies. Vessels are stored in Engram as graph nodes, not files in a registry index — `epm` reads the local `manifest.el`, talks to Engram over HTTP, and writes resolved vessels to `.epm/vessels/`. Source: `registry.el`, `install.el`, `update.el`, `manifest.el`.
### [ide/](ide/) — El IDE
Three vessels: **el-ide-server** (HTTP backend — file ops, build/run, LSP bridge, plugin host, settings), **el-lsp** (the language server — completion, hover, diagnostics, outline, format, type graph), and **el-plugin-host** (first-party plugin lifecycle: install/remove/enable/disable). `ide/projects/` and `ide/examples/` hold sample projects, including the canonical `hello-friends` first-program walkthrough.
### [ql/](ql/) — engram-el
The El-native integration layer for a *live* Engram server — not a library (no importable modules, no build artifact), a set of standalone `.el` programs run directly via `el run-file`. Three components: **Studio** (`studio/studio.el`, a full terminal graph explorer), a **Hebbian field-model** proof of concept, and El builtin / LLM-builtin smoke test suites. This is the reference for correct patterns when an El program uses Engram as its substrate. Spec: [ql/spec/elql.md](ql/spec/elql.md).
### [ui/](ui/) — el-ui
A frontend framework where **component state is an Engram graph and reactivity is spreading activation** — not virtual-DOM diffing (React), Proxy-based dependency tracking (Vue), or compile-time analysis (Svelte). Re-renders are activated and propagated the same way associative memory retrieval works in `engram/`.
~15 vessels covering the full frontend surface: `el-platform` (env/fs/network/clock abstraction), `el-config`, `el-html` (SSR emit primitives), `el-layout`, `el-style` (design tokens/themes), `el-i18n`, `el-auth` / `el-identity` (JWT, sessions, OAuth PKCE — Engram-native), `el-services` (REST/gRPC/WebSocket bindings), `el-aop` (`@authenticate`/`@authorize`/`@cache`/`@rate_limit` decorators), `el-secrets`, `el-graph` (graph rendering/editor), `el-publish` (App Store / Play Store automation), and `el-ui-compiler` (El→JS component compiler; currently a stub pending a JS backend in `elc`). Spec: [ui/spec/framework.md](ui/spec/framework.md).
### [arbor/](arbor/) — diagram language
A `.arbor` diagram language and toolchain: `arbor-core` (NodeId/shape/edge-kind types), `arbor-parse` (recursive-descent parser), `arbor-diagram` (IR + Mermaid serializer + architecture-diagram builders), `arbor-layout` (hierarchical layout — rank assignment, positioning, group bounds), `arbor-render` (SVG renderer), `arbor-cli`. (The architecture map above is the kind of diagram this is for.)
---
## Getting started
Install the El SDK from the latest release:
```bash
bash lang/install.sh
# EL_VERSION=v1.0.0 bash lang/install.sh # pin a specific release tag
# EL_PREFIX=/opt/el bash lang/install.sh # custom install prefix
```
Or build the compiler from source and verify the self-hosting chain:
```bash
cd lang
./dist/platform/elc elc-cli.el > elc-new.c
cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \
-o dist/platform/elc-new \
elc-new.c el-compiler/runtime/el_seed.c
# Confirm the new binary reproduces itself exactly
./dist/platform/elc-new elc-cli.el > elc-verify.c
diff elc-new.c elc-verify.c # should be identical
mv dist/platform/elc-new dist/platform/elc
```
Run your first program:
```bash
./lang/dist/platform/elc lang/examples/hello.el > hello.c
cc -std=c11 -I lang/el-compiler/runtime -lcurl -lpthread \
-o hello hello.c lang/el-compiler/runtime/el_seed.c
./hello
```
More examples in [lang/examples/](lang/examples/), including a full starter project at `lang/examples/hello-project/`.
If the compiler binary is ever lost or corrupted, [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) is the authoritative recovery path.
---
## Development workflow
Branching follows `dev → stage → main`: work lands on `dev`, promotes to `stage` for integration testing, and is promoted to `main` for release (visible directly in the git history of this repo). CI is defined per-subproject under `.gitea/workflows/``lang`/`epm`/`ide` share the root pipeline; `engram` and `ql` carry their own (`ci-dev`, `ci-stage`, and a release workflow each).
- Language/runtime specs live at `*/spec/*.md` (`lang/spec/`, `ql/spec/`, `ui/spec/`) and are the single source of truth for implemented-vs-planned status — code and docs are expected to agree with the spec's status markers, not the other way around.
- Agent-facing orientation guides live at `*/AGENTS.md` (currently `lang/AGENTS.md`); more subprojects may grow their own as they need agent-specific conventions documented.
- Tagged releases live under `lang/releases/`, each with its own `RELEASE.md`.
---
## Status
This is an actively developed, internal monorepo — not yet published under an open license. Treat everything here as proprietary to Neuron Technologies unless told otherwise.
+1 -1
View File
@@ -81,7 +81,7 @@ jobs:
# Link to produce the engram binary # Link to produce the engram binary
- name: Link engram binary - name: Link engram binary
run: | run: |
cc -std=c11 -O2 -DHAVE_CURL \ cc -std=c11 -O2 \
-I /usr/local/lib/el \ -I /usr/local/lib/el \
-o dist/engram \ -o dist/engram \
dist/engram.c \ dist/engram.c \
+1 -1
View File
@@ -88,7 +88,7 @@ jobs:
# Link to produce the engram binary # Link to produce the engram binary
- name: Link engram binary - name: Link engram binary
run: | run: |
cc -std=c11 -O2 -DHAVE_CURL \ cc -std=c11 -O2 \
-I /usr/local/lib/el \ -I /usr/local/lib/el \
-o dist/engram \ -o dist/engram \
dist/engram.c \ dist/engram.c \
+1 -1
View File
@@ -62,7 +62,7 @@ jobs:
# Link to produce the engram binary # Link to produce the engram binary
- name: Link engram binary - name: Link engram binary
run: | run: |
cc -std=c11 -O2 -DHAVE_CURL \ cc -std=c11 -O2 \
-I /usr/local/lib/el \ -I /usr/local/lib/el \
-o dist/engram \ -o dist/engram \
dist/engram.c \ dist/engram.c \
BIN
View File
Binary file not shown.
+95 -142
View File
@@ -10,7 +10,6 @@ el_val_t query_param(el_val_t path, el_val_t key);
el_val_t query_int(el_val_t path, el_val_t key, el_val_t default_val); el_val_t query_int(el_val_t path, el_val_t key, el_val_t default_val);
el_val_t extract_id(el_val_t path, el_val_t prefix); el_val_t extract_id(el_val_t path, el_val_t prefix);
el_val_t route_stats(el_val_t method, el_val_t path, el_val_t body); el_val_t route_stats(el_val_t method, el_val_t path, el_val_t body);
el_val_t persist_canonical(void);
el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body); el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_get_node(el_val_t method, el_val_t path, el_val_t body); el_val_t route_get_node(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_scan_nodes(el_val_t method, el_val_t path, el_val_t body); el_val_t route_scan_nodes(el_val_t method, el_val_t path, el_val_t body);
@@ -21,23 +20,18 @@ el_val_t route_create_edge(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neighbors(el_val_t method, el_val_t path, el_val_t body); el_val_t route_neighbors(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_strengthen(el_val_t method, el_val_t path, el_val_t body); el_val_t route_strengthen(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_forget(el_val_t method, el_val_t path, el_val_t body); el_val_t route_forget(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_create_ise(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_save(el_val_t method, el_val_t path, el_val_t body); el_val_t route_save(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_load(el_val_t method, el_val_t path, el_val_t body); el_val_t route_load(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_health(el_val_t method, el_val_t path, el_val_t body); el_val_t route_health(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_load_merge(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_emit_ise(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_capture_knowledge(el_val_t method, el_val_t path, el_val_t body);
el_val_t check_auth_ok(el_val_t method, el_val_t body); el_val_t check_auth_ok(el_val_t method, el_val_t body);
el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body); el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body);
el_val_t bind_raw;
el_val_t bind_str; el_val_t bind_str;
el_val_t port; el_val_t port;
el_val_t data_dir_raw;
el_val_t data_dir; el_val_t data_dir;
el_val_t snapshot_path; el_val_t snapshot_path;
el_val_t boot_snap;
el_val_t parse_port(el_val_t bind) { el_val_t parse_port(el_val_t bind) {
el_val_t colon = str_index_of(bind, EL_STR(":")); el_val_t colon = str_index_of(bind, EL_STR(":"));
@@ -116,22 +110,17 @@ el_val_t route_stats(el_val_t method, el_val_t path, el_val_t body) {
return 0; return 0;
} }
el_val_t persist_canonical(void) {
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_1 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_1 = (EL_STR("/tmp/engram")); } else { _if_result_1 = (dir_raw); } _if_result_1; });
engram_save(el_str_concat(dir, EL_STR("/snapshot.json")));
return 1;
return 0;
}
el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body) {
el_val_t content = json_get_string(body, EL_STR("content")); el_val_t content = json_get_string(body, EL_STR("content"));
el_val_t nt_raw = json_get_string(body, EL_STR("node_type")); el_val_t node_type = json_get_string(body, EL_STR("node_type"));
el_val_t node_type = ({ el_val_t _if_result_2 = 0; if (str_eq(nt_raw, EL_STR(""))) { _if_result_2 = (EL_STR("Memory")); } else { _if_result_2 = (nt_raw); } _if_result_2; }); if (str_eq(node_type, EL_STR(""))) {
el_val_t sal_raw = json_get_float(body, EL_STR("salience")); node_type = EL_STR("Memory");
el_val_t salience = ({ el_val_t _if_result_3 = 0; if ((sal_raw == el_from_float(0.0))) { _if_result_3 = (el_from_float(0.5)); } else { _if_result_3 = (sal_raw); } _if_result_3; }); }
el_val_t salience = json_get_float(body, EL_STR("salience"));
if (salience == el_from_float(0.0)) {
salience = el_from_float(0.5);
}
el_val_t id = engram_node(content, node_type, salience); el_val_t id = engram_node(content, node_type, salience);
el_val_t saved = persist_canonical();
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"id\":\""), id), EL_STR("\",\"content\":\"")), content), EL_STR("\",\"node_type\":\"")), node_type), EL_STR("\"}")); return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"id\":\""), id), EL_STR("\",\"content\":\"")), content), EL_STR("\",\"node_type\":\"")), node_type), EL_STR("\"}"));
return 0; return 0;
} }
@@ -157,9 +146,11 @@ el_val_t route_scan_nodes(el_val_t method, el_val_t path, el_val_t body) {
} }
el_val_t route_scan_edges(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_scan_edges(el_val_t method, el_val_t path, el_val_t body) {
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR")); el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_4 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_4 = (EL_STR("/tmp/engram")); } else { _if_result_4 = (dir_raw); } _if_result_4; }); if (str_eq(dir, EL_STR(""))) {
el_val_t snap_path = el_str_concat(dir, EL_STR("/.scan-export.json")); dir = EL_STR("/tmp/engram");
}
el_val_t snap_path = el_str_concat(dir, EL_STR("/snapshot.json"));
engram_save(snap_path); engram_save(snap_path);
el_val_t snap = fs_read(snap_path); el_val_t snap = fs_read(snap_path);
if (str_eq(snap, EL_STR(""))) { if (str_eq(snap, EL_STR(""))) {
@@ -174,22 +165,36 @@ el_val_t route_scan_edges(el_val_t method, el_val_t path, el_val_t body) {
} }
el_val_t route_search(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_search(el_val_t method, el_val_t path, el_val_t body) {
el_val_t q = ({ el_val_t _if_result_5 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_5 = (query_param(path, EL_STR("q"))); } else { _if_result_5 = (json_get_string(body, EL_STR("query"))); } _if_result_5; }); el_val_t q = EL_STR("");
el_val_t lim_url = query_int(path, EL_STR("limit"), 0); if (str_eq(method, EL_STR("GET"))) {
el_val_t lim_body = json_get_int(body, EL_STR("limit")); q = query_param(path, EL_STR("q"));
el_val_t lim_either = ({ el_val_t _if_result_6 = 0; if ((lim_url > 0)) { _if_result_6 = (lim_url); } else { _if_result_6 = (lim_body); } _if_result_6; }); } else {
el_val_t limit = ({ el_val_t _if_result_7 = 0; if ((lim_either > 0)) { _if_result_7 = (lim_either); } else { _if_result_7 = (20); } _if_result_7; }); q = json_get_string(body, EL_STR("query"));
}
el_val_t limit = query_int(path, EL_STR("limit"), 20);
if (limit == 0) {
limit = json_get_int(body, EL_STR("limit"));
}
if (limit == 0) {
limit = 20;
}
return engram_search_json(q, limit); return engram_search_json(q, limit);
return 0; return 0;
} }
el_val_t route_activate(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_activate(el_val_t method, el_val_t path, el_val_t body) {
el_val_t q = ({ el_val_t _if_result_8 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_8 = (query_param(path, EL_STR("q"))); } else { _if_result_8 = (json_get_string(body, EL_STR("query"))); } _if_result_8; }); el_val_t q = EL_STR("");
if (str_eq(q, EL_STR(""))) { el_val_t depth = 3;
return err_json(EL_STR("missing query")); if (str_eq(method, EL_STR("GET"))) {
q = query_param(path, EL_STR("q"));
depth = query_int(path, EL_STR("depth"), 3);
} else {
q = json_get_string(body, EL_STR("query"));
el_val_t bd = json_get_int(body, EL_STR("depth"));
if (bd > 0) {
depth = bd;
}
} }
el_val_t d_raw = ({ el_val_t _if_result_9 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_9 = (query_int(path, EL_STR("depth"), 3)); } else { _if_result_9 = (json_get_int(body, EL_STR("depth"))); } _if_result_9; });
el_val_t depth = ({ el_val_t _if_result_10 = 0; if ((d_raw > 0)) { _if_result_10 = (d_raw); } else { _if_result_10 = (3); } _if_result_10; });
return el_str_concat(el_str_concat(EL_STR("{\"results\":"), engram_activate_json(q, depth)), EL_STR("}")); return el_str_concat(el_str_concat(EL_STR("{\"results\":"), engram_activate_json(q, depth)), EL_STR("}"));
return 0; return 0;
} }
@@ -197,12 +202,15 @@ el_val_t route_activate(el_val_t method, el_val_t path, el_val_t body) {
el_val_t route_create_edge(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_create_edge(el_val_t method, el_val_t path, el_val_t body) {
el_val_t from_id = json_get_string(body, EL_STR("from_id")); el_val_t from_id = json_get_string(body, EL_STR("from_id"));
el_val_t to_id = json_get_string(body, EL_STR("to_id")); el_val_t to_id = json_get_string(body, EL_STR("to_id"));
el_val_t rel_raw = json_get_string(body, EL_STR("relation")); el_val_t relation = json_get_string(body, EL_STR("relation"));
el_val_t relation = ({ el_val_t _if_result_11 = 0; if (str_eq(rel_raw, EL_STR(""))) { _if_result_11 = (EL_STR("associates")); } else { _if_result_11 = (rel_raw); } _if_result_11; }); if (str_eq(relation, EL_STR(""))) {
el_val_t w_raw = json_get_float(body, EL_STR("weight")); relation = EL_STR("associates");
el_val_t weight = ({ el_val_t _if_result_12 = 0; if ((w_raw == el_from_float(0.0))) { _if_result_12 = (el_from_float(0.5)); } else { _if_result_12 = (w_raw); } _if_result_12; }); }
el_val_t weight = json_get_float(body, EL_STR("weight"));
if (weight == el_from_float(0.0)) {
weight = el_from_float(0.5);
}
engram_connect(from_id, to_id, weight, relation); engram_connect(from_id, to_id, weight, relation);
el_val_t saved = persist_canonical();
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"from_id\":\""), from_id), EL_STR("\",\"to_id\":\"")), to_id), EL_STR("\",\"relation\":\"")), relation), EL_STR("\"}")); return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"from_id\":\""), from_id), EL_STR("\",\"to_id\":\"")), to_id), EL_STR("\",\"relation\":\"")), relation), EL_STR("\"}"));
return 0; return 0;
} }
@@ -223,7 +231,6 @@ el_val_t route_strengthen(el_val_t method, el_val_t path, el_val_t body) {
return err_json(EL_STR("missing node_id")); return err_json(EL_STR("missing node_id"));
} }
engram_strengthen(id); engram_strengthen(id);
el_val_t saved = persist_canonical();
return ok_json(); return ok_json();
return 0; return 0;
} }
@@ -234,40 +241,29 @@ el_val_t route_forget(el_val_t method, el_val_t path, el_val_t body) {
return err_json(EL_STR("missing id")); return err_json(EL_STR("missing id"));
} }
engram_forget(id); engram_forget(id);
el_val_t saved = persist_canonical();
return ok_json(); return ok_json();
return 0; return 0;
} }
el_val_t route_save(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_create_ise(el_val_t method, el_val_t path, el_val_t body) {
el_val_t p_raw = json_get_string(body, EL_STR("path")); el_val_t content = json_get_string(body, EL_STR("content"));
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR")); if (str_eq(content, EL_STR(""))) {
el_val_t dir = ({ el_val_t _if_result_13 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_13 = (EL_STR("/tmp/engram")); } else { _if_result_13 = (dir_raw); } _if_result_13; }); return err_json(EL_STR("missing content"));
el_val_t p = ({ el_val_t _if_result_14 = 0; if (str_eq(p_raw, EL_STR(""))) { _if_result_14 = (el_str_concat(dir, EL_STR("/snapshot.json"))); } else { _if_result_14 = (p_raw); } _if_result_14; }); }
engram_save(p); el_val_t sal = el_from_float(0.3);
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"path\":\""), p), EL_STR("\"}")); el_val_t imp = el_from_float(0.3);
return 0; el_val_t conf = el_from_float(0.8);
} el_val_t id = engram_node_full(content, EL_STR("InternalStateEvent"), EL_STR("state-event"), sal, imp, conf, EL_STR("Episodic"), EL_STR("[\"internal-state\",\"InternalStateEvent\"]"));
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\"}"));
el_val_t route_load(el_val_t method, el_val_t path, el_val_t body) {
el_val_t p_raw = json_get_string(body, EL_STR("path"));
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_15 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_15 = (EL_STR("/tmp/engram")); } else { _if_result_15 = (dir_raw); } _if_result_15; });
el_val_t p = ({ el_val_t _if_result_16 = 0; if (str_eq(p_raw, EL_STR(""))) { _if_result_16 = (el_str_concat(dir, EL_STR("/snapshot.json"))); } else { _if_result_16 = (p_raw); } _if_result_16; });
engram_load(p);
return ok_json();
return 0;
}
el_val_t route_health(el_val_t method, el_val_t path, el_val_t body) {
return EL_STR("{\"status\":\"ok\",\"engine\":\"engram-runtime-native\"}");
return 0; return 0;
} }
el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body) {
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR")); el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_17 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_17 = (EL_STR("/tmp/engram")); } else { _if_result_17 = (dir_raw); } _if_result_17; }); if (str_eq(dir, EL_STR(""))) {
el_val_t snap_path = el_str_concat(dir, EL_STR("/.sync-export.json")); dir = EL_STR("/tmp/engram");
}
el_val_t snap_path = el_str_concat(dir, EL_STR("/sync-export.json"));
engram_save(snap_path); engram_save(snap_path);
el_val_t snap = fs_read(snap_path); el_val_t snap = fs_read(snap_path);
if (str_eq(snap, EL_STR(""))) { if (str_eq(snap, EL_STR(""))) {
@@ -277,68 +273,36 @@ el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body) {
return 0; return 0;
} }
el_val_t route_load_merge(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_save(el_val_t method, el_val_t path, el_val_t body) {
el_val_t p = json_get_string(body, EL_STR("path")); el_val_t p = json_get_string(body, EL_STR("path"));
if (str_eq(p, EL_STR(""))) { if (str_eq(p, EL_STR(""))) {
return err_json(EL_STR("path is required")); el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
p = el_str_concat(dir, EL_STR("/snapshot.json"));
} }
if (str_eq(fs_read(p), EL_STR(""))) { engram_save(p);
return err_json(EL_STR("file missing or empty")); return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"path\":\""), p), EL_STR("\"}"));
}
el_val_t before_n = engram_node_count();
el_val_t before_e = engram_edge_count();
engram_load_merge(p);
el_val_t added_n = (engram_node_count() - before_n);
el_val_t added_e = (engram_edge_count() - before_e);
el_val_t saved = persist_canonical();
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"nodes_added\":"), int_to_str(added_n)), EL_STR(",\"edges_added\":")), int_to_str(added_e)), EL_STR(",\"node_count\":")), int_to_str(engram_node_count())), EL_STR("}"));
return 0; return 0;
} }
el_val_t route_emit_ise(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_load(el_val_t method, el_val_t path, el_val_t body) {
el_val_t content = json_get_string(body, EL_STR("content")); el_val_t p = json_get_string(body, EL_STR("path"));
if (str_eq(content, EL_STR(""))) { if (str_eq(p, EL_STR(""))) {
return err_json(EL_STR("missing content")); el_val_t dir = env(EL_STR("ENGRAM_DATA_DIR"));
if (str_eq(dir, EL_STR(""))) {
dir = EL_STR("/tmp/engram");
}
p = el_str_concat(dir, EL_STR("/snapshot.json"));
} }
el_val_t sal = el_from_float(0.3); engram_load(p);
el_val_t imp = el_from_float(0.3); return ok_json();
el_val_t conf = el_from_float(0.8);
el_val_t id = engram_node_full(content, EL_STR("InternalStateEvent"), EL_STR("state-event"), sal, imp, conf, EL_STR("Episodic"), EL_STR("[\"internal-state\",\"InternalStateEvent\"]"));
el_val_t ret_raw = env(EL_STR("ENGRAM_ISE_RETENTION_MS"));
el_val_t ret_ms = ({ el_val_t _if_result_18 = 0; if (str_eq(ret_raw, EL_STR(""))) { _if_result_18 = (172800000); } else { _if_result_18 = (str_to_int(ret_raw)); } _if_result_18; });
el_val_t pruned = engram_prune_telemetry(ret_ms);
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\",\"pruned\":")), int_to_str(pruned)), EL_STR("}"));
return 0; return 0;
} }
el_val_t route_capture_knowledge(el_val_t method, el_val_t path, el_val_t body) { el_val_t route_health(el_val_t method, el_val_t path, el_val_t body) {
el_val_t content = json_get_string(body, EL_STR("content")); return EL_STR("{\"status\":\"ok\",\"engine\":\"engram-runtime-native\"}");
if (str_eq(content, EL_STR(""))) {
return err_json(EL_STR("missing content"));
}
el_val_t title = json_get_string(body, EL_STR("title"));
el_val_t label = ({ el_val_t _if_result_19 = 0; if (str_eq(title, EL_STR(""))) { _if_result_19 = (str_slice(content, 0, 60)); } else { _if_result_19 = (title); } _if_result_19; });
el_val_t category_raw = json_get_string(body, EL_STR("category"));
el_val_t category = ({ el_val_t _if_result_20 = 0; if (str_eq(category_raw, EL_STR(""))) { _if_result_20 = (EL_STR("other")); } else { _if_result_20 = (category_raw); } _if_result_20; });
el_val_t ktier_raw = json_get_string(body, EL_STR("tier"));
el_val_t ktier = ({ el_val_t _if_result_21 = 0; if (str_eq(ktier_raw, EL_STR(""))) { _if_result_21 = (EL_STR("note")); } else { _if_result_21 = (ktier_raw); } _if_result_21; });
el_val_t project = json_get_string(body, EL_STR("project"));
el_val_t tags_raw = json_get_raw(body, EL_STR("tags"));
el_val_t tags_base = ({ el_val_t _if_result_22 = 0; if (str_eq(tags_raw, EL_STR(""))) { _if_result_22 = (EL_STR("[]")); } else { _if_result_22 = (tags_raw); } _if_result_22; });
el_val_t base_len = str_len(tags_base);
el_val_t head = str_slice(tags_base, 0, (base_len - 1));
el_val_t sep = ({ el_val_t _if_result_23 = 0; if (str_eq(head, EL_STR("["))) { _if_result_23 = (EL_STR("")); } else { _if_result_23 = (EL_STR(",")); } _if_result_23; });
el_val_t safe_cat = str_replace(category, EL_STR("\""), EL_STR("'"));
el_val_t safe_tier = str_replace(ktier, EL_STR("\""), EL_STR("'"));
el_val_t safe_proj = str_replace(project, EL_STR("\""), EL_STR("'"));
el_val_t proj_tag = ({ el_val_t _if_result_24 = 0; if (str_eq(safe_proj, EL_STR(""))) { _if_result_24 = (EL_STR("")); } else { _if_result_24 = (el_str_concat(el_str_concat(EL_STR(",\"project:"), safe_proj), EL_STR("\""))); } _if_result_24; });
el_val_t tags = el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(head, sep), EL_STR("\"category:")), safe_cat), EL_STR("\",\"tier:")), safe_tier), EL_STR("\"")), proj_tag), EL_STR("]"));
el_val_t sal = el_from_float(0.5);
el_val_t imp = el_from_float(0.5);
el_val_t conf = el_from_float(0.9);
el_val_t id = engram_node_full(content, EL_STR("Knowledge"), label, sal, imp, conf, EL_STR("Semantic"), tags);
el_val_t saved = persist_canonical();
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\"}"));
return 0; return 0;
} }
@@ -365,15 +329,12 @@ el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body) {
return route_health(method, path, body); return route_health(method, path, body);
} }
} }
if (str_eq(method, EL_STR("POST")) && str_eq(clean, EL_STR("/api/neuron/state-events"))) { if (str_eq(method, EL_STR("POST")) && str_starts_with(clean, EL_STR("/api/neuron/state-events"))) {
return route_emit_ise(method, path, body); return route_create_ise(method, path, body);
} }
if (!check_auth_ok(method, body)) { if (!check_auth_ok(method, body)) {
return err_json(EL_STR("unauthorized")); return err_json(EL_STR("unauthorized"));
} }
if (str_eq(method, EL_STR("POST")) && str_eq(clean, EL_STR("/api/neuron/knowledge/capture"))) {
return route_capture_knowledge(method, path, body);
}
if (str_eq(method, EL_STR("GET")) && (str_eq(clean, EL_STR("/api/stats")) || str_eq(clean, EL_STR("/stats")))) { if (str_eq(method, EL_STR("GET")) && (str_eq(clean, EL_STR("/api/stats")) || str_eq(clean, EL_STR("/stats")))) {
return route_stats(method, path, body); return route_stats(method, path, body);
} }
@@ -413,40 +374,32 @@ el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body) {
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/strengthen")) || str_eq(clean, EL_STR("/strengthen")))) { if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/strengthen")) || str_eq(clean, EL_STR("/strengthen")))) {
return route_strengthen(method, path, body); return route_strengthen(method, path, body);
} }
if (str_eq(method, EL_STR("GET")) && (str_eq(clean, EL_STR("/api/sync")) || str_eq(clean, EL_STR("/sync")))) {
return route_sync(method, path, body);
}
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/save")) || str_eq(clean, EL_STR("/save")))) { if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/save")) || str_eq(clean, EL_STR("/save")))) {
return route_save(method, path, body); return route_save(method, path, body);
} }
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/load")) || str_eq(clean, EL_STR("/load")))) { if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/load")) || str_eq(clean, EL_STR("/load")))) {
return route_load(method, path, body); return route_load(method, path, body);
} }
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/load-merge")) || str_eq(clean, EL_STR("/load-merge")))) {
return route_load_merge(method, path, body);
}
if (str_eq(method, EL_STR("GET")) && str_eq(clean, EL_STR("/api/sync"))) {
return route_sync(method, path, body);
}
return el_str_concat(el_str_concat(EL_STR("{\"error\":\"not found\",\"path\":\""), clean), EL_STR("\"}")); return el_str_concat(el_str_concat(EL_STR("{\"error\":\"not found\",\"path\":\""), clean), EL_STR("\"}"));
return 0; return 0;
} }
int main(int _argc, char** _argv) { int main(int _argc, char** _argv) {
el_runtime_init_args(_argc, _argv); el_runtime_init_args(_argc, _argv);
bind_raw = env(EL_STR("ENGRAM_BIND")); bind_str = env(EL_STR("ENGRAM_BIND"));
bind_str = ({ el_val_t _if_result_25 = 0; if (str_eq(bind_raw, EL_STR(""))) { _if_result_25 = (EL_STR(":8742")); } else { _if_result_25 = (bind_raw); } _if_result_25; }); if (str_eq(bind_str, EL_STR(""))) {
bind_str = EL_STR(":8742");
}
port = parse_port(bind_str); port = parse_port(bind_str);
data_dir_raw = env(EL_STR("ENGRAM_DATA_DIR")); data_dir = env(EL_STR("ENGRAM_DATA_DIR"));
data_dir = ({ el_val_t _if_result_26 = 0; if (str_eq(data_dir_raw, EL_STR(""))) { _if_result_26 = (EL_STR("/tmp/engram")); } else { _if_result_26 = (data_dir_raw); } _if_result_26; }); if (str_eq(data_dir, EL_STR(""))) {
data_dir = EL_STR("/tmp/engram");
}
snapshot_path = el_str_concat(data_dir, EL_STR("/snapshot.json")); snapshot_path = el_str_concat(data_dir, EL_STR("/snapshot.json"));
engram_load(snapshot_path); engram_load(snapshot_path);
boot_snap = fs_read(snapshot_path);
if (!str_eq(boot_snap, EL_STR(""))) {
if (engram_node_count() == 0) {
println(EL_STR("[engram] WARNING: snapshot.json is non-empty but load produced 0 nodes \xe2\x80\x94 preserving copy at snapshot.failed-load.json"));
fs_write(el_str_concat(data_dir, EL_STR("/snapshot.failed-load.json")), boot_snap);
} else {
fs_write(el_str_concat(data_dir, EL_STR("/snapshot.boot-backup.json")), boot_snap);
}
}
println(EL_STR("[engram] runtime-native graph engine")); println(EL_STR("[engram] runtime-native graph engine"));
println(el_str_concat(EL_STR("[engram] data_dir="), data_dir)); println(el_str_concat(EL_STR("[engram] data_dir="), data_dir));
println(el_str_concat(EL_STR("[engram] node_count="), int_to_str(engram_node_count()))); println(el_str_concat(EL_STR("[engram] node_count="), int_to_str(engram_node_count())));
+60 -192
View File
@@ -50,8 +50,12 @@ fn query_param(path: String, key: String) -> String {
if pos < 0 { return "" } if pos < 0 { return "" }
let after: String = str_slice(qs, pos + str_len(needle), str_len(qs)) let after: String = str_slice(qs, pos + str_len(needle), str_len(qs))
let amp: Int = str_index_of(after, "&") let amp: Int = str_index_of(after, "&")
if amp < 0 { return after } // SPEC-SEARCH-UPGRADE 2026-07-14: URL-decode the extracted value (%XX and
str_slice(after, 0, amp) // '+' were previously passed through literally, so an encoded multi-word
// query arrived as junk tokens pre-existing GET-path defect, masked
// until search could actually rank multi-word queries).
if amp < 0 { return url_decode(after) }
url_decode(str_slice(after, 0, amp))
} }
fn query_int(path: String, key: String, default_val: Int) -> Int { fn query_int(path: String, key: String, default_val: Int) -> Int {
@@ -76,43 +80,13 @@ fn route_stats(method: String, path: String, body: String) -> String {
engram_stats_json() engram_stats_json()
} }
// (2026-07-18 self-review) Scoping sweep: `let` inside an if-block creates an
// inner scope only it does NOT mutate the outer binding (documented with
// evidence in awareness.el, 2026-05-25). Every default/reassignment below used
// that broken pattern, so defaults never applied: nodes were created with
// node_type="" and salience=0.0, /api/search and /api/activate ALWAYS ran with
// q="" regardless of input, edges defaulted to relation=""/weight=0.0, and
// save/load with no "path" hit engram_save(""). Rewritten to the
// `let x = if cond { a } else { b }` expression form (the pattern the newer
// routes route_emit_ise/route_capture_knowledge already use correctly).
// persist_canonical save the canonical snapshot after a durable write.
//
// WHY (2026-07-22 self-review): the 2026-07-21 fix correctly stopped READ
// routes from writing the canonical snapshot.json but nothing was left
// that saved it on WRITE. Every mutation (node create, edge create,
// knowledge capture, forget, merge) lived only in RAM until someone POSTed
// /api/save manually; a process restart silently discarded everything since
// the last manual save. Observed live: two engram restarts during the
// 2026-07-22 review reverted the store to a ~17h-old snapshot, destroying
// same-day writes. Reads must never write the canonical; writes must always
// persist it. ISE telemetry is deliberately excluded (48h-pruned, loss-
// tolerant, ~2/min snapshotting the whole store per heartbeat is waste;
// any durable write that follows persists the pruning too).
fn persist_canonical() -> Int {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw }
engram_save(dir + "/snapshot.json")
return 1
}
fn route_create_node(method: String, path: String, body: String) -> String { fn route_create_node(method: String, path: String, body: String) -> String {
let content: String = json_get_string(body, "content") let content: String = json_get_string(body, "content")
let nt_raw: String = json_get_string(body, "node_type") let node_type: String = json_get_string(body, "node_type")
let node_type: String = if str_eq(nt_raw, "") { "Memory" } else { nt_raw } if str_eq(node_type, "") { let node_type = "Memory" }
let sal_raw: Float = json_get_float(body, "salience") let salience: Float = json_get_float(body, "salience")
let salience: Float = if sal_raw == 0.0 { 0.5 } else { sal_raw } if salience == 0.0 { let salience = 0.5 }
let id: String = engram_node(content, node_type, salience) let id: String = engram_node(content, node_type, salience)
let saved: Int = persist_canonical()
"{\"id\":\"" + id + "\",\"content\":\"" + content + "\",\"node_type\":\"" + node_type + "\"}" "{\"id\":\"" + id + "\",\"content\":\"" + content + "\",\"node_type\":\"" + node_type + "\"}"
} }
@@ -133,14 +107,13 @@ fn route_scan_nodes(method: String, path: String, body: String) -> String {
} }
// route_scan_edges bulk export of all edges as a JSON array. Implemented // route_scan_edges bulk export of all edges as a JSON array. Implemented
// via engram_save fs_read of a SCRATCH export path. (2026-07-21 self-review: // via engram_save fs_read of the canonical on-disk snapshot, which the
// previously this saved over the canonical snapshot.json on every GET if the // runtime keeps in lockstep with the in-memory graph. Live against the
// process ever booted with a partial/empty store, the first read request // running graph, not a stale export.
// clobbered the good snapshot. Read routes must never write the canonical path.)
fn route_scan_edges(method: String, path: String, body: String) -> String { fn route_scan_edges(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR") let dir: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw } if str_eq(dir, "") { let dir = "/tmp/engram" }
let snap_path: String = dir + "/.scan-export.json" let snap_path: String = dir + "/snapshot.json"
engram_save(snap_path) engram_save(snap_path)
let snap: String = fs_read(snap_path) let snap: String = fs_read(snap_path)
if str_eq(snap, "") { return "[]" } if str_eq(snap, "") { return "[]" }
@@ -153,34 +126,40 @@ fn route_scan_edges(method: String, path: String, body: String) -> String {
} }
fn route_search(method: String, path: String, body: String) -> String { fn route_search(method: String, path: String, body: String) -> String {
let q: String = if str_eq(method, "GET") { query_param(path, "q") } else { json_get_string(body, "query") } let q: String = ""
let lim_url: Int = query_int(path, "limit", 0) if str_eq(method, "GET") {
let lim_body: Int = json_get_int(body, "limit") let q = query_param(path, "q")
let lim_either: Int = if lim_url > 0 { lim_url } else { lim_body } } else {
let limit: Int = if lim_either > 0 { lim_either } else { 20 } let q = json_get_string(body, "query")
}
let limit: Int = query_int(path, "limit", 20)
if limit == 0 { let limit = json_get_int(body, "limit") }
if limit == 0 { let limit = 20 }
return engram_search_json(q, limit) return engram_search_json(q, limit)
} }
fn route_activate(method: String, path: String, body: String) -> String { fn route_activate(method: String, path: String, body: String) -> String {
let q: String = if str_eq(method, "GET") { query_param(path, "q") } else { json_get_string(body, "query") } let q: String = ""
// Guard: engram_activate with an empty query matches zero seeds, which let depth: Int = 3
// zeroes ALL carried working-memory weights (documented in awareness.el if str_eq(method, "GET") {
// perceive()). Never let an empty activation through to wipe WM. let q = query_param(path, "q")
if str_eq(q, "") { return err_json("missing query") } let depth = query_int(path, "depth", 3)
let d_raw: Int = if str_eq(method, "GET") { query_int(path, "depth", 3) } else { json_get_int(body, "depth") } } else {
let depth: Int = if d_raw > 0 { d_raw } else { 3 } let q = json_get_string(body, "query")
let bd: Int = json_get_int(body, "depth")
if bd > 0 { let depth = bd }
}
return "{\"results\":" + engram_activate_json(q, depth) + "}" return "{\"results\":" + engram_activate_json(q, depth) + "}"
} }
fn route_create_edge(method: String, path: String, body: String) -> String { fn route_create_edge(method: String, path: String, body: String) -> String {
let from_id: String = json_get_string(body, "from_id") let from_id: String = json_get_string(body, "from_id")
let to_id: String = json_get_string(body, "to_id") let to_id: String = json_get_string(body, "to_id")
let rel_raw: String = json_get_string(body, "relation") let relation: String = json_get_string(body, "relation")
let relation: String = if str_eq(rel_raw, "") { "associates" } else { rel_raw } if str_eq(relation, "") { let relation = "associates" }
let w_raw: Float = json_get_float(body, "weight") let weight: Float = json_get_float(body, "weight")
let weight: Float = if w_raw == 0.0 { 0.5 } else { w_raw } if weight == 0.0 { let weight = 0.5 }
engram_connect(from_id, to_id, weight, relation) engram_connect(from_id, to_id, weight, relation)
let saved: Int = persist_canonical()
"{\"ok\":true,\"from_id\":\"" + from_id + "\",\"to_id\":\"" + to_id + "\",\"relation\":\"" + relation + "\"}" "{\"ok\":true,\"from_id\":\"" + from_id + "\",\"to_id\":\"" + to_id + "\",\"relation\":\"" + relation + "\"}"
} }
@@ -195,7 +174,6 @@ fn route_strengthen(method: String, path: String, body: String) -> String {
let id: String = json_get_string(body, "node_id") let id: String = json_get_string(body, "node_id")
if str_eq(id, "") { return err_json("missing node_id") } if str_eq(id, "") { return err_json("missing node_id") }
engram_strengthen(id) engram_strengthen(id)
let saved: Int = persist_canonical()
ok_json() ok_json()
} }
@@ -203,24 +181,27 @@ fn route_forget(method: String, path: String, body: String) -> String {
let id: String = extract_id(path, "/api/nodes/") let id: String = extract_id(path, "/api/nodes/")
if str_eq(id, "") { return err_json("missing id") } if str_eq(id, "") { return err_json("missing id") }
engram_forget(id) engram_forget(id)
let saved: Int = persist_canonical()
ok_json() ok_json()
} }
fn route_save(method: String, path: String, body: String) -> String { fn route_save(method: String, path: String, body: String) -> String {
let p_raw: String = json_get_string(body, "path") let p: String = json_get_string(body, "path")
let dir_raw: String = env("ENGRAM_DATA_DIR") if str_eq(p, "") {
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw } let dir: String = env("ENGRAM_DATA_DIR")
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw } if str_eq(dir, "") { let dir = "/tmp/engram" }
let p = dir + "/snapshot.json"
}
engram_save(p) engram_save(p)
"{\"ok\":true,\"path\":\"" + p + "\"}" "{\"ok\":true,\"path\":\"" + p + "\"}"
} }
fn route_load(method: String, path: String, body: String) -> String { fn route_load(method: String, path: String, body: String) -> String {
let p_raw: String = json_get_string(body, "path") let p: String = json_get_string(body, "path")
let dir_raw: String = env("ENGRAM_DATA_DIR") if str_eq(p, "") {
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw } let dir: String = env("ENGRAM_DATA_DIR")
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw } if str_eq(dir, "") { let dir = "/tmp/engram" }
let p = dir + "/snapshot.json"
}
engram_load(p) engram_load(p)
ok_json() ok_json()
} }
@@ -242,36 +223,15 @@ fn route_health(method: String, path: String, body: String) -> String {
// (it skips nodes already present by ID). Auth-exempt: same-host internal call. // (it skips nodes already present by ID). Auth-exempt: same-host internal call.
// (2026-06-27 self-review: added this route to fix silent 10-min sync failures) // (2026-06-27 self-review: added this route to fix silent 10-min sync failures)
fn route_sync(method: String, path: String, body: String) -> String { fn route_sync(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR") let dir: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw } if str_eq(dir, "") { let dir = "/tmp/engram" }
// 2026-07-21 self-review: export to a scratch path, never the canonical let snap_path: String = dir + "/snapshot.json"
// snapshot.json read routes must not be able to clobber the good snapshot.
let snap_path: String = dir + "/.sync-export.json"
engram_save(snap_path) engram_save(snap_path)
let snap: String = fs_read(snap_path) let snap: String = fs_read(snap_path)
if str_eq(snap, "") { return "{\"nodes\":[],\"edges\":[]}" } if str_eq(snap, "") { return "{\"nodes\":[],\"edges\":[]}" }
return snap return snap
} }
// route_load_merge POST /api/load-merge {"path": "..."} merge a snapshot
// file into the live store WITHOUT resetting it (engram_load_merge skips nodes
// already present by id). Added 2026-07-21 self-review to restore the 244 kn-
// identity Knowledge nodes lost from the snapshot lineage between 05-13 and
// 07-13. Requires an explicit path: refuses to run without one so it can never
// be triggered accidentally against a default.
fn route_load_merge(method: String, path: String, body: String) -> String {
let p: String = json_get_string(body, "path")
if str_eq(p, "") { return err_json("path is required") }
if str_eq(fs_read(p), "") { return err_json("file missing or empty") }
let before_n: Int = engram_node_count()
let before_e: Int = engram_edge_count()
engram_load_merge(p)
let added_n: Int = engram_node_count() - before_n
let added_e: Int = engram_edge_count() - before_e
let saved: Int = persist_canonical()
"{\"ok\":true,\"nodes_added\":" + int_to_str(added_n) + ",\"edges_added\":" + int_to_str(added_e) + ",\"node_count\":" + int_to_str(engram_node_count()) + "}"
}
// route_emit_ise write an InternalStateEvent node from the soul daemon. // route_emit_ise write an InternalStateEvent node from the soul daemon.
// //
// Endpoint: POST /api/neuron/state-events // Endpoint: POST /api/neuron/state-events
@@ -285,20 +245,10 @@ fn route_load_merge(method: String, path: String, body: String) -> String {
// //
// Salience/importance set to match engram_node_full ISE defaults used by the // Salience/importance set to match engram_node_full ISE defaults used by the
// in-process fallback path in awareness.el (salience=0.3, importance=0.3, // in-process fallback path in awareness.el (salience=0.3, importance=0.3,
// confidence=0.8, tier=Episodic). // confidence=0.8, tier=Episodic). High temporal_decay_rate (1.617) ISEs
// are inherently transient; they should decay faster than structural knowledge.
// (2026-06-26 self-review: added this route after discovering ise_post was // (2026-06-26 self-review: added this route after discovering ise_post was
// silently failing the soul posts here but the endpoint didn't exist.) // silently failing the soul posts here but the endpoint didn't exist.)
//
// Retention (2026-07-16 self-review): an earlier comment here claimed ISEs
// got temporal_decay_rate=1.617 that was never implemented (engram_node_full
// hardcodes 0.0), and per-node decay only dampens activation anyway; it never
// removes nodes. By 2026-07-16 ISEs were 75% of the store (10,175 of 13,522
// nodes, ~4,300/day, unbounded). ISEs are already WM-excluded in
// engram_activate, so the fix is retention, not decay: every insert calls
// engram_prune_telemetry(), a single O(nodes+edges) compaction pass that
// removes ISEs older than ENGRAM_ISE_RETENTION_MS (default 48h), protecting
// "session-start" labels and self_review events as durable history. At
// ~3 ISEs/min this bounds telemetry at ~8.6k nodes instead of growing forever.
fn route_emit_ise(method: String, path: String, body: String) -> String { fn route_emit_ise(method: String, path: String, body: String) -> String {
let content: String = json_get_string(body, "content") let content: String = json_get_string(body, "content")
if str_eq(content, "") { return err_json("missing content") } if str_eq(content, "") { return err_json("missing content") }
@@ -310,64 +260,6 @@ fn route_emit_ise(method: String, path: String, body: String) -> String {
sal, imp, conf, sal, imp, conf,
"Episodic", "[\"internal-state\",\"InternalStateEvent\"]" "Episodic", "[\"internal-state\",\"InternalStateEvent\"]"
) )
let ret_raw: String = env("ENGRAM_ISE_RETENTION_MS")
let ret_ms: Int = if str_eq(ret_raw, "") { 172800000 } else { str_to_int(ret_raw) }
let pruned: Int = engram_prune_telemetry(ret_ms)
"{\"ok\":true,\"id\":\"" + id + "\",\"pruned\":" + int_to_str(pruned) + "}"
}
// Knowledge capture
//
// route_capture_knowledge direct Knowledge-node capture over HTTP.
//
// Endpoint: POST /api/neuron/knowledge/capture (auth required: "_auth" in body)
// Body: {"content": "...", "title": "...", "category": "...",
// "tier": "note|lesson|canonical", "tags": [...], "project": "...",
// "_auth": "<key>"}
//
// WHY (2026-07-15 self-review): the world-ingestor integrator was designed
// against this endpoint (its MCP-unavailable fallback), but the route never
// existed every direct push 404'd, and because the auth gate ran before
// routing, the failure surfaced as {"error":"unauthorized"} and was
// misdiagnosed for two weeks while world knowledge silently dropped.
// POST /api/nodes was no substitute: it discards label/tags/tier, which
// makes captured knowledge invisible to tag-scoped search and curiosity.
//
// The incoming knowledge tier (note/lesson/canonical) is preserved as a
// "tier:<x>" tag rather than mapped onto Engram's cognitive tiers Knowledge
// nodes land in Semantic (stable reference), and the epistemic tier stays
// queryable without inventing a lossy mapping.
fn route_capture_knowledge(method: String, path: String, body: String) -> String {
let content: String = json_get_string(body, "content")
if str_eq(content, "") { return err_json("missing content") }
let title: String = json_get_string(body, "title")
let label: String = if str_eq(title, "") { str_slice(content, 0, 60) } else { title }
let category_raw: String = json_get_string(body, "category")
let category: String = if str_eq(category_raw, "") { "other" } else { category_raw }
let ktier_raw: String = json_get_string(body, "tier")
let ktier: String = if str_eq(ktier_raw, "") { "note" } else { ktier_raw }
let project: String = json_get_string(body, "project")
let tags_raw: String = json_get_raw(body, "tags")
let tags_base: String = if str_eq(tags_raw, "") { "[]" } else { tags_raw }
// Merge category/tier/project markers into the tag array. Search matches
// against the tags string, so these make captures findable by facet.
let base_len: Int = str_len(tags_base)
let head: String = str_slice(tags_base, 0, base_len - 1)
let sep: String = if str_eq(head, "[") { "" } else { "," }
let safe_cat: String = str_replace(category, "\"", "'")
let safe_tier: String = str_replace(ktier, "\"", "'")
let safe_proj: String = str_replace(project, "\"", "'")
let proj_tag: String = if str_eq(safe_proj, "") { "" } else { ",\"project:" + safe_proj + "\"" }
let tags: String = head + sep + "\"category:" + safe_cat + "\",\"tier:" + safe_tier + "\"" + proj_tag + "]"
let sal: Float = 0.5
let imp: Float = 0.5
let conf: Float = 0.9
let id: String = engram_node_full(
content, "Knowledge", label,
sal, imp, conf,
"Semantic", tags
)
let saved: Int = persist_canonical()
"{\"ok\":true,\"id\":\"" + id + "\"}" "{\"ok\":true,\"id\":\"" + id + "\"}"
} }
@@ -407,12 +299,6 @@ fn handle_request(method: String, path: String, body: String) -> String {
return err_json("unauthorized") return err_json("unauthorized")
} }
// Knowledge capture (auth enforced above; the world-ingestor integrator
// and any headless session without MCP push knowledge through this)
if str_eq(method, "POST") && str_eq(clean, "/api/neuron/knowledge/capture") {
return route_capture_knowledge(method, path, body)
}
// Stats // Stats
if str_eq(method, "GET") && (str_eq(clean, "/api/stats") || str_eq(clean, "/stats")) { if str_eq(method, "GET") && (str_eq(clean, "/api/stats") || str_eq(clean, "/stats")) {
return route_stats(method, path, body) return route_stats(method, path, body)
@@ -469,9 +355,6 @@ fn handle_request(method: String, path: String, body: String) -> String {
if str_eq(method, "POST") && (str_eq(clean, "/api/load") || str_eq(clean, "/load")) { if str_eq(method, "POST") && (str_eq(clean, "/api/load") || str_eq(clean, "/load")) {
return route_load(method, path, body) return route_load(method, path, body)
} }
if str_eq(method, "POST") && (str_eq(clean, "/api/load-merge") || str_eq(clean, "/load-merge")) {
return route_load_merge(method, path, body)
}
// Sync soul daemon periodic pull of non-ISE knowledge into in-process graph // Sync soul daemon periodic pull of non-ISE knowledge into in-process graph
if str_eq(method, "GET") && str_eq(clean, "/api/sync") { if str_eq(method, "GET") && str_eq(clean, "/api/sync") {
@@ -483,31 +366,16 @@ fn handle_request(method: String, path: String, body: String) -> String {
// Entry // Entry
let bind_raw: String = env("ENGRAM_BIND") let bind_str: String = env("ENGRAM_BIND")
let bind_str: String = if str_eq(bind_raw, "") { ":8742" } else { bind_raw } if str_eq(bind_str, "") { let bind_str = ":8742" }
let port: Int = parse_port(bind_str) let port: Int = parse_port(bind_str)
// On startup, try to load any existing snapshot (best effort). // On startup, try to load any existing snapshot (best effort).
let data_dir_raw: String = env("ENGRAM_DATA_DIR") let data_dir: String = env("ENGRAM_DATA_DIR")
let data_dir: String = if str_eq(data_dir_raw, "") { "/tmp/engram" } else { data_dir_raw } if str_eq(data_dir, "") { let data_dir = "/tmp/engram" }
let snapshot_path: String = data_dir + "/snapshot.json" let snapshot_path: String = data_dir + "/snapshot.json"
engram_load(snapshot_path) engram_load(snapshot_path)
// 2026-07-21 self-review boot guard: if the snapshot file has content but the
// load produced 0 nodes, something is wrong (corrupt file / parse failure).
// Preserve the evidence and warn loudly and since read routes no longer write
// the canonical path, a bad boot can no longer clobber the good snapshot.
let boot_snap: String = fs_read(snapshot_path)
if !str_eq(boot_snap, "") {
if engram_node_count() == 0 {
println("[engram] WARNING: snapshot.json is non-empty but load produced 0 nodes — preserving copy at snapshot.failed-load.json")
fs_write(data_dir + "/snapshot.failed-load.json", boot_snap)
} else {
// Good load: keep a boot-time backup of the snapshot as loaded.
fs_write(data_dir + "/snapshot.boot-backup.json", boot_snap)
}
}
println("[engram] runtime-native graph engine") println("[engram] runtime-native graph engine")
println("[engram] data_dir=" + data_dir) println("[engram] data_dir=" + data_dir)
println("[engram] node_count=" + int_to_str(engram_node_count())) println("[engram] node_count=" + int_to_str(engram_node_count()))
-10
View File
@@ -17,16 +17,6 @@
// 4. Append dep to order after all its transitive deps // 4. Append dep to order after all its transitive deps
// 5. Deduplicate: skip already-ordered vessels // 5. Deduplicate: skip already-ordered vessels
// Cross-module forward declarations
// Defined in sibling epm modules; resolved at link time. The `extern fn` decls
// give elc the C prototypes so generated install.c compiles cleanly under strict
// compilers (gcc>=14 / clang) that reject implicit function declarations.
extern fn manifest_name(src: String) -> String // manifest.el
extern fn manifest_deps(src: String) -> String // manifest.el
extern fn registry_token() -> String // registry.el
extern fn registry_find(name: String, version: String) -> String // registry.el
extern fn registry_latest_version(name: String) -> String // registry.el
// Install paths // Install paths
// packages_dir returns the root directory for installed vessels. // packages_dir returns the root directory for installed vessels.
-9
View File
@@ -14,15 +14,6 @@
// EPM_REGISTRY_ORG org name that hosts vessel repos (default: neuron-technologies) // EPM_REGISTRY_ORG org name that hosts vessel repos (default: neuron-technologies)
// EPM_TOKEN Gitea personal access token (required for publish) // EPM_TOKEN Gitea personal access token (required for publish)
// Cross-module forward declarations
// These symbols are defined in sibling epm modules or the El runtime and are
// resolved at link time. The `extern fn` decls give elc the C prototype so the
// generated registry.c compiles cleanly under strict compilers (gcc>=14 / clang)
// that reject implicit function declarations. Signature arity must match the
// definition; return/param types are informational (all lower to el_val_t).
extern fn config(key: String) -> String // El runtime builtin
extern fn read_installed() -> String // install.el
// Config helpers // Config helpers
// registry_api_url returns the Gitea API base URL with no trailing slash. // registry_api_url returns the Gitea API base URL with no trailing slash.
-9
View File
@@ -6,15 +6,6 @@
// Depends on: registry.el (registry_latest_version, registry_find), // Depends on: registry.el (registry_latest_version, registry_find),
// install.el (read_installed, install_vessel, installed_version) // install.el (read_installed, install_vessel, installed_version)
// Cross-module forward declarations
// Defined in sibling epm modules; resolved at link time. The `extern fn` decls
// give elc the C prototypes so generated update.c compiles cleanly under strict
// compilers (gcc>=14 / clang) that reject implicit function declarations.
extern fn read_installed() -> String // install.el
extern fn installed_version(name: String) -> String // install.el
extern fn install_vessel(name: String, version: String) -> Bool // install.el
extern fn registry_latest_version(name: String) -> String // registry.el
// Semver helpers // Semver helpers
// semver_part extracts the Nth dot-separated component from a semver string. // semver_part extracts the Nth dot-separated component from a semver string.
@@ -75,7 +75,6 @@ static inline void* el_win_dlsym(void* handle, const char* name) {
#include <direct.h> /* _mkdir */ #include <direct.h> /* _mkdir */
#define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */ #define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */
#define timegm _mkgmtime /* UTC tm → time_t */ #define timegm _mkgmtime /* UTC tm → time_t */
#define fsync(fd) _commit(fd) /* no fsync() on Windows; _commit() (<io.h>) is the equiv */
/* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */ /* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */
static inline int setenv(const char* name, const char* value, int overwrite) { static inline int setenv(const char* name, const char* value, int overwrite) {
+128 -476
View File
@@ -82,14 +82,8 @@ static _Thread_local ElArena _tl_arena = {NULL, 0, 0};
static _Thread_local int _tl_arena_active = 0; static _Thread_local int _tl_arena_active = 0;
/* Binary-safe fs_read length — set by fs_read, consumed by http_send_response. /* Binary-safe fs_read length — set by fs_read, consumed by http_send_response.
* Allows serving PNGs and other binary files without strlen truncation. * Allows serving PNGs and other binary files without strlen truncation. */
* PAIRED with the buffer pointer it describes: the length may only be applied static _Thread_local size_t _tl_fs_read_len = 0;
* to the exact buffer fs_read returned. Without the pairing, any handler that
* fs_read a file and then WRAPPED it into a larger response had that response
* truncated to the file's length (Content-Length lied AND the send stopped
* short) the safety-contact onboarding trap, 2026-07-17. */
static _Thread_local size_t _tl_fs_read_len = 0;
static _Thread_local const char* _tl_fs_read_buf = NULL;
static void el_arena_track(char* p) { static void el_arena_track(char* p) {
if (!_tl_arena_active || !p) return; if (!_tl_arena_active || !p) return;
@@ -107,8 +101,6 @@ static void el_arena_track(char* p) {
void el_request_start(void) { void el_request_start(void) {
_tl_arena.count = 0; _tl_arena.count = 0;
_tl_arena_active = 1; _tl_arena_active = 1;
_tl_fs_read_len = 0; /* never let a previous request's file length */
_tl_fs_read_buf = NULL; /* leak into this response's byte accounting */
} }
/* Called by http_worker after the El handler returns and the response is sent. /* Called by http_worker after the El handler returns and the response is sent.
@@ -1492,14 +1484,11 @@ static void http_send_response(int fd, const char* body) {
} }
const char* eff_body = is_envelope ? env_body : body; const char* eff_body = is_envelope ? env_body : body;
/* Use the real byte count from fs_read ONLY when this body IS the exact /* Use the real byte count from fs_read if available (handles binary files
* buffer fs_read returned (binary files with embedded null bytes PNG, * with embedded null bytes PNG, WOFF2, etc.). Fall back to strlen for
* WOFF2, etc.). Any other body wrapped, enveloped, or derived must be * normal text/JSON responses where _tl_fs_read_len is 0. */
* measured with strlen, or it is truncated/over-read to the file's size. */ size_t blen = (_tl_fs_read_len > 0) ? _tl_fs_read_len : strlen(eff_body);
size_t blen = (_tl_fs_read_len > 0 && eff_body == _tl_fs_read_buf)
? _tl_fs_read_len : strlen(eff_body);
_tl_fs_read_len = 0; /* consume — one-shot per response */ _tl_fs_read_len = 0; /* consume — one-shot per response */
_tl_fs_read_buf = NULL;
int head_only = _tl_http_head_only; int head_only = _tl_http_head_only;
JsonBuf hdrs; jb_init(&hdrs); JsonBuf hdrs; jb_init(&hdrs);
@@ -1579,22 +1568,11 @@ static void* http_worker(void* arg) {
const char* rs = EL_CSTR(r); const char* rs = EL_CSTR(r);
/* Copy response out BEFORE arena teardown. /* Copy response out BEFORE arena teardown.
* For binary files, _tl_fs_read_len holds the real byte count * For binary files, _tl_fs_read_len holds the real byte count
* use memcpy instead of strdup so null bytes are preserved. * use memcpy instead of strdup so null bytes are preserved. */
* The stored length applies ONLY when the response IS the exact size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0);
* fs_read buffer; a wrapped/derived response must use strlen or
* it gets truncated (or over-read) to the file's length. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1); response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; } if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; } else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
} else { } else {
response = el_strdup_persist("el-runtime: no http handler registered"); response = el_strdup_persist("el-runtime: no http handler registered");
} }
@@ -1844,20 +1822,10 @@ static void* http_worker_v2(void* arg) {
el_val_t hmap = http_build_headers_map(hdr_block ? hdr_block : ""); el_val_t hmap = http_build_headers_map(hdr_block ? hdr_block : "");
el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), hmap, EL_STR(body)); el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), hmap, EL_STR(body));
const char* rs = EL_CSTR(r); const char* rs = EL_CSTR(r);
/* Same pairing rule as the v1 worker: the fs_read length is only size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0);
* trustworthy for the exact buffer fs_read returned. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1); response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; } if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; } else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
el_release(hmap); el_release(hmap);
} else { } else {
response = el_strdup_persist( response = el_strdup_persist(
@@ -1995,9 +1963,8 @@ void http_serve_async(el_val_t port, el_val_t handler) {
int sock = socket(AF_INET6, SOCK_STREAM, 0); int sock = socket(AF_INET6, SOCK_STREAM, 0);
if (sock < 0) { perror("socket"); return; } if (sock < 0) { perror("socket"); return; }
int yes = 1; int no = 0; int yes = 1; int no = 0;
/* Win32/mingw setsockopt takes optval as (const char*); the cast is portable on POSIX too. */ setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes));
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (const char*)&yes, sizeof(yes)); setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, &no, sizeof(no));
setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, (const char*)&no, sizeof(no));
struct sockaddr_in6 addr; struct sockaddr_in6 addr;
memset(&addr, 0, sizeof(addr)); memset(&addr, 0, sizeof(addr));
addr.sin6_family = AF_INET6; addr.sin6_family = AF_INET6;
@@ -2056,7 +2023,6 @@ el_val_t http_response(el_val_t status, el_val_t headers_json, el_val_t body) {
el_val_t fs_read(el_val_t pathv) { el_val_t fs_read(el_val_t pathv) {
const char* path = EL_CSTR(pathv); const char* path = EL_CSTR(pathv);
_tl_fs_read_len = 0; _tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!path) return el_wrap_str(el_strdup("")); if (!path) return el_wrap_str(el_strdup(""));
FILE* f = fopen(path, "rb"); FILE* f = fopen(path, "rb");
if (!f) return el_wrap_str(el_strdup("")); if (!f) return el_wrap_str(el_strdup(""));
@@ -2068,7 +2034,6 @@ el_val_t fs_read(el_val_t pathv) {
size_t got = fread(buf, 1, (size_t)sz, f); size_t got = fread(buf, 1, (size_t)sz, f);
buf[got] = '\0'; buf[got] = '\0';
_tl_fs_read_len = got; /* store real byte count for binary-safe send */ _tl_fs_read_len = got; /* store real byte count for binary-safe send */
_tl_fs_read_buf = buf; /* ...valid ONLY for this exact buffer */
fclose(f); fclose(f);
return el_wrap_str(buf); return el_wrap_str(buf);
} }
@@ -3611,10 +3576,8 @@ el_val_t json_get_raw(el_val_t json_str, el_val_t key) {
const char* k = EL_CSTR(key); const char* k = EL_CSTR(key);
const char* p = json_find_key(json, k); const char* p = json_find_key(json, k);
/* Clear fs_read binary-length hint — result is a fresh null-terminated /* Clear fs_read binary-length hint — result is a fresh null-terminated
* string, not the raw file bytes, so Content-Length must use strlen. * string, not the raw file bytes, so Content-Length must use strlen. */
* (Kept although the pointer pairing now makes this redundant.) */
_tl_fs_read_len = 0; _tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!p) return el_wrap_str(el_strdup("")); if (!p) return el_wrap_str(el_strdup(""));
const char* end = json_skip_value(p); const char* end = json_skip_value(p);
size_t n = (size_t)(end - p); size_t n = (size_t)(end - p);
@@ -6863,312 +6826,116 @@ static int istr_contains(const char* hay, const char* needle) {
return 0; return 0;
} }
/* ── Tokenized query matching ─────────────────────────────────────────── /* ---- SPEC-SEARCH-UPGRADE-OURS-2026-07-14: ranked search (BM25 + recency) ----
* The engram query surface (search / activate / goal-bias) historically * Replaces first-N-in-storage-order substring matching (measured 13% hit@5 on
* matched the ENTIRE raw query string as a single case-insensitive * the 15-query pinned eval; ranked model measured 93% offline). Deterministic,
* substring via istr_contains(field, q). That is Ctrl-F, not search: * local, transparent no model call on the hot path. Multi-word queries score
* a multi-word query like "windows msi signing" only matched a node whose * per-token (rare+concentrated terms weigh most); ties break newest-first so
* text contained that exact contiguous run, so real multi-word queries * fresh memories stop losing to storage order. The transparent-layer identity
* returned zero. istr_contains stays as the per-TOKEN primitive; these * filter is preserved unchanged: hidden self layers stay invisible here and
* helpers split the query on whitespace and match ANY token, then rank by * surface only via engram_activate the legitimate path. */
* how many DISTINCT tokens a node covers. Single-token queries are a strict
* special case (score is 0 or 1) so single-word callers never regress. */
#define ENGRAM_MAX_QTOKENS 32
#define ENGRAM_QTOK_LEN 256
/* Split q on whitespace into up to ENGRAM_MAX_QTOKENS distinct #define ENGRAM_BM25_MAX_QTOK 16
* (case-insensitive) tokens. Returns the token count. Over-long tokens are #define ENGRAM_BM25_TOKLEN 48
* truncated to ENGRAM_QTOK_LEN-1; over-count tokens are ignored. */
static int engram_tokenize_query(const char* q, static int engram_tok_next(const char** ps, char* out, int cap) {
char toks[][ENGRAM_QTOK_LEN], int maxtok) { const char* s = *ps;
while (*s && !isalnum((unsigned char)*s)) s++;
if (!*s) { *ps = s; return 0; }
int n = 0; int n = 0;
if (!q) return 0; while (*s && isalnum((unsigned char)*s)) {
const char* p = q; if (n < cap - 1) out[n++] = (char)tolower((unsigned char)*s);
while (*p && n < maxtok) { s++;
while (*p && isspace((unsigned char)*p)) p++;
if (!*p) break;
char buf[ENGRAM_QTOK_LEN];
size_t tl = 0;
while (*p && !isspace((unsigned char)*p)) {
if (tl < sizeof(buf) - 1) buf[tl++] = *p;
p++;
}
buf[tl] = '\0';
if (tl == 0) continue;
int dup = 0;
for (int s = 0; s < n; s++) {
if (strcasecmp(toks[s], buf) == 0) { dup = 1; break; }
}
if (dup) continue;
memcpy(toks[n], buf, tl + 1);
n++;
} }
return n; out[n] = 0; *ps = s; return 1;
} }
/* Count how many of the ntok distinct query tokens appear (case-insensitive) static void engram_field_stats(const char* field,
* in the node's content, label, or tags. 0 == no match. */ char qtok[][ENGRAM_BM25_TOKLEN], int nq,
static int engram_node_match_score(const EngramNode* n, int64_t* tf, int64_t* doclen) {
char toks[][ENGRAM_QTOK_LEN], int ntok) { if (!field) return;
int score = 0; char buf[ENGRAM_BM25_TOKLEN];
for (int t = 0; t < ntok; t++) { const char* p = field;
if (istr_contains(n->content, toks[t]) || while (engram_tok_next(&p, buf, sizeof buf)) {
istr_contains(n->label, toks[t]) || (*doclen)++;
istr_contains(n->tags, toks[t])) for (int t = 0; t < nq; t++)
score++; if (strcmp(buf, qtok[t]) == 0) tf[t]++;
} }
return score;
} }
/* Rank entry: distinct-token match count (primary, desc) then salience typedef struct { double score; int64_t created; int64_t idx; } EngramHit;
* (tiebreak, desc). */
typedef struct { int64_t idx; int score; double salience; } EngramRankEntry; static int engram_hit_cmp(const void* a, const void* b) {
static int engram_rank_cmp(const void* a, const void* b) { const EngramHit* x = (const EngramHit*)a;
const EngramRankEntry* ea = (const EngramRankEntry*)a; const EngramHit* y = (const EngramHit*)b;
const EngramRankEntry* eb = (const EngramRankEntry*)b; if (x->score != y->score) return (x->score < y->score) ? 1 : -1;
if (ea->score != eb->score) return eb->score - ea->score; /* desc */ if (x->created != y->created) return (x->created < y->created) ? 1 : -1;
if (ea->salience < eb->salience) return 1;
if (ea->salience > eb->salience) return -1;
return 0; return 0;
} }
/* ══════════════════════════════════════════════════════════════════════════ /* Scores every visible node against the query; writes ranked hits into `out`
* SEMANTIC SEARCH LAYER nomic-embed-text via Ollama /api/embeddings * (caller allocates g->node_count entries). Returns min(hits, lim). */
* static int64_t engram_search_ranked(EngramStore* g, const char* q, int64_t lim,
* Augments the lexical (istr_contains) matcher with dense-vector retrieval. EngramHit* out) {
* Node content and the query are embedded through a local Ollama server; char qtok[ENGRAM_BM25_MAX_QTOK][ENGRAM_BM25_TOKLEN];
* nodes are ranked by cosine similarity and UNIONED with lexical hits. This int nq = 0;
* lets a paraphrase query surface a node whose words never appear in it. {
* const char* p = q; char buf[ENGRAM_BM25_TOKLEN];
* DEGRADABLE BY DESIGN. The whole layer is gated on HAVE_CURL plus a one-shot while (nq < ENGRAM_BM25_MAX_QTOK && engram_tok_next(&p, buf, sizeof buf)) {
* runtime probe of the embedding endpoint. If curl is not compiled in, or int dup = 0;
* Ollama is unreachable, or ENGRAM_SEMANTIC=0, every entry point returns for (int t = 0; t < nq; t++)
* "no semantic signal" and callers fall back to pure lexical behaviour if (strcmp(qtok[t], buf) == 0) { dup = 1; break; }
* byte-for-byte the pre-existing search. if (!dup) { strcpy(qtok[nq], buf); nq++; }
*
* CACHE. Node embeddings are computed lazily on first use and cached in
* process memory keyed by node id, with an FNV-1a content hash for
* invalidation (edited content re-embeds). The query is embedded once per
* search call. This is what "avoid re-embedding the whole graph every query"
* buys us: a warm cache serves cosine from RAM. (A cold process still pays
* O(N) embed calls the first time each node is scanned persisting the cache
* to a snapshot sidecar is the documented next step, not done here.)
*
* nomic task prefixes ("search_query:" / "search_document:") are applied
* because nomic-embed-text is trained with them; they materially improve
* retrieval separation (empirically: paraphrase 0.72 vs distractors <0.48).
*
* ENV:
* ENGRAM_SEMANTIC "0" disables; unset/other = auto-probe
* ENGRAM_EMBED_URL default http://localhost:11434/api/embeddings
* ENGRAM_EMBED_MODEL default nomic-embed-text
* ENGRAM_SEMANTIC_MIN cosine threshold for a pure-semantic match (def 0.6)
* */
static double engram_semantic_min(void) {
static double v = -1.0;
if (v >= 0.0) return v;
const char* s = getenv("ENGRAM_SEMANTIC_MIN");
double d = 0.6;
if (s && *s) { char* e = NULL; double t = strtod(s, &e);
if (e != s && t >= 0.0 && t <= 1.0) d = t; }
v = d; return v;
}
#ifdef HAVE_CURL
typedef struct { char* id; uint64_t hash; float* vec; int dim; } EngramEmbEntry;
static EngramEmbEntry* g_emb_items = NULL;
static int64_t g_emb_count = 0, g_emb_cap = 0;
static int g_emb_state = 0; /* 0=unprobed, 1=available, -1=disabled */
static uint64_t engram_fnv1a(const char* s) {
uint64_t h = 1469598103934665603ULL;
if (s) for (const unsigned char* p = (const unsigned char*)s; *p; p++) {
h ^= *p; h *= 1099511628211ULL;
}
return h;
}
/* Parse "embedding":[f,f,...] from an Ollama response. malloc'd vec, or NULL. */
static float* engram_parse_embedding(const char* json, int* out_dim) {
if (!json) return NULL;
const char* p = strstr(json, "\"embedding\"");
if (!p) return NULL;
p = strchr(p, '[');
if (!p) return NULL;
p++;
int cap = 1024, n = 0;
float* v = malloc((size_t)cap * sizeof(float));
if (!v) return NULL;
while (*p && *p != ']') {
while (*p == ' ' || *p == '\t' || *p == '\n' || *p == '\r' || *p == ',') p++;
if (*p == ']' || !*p) break;
char* e = NULL;
double d = strtod(p, &e);
if (e == p) break;
if (n >= cap) { cap *= 2; float* nv = realloc(v, (size_t)cap * sizeof(float));
if (!nv) { free(v); return NULL; } v = nv; }
v[n++] = (float)d;
p = e;
}
if (n == 0) { free(v); return NULL; }
*out_dim = n;
return v;
}
/* JSON-escape src into a malloc'd buffer (no surrounding quotes). */
static char* engram_json_escape(const char* src) {
if (!src) src = "";
size_t n = strlen(src);
char* out = malloc(n * 2 + 1);
if (!out) return NULL;
size_t j = 0;
for (size_t i = 0; i < n; i++) {
unsigned char c = (unsigned char)src[i];
if (c == '"') { out[j++] = '\\'; out[j++] = '"'; }
else if (c == '\\') { out[j++] = '\\'; out[j++] = '\\'; }
else if (c == '\n') { out[j++] = '\\'; out[j++] = 'n'; }
else if (c == '\r') { out[j++] = '\\'; out[j++] = 'r'; }
else if (c == '\t') { out[j++] = '\\'; out[j++] = 't'; }
else if (c < 0x20) { /* drop other control bytes */ }
else { out[j++] = (char)c; }
}
out[j] = '\0';
return out;
}
/* Embed `prefix+text` via Ollama. Returns malloc'd vec (caller frees), or NULL. */
static float* engram_embed_raw(const char* prefix, const char* text, int* out_dim) {
if (!text) return NULL;
const char* url = getenv("ENGRAM_EMBED_URL");
if (!url || !*url) url = "http://localhost:11434/api/embeddings";
const char* model = getenv("ENGRAM_EMBED_MODEL");
if (!model || !*model) model = "nomic-embed-text";
/* Bound content length to keep latency/memory sane on huge nodes. */
char* trunc = NULL;
size_t maxlen = 8192;
if (strlen(text) > maxlen) {
trunc = malloc(maxlen + 1);
if (trunc) { memcpy(trunc, text, maxlen); trunc[maxlen] = '\0'; text = trunc; }
}
char* esc_prefix = engram_json_escape(prefix ? prefix : "");
char* esc = engram_json_escape(text);
free(trunc);
if (!esc || !esc_prefix) { free(esc); free(esc_prefix); return NULL; }
size_t blen = strlen(esc) + strlen(esc_prefix) + strlen(model) + 64;
char* body = malloc(blen);
if (!body) { free(esc); free(esc_prefix); return NULL; }
snprintf(body, blen, "{\"model\":\"%s\",\"prompt\":\"%s%s\"}", model, esc_prefix, esc);
free(esc); free(esc_prefix);
CURL* c = curl_easy_init();
if (!c) { free(body); return NULL; }
HttpBuf rb; httpbuf_init(&rb);
struct curl_slist* h = curl_slist_append(NULL, "Content-Type: application/json");
char errbuf[CURL_ERROR_SIZE]; errbuf[0] = '\0';
curl_easy_setopt(c, CURLOPT_URL, url);
curl_easy_setopt(c, CURLOPT_WRITEFUNCTION, http_write_cb);
curl_easy_setopt(c, CURLOPT_WRITEDATA, &rb);
curl_easy_setopt(c, CURLOPT_POST, 1L);
curl_easy_setopt(c, CURLOPT_POSTFIELDS, body);
curl_easy_setopt(c, CURLOPT_POSTFIELDSIZE, (long)strlen(body));
curl_easy_setopt(c, CURLOPT_HTTPHEADER, h);
curl_easy_setopt(c, CURLOPT_TIMEOUT_MS, el_http_timeout_ms());
curl_easy_setopt(c, CURLOPT_NOSIGNAL, 1L);
curl_easy_setopt(c, CURLOPT_ERRORBUFFER, errbuf);
CURLcode rc = curl_easy_perform(c);
curl_slist_free_all(h);
curl_easy_cleanup(c);
free(body);
if (rc != CURLE_OK) { free(rb.data); return NULL; }
float* v = engram_parse_embedding(rb.data, out_dim);
free(rb.data);
return v;
}
/* One-shot probe: is semantic search available? Caches the verdict. */
static int engram_semantic_enabled(void) {
if (g_emb_state != 0) return g_emb_state == 1;
const char* s = getenv("ENGRAM_SEMANTIC");
if (s && strcmp(s, "0") == 0) { g_emb_state = -1; return 0; }
int dim = 0;
float* v = engram_embed_raw("search_query: ", "probe", &dim);
if (v && dim > 0) { free(v); g_emb_state = 1; return 1; }
free(v);
g_emb_state = -1; return 0;
}
/* Embed the query. Returns malloc'd vec (caller frees), or NULL if semantic off. */
static float* engram_embed_query(const char* q, int* dim) {
if (!engram_semantic_enabled()) return NULL;
if (!q || !*q) return NULL;
return engram_embed_raw("search_query: ", q, dim);
}
/* Cached node embedding. Returns a pointer OWNED BY THE CACHE — do not free. */
static const float* engram_node_vec(EngramNode* n, int* out_dim) {
if (!n || !n->id) return NULL;
uint64_t h = engram_fnv1a(n->content);
for (int64_t i = 0; i < g_emb_count; i++) {
if (g_emb_items[i].id && strcmp(g_emb_items[i].id, n->id) == 0) {
if (g_emb_items[i].hash == h && g_emb_items[i].vec) {
*out_dim = g_emb_items[i].dim; return g_emb_items[i].vec;
}
/* content changed → re-embed in place */
int dim = 0;
float* v = engram_embed_raw("search_document: ", n->content ? n->content : "", &dim);
if (!v) return NULL;
free(g_emb_items[i].vec);
g_emb_items[i].vec = v; g_emb_items[i].dim = dim; g_emb_items[i].hash = h;
*out_dim = dim; return v;
} }
} }
int dim = 0; if (nq == 0) return 0;
float* v = engram_embed_raw("search_document: ", n->content ? n->content : "", &dim);
if (!v) return NULL; int64_t N = g->node_count;
if (g_emb_count >= g_emb_cap) { int64_t* tfm = (int64_t*)calloc((size_t)(N * nq), sizeof(int64_t));
int64_t nc = g_emb_cap ? g_emb_cap * 2 : 256; int64_t* dlen = (int64_t*)calloc((size_t)N, sizeof(int64_t));
EngramEmbEntry* ni = realloc(g_emb_items, (size_t)nc * sizeof(EngramEmbEntry)); if (!tfm || !dlen) { free(tfm); free(dlen); return 0; }
if (!ni) { free(v); return NULL; } int64_t df[ENGRAM_BM25_MAX_QTOK] = {0};
g_emb_items = ni; g_emb_cap = nc; double total_len = 0.0; int64_t live = 0;
for (int64_t i = 0; i < N; i++) {
EngramNode* n = &g->nodes[i];
if (engram_layer_is_transparent(n->layer_id)) continue;
live++;
int64_t* tf = &tfm[i * nq];
engram_field_stats(n->content, qtok, nq, tf, &dlen[i]);
engram_field_stats(n->label, qtok, nq, tf, &dlen[i]);
engram_field_stats(n->tags, qtok, nq, tf, &dlen[i]);
total_len += (double)dlen[i];
for (int t = 0; t < nq; t++) if (tf[t] > 0) df[t]++;
} }
g_emb_items[g_emb_count].id = strdup(n->id); double avg = (live > 0) ? total_len / (double)live : 1.0;
g_emb_items[g_emb_count].hash = h; if (avg <= 0.0) avg = 1.0;
g_emb_items[g_emb_count].vec = v; const double k1 = 1.2, b = 0.75;
g_emb_items[g_emb_count].dim = dim; int64_t nhits = 0;
g_emb_count++; for (int64_t i = 0; i < N; i++) {
*out_dim = dim; return v; EngramNode* n = &g->nodes[i];
if (engram_layer_is_transparent(n->layer_id)) continue;
int64_t* tf = &tfm[i * nq];
double s = 0.0;
for (int t = 0; t < nq; t++) {
if (tf[t] == 0) continue;
double idf = log(((double)live - (double)df[t] + 0.5) /
((double)df[t] + 0.5) + 1.0);
double tfd = (double)tf[t];
s += idf * (tfd * (k1 + 1.0)) /
(tfd + k1 * (1.0 - b + b * (double)dlen[i] / avg));
}
if (s > 0.0) {
out[nhits].score = s;
out[nhits].created = n->created_at;
out[nhits].idx = i;
nhits++;
}
}
free(tfm); free(dlen);
qsort(out, (size_t)nhits, sizeof(EngramHit), engram_hit_cmp);
return (nhits < lim) ? nhits : lim;
} }
static double engram_cosine(const float* a, const float* b, int dim) {
double dot = 0, na = 0, nb = 0;
for (int i = 0; i < dim; i++) { dot += (double)a[i] * b[i];
na += (double)a[i] * a[i];
nb += (double)b[i] * b[i]; }
if (na <= 0 || nb <= 0) return 0.0;
return dot / (sqrt(na) * sqrt(nb));
}
/* Cosine of node n against the query vector; 0 if unavailable / dim mismatch. */
static double engram_node_cosine(EngramNode* n, const float* qvec, int qdim) {
if (!qvec || qdim <= 0) return 0.0;
int ndim = 0;
const float* nv = engram_node_vec(n, &ndim);
if (!nv || ndim != qdim) return 0.0;
return engram_cosine(qvec, nv, qdim);
}
#else /* !HAVE_CURL — semantic layer compiled out; callers stay pure-lexical.
* Only the two boundary functions the always-compiled search/activate
* code calls are stubbed; the query embed always yields NULL so every
* cosine is 0 and every caller collapses to lexical-only. */
static float* engram_embed_query(const char* q, int* dim) { (void)q; (void)dim; return NULL; }
static double engram_node_cosine(EngramNode* n, const float* qvec, int qdim) {
(void)n; (void)qvec; (void)qdim; return 0.0;
}
#endif /* HAVE_CURL */
el_val_t engram_search(el_val_t query, el_val_t limit) { el_val_t engram_search(el_val_t query, el_val_t limit) {
EngramStore* g = engram_get(); EngramStore* g = engram_get();
const char* q = EL_CSTR(query); const char* q = EL_CSTR(query);
@@ -7176,45 +6943,13 @@ el_val_t engram_search(el_val_t query, el_val_t limit) {
if (lim <= 0) lim = 100; if (lim <= 0) lim = 100;
el_val_t lst = el_list_empty(); el_val_t lst = el_list_empty();
if (!q || !*q) return lst; if (!q || !*q) return lst;
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN]; if (g->node_count == 0) return lst;
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS); EngramHit* hits = (EngramHit*)malloc((size_t)g->node_count * sizeof(EngramHit));
if (ntok == 0) return lst; if (!hits) return lst;
/* Semantic augmentation: embed the query once; a node is a hit if it covers int64_t k = engram_search_ranked(g, q, lim, hits);
* >=1 query token (tokenized-lexical, #66) OR its cosine clears the for (int64_t i = 0; i < k; i++)
* threshold (#67). qvec is NULL (cosine 0) when semantic is unavailable lst = el_list_append(lst, engram_node_to_map(&g->nodes[hits[i].idx]));
* pure tokenized-lexical, byte-identical to the lexical-only behaviour. */
int qdim = 0;
float* qvec = engram_embed_query(q, &qdim);
double sem_min = engram_semantic_min();
EngramRankEntry* hits = malloc((size_t)g->node_count * sizeof(EngramRankEntry));
if (!hits) { free(qvec); return lst; }
int64_t nhits = 0;
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
/* Filter transparent layers: nodes whose layer is `transparent=1`
* shape output but are invisible to introspection ("what do you
* know about yourself"). They still surface via engram_activate
* + engram_compile_layered_json that's the legitimate path. */
if (engram_layer_is_transparent(n->layer_id)) continue;
int sc = engram_node_match_score(n, toks, ntok);
double sem = qvec ? engram_node_cosine(n, qvec, qdim) : 0.0;
if (sc > 0 || sem >= sem_min) {
hits[nhits].idx = i;
hits[nhits].score = sc;
hits[nhits].salience = n->salience;
nhits++;
}
}
/* Rank by distinct tokens matched (desc) then salience (desc), then cap.
* Pure-semantic hits (token score 0) sort after every lexical hit a
* lexical semantic union with lexical precedence. */
qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp);
int64_t end = nhits < lim ? nhits : lim;
for (int64_t k = 0; k < end; k++) {
lst = el_list_append(lst, engram_node_to_map(&g->nodes[hits[k].idx]));
}
free(hits); free(hits);
free(qvec);
return lst; return lst;
} }
@@ -7491,14 +7226,10 @@ static double engram_temporal_proximity_bonus(int64_t node_created,
static double engram_goal_bias(const EngramNode* n, const char* query) { static double engram_goal_bias(const EngramNode* n, const char* query) {
if (!query || !*query) return 1.0; if (!query || !*query) return 1.0;
double bias = 1.0; double bias = 1.0;
/* Direct lexical overlap, graded by token coverage: a node covering all /* Direct lexical overlap: node content/label/tags share text with query. */
* query tokens gets the full +0.5; partial coverage gets a proportional if (istr_contains(n->content, query) || istr_contains(n->label, query) ||
* share. Single-token queries full +0.5 on match, identical to before. */ istr_contains(n->tags, query)) {
{ bias += 0.5;
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(query, toks, ENGRAM_MAX_QTOKENS);
int sc = engram_node_match_score(n, toks, ntok);
if (sc > 0 && ntok > 0) bias += 0.5 * ((double)sc / (double)ntok);
} }
/* Node-type resonance with query intent. */ /* Node-type resonance with query intent. */
int technical_query = istr_contains(query, "code") || int technical_query = istr_contains(query, "code") ||
@@ -7564,31 +7295,14 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
if (!seeds) { if (!seeds) {
free(best_bg); free(best_hops); free(reached); return out; free(best_bg); free(best_hops); free(reached); return out;
} }
/* Tokenized + semantic seeding: a node seeds if it covers >=1 query token
* (tokenized-lexical, #66) OR its cosine clears the threshold (#67). A
* lexical seed's activation is scaled by token coverage (fraction of
* distinct query tokens covered) so a node matching all words seeds more
* strongly than one matching a single word; single-word queries coverage
* 1.0. A pure-semantic seed (no token match) is instead down-weighted by
* its cosine so paraphrase matches spread without overpowering exact seeds.
* q_vec is NULL (cosine 0) when semantic is unavailable the seed set is
* exactly the tokenized-lexical one. q_vec is freed right after this loop
* so the many downstream early-returns need no cleanup change. */
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
int q_dim = 0;
float* q_vec = engram_embed_query(q, &q_dim);
double q_sem_min = engram_semantic_min();
for (int64_t i = 0; i < g->node_count; i++) { for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i]; EngramNode* n = &g->nodes[i];
int sc = engram_node_match_score(n, toks, ntok); if (istr_contains(n->content, q) ||
double sem = q_vec ? engram_node_cosine(n, q_vec, q_dim) : 0.0; istr_contains(n->label, q) ||
if (sc > 0 || sem >= q_sem_min) { istr_contains(n->tags, q)) {
double tdecay = engram_temporal_decay(n, now_ms); double tdecay = engram_temporal_decay(n, now_ms);
double dampen = engram_activation_dampen(n); double dampen = engram_activation_dampen(n);
double act = n->salience * tdecay * dampen; double act = n->salience * tdecay * dampen;
if (sc > 0) act *= (ntok > 0 ? (double)sc / (double)ntok : 1.0);
else act *= sem; /* pure-semantic seed: down-weight by cosine */
seeds[seed_count].idx = i; seeds[seed_count].idx = i;
seeds[seed_count].act = act; seeds[seed_count].act = act;
seeds[seed_count].created_at = n->created_at; seeds[seed_count].created_at = n->created_at;
@@ -7598,7 +7312,6 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
reached[i] = 1; reached[i] = 1;
} }
} }
free(q_vec);
/* Compute mean seed created_at for temporal proximity bonus. */ /* Compute mean seed created_at for temporal proximity bonus. */
int64_t seed_epoch = 0; int64_t seed_epoch = 0;
if (seed_count > 0) { if (seed_count > 0) {
@@ -8150,36 +7863,9 @@ el_val_t engram_get_node_json(el_val_t id) {
return el_wrap_str(jb_finish(&b)); return el_wrap_str(jb_finish(&b));
} }
/* engram_get_node_by_label — find the first node whose label field exactly
* matches the given string. Returns the node as a JSON object string, or "{}"
* if no match is found.
*
* Used by chat.el to retrieve well-known nodes (e.g. "conv:history",
* "session:summary") by their stable label rather than by ID, which is immune
* to vector index drift across restarts.
*
* Exact match (strcmp, not istr_contains) because labels like "conv:history"
* must not collide with nodes whose content happens to contain that substring.
*
* Backported verbatim (idiom-adapted to jb_finish) from release runtime
* v1.0.0-20260501 to unblock the soul regen link: chat.el references this
* native but the current runtime lacked its definition. */
el_val_t engram_get_node_by_label(el_val_t label) {
const char* lbl = EL_CSTR(label);
if (!lbl || !*lbl) return el_wrap_str(el_strdup("{}"));
EngramStore* g = engram_get();
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
if (n->label && strcmp(n->label, lbl) == 0) {
JsonBuf b; jb_init(&b);
engram_emit_node_json(&b, n);
return el_wrap_str(jb_finish(&b));
}
}
return el_wrap_str(el_strdup("{}"));
}
el_val_t engram_search_json(el_val_t query, el_val_t limit) { el_val_t engram_search_json(el_val_t query, el_val_t limit) {
/* SPEC-SEARCH-UPGRADE 2026-07-14: same ranked BM25+recency core as
* engram_search; transparent-layer identity filter enforced inside it. */
EngramStore* g = engram_get(); EngramStore* g = engram_get();
const char* q = EL_CSTR(query); const char* q = EL_CSTR(query);
int64_t lim = (int64_t)limit; int64_t lim = (int64_t)limit;
@@ -8187,49 +7873,15 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) {
JsonBuf b; jb_init(&b); JsonBuf b; jb_init(&b);
jb_putc(&b, '['); jb_putc(&b, '[');
if (q && *q && g->node_count > 0) { if (q && *q && g->node_count > 0) {
/* Collect candidates from the UNION of tokenized-lexical and semantic EngramHit* hits = (EngramHit*)malloc((size_t)g->node_count * sizeof(EngramHit));
* matches, score each, rank by score, emit the top `lim`. A node is a if (hits) {
* candidate if it covers >=1 query token (tokenized-lexical, #66) OR its int64_t k = engram_search_ranked(g, q, lim, hits);
* query cosine clears the threshold (#67). Lexical score is the distinct for (int64_t i = 0; i < k; i++) {
* token count (>=1), so any lexical hit outranks a pure-semantic hit if (i) jb_putc(&b, ',');
* (cosine < 1); pure-semantic hits are scored by cosine alone. When engram_emit_node_json(&b, &g->nodes[hits[i].idx]);
* semantic is unavailable qvec is NULL, sem is 0, only tokenized-lexical
* hits are collected, and the stable insertion sort preserves order. */
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
int qdim = 0;
float* qvec = engram_embed_query(q, &qdim);
double sem_min = engram_semantic_min();
typedef struct { int64_t idx; double score; } Cand;
Cand* cand = malloc((size_t)g->node_count * sizeof(Cand));
if (cand) {
int64_t nc = 0;
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
if (engram_layer_is_transparent(n->layer_id)) continue;
int sc = engram_node_match_score(n, toks, ntok);
double sem = qvec ? engram_node_cosine(n, qvec, qdim) : 0.0;
if (sc > 0 || sem >= sem_min) {
cand[nc].idx = i;
cand[nc].score = (double)sc + sem;
nc++;
}
} }
/* Insertion sort by score desc; stable for equal scores. */ free(hits);
for (int64_t i = 1; i < nc; i++) {
Cand k = cand[i]; int64_t j = i - 1;
while (j >= 0 && cand[j].score < k.score) { cand[j + 1] = cand[j]; j--; }
cand[j + 1] = k;
}
int first = 1;
for (int64_t i = 0; i < nc && i < lim; i++) {
if (!first) jb_putc(&b, ',');
engram_emit_node_json(&b, &g->nodes[cand[i].idx]);
first = 0;
}
free(cand);
} }
free(qvec);
} }
jb_putc(&b, ']'); jb_putc(&b, ']');
return el_wrap_str(jb_finish(&b)); return el_wrap_str(jb_finish(&b));
-1
View File
@@ -632,7 +632,6 @@ el_val_t engram_load(el_val_t path);
* can pass results straight through without round-tripping ElList/ElMap * can pass results straight through without round-tripping ElList/ElMap
* through json_stringify. */ * through json_stringify. */
el_val_t engram_get_node_json(el_val_t id); el_val_t engram_get_node_json(el_val_t id);
el_val_t engram_get_node_by_label(el_val_t label);
el_val_t engram_search_json(el_val_t query, el_val_t limit); el_val_t engram_search_json(el_val_t query, el_val_t limit);
el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset); el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset);
el_val_t engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset); el_val_t engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset);
-1
View File
@@ -1072,7 +1072,6 @@ el_val_t __engram_save(el_val_t path) { return engram_save
el_val_t __engram_load(el_val_t path) { return engram_load(path); } el_val_t __engram_load(el_val_t path) { return engram_load(path); }
el_val_t __engram_get_node_json(el_val_t id) { return engram_get_node_json(id); } el_val_t __engram_get_node_json(el_val_t id) { return engram_get_node_json(id); }
el_val_t __engram_get_node_by_label(el_val_t label) { return engram_get_node_by_label(label); }
el_val_t __engram_search_json(el_val_t query, el_val_t limit) { el_val_t __engram_search_json(el_val_t query, el_val_t limit) {
return engram_search_json(query, limit); return engram_search_json(query, limit);
-1
View File
@@ -226,7 +226,6 @@ el_val_t __engram_activate(el_val_t query, el_val_t depth);
el_val_t __engram_save(el_val_t path); el_val_t __engram_save(el_val_t path);
el_val_t __engram_load(el_val_t path); el_val_t __engram_load(el_val_t path);
el_val_t __engram_get_node_json(el_val_t id); el_val_t __engram_get_node_json(el_val_t id);
el_val_t __engram_get_node_by_label(el_val_t label);
el_val_t __engram_search_json(el_val_t query, el_val_t limit); el_val_t __engram_search_json(el_val_t query, el_val_t limit);
el_val_t __engram_scan_nodes_json(el_val_t limit, el_val_t offset); el_val_t __engram_scan_nodes_json(el_val_t limit, el_val_t offset);
el_val_t __engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset); el_val_t __engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset);
+43 -28
View File
@@ -2670,7 +2670,6 @@ fn builtin_arity(name: String) -> Int {
if str_eq(name, "engram_save") { return 1 } if str_eq(name, "engram_save") { return 1 }
if str_eq(name, "engram_load") { return 1 } if str_eq(name, "engram_load") { return 1 }
if str_eq(name, "engram_get_node_json") { return 1 } if str_eq(name, "engram_get_node_json") { return 1 }
if str_eq(name, "engram_get_node_by_label") { return 1 }
if str_eq(name, "engram_search_json") { return 2 } if str_eq(name, "engram_search_json") { return 2 }
if str_eq(name, "engram_scan_nodes_json") { return 2 } if str_eq(name, "engram_scan_nodes_json") { return 2 }
if str_eq(name, "engram_neighbors_json") { return 3 } if str_eq(name, "engram_neighbors_json") { return 3 }
@@ -3627,6 +3626,24 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
let pos: Int = 0 let pos: Int = 0
let el_main_body: [Map<String, Any>] = native_list_empty() let el_main_body: [Map<String, Any>] = native_list_empty()
let toplevel_exec_stmts: [Map<String, Any>] = native_list_empty() let toplevel_exec_stmts: [Map<String, Any>] = native_list_empty()
// CGI IDENTITY CAPTURE (2026-08-09). A cgi block is a top-level DECLARATION, so
// the classifier below correctly excludes it from toplevel_exec_stmts and calls
// el_release on it. The identity emission further down then searched
// toplevel_exec_stmts for it a list that structurally can never contain it
// found nothing, and emitted nothing, silently. Measured: that search sees only
// [Let, Expr] for a program whose first statement is a cgi block.
// Fix: copy the values out BEFORE the release (strings, so no dangling reference)
// and emit from these. No search, so the failure mode is removed rather than moved.
let cgi_have: Bool = false
let cgi_name_v: String = ""
let cgi_did_v: String = ""
let cgi_prin_v: String = ""
let cgi_net_v: String = ""
let cgi_eng_v: String = ""
let cgi_has_did: Bool = false
let cgi_has_prin: Bool = false
let cgi_has_net: Bool = false
let cgi_has_eng: Bool = false
let has_toplevel_exec: Bool = false let has_toplevel_exec: Bool = false
let stream_running: Bool = true let stream_running: Bool = true
@@ -3737,6 +3754,20 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
if is_top_level_decl(stmt) { if is_top_level_decl(stmt) {
// Import, TypeDef, EnumDef, CgiBlock, ServiceBlock, ExternFn // Import, TypeDef, EnumDef, CgiBlock, ServiceBlock, ExternFn
// These are no-ops in codegen (forward decls already emitted) // These are no-ops in codegen (forward decls already emitted)
// except a CgiBlock, whose declared identity must survive
// this release to be emitted as a compiled constant.
if str_eq(sk, "CgiBlock") {
let cgi_have = true
let cgi_name_v = stmt["name"]
let cgi_did_v = stmt["dharma_id"]
let cgi_prin_v = stmt["principal"]
let cgi_net_v = stmt["network"]
let cgi_eng_v = stmt["engram"]
let cgi_has_did = stmt["has_dharma_id"]
let cgi_has_prin = stmt["has_principal"]
let cgi_has_net = stmt["has_network"]
let cgi_has_eng = stmt["has_engram"]
}
el_release(stmt) el_release(stmt)
} else { } else {
if str_eq(sk, "Let") { if str_eq(sk, "Let") {
@@ -3815,33 +3846,17 @@ fn codegen_streaming(tokens: [Any], sigs: [Map<String, Any>], source: String) ->
let sig2 = native_list_get(sigs, si2) let sig2 = native_list_get(sigs, si2)
let sk3: String = sig2["kind"] let sk3: String = sig2["kind"]
if str_eq(sk3, "cgi_block") { if str_eq(sk3, "cgi_block") {
// We need the full cgi_block data it was parsed by scan_fn_sigs // Emit from the values captured before the declaration was released.
// but scan only stored the name. For cgi_init we need dharma_id etc. // The previous implementation searched toplevel_exec_stmts, which by
// Since cgi blocks are rare and small, they end up in toplevel_exec_stmts. // construction never contains a declaration so it emitted nothing and
// Find the CgiBlock in toplevel_exec_stmts. // said nothing. See the capture block near toplevel_exec_stmts init.
let tes_n: Int = native_list_len(toplevel_exec_stmts) if cgi_have {
let tes_i: Int = 0 let arg_name2: String = "EL_STR(" + c_str_lit(cgi_name_v) + ")"
while tes_i < tes_n { let arg_did2: String = cgi_arg(cgi_did_v, cgi_has_did)
let tes = native_list_get(toplevel_exec_stmts, tes_i) let arg_prin2: String = cgi_arg(cgi_prin_v, cgi_has_prin)
let tes_k: String = tes["stmt"] let arg_net2: String = cgi_arg(cgi_net_v, cgi_has_net)
if str_eq(tes_k, "CgiBlock") { let arg_eng2: String = cgi_arg(cgi_eng_v, cgi_has_eng)
let cname2: String = tes["name"] emit_line(" el_cgi_init(" + arg_name2 + ", " + arg_did2 + ", " + arg_prin2 + ", " + arg_net2 + ", " + arg_eng2 + ");")
let cdid2: String = tes["dharma_id"]
let cprin2: String = tes["principal"]
let cnet2: String = tes["network"]
let ceng2: String = tes["engram"]
let has_did2: Bool = tes["has_dharma_id"]
let has_prin2: Bool = tes["has_principal"]
let has_net2: Bool = tes["has_network"]
let has_eng2: Bool = tes["has_engram"]
let arg_name2: String = "EL_STR(" + c_str_lit(cname2) + ")"
let arg_did2: String = cgi_arg(cdid2, has_did2)
let arg_prin2: String = cgi_arg(cprin2, has_prin2)
let arg_net2: String = cgi_arg(cnet2, has_net2)
let arg_eng2: String = cgi_arg(ceng2, has_eng2)
emit_line(" el_cgi_init(" + arg_name2 + ", " + arg_did2 + ", " + arg_prin2 + ", " + arg_net2 + ", " + arg_eng2 + ");")
}
let tes_i = tes_i + 1
} }
} }
let si2 = si2 + 1 let si2 = si2 + 1
+1 -25
View File
@@ -23,29 +23,10 @@ fn tok_at(tokens: [Any], pos: Int) -> Map<String, Any> {
} }
fn tok_kind(tokens: [Any], pos: Int) -> String { fn tok_kind(tokens: [Any], pos: Int) -> String {
// Out-of-range reads must report the Eof sentinel so every `== "Eof"`
// termination guard in the parser fires. Without this, reading past the
// single trailing Eof token returns runtime null (el_list_get OOB -> 0),
// which matches no delimiter, letting inner parse loops append AST nodes
// forever on malformed input -> unbounded allocation -> OOM.
let n: Int = native_list_len(tokens) / 2
if pos < 0 {
return "Eof"
}
if pos >= n {
return "Eof"
}
native_list_get(tokens, pos * 2) native_list_get(tokens, pos * 2)
} }
fn tok_value(tokens: [Any], pos: Int) -> String { fn tok_value(tokens: [Any], pos: Int) -> String {
let n: Int = native_list_len(tokens) / 2
if pos < 0 {
return ""
}
if pos >= n {
return ""
}
native_list_get(tokens, pos * 2 + 1) native_list_get(tokens, pos * 2 + 1)
} }
@@ -54,12 +35,7 @@ fn expect(tokens: [Any], pos: Int, kind: String) -> Int {
if k == kind { if k == kind {
return pos + 1 return pos + 1
} }
// On mismatch, error recovery is best-effort. But never step PAST the Eof // On mismatch just advance; error recovery is best-effort
// sentinel: once at Eof a mismatch means the input ended early, and
// advancing would run the cursor off the token list.
if k == "Eof" {
return pos
}
pos + 1 pos + 1
} }
@@ -1,186 +0,0 @@
#ifndef EL_PLATFORM_WIN_H
#define EL_PLATFORM_WIN_H
/*
* el_platform_win.h Windows OS-boundary shim for el_runtime.c.
*
* Branch: feat/windows-el-runtime. Included ONLY when _WIN32 is defined; the POSIX build is
* untouched. Goal: let el_runtime.c (a BSD-sockets / dlfcn / fork host) compile and link with
* mingw-w64 into a native neuron.exe, with no behavioural change to the Linux/macOS build.
*
* What it maps:
* - sockets : winsock2 (same call names: socket/bind/listen/accept/recv/send/setsockopt).
* Sockets close with closesocket() (see el_closesocket), and the stack must be
* started once with WSAStartup done automatically via a load-time constructor.
* - dlsym : el_runtime.c uses dlsym(RTLD_DEFAULT, name) to resolve callback/tool symbols
* exported by the main module. Windows equivalent: GetProcAddress on the process
* module. Link the soul with -Wl,--export-all-symbols so the symbols are findable.
* - popen : mapped to _popen/_pclose.
* - threads : UNCHANGED. mingw-w64 ships winpthreads, so <pthread.h> + -lpthread just work.
*/
#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN
#endif
#include <winsock2.h>
#include <ws2tcpip.h>
#include <windows.h>
#include <io.h>
#include <process.h>
/* Portable headers mingw-w64 provides (verified present). */
#include <stdarg.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h> /* strcasecmp */
#include <ctype.h>
#include <math.h>
#include <time.h>
#include <sys/time.h> /* mingw-w64 provides gettimeofday here */
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <dirent.h>
#include <errno.h>
#include <pthread.h>
/* ── socket close ─────────────────────────────────────────────────────────── */
/* Winsock closes sockets with closesocket(), not close() (close() is for file fds). The POSIX
build defines the same helper as close() so the call sites are identical across platforms. */
static inline int el_closesocket(SOCKET s) { return closesocket(s); }
/* ── setsockopt optval type ───────────────────────────────────────────────── */
/* Winsock's setsockopt takes optval as (const char*); POSIX takes (const void*), so el_runtime.c
passes &int directly. GCC 14+ makes that an error under -Wincompatible-pointer-types. Wrap it so
the runtime's POSIX-style call sites compile unchanged (defined before the macro so the wrapper
itself resolves to the real winsock setsockopt). */
static inline int el_setsockopt(SOCKET s, int level, int optname, const void* optval, int optlen) {
return setsockopt(s, level, optname, (const char*)optval, optlen);
}
#define setsockopt(s, l, o, v, n) el_setsockopt((s), (l), (o), (v), (int)(n))
/* ── winsock init (once, at load) ─────────────────────────────────────────── */
static void el__win_net_init(void) {
static int inited = 0;
if (!inited) { WSADATA w; WSAStartup(MAKEWORD(2, 2), &w); inited = 1; }
}
__attribute__((constructor)) static void el__win_ctor(void) { el__win_net_init(); }
/* ── dlsym → GetProcAddress ───────────────────────────────────────────────── */
#ifndef RTLD_DEFAULT
#define RTLD_DEFAULT ((void*)0)
#endif
static inline void* el_win_dlsym(void* handle, const char* name) {
(void)handle;
return (void*)(uintptr_t)GetProcAddress(GetModuleHandleA(NULL), name);
}
#define dlsym(h, n) el_win_dlsym((h), (n))
/* ── popen / pclose ───────────────────────────────────────────────────────── */
#define popen _popen
#define pclose _pclose
/* ── misc POSIX → Win32 shims ─────────────────────────────────────────────── */
#include <direct.h> /* _mkdir */
#define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */
#define timegm _mkgmtime /* UTC tm → time_t */
/* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */
static inline int setenv(const char* name, const char* value, int overwrite) {
(void)overwrite;
return _putenv_s(name, value ? value : "");
}
static inline int unsetenv(const char* name) {
/* _putenv_s(name, "") sets VAR="" rather than removing it.
* SetEnvironmentVariableA(name, NULL) truly deletes it from the Win32
* env block; then we sync the CRT cache with _putenv("NAME="). */
SetEnvironmentVariableA(name, NULL);
size_t len = strlen(name);
char *buf = (char*)malloc(len + 2);
if (!buf) return -1;
memcpy(buf, name, len);
buf[len] = '=';
buf[len + 1] = '\0';
_putenv(buf);
free(buf);
return 0;
}
/* nanosleep — not available in MSVC/UCRT; approximate with Sleep(). */
static inline int el_nanosleep(const struct timespec *req, struct timespec *rem) {
(void)rem;
DWORD ms = (DWORD)((req->tv_sec * 1000ULL) + (req->tv_nsec / 1000000ULL));
Sleep(ms ? ms : 1);
return 0;
}
#define nanosleep(req, rem) el_nanosleep((req), (rem))
/* localtime_r/gmtime_r: Windows offers localtime_s/gmtime_s with reversed arg order. */
static inline struct tm* localtime_r(const time_t* t, struct tm* out) {
return localtime_s(out, t) == 0 ? out : (struct tm*)0;
}
static inline struct tm* gmtime_r(const time_t* t, struct tm* out) {
return gmtime_s(out, t) == 0 ? out : (struct tm*)0;
}
/* ── libcurl: degradable stubs for the curl-less Windows build ─────────────── */
/* The curl-less validation build (WITH_CURL=0) links no libcurl. el_runtime.c uses libcurl
* unconditionally for its HTTP client / LLM layer; these stubs let it compile and link so the
* runtime, HTTP *server*, graph and memory work natively on Windows. Live outbound HTTP/LLM calls
* degrade to a runtime error (curl_easy_perform returns an error) matching the documented
* curl-less contract. When HAVE_CURL is defined (WITH_CURL=1) the real <curl/curl.h> is used and
* this whole block is compiled out. POSIX never sees this header, so the POSIX build is untouched. */
#ifndef HAVE_CURL
typedef void CURL;
typedef int CURLcode;
#define CURLE_OK 0
#define CURLE_HTTP_RETURNED_ERROR 22
#define CURL_ERROR_SIZE 256
/* Option ids: values are irrelevant to the no-op setopt below; kept distinct for readability. */
#define CURLOPT_URL 10002
#define CURLOPT_WRITEFUNCTION 20011
#define CURLOPT_WRITEDATA 10001
#define CURLOPT_POSTFIELDS 10015
#define CURLOPT_POSTFIELDSIZE 120
#define CURLOPT_POST 47
#define CURLOPT_HTTPHEADER 10023
#define CURLOPT_TIMEOUT_MS 155
#define CURLOPT_NOSIGNAL 99
#define CURLOPT_USERAGENT 10018
#define CURLOPT_FOLLOWLOCATION 52
#define CURLOPT_ERRORBUFFER 10010
#define CURLOPT_CUSTOMREQUEST 10036
#define CURLOPT_FAILONERROR 45
struct curl_slist { char* data; struct curl_slist* next; };
static inline struct curl_slist* curl_slist_append(struct curl_slist* list, const char* s) {
struct curl_slist* node = (struct curl_slist*)malloc(sizeof(struct curl_slist));
if (!node) return list;
node->data = s ? strdup(s) : NULL;
node->next = NULL;
if (!list) return node;
struct curl_slist* p = list;
while (p->next) p = p->next;
p->next = node;
return list;
}
static inline void curl_slist_free_all(struct curl_slist* list) {
while (list) { struct curl_slist* n = list->next; free(list->data); free(list); list = n; }
}
static inline CURL* curl_easy_init(void) { return (CURL*)malloc(1); }
static inline CURLcode curl_easy_setopt(CURL* h, int opt, ...) { (void)h; (void)opt; return CURLE_OK; }
static inline CURLcode curl_easy_perform(CURL* h) { (void)h; return 7 /* CURLE_COULDNT_CONNECT */; }
static inline void curl_easy_cleanup(CURL* h) { free(h); }
static inline const char* curl_easy_strerror(CURLcode c) {
(void)c; return "libcurl not built in (curl-less build)";
}
#endif /* !HAVE_CURL */
#endif /* EL_PLATFORM_WIN_H */
File diff suppressed because it is too large Load Diff
@@ -758,18 +758,6 @@ el_val_t trace_span_start(el_val_t name);
el_val_t trace_span_end(el_val_t span_handle); el_val_t trace_span_end(el_val_t span_handle);
el_val_t emit_event(el_val_t name, el_val_t duration_ms); el_val_t emit_event(el_val_t name, el_val_t duration_ms);
/* ── Runtime symbols required by the soul modules ──────────────────────────── */
/* All implemented in el_runtime.c but omitted from this release header; the soul dist modules
* reference them directly, so the public header must export them. Declarations only mirrors the
* mainline el_runtime.h and is platform-independent (no behavioural change to the POSIX build). */
typedef el_val_t (*http_handler_fn)(el_val_t method, el_val_t path, el_val_t body);
typedef el_val_t (*http_handler4_fn)(el_val_t method, el_val_t path, el_val_t body, el_val_t headers);
el_val_t el_arena_push(void);
el_val_t el_arena_pop(el_val_t mark);
void http_serve_async(el_val_t port, el_val_t handler);
el_val_t engram_get_node_by_label(el_val_t label);
el_val_t engram_prune_telemetry(el_val_t older_than_ms);
#ifdef __cplusplus #ifdef __cplusplus
} }
#endif #endif