Compare commits

..

31 Commits

Author SHA1 Message Date
Neuron ac248887c2 test: empty commit — is dev CI red independent of any change?
El SDK CI - dev / build-and-test (pull_request) Failing after 12m14s
Baseline probe. #89 shows red on dev CI and this Gitea instance does not serve
Actions logs (404/500 on every route), so the only way to learn whether dev CI is
broken on its own is to run it against dev with no change at all.

Reproduced locally first: control (dev head) and treatment (dev + #89) produce
BYTE-IDENTICAL test results — both build the self-hosted compiler and elb, both
fail the same three timezone tests (earth-zone, dst-spring-forward,
rhythm-grounding). That exonerates #89 locally. This probe asks the same question
of the real runner.

Close once it reports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 14:45:39 -05:00
will.anderson 5d0d4555ae Sync main into dev (GitOps: keep dev current; main authoritative) (#84)
El SDK CI - dev / build-and-test (push) Successful in 8m32s
2026-08-03 15:38:40 +00:00
will.anderson 8347a2f1c0 Merge pull request 'docs: add root README mapping the El monorepo' (#83) from feat/AddingReadme into dev
El SDK CI - dev / build-and-test (push) Successful in 8m22s
2026-07-31 04:25:44 +00:00
Andre Botelho Rodrigues Almeida b97b644799 Addind readme.md file to start documenting the repo
El SDK CI - dev / build-and-test (pull_request) Successful in 8m18s
2026-07-23 16:41:51 -03:00
will.anderson d71fc4c1c0 Merge pull request 'promote stage -> main: reconciled el runtime (engram search + natives + durable truncation fix + Windows port)' (#82) from stage into main
El SDK Release / build-and-release (push) Successful in 8m31s
El SDK CI - dev / build-and-test (pull_request) Successful in 8m41s
2026-07-22 21:44:01 +00:00
will.anderson a118d19393 Merge pull request 'promote dev -> stage: el cluster (#66 engram + #79 truncation fix + release-runtime Windows port)' (#81) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 7m58s
El SDK Release / build-and-release (pull_request) Successful in 4m16s
2026-07-22 21:20:17 +00:00
will.anderson c6aa1e5c53 Merge pull request 'Land el cluster: #66 engram search + natives, #79 truncation fix, + release-runtime Windows port (reconciled)' (#80) from reconcile/el-cluster-windows-runtime into dev
El SDK CI - dev / build-and-test (push) Successful in 8m3s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m25s
Land el cluster (#66 + #79 + release-runtime Windows-port reconciliation) into dev
2026-07-22 21:06:36 +00:00
will.anderson ff577391f2 reconcile(release-runtime): Windows-port + complete v1.0.0 release runtime so the desktop soul cross-compiles
El SDK CI - dev / build-and-test (pull_request) Successful in 7m7s
The desktop soul (neuron/dist) compiles against the v1.0.0-20260501 release
runtime. After #66 landed the engram natives (tokenized/ranked search,
engram_prune_telemetry) and #79 the durable truncation fix into this runtime,
two gaps remained before it could cross-compile the Windows brain:

1. Windows OS boundary: the release runtime had no Win32 path. Ported the same
   _WIN32-guarded shim the mainline runtime carries (#69): #ifdef _WIN32 ->
   el_platform_win.h (winsock/dlsym/popen + WSAStartup ctor), SOCKET fd guards
   and el_closesocket() at every socket site, CreateProcessA for exec_bg, the
   tm_zone/mingw guard, an el_setsockopt optval wrapper (GCC14), and curl-less
   libcurl stubs. Every change is _WIN32/HAVE_CURL-gated — the POSIX build is
   byte-identical (gcc -fsyntax-only clean; native behaviour unchanged).

2. Header exports: the release el_runtime.h omitted symbols the soul dist calls
   that are defined in this runtime's .c — the http_handler_fn/http_handler4_fn
   typedefs and el_arena_push/pop, engram_prune_telemetry, engram_get_node_by_label.
   Declaration-only, POSIX-neutral; fixes implicit-declaration/unknown-type
   errors under the C11 mingw build.

Result: x86_64-w64-mingw32-gcc compiles el_runtime.c + all 48 soul modules
clean; POSIX gcc -fsyntax-only clean. This is the Windows-port PR the runtime
needed on main (the release-runtime counterpart to #69), landed via stage.
2026-07-22 15:56:08 -05:00
will.anderson ee0d5f9b97 Merge #79: durable HTTP response-truncation fix, both runtimes (via stage) 2026-07-22 15:46:02 -05:00
will.anderson 391bd818ea Merge #66: tokenized+ranked engram lexical search + engram natives (via stage) 2026-07-22 15:45:54 -05:00
will.anderson 43636aed99 runtime: pair fs_read length hint with its buffer in BOTH runtimes — kill response truncation for good
El SDK Release / build-and-release (pull_request) Failing after 7s
The binary-safe fs_read length (_tl_fs_read_len) was consumed by the HTTP
response path for ANY body, even when a handler wrapped a smaller file into a
larger reply. Content-Length then lied AND the send stopped short: the
safety-contact (988) routes returned 178 of 208/218 bytes, cut mid-'set_at' —
unparseable JSON. The desktop app read that as failure. On Windows the shipped
brain is an OLD build without even the per-handler workaround, so EVERY reply
truncated: the app can't read confirmations and refuses the new user.

Durable fix: pair the length hint with the exact buffer pointer it describes
(_tl_fs_read_buf). Apply the raw byte count ONLY when the response IS that
buffer (binary file serving stays correct); every wrapped/enveloped/derived
body is measured with strlen. Reset both at request start and in fs_read /
json_get_raw. This also closes the stale-hint heap over-read (a length larger
than a later body would read past it out the socket) that a plain max() leaves
open — so this class of bug dies on every platform, not just where a handler
happened to be patched.

Applied identically to the mainline runtime (lang/el-compiler/runtime) AND the
frozen release runtime (lang/releases/v1.0.0-20260501) the desktop souls
compile against — the release copy still carried the raw leak, which is why the
Windows brain kept truncating. Same proven approach as PR #78 (Tim Lingo),
extended to cover the release runtime and rebased onto current main.

Both runtimes: gcc -fsyntax-only clean.
2026-07-22 15:04:25 -05:00
will.anderson 2baa0b9a41 Merge pull request 'release: promote stage -> main (ci publish hardening for sdk-release)' (#77) from stage into main
El SDK Release / build-and-release (push) Successful in 7m55s
2026-07-15 21:21:28 +00:00
will.anderson 6a8b2461cd Merge pull request 'release: promote dev -> stage (ci publish hardening for stage/main)' (#76) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 8m19s
El SDK Release / build-and-release (pull_request) Failing after 13m1s
2026-07-15 21:16:11 +00:00
will.anderson bcb356fe69 Merge pull request 'ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly' (#75) from hotfix/ci-stage-main-publish-hardening into dev
El SDK CI - stage / build-and-test (pull_request) Successful in 4m27s
El SDK CI - dev / build-and-test (push) Failing after 14m3s
2026-07-15 21:15:27 +00:00
will.anderson dd7827059a ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly
El SDK CI - dev / build-and-test (pull_request) Failing after 14m30s
Mirror the PR #72 fix (applied to ci-dev.yaml) onto ci-stage.yaml and
sdk-release.yaml. The stage and prod release jobs reported FAILURE even
when the el-runtime-c/-h publish SUCCEEDED, because the ancillary ci-base
Docker rebuild (a CI-cache optimization on the fragile host-mode GCE
runner) reddened the whole job.

- Rebuild ci-base step: continue-on-error: true — never blocks/reddens
  the job; the SDK publish is the deliverable.
- Publish step: set -euo pipefail + empty-key guard + active-account echo
  so a real publish failure still fails loud and is diagnosable.
2026-07-15 16:14:50 -05:00
will.anderson 208e36c899 Merge pull request 'release: promote stage -> main (tokenized search, get_node_by_label, epm fix, win portability)' (#74) from stage into main
El SDK Release / build-and-release (push) Successful in 8m23s
2026-07-15 18:24:39 +00:00
will.anderson b97ce74d1f Merge pull request 'release: promote dev -> stage (tokenized search, get_node_by_label, epm fix)' (#73) from dev into stage
El SDK CI - stage / build-and-test (push) Failing after 8m45s
El SDK Release / build-and-release (pull_request) Successful in 4m1s
2026-07-15 17:20:11 +00:00
will.anderson 155a449c4e Merge pull request 'ci(dev): make SDK publish fail loudly, decouple ci-base rebuild' (#72) from hotfix/ci-dev-publish-hardening into dev
El SDK CI - dev / build-and-test (push) Successful in 8m56s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m10s
2026-07-15 16:34:14 +00:00
will.anderson 4696fd6833 ci(dev): make SDK publish fail loudly, decouple ci-base rebuild
El SDK CI - dev / build-and-test (pull_request) Successful in 8m51s
The dev push build went green-then-red while nothing published: the
Publish step had no set -e, so an auth/upload failure exited 0 (silent
no-publish), while the ci-base rebuild (set -euo pipefail + Docker on the
host-mode runner) hard-failed the job. Add set -euo pipefail + an empty-key
guard + active-account echo to the Publish step so failures surface with a
retrievable log, and mark the ci-base cache rebuild continue-on-error so
the fragile Docker step can never block the actual SDK artifact publish.
2026-07-15 11:33:37 -05:00
will.anderson 581a351fb1 Merge pull request 'integrate: stack PRs #65–#69 (elc OOM guard, tokenized+semantic engram search, get_node_by_label, win portability) for green CI' (#71) from hotfix/stage-elc-engram-integration into dev
El SDK CI - dev / build-and-test (push) Failing after 14m31s
2026-07-15 15:49:57 +00:00
will.anderson 8ce8656de2 epm: declare cross-module callees as extern fn so strict compilers accept generated C
El SDK CI - dev / build-and-test (pull_request) Successful in 7m33s
epm's sibling modules (registry/install/update) call functions defined in other
modules and in the El runtime (config, read_installed, registry_find,
manifest_deps, manifest_name, registry_latest_version, registry_token,
install_vessel, installed_version) without importing them, so elc emits no C
prototype for those calls. gcc<=13 treated the resulting implicit declarations
as warnings; gcc>=14 and clang reject them as hard errors, which is why the
"Build epm" CI step fails and blocks the whole dev/stage pipeline.

Add `extern fn` forward declarations -- El's own separate-compilation mechanism
-- for each cross-module callee at the top of registry/install/update. This
gives elc the correct C prototype in every generated translation unit, so the
calls compile cleanly and still resolve at link time. Simply suppressing
-Wimplicit-function-declaration would be unsafe: an implicit int return
truncates the 64-bit pointer returns of config/registry_find into a latent
crash, so declaring the true signatures is the correct fix. Localized to epm;
touches neither elc nor the runtime.
2026-07-15 10:14:43 -05:00
will.anderson 1e49560f1f Merge remote-tracking branch 'origin/feat/engram-semantic-search' into hotfix/stage-elc-engram-integration
El SDK CI - dev / build-and-test (pull_request) Failing after 14m39s
# Conflicts:
#	lang/el-compiler/runtime/el_runtime.c
2026-07-15 09:33:05 -05:00
will.anderson e8f0b5a9de Merge remote-tracking branch 'origin/fix/engram-lexical-tokenized-search' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 40287c4cfc Merge remote-tracking branch 'origin/hotfix/win-runtime-portability' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 0481bea44d Merge remote-tracking branch 'origin/hotfix/runtime-engram-get-node-by-label' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 9d565ca080 Merge remote-tracking branch 'origin/hotfix/elc-fixes' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 4773dd0aa2 runtime: make Windows soul reproducible from a clean el checkout
El SDK Release / build-and-release (pull_request) Failing after 16s
Two el_runtime portability defects only ever lived in staged local copies
used to hand-build neuron-ui PR #136's curl-enabled Windows neuron.exe.
gcc 15 promotes both to hard errors, so a clean el checkout cannot rebuild
that soul. Upstream the minimal fixes so the build is reproducible:

- http_serve_async: cast setsockopt optval to (const char*). Win32/mingw
  setsockopt wants const char*, not int*; the cast is a no-op on POSIX and
  matches the four already-cast sites elsewhere in this file.
- engram_save persist path: map fsync -> _commit in the _WIN32-only
  el_platform_win.h (io.h already included). Windows has no fsync(); the
  POSIX path is untouched.
2026-07-15 04:24:08 -05:00
will.anderson 6b9d9e6c4a Add engram_get_node_by_label runtime native to unblock soul link
El SDK Release / build-and-release (pull_request) Failing after 22s
chat.el calls the runtime native engram_get_node_by_label to fetch
well-known nodes (conv:history, session:summary) by stable label rather
than by ID — immune to vector-index drift across restarts. The current
runtime never defined it, so the regenerated dist/soul.c fails to link.

Backport the function verbatim (idiom-adapted to jb_finish) from release
runtime v1.0.0-20260501 and register it as an EL builtin exactly like its
siblings: runtime definition + prototype, __-prefixed seed wrapper +
prototype, and codegen arity entry. No search-site code is touched.
2026-07-15 04:07:33 -05:00
will.anderson b4967af13e feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical)
Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.

Wired into all three query entry points in el_runtime.c:
  - engram_search_json (HTTP /api/search): collect lexical ∪ semantic
    candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
    rank, emit top-N. Stable sort preserves old order when semantic is off.
  - engram_search (internal el_val twin): lexical ∪ semantic union.
  - engram_activate seed loop (HTTP /api/activate): a node seeds if it
    lexically matches OR clears the cosine threshold; pure-semantic seeds
    enter scaled by cosine so paraphrase spreads without overpowering.

Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.

Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.

Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).
2026-07-14 18:48:16 -05:00
will.anderson 2b2a1246e7 Merge pull request 'runtime: fix the memory-leak + write-corruption pair in el_runtime.c' (#64) from hotfix/el-runtime-leak-and-persist into main
El SDK Release / build-and-release (push) Failing after 10m52s
2026-07-13 21:23:31 +00:00
will.anderson 5c41c66a0f Merge pull request 'fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only' (#60) from fix/windows-rusage-guard into stage
El SDK CI - stage / build-and-test (push) Failing after 13m21s
fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only
2026-06-25 16:48:13 +00:00
19 changed files with 934 additions and 76 deletions
+15
View File
@@ -214,9 +214,18 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -268,6 +277,12 @@ jobs:
# Patches ci-base:dev in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+15
View File
@@ -212,12 +212,21 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -253,6 +262,12 @@ jobs:
# Patches ci-base:stage in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+15
View File
@@ -288,12 +288,21 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -345,6 +354,12 @@ jobs:
# Patches ci-base:latest in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+154
View File
@@ -0,0 +1,154 @@
# El
**A self-hosting, statically-typed language that compiles to C — built around a graph-native runtime instead of a database driver.**
El is the execution substrate for the Neuron agent runtime, the DHARMA network, and the Engram knowledge graph. This repository is the monorepo for the whole stack: the language itself, the graph memory engine it's built to talk to natively, and the tools (package manager, IDE, UI framework, diagramming) built on top of it.
---
## Why El exists
Every other language treats persistent, associative state as something you reach for through a driver — a SQL client, an ORM, a Redis library bolted on from outside. El inverts that: graph operations (`engram_*`) are runtime primitives, on the same footing as string or list operations. There is no separate database driver because the database is not separate.
El has four defining properties:
1. **Self-hosting compiler.** The compiler (`lexer.el`, `parser.el`, `codegen.el`, `compiler.el`) is written in El. It compiles El source to C, which `cc` compiles against a fixed runtime into a native binary. A Rust genesis compiler bootstrapped the first iteration; the self-hosted binary at `lang/dist/platform/elc` has been the canonical compiler ever since — every binary in `dist/platform/` was produced by an earlier version of itself compiling `el-compiler/src/`. The chain is auditable: source is the ground truth, not the binary. See [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) for the full recovery path if that binary is ever lost.
2. **C compilation target.** Every compiled program is plain C11. Every El value is `el_val_t` (`int64_t`); strings are heap pointers cast through it. Functions become C functions; top-level statements become `main()`.
3. **Graph-native runtime.** The runtime provides first-class graph operations over an in-process Engram store — no separate DB driver, no ORM.
4. **DHARMA-aware identity.** A `cgi` block declares a program's DHARMA identity at compile time. The runtime resolves identity before user code runs, so `dharma_*` calls have a stable principal and channel surface throughout.
---
## Architecture map
```
┌─────────────┐
│ lang │ El compiler + C runtime
│ (El itself) │ everything below is written in it,
└──────┬──────┘ or compiles down through it
┌─────────────┼─────────────┐
│ │ │
┌──────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
│ engram │ │ epm │ │ ide │
│ graph/mem │ │ package │ │ editor + │
│ substrate │ │ manager │ │ LSP │
└──────┬─────┘ └───────────┘ └───────────┘
┌───────┼────────────────┬─────────────────────┐
│ │ │ │
┌─────▼───┐ ┌─▼──────────┐ ┌──▼──────────┐ ┌─────▼──────┐
│ elp │ │ ql │ │ ui │ │ arbor │
│ NLG / │ │engram-el. │ |spreading- │ |arbor │
│ 31 langs│ │studio+tests│ |activation UI│ |diagram lang│
└─────────┘ └────────────┘ └─────────────┘ └────────────┘
```
`lang` is the foundation — the compiler and C runtime everything else builds on. `engram` is the graph-native memory/state engine that gives El its identity (property 3 above). Everything else is either a tool for working with El (`epm`, `ide`) or a system built on top of Engram's graph model (`elp`, `ql`, `ui`, `arbor`).
---
## Repository layout
### [lang/](lang/) — the El language
The compiler and runtime. Self-hosting: `elc-cli.el``compiler.el``lexer.el` / `parser.el` / `codegen.el` / `codegen-js.el`, textually inlined and compiled in one pass. Compiles to C11 and links against `el-compiler/runtime/el_seed.c`, a hand-maintained OS-boundary layer (libcurl HTTP, pthreads, filesystem, arena allocation) — everything else in the runtime is native El (`runtime/*.el`).
Two layers to know: **El programs** (`.el` files — where nearly all work belongs) and **the C seed** (`el_seed.c` — edit only for genuine OS-level access; never re-implement what El can already express).
Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented. In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), the real `engram_*` and `dharma_*` runtimes (currently stubs), and libcurl-backed `http_get`/`http_post`/`http_serve`. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language.
Key docs: [AGENTS.md](lang/AGENTS.md) (agent-facing orientation), [BOOTSTRAP.md](lang/BOOTSTRAP.md) (compiler recovery from scratch), [spec/language.md](lang/spec/language.md), [spec/codegen-js.md](lang/spec/codegen-js.md).
### [engram/](engram/) — graph intelligence substrate
**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. Rust core (`engram-core`, `engram-ffi`) exposed to El and other languages (Kotlin, TypeScript/WASM, Go bindings).
The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop (`strength = parent_strength × edge_weight × target_salience × cosine_sim`), gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works.
Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay**`importance × recency-decay × log(activation_count)`. Forgetting is adaptive pruning, not a bug: unreinforced memories stop competing for attention without being deleted.
Backed by `sled` (embedded, local-first, no daemon) with flat cosine scan for vector search — deliberately simple until scale demands an HNSW layer. Full API and design rationale in [engram/README.md](engram/README.md).
### [elp/](elp/) — Engram Language Protocol
Bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*``grammar``realizer``semantics``elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages.
### [epm/](epm/) — El Package Manager
Manages **vessels** (El's package unit): publish, install, resolve dependencies. Vessels are stored in Engram as graph nodes, not files in a registry index — `epm` reads the local `manifest.el`, talks to Engram over HTTP, and writes resolved vessels to `.epm/vessels/`. Source: `registry.el`, `install.el`, `update.el`, `manifest.el`.
### [ide/](ide/) — El IDE
Three vessels: **el-ide-server** (HTTP backend — file ops, build/run, LSP bridge, plugin host, settings), **el-lsp** (the language server — completion, hover, diagnostics, outline, format, type graph), and **el-plugin-host** (first-party plugin lifecycle: install/remove/enable/disable). `ide/projects/` and `ide/examples/` hold sample projects, including the canonical `hello-friends` first-program walkthrough.
### [ql/](ql/) — engram-el
The El-native integration layer for a *live* Engram server — not a library (no importable modules, no build artifact), a set of standalone `.el` programs run directly via `el run-file`. Three components: **Studio** (`studio/studio.el`, a full terminal graph explorer), a **Hebbian field-model** proof of concept, and El builtin / LLM-builtin smoke test suites. This is the reference for correct patterns when an El program uses Engram as its substrate. Spec: [ql/spec/elql.md](ql/spec/elql.md).
### [ui/](ui/) — el-ui
A frontend framework where **component state is an Engram graph and reactivity is spreading activation** — not virtual-DOM diffing (React), Proxy-based dependency tracking (Vue), or compile-time analysis (Svelte). Re-renders are activated and propagated the same way associative memory retrieval works in `engram/`.
~15 vessels covering the full frontend surface: `el-platform` (env/fs/network/clock abstraction), `el-config`, `el-html` (SSR emit primitives), `el-layout`, `el-style` (design tokens/themes), `el-i18n`, `el-auth` / `el-identity` (JWT, sessions, OAuth PKCE — Engram-native), `el-services` (REST/gRPC/WebSocket bindings), `el-aop` (`@authenticate`/`@authorize`/`@cache`/`@rate_limit` decorators), `el-secrets`, `el-graph` (graph rendering/editor), `el-publish` (App Store / Play Store automation), and `el-ui-compiler` (El→JS component compiler; currently a stub pending a JS backend in `elc`). Spec: [ui/spec/framework.md](ui/spec/framework.md).
### [arbor/](arbor/) — diagram language
A `.arbor` diagram language and toolchain: `arbor-core` (NodeId/shape/edge-kind types), `arbor-parse` (recursive-descent parser), `arbor-diagram` (IR + Mermaid serializer + architecture-diagram builders), `arbor-layout` (hierarchical layout — rank assignment, positioning, group bounds), `arbor-render` (SVG renderer), `arbor-cli`. (The architecture map above is the kind of diagram this is for.)
---
## Getting started
Install the El SDK from the latest release:
```bash
bash lang/install.sh
# EL_VERSION=v1.0.0 bash lang/install.sh # pin a specific release tag
# EL_PREFIX=/opt/el bash lang/install.sh # custom install prefix
```
Or build the compiler from source and verify the self-hosting chain:
```bash
cd lang
./dist/platform/elc elc-cli.el > elc-new.c
cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \
-o dist/platform/elc-new \
elc-new.c el-compiler/runtime/el_seed.c
# Confirm the new binary reproduces itself exactly
./dist/platform/elc-new elc-cli.el > elc-verify.c
diff elc-new.c elc-verify.c # should be identical
mv dist/platform/elc-new dist/platform/elc
```
Run your first program:
```bash
./lang/dist/platform/elc lang/examples/hello.el > hello.c
cc -std=c11 -I lang/el-compiler/runtime -lcurl -lpthread \
-o hello hello.c lang/el-compiler/runtime/el_seed.c
./hello
```
More examples in [lang/examples/](lang/examples/), including a full starter project at `lang/examples/hello-project/`.
If the compiler binary is ever lost or corrupted, [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) is the authoritative recovery path.
---
## Development workflow
Branching follows `dev → stage → main`: work lands on `dev`, promotes to `stage` for integration testing, and is promoted to `main` for release (visible directly in the git history of this repo). CI is defined per-subproject under `.gitea/workflows/``lang`/`epm`/`ide` share the root pipeline; `engram` and `ql` carry their own (`ci-dev`, `ci-stage`, and a release workflow each).
- Language/runtime specs live at `*/spec/*.md` (`lang/spec/`, `ql/spec/`, `ui/spec/`) and are the single source of truth for implemented-vs-planned status — code and docs are expected to agree with the spec's status markers, not the other way around.
- Agent-facing orientation guides live at `*/AGENTS.md` (currently `lang/AGENTS.md`); more subprojects may grow their own as they need agent-specific conventions documented.
- Tagged releases live under `lang/releases/`, each with its own `RELEASE.md`.
---
## Status
This is an actively developed, internal monorepo — not yet published under an open license. Treat everything here as proprietary to Neuron Technologies unless told otherwise.
+1 -1
View File
@@ -81,7 +81,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+1 -1
View File
@@ -88,7 +88,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+1 -1
View File
@@ -62,7 +62,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+10
View File
@@ -17,6 +17,16 @@
// 4. Append dep to order after all its transitive deps
// 5. Deduplicate: skip already-ordered vessels
// Cross-module forward declarations
// Defined in sibling epm modules; resolved at link time. The `extern fn` decls
// give elc the C prototypes so generated install.c compiles cleanly under strict
// compilers (gcc>=14 / clang) that reject implicit function declarations.
extern fn manifest_name(src: String) -> String // manifest.el
extern fn manifest_deps(src: String) -> String // manifest.el
extern fn registry_token() -> String // registry.el
extern fn registry_find(name: String, version: String) -> String // registry.el
extern fn registry_latest_version(name: String) -> String // registry.el
// Install paths
// packages_dir returns the root directory for installed vessels.
+9
View File
@@ -14,6 +14,15 @@
// EPM_REGISTRY_ORG org name that hosts vessel repos (default: neuron-technologies)
// EPM_TOKEN Gitea personal access token (required for publish)
// Cross-module forward declarations
// These symbols are defined in sibling epm modules or the El runtime and are
// resolved at link time. The `extern fn` decls give elc the C prototype so the
// generated registry.c compiles cleanly under strict compilers (gcc>=14 / clang)
// that reject implicit function declarations. Signature arity must match the
// definition; return/param types are informational (all lower to el_val_t).
extern fn config(key: String) -> String // El runtime builtin
extern fn read_installed() -> String // install.el
// Config helpers
// registry_api_url returns the Gitea API base URL with no trailing slash.
+9
View File
@@ -6,6 +6,15 @@
// Depends on: registry.el (registry_latest_version, registry_find),
// install.el (read_installed, install_vessel, installed_version)
// Cross-module forward declarations
// Defined in sibling epm modules; resolved at link time. The `extern fn` decls
// give elc the C prototypes so generated update.c compiles cleanly under strict
// compilers (gcc>=14 / clang) that reject implicit function declarations.
extern fn read_installed() -> String // install.el
extern fn installed_version(name: String) -> String // install.el
extern fn install_vessel(name: String, version: String) -> Bool // install.el
extern fn registry_latest_version(name: String) -> String // registry.el
// Semver helpers
// semver_part extracts the Nth dot-separated component from a semver string.
@@ -75,6 +75,7 @@ static inline void* el_win_dlsym(void* handle, const char* name) {
#include <direct.h> /* _mkdir */
#define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */
#define timegm _mkgmtime /* UTC tm → time_t */
#define fsync(fd) _commit(fd) /* no fsync() on Windows; _commit() (<io.h>) is the equiv */
/* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */
static inline int setenv(const char* name, const char* value, int overwrite) {
+387 -51
View File
@@ -82,8 +82,14 @@ static _Thread_local ElArena _tl_arena = {NULL, 0, 0};
static _Thread_local int _tl_arena_active = 0;
/* Binary-safe fs_read length — set by fs_read, consumed by http_send_response.
* Allows serving PNGs and other binary files without strlen truncation. */
static _Thread_local size_t _tl_fs_read_len = 0;
* Allows serving PNGs and other binary files without strlen truncation.
* PAIRED with the buffer pointer it describes: the length may only be applied
* to the exact buffer fs_read returned. Without the pairing, any handler that
* fs_read a file and then WRAPPED it into a larger response had that response
* truncated to the file's length (Content-Length lied AND the send stopped
* short) the safety-contact onboarding trap, 2026-07-17. */
static _Thread_local size_t _tl_fs_read_len = 0;
static _Thread_local const char* _tl_fs_read_buf = NULL;
static void el_arena_track(char* p) {
if (!_tl_arena_active || !p) return;
@@ -101,6 +107,8 @@ static void el_arena_track(char* p) {
void el_request_start(void) {
_tl_arena.count = 0;
_tl_arena_active = 1;
_tl_fs_read_len = 0; /* never let a previous request's file length */
_tl_fs_read_buf = NULL; /* leak into this response's byte accounting */
}
/* Called by http_worker after the El handler returns and the response is sent.
@@ -1484,11 +1492,14 @@ static void http_send_response(int fd, const char* body) {
}
const char* eff_body = is_envelope ? env_body : body;
/* Use the real byte count from fs_read if available (handles binary files
* with embedded null bytes PNG, WOFF2, etc.). Fall back to strlen for
* normal text/JSON responses where _tl_fs_read_len is 0. */
size_t blen = (_tl_fs_read_len > 0) ? _tl_fs_read_len : strlen(eff_body);
/* Use the real byte count from fs_read ONLY when this body IS the exact
* buffer fs_read returned (binary files with embedded null bytes PNG,
* WOFF2, etc.). Any other body wrapped, enveloped, or derived must be
* measured with strlen, or it is truncated/over-read to the file's size. */
size_t blen = (_tl_fs_read_len > 0 && eff_body == _tl_fs_read_buf)
? _tl_fs_read_len : strlen(eff_body);
_tl_fs_read_len = 0; /* consume — one-shot per response */
_tl_fs_read_buf = NULL;
int head_only = _tl_http_head_only;
JsonBuf hdrs; jb_init(&hdrs);
@@ -1568,11 +1579,22 @@ static void* http_worker(void* arg) {
const char* rs = EL_CSTR(r);
/* Copy response out BEFORE arena teardown.
* For binary files, _tl_fs_read_len holds the real byte count
* use memcpy instead of strdup so null bytes are preserved. */
size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0);
* use memcpy instead of strdup so null bytes are preserved.
* The stored length applies ONLY when the response IS the exact
* fs_read buffer; a wrapped/derived response must use strlen or
* it gets truncated (or over-read) to the file's length. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
} else {
response = el_strdup_persist("el-runtime: no http handler registered");
}
@@ -1822,10 +1844,20 @@ static void* http_worker_v2(void* arg) {
el_val_t hmap = http_build_headers_map(hdr_block ? hdr_block : "");
el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), hmap, EL_STR(body));
const char* rs = EL_CSTR(r);
size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0);
/* Same pairing rule as the v1 worker: the fs_read length is only
* trustworthy for the exact buffer fs_read returned. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
el_release(hmap);
} else {
response = el_strdup_persist(
@@ -1963,8 +1995,9 @@ void http_serve_async(el_val_t port, el_val_t handler) {
int sock = socket(AF_INET6, SOCK_STREAM, 0);
if (sock < 0) { perror("socket"); return; }
int yes = 1; int no = 0;
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes));
setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, &no, sizeof(no));
/* Win32/mingw setsockopt takes optval as (const char*); the cast is portable on POSIX too. */
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (const char*)&yes, sizeof(yes));
setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, (const char*)&no, sizeof(no));
struct sockaddr_in6 addr;
memset(&addr, 0, sizeof(addr));
addr.sin6_family = AF_INET6;
@@ -2023,6 +2056,7 @@ el_val_t http_response(el_val_t status, el_val_t headers_json, el_val_t body) {
el_val_t fs_read(el_val_t pathv) {
const char* path = EL_CSTR(pathv);
_tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!path) return el_wrap_str(el_strdup(""));
FILE* f = fopen(path, "rb");
if (!f) return el_wrap_str(el_strdup(""));
@@ -2034,6 +2068,7 @@ el_val_t fs_read(el_val_t pathv) {
size_t got = fread(buf, 1, (size_t)sz, f);
buf[got] = '\0';
_tl_fs_read_len = got; /* store real byte count for binary-safe send */
_tl_fs_read_buf = buf; /* ...valid ONLY for this exact buffer */
fclose(f);
return el_wrap_str(buf);
}
@@ -3576,8 +3611,10 @@ el_val_t json_get_raw(el_val_t json_str, el_val_t key) {
const char* k = EL_CSTR(key);
const char* p = json_find_key(json, k);
/* Clear fs_read binary-length hint — result is a fresh null-terminated
* string, not the raw file bytes, so Content-Length must use strlen. */
* string, not the raw file bytes, so Content-Length must use strlen.
* (Kept although the pointer pairing now makes this redundant.) */
_tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!p) return el_wrap_str(el_strdup(""));
const char* end = json_skip_value(p);
size_t n = (size_t)(end - p);
@@ -6895,6 +6932,243 @@ static int engram_rank_cmp(const void* a, const void* b) {
return 0;
}
/* ══════════════════════════════════════════════════════════════════════════
* SEMANTIC SEARCH LAYER nomic-embed-text via Ollama /api/embeddings
*
* Augments the lexical (istr_contains) matcher with dense-vector retrieval.
* Node content and the query are embedded through a local Ollama server;
* nodes are ranked by cosine similarity and UNIONED with lexical hits. This
* lets a paraphrase query surface a node whose words never appear in it.
*
* DEGRADABLE BY DESIGN. The whole layer is gated on HAVE_CURL plus a one-shot
* runtime probe of the embedding endpoint. If curl is not compiled in, or
* Ollama is unreachable, or ENGRAM_SEMANTIC=0, every entry point returns
* "no semantic signal" and callers fall back to pure lexical behaviour
* byte-for-byte the pre-existing search.
*
* CACHE. Node embeddings are computed lazily on first use and cached in
* process memory keyed by node id, with an FNV-1a content hash for
* invalidation (edited content re-embeds). The query is embedded once per
* search call. This is what "avoid re-embedding the whole graph every query"
* buys us: a warm cache serves cosine from RAM. (A cold process still pays
* O(N) embed calls the first time each node is scanned persisting the cache
* to a snapshot sidecar is the documented next step, not done here.)
*
* nomic task prefixes ("search_query:" / "search_document:") are applied
* because nomic-embed-text is trained with them; they materially improve
* retrieval separation (empirically: paraphrase 0.72 vs distractors <0.48).
*
* ENV:
* ENGRAM_SEMANTIC "0" disables; unset/other = auto-probe
* ENGRAM_EMBED_URL default http://localhost:11434/api/embeddings
* ENGRAM_EMBED_MODEL default nomic-embed-text
* ENGRAM_SEMANTIC_MIN cosine threshold for a pure-semantic match (def 0.6)
* */
static double engram_semantic_min(void) {
static double v = -1.0;
if (v >= 0.0) return v;
const char* s = getenv("ENGRAM_SEMANTIC_MIN");
double d = 0.6;
if (s && *s) { char* e = NULL; double t = strtod(s, &e);
if (e != s && t >= 0.0 && t <= 1.0) d = t; }
v = d; return v;
}
#ifdef HAVE_CURL
typedef struct { char* id; uint64_t hash; float* vec; int dim; } EngramEmbEntry;
static EngramEmbEntry* g_emb_items = NULL;
static int64_t g_emb_count = 0, g_emb_cap = 0;
static int g_emb_state = 0; /* 0=unprobed, 1=available, -1=disabled */
static uint64_t engram_fnv1a(const char* s) {
uint64_t h = 1469598103934665603ULL;
if (s) for (const unsigned char* p = (const unsigned char*)s; *p; p++) {
h ^= *p; h *= 1099511628211ULL;
}
return h;
}
/* Parse "embedding":[f,f,...] from an Ollama response. malloc'd vec, or NULL. */
static float* engram_parse_embedding(const char* json, int* out_dim) {
if (!json) return NULL;
const char* p = strstr(json, "\"embedding\"");
if (!p) return NULL;
p = strchr(p, '[');
if (!p) return NULL;
p++;
int cap = 1024, n = 0;
float* v = malloc((size_t)cap * sizeof(float));
if (!v) return NULL;
while (*p && *p != ']') {
while (*p == ' ' || *p == '\t' || *p == '\n' || *p == '\r' || *p == ',') p++;
if (*p == ']' || !*p) break;
char* e = NULL;
double d = strtod(p, &e);
if (e == p) break;
if (n >= cap) { cap *= 2; float* nv = realloc(v, (size_t)cap * sizeof(float));
if (!nv) { free(v); return NULL; } v = nv; }
v[n++] = (float)d;
p = e;
}
if (n == 0) { free(v); return NULL; }
*out_dim = n;
return v;
}
/* JSON-escape src into a malloc'd buffer (no surrounding quotes). */
static char* engram_json_escape(const char* src) {
if (!src) src = "";
size_t n = strlen(src);
char* out = malloc(n * 2 + 1);
if (!out) return NULL;
size_t j = 0;
for (size_t i = 0; i < n; i++) {
unsigned char c = (unsigned char)src[i];
if (c == '"') { out[j++] = '\\'; out[j++] = '"'; }
else if (c == '\\') { out[j++] = '\\'; out[j++] = '\\'; }
else if (c == '\n') { out[j++] = '\\'; out[j++] = 'n'; }
else if (c == '\r') { out[j++] = '\\'; out[j++] = 'r'; }
else if (c == '\t') { out[j++] = '\\'; out[j++] = 't'; }
else if (c < 0x20) { /* drop other control bytes */ }
else { out[j++] = (char)c; }
}
out[j] = '\0';
return out;
}
/* Embed `prefix+text` via Ollama. Returns malloc'd vec (caller frees), or NULL. */
static float* engram_embed_raw(const char* prefix, const char* text, int* out_dim) {
if (!text) return NULL;
const char* url = getenv("ENGRAM_EMBED_URL");
if (!url || !*url) url = "http://localhost:11434/api/embeddings";
const char* model = getenv("ENGRAM_EMBED_MODEL");
if (!model || !*model) model = "nomic-embed-text";
/* Bound content length to keep latency/memory sane on huge nodes. */
char* trunc = NULL;
size_t maxlen = 8192;
if (strlen(text) > maxlen) {
trunc = malloc(maxlen + 1);
if (trunc) { memcpy(trunc, text, maxlen); trunc[maxlen] = '\0'; text = trunc; }
}
char* esc_prefix = engram_json_escape(prefix ? prefix : "");
char* esc = engram_json_escape(text);
free(trunc);
if (!esc || !esc_prefix) { free(esc); free(esc_prefix); return NULL; }
size_t blen = strlen(esc) + strlen(esc_prefix) + strlen(model) + 64;
char* body = malloc(blen);
if (!body) { free(esc); free(esc_prefix); return NULL; }
snprintf(body, blen, "{\"model\":\"%s\",\"prompt\":\"%s%s\"}", model, esc_prefix, esc);
free(esc); free(esc_prefix);
CURL* c = curl_easy_init();
if (!c) { free(body); return NULL; }
HttpBuf rb; httpbuf_init(&rb);
struct curl_slist* h = curl_slist_append(NULL, "Content-Type: application/json");
char errbuf[CURL_ERROR_SIZE]; errbuf[0] = '\0';
curl_easy_setopt(c, CURLOPT_URL, url);
curl_easy_setopt(c, CURLOPT_WRITEFUNCTION, http_write_cb);
curl_easy_setopt(c, CURLOPT_WRITEDATA, &rb);
curl_easy_setopt(c, CURLOPT_POST, 1L);
curl_easy_setopt(c, CURLOPT_POSTFIELDS, body);
curl_easy_setopt(c, CURLOPT_POSTFIELDSIZE, (long)strlen(body));
curl_easy_setopt(c, CURLOPT_HTTPHEADER, h);
curl_easy_setopt(c, CURLOPT_TIMEOUT_MS, el_http_timeout_ms());
curl_easy_setopt(c, CURLOPT_NOSIGNAL, 1L);
curl_easy_setopt(c, CURLOPT_ERRORBUFFER, errbuf);
CURLcode rc = curl_easy_perform(c);
curl_slist_free_all(h);
curl_easy_cleanup(c);
free(body);
if (rc != CURLE_OK) { free(rb.data); return NULL; }
float* v = engram_parse_embedding(rb.data, out_dim);
free(rb.data);
return v;
}
/* One-shot probe: is semantic search available? Caches the verdict. */
static int engram_semantic_enabled(void) {
if (g_emb_state != 0) return g_emb_state == 1;
const char* s = getenv("ENGRAM_SEMANTIC");
if (s && strcmp(s, "0") == 0) { g_emb_state = -1; return 0; }
int dim = 0;
float* v = engram_embed_raw("search_query: ", "probe", &dim);
if (v && dim > 0) { free(v); g_emb_state = 1; return 1; }
free(v);
g_emb_state = -1; return 0;
}
/* Embed the query. Returns malloc'd vec (caller frees), or NULL if semantic off. */
static float* engram_embed_query(const char* q, int* dim) {
if (!engram_semantic_enabled()) return NULL;
if (!q || !*q) return NULL;
return engram_embed_raw("search_query: ", q, dim);
}
/* Cached node embedding. Returns a pointer OWNED BY THE CACHE — do not free. */
static const float* engram_node_vec(EngramNode* n, int* out_dim) {
if (!n || !n->id) return NULL;
uint64_t h = engram_fnv1a(n->content);
for (int64_t i = 0; i < g_emb_count; i++) {
if (g_emb_items[i].id && strcmp(g_emb_items[i].id, n->id) == 0) {
if (g_emb_items[i].hash == h && g_emb_items[i].vec) {
*out_dim = g_emb_items[i].dim; return g_emb_items[i].vec;
}
/* content changed → re-embed in place */
int dim = 0;
float* v = engram_embed_raw("search_document: ", n->content ? n->content : "", &dim);
if (!v) return NULL;
free(g_emb_items[i].vec);
g_emb_items[i].vec = v; g_emb_items[i].dim = dim; g_emb_items[i].hash = h;
*out_dim = dim; return v;
}
}
int dim = 0;
float* v = engram_embed_raw("search_document: ", n->content ? n->content : "", &dim);
if (!v) return NULL;
if (g_emb_count >= g_emb_cap) {
int64_t nc = g_emb_cap ? g_emb_cap * 2 : 256;
EngramEmbEntry* ni = realloc(g_emb_items, (size_t)nc * sizeof(EngramEmbEntry));
if (!ni) { free(v); return NULL; }
g_emb_items = ni; g_emb_cap = nc;
}
g_emb_items[g_emb_count].id = strdup(n->id);
g_emb_items[g_emb_count].hash = h;
g_emb_items[g_emb_count].vec = v;
g_emb_items[g_emb_count].dim = dim;
g_emb_count++;
*out_dim = dim; return v;
}
static double engram_cosine(const float* a, const float* b, int dim) {
double dot = 0, na = 0, nb = 0;
for (int i = 0; i < dim; i++) { dot += (double)a[i] * b[i];
na += (double)a[i] * a[i];
nb += (double)b[i] * b[i]; }
if (na <= 0 || nb <= 0) return 0.0;
return dot / (sqrt(na) * sqrt(nb));
}
/* Cosine of node n against the query vector; 0 if unavailable / dim mismatch. */
static double engram_node_cosine(EngramNode* n, const float* qvec, int qdim) {
if (!qvec || qdim <= 0) return 0.0;
int ndim = 0;
const float* nv = engram_node_vec(n, &ndim);
if (!nv || ndim != qdim) return 0.0;
return engram_cosine(qvec, nv, qdim);
}
#else /* !HAVE_CURL — semantic layer compiled out; callers stay pure-lexical.
* Only the two boundary functions the always-compiled search/activate
* code calls are stubbed; the query embed always yields NULL so every
* cosine is 0 and every caller collapses to lexical-only. */
static float* engram_embed_query(const char* q, int* dim) { (void)q; (void)dim; return NULL; }
static double engram_node_cosine(EngramNode* n, const float* qvec, int qdim) {
(void)n; (void)qvec; (void)qdim; return 0.0;
}
#endif /* HAVE_CURL */
el_val_t engram_search(el_val_t query, el_val_t limit) {
EngramStore* g = engram_get();
const char* q = EL_CSTR(query);
@@ -6905,8 +7179,15 @@ el_val_t engram_search(el_val_t query, el_val_t limit) {
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
if (ntok == 0) return lst;
/* Semantic augmentation: embed the query once; a node is a hit if it covers
* >=1 query token (tokenized-lexical, #66) OR its cosine clears the
* threshold (#67). qvec is NULL (cosine 0) when semantic is unavailable
* pure tokenized-lexical, byte-identical to the lexical-only behaviour. */
int qdim = 0;
float* qvec = engram_embed_query(q, &qdim);
double sem_min = engram_semantic_min();
EngramRankEntry* hits = malloc((size_t)g->node_count * sizeof(EngramRankEntry));
if (!hits) return lst;
if (!hits) { free(qvec); return lst; }
int64_t nhits = 0;
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
@@ -6916,20 +7197,24 @@ el_val_t engram_search(el_val_t query, el_val_t limit) {
* + engram_compile_layered_json that's the legitimate path. */
if (engram_layer_is_transparent(n->layer_id)) continue;
int sc = engram_node_match_score(n, toks, ntok);
if (sc > 0) {
double sem = qvec ? engram_node_cosine(n, qvec, qdim) : 0.0;
if (sc > 0 || sem >= sem_min) {
hits[nhits].idx = i;
hits[nhits].score = sc;
hits[nhits].salience = n->salience;
nhits++;
}
}
/* Rank by distinct tokens matched (desc) then salience (desc), then cap. */
/* Rank by distinct tokens matched (desc) then salience (desc), then cap.
* Pure-semantic hits (token score 0) sort after every lexical hit a
* lexical semantic union with lexical precedence. */
qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp);
int64_t end = nhits < lim ? nhits : lim;
for (int64_t k = 0; k < end; k++) {
lst = el_list_append(lst, engram_node_to_map(&g->nodes[hits[k].idx]));
}
free(hits);
free(qvec);
return lst;
}
@@ -7279,21 +7564,31 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
if (!seeds) {
free(best_bg); free(best_hops); free(reached); return out;
}
/* Tokenize once: a node seeds if it matches ANY query token, and its seed
* activation is scaled by token coverage (fraction of distinct query
* tokens it contains) so a node matching all words seeds more strongly
* than one matching a single word. Single-word queries coverage 1.0,
* identical to the prior whole-query behavior. */
/* Tokenized + semantic seeding: a node seeds if it covers >=1 query token
* (tokenized-lexical, #66) OR its cosine clears the threshold (#67). A
* lexical seed's activation is scaled by token coverage (fraction of
* distinct query tokens covered) so a node matching all words seeds more
* strongly than one matching a single word; single-word queries coverage
* 1.0. A pure-semantic seed (no token match) is instead down-weighted by
* its cosine so paraphrase matches spread without overpowering exact seeds.
* q_vec is NULL (cosine 0) when semantic is unavailable the seed set is
* exactly the tokenized-lexical one. q_vec is freed right after this loop
* so the many downstream early-returns need no cleanup change. */
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
int q_dim = 0;
float* q_vec = engram_embed_query(q, &q_dim);
double q_sem_min = engram_semantic_min();
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
int sc = engram_node_match_score(n, toks, ntok);
if (sc > 0) {
double sem = q_vec ? engram_node_cosine(n, q_vec, q_dim) : 0.0;
if (sc > 0 || sem >= q_sem_min) {
double tdecay = engram_temporal_decay(n, now_ms);
double dampen = engram_activation_dampen(n);
double cover = ntok > 0 ? (double)sc / (double)ntok : 1.0;
double act = n->salience * tdecay * dampen * cover;
double act = n->salience * tdecay * dampen;
if (sc > 0) act *= (ntok > 0 ? (double)sc / (double)ntok : 1.0);
else act *= sem; /* pure-semantic seed: down-weight by cosine */
seeds[seed_count].idx = i;
seeds[seed_count].act = act;
seeds[seed_count].created_at = n->created_at;
@@ -7303,6 +7598,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
reached[i] = 1;
}
}
free(q_vec);
/* Compute mean seed created_at for temporal proximity bonus. */
int64_t seed_epoch = 0;
if (seed_count > 0) {
@@ -7854,6 +8150,35 @@ el_val_t engram_get_node_json(el_val_t id) {
return el_wrap_str(jb_finish(&b));
}
/* engram_get_node_by_label — find the first node whose label field exactly
* matches the given string. Returns the node as a JSON object string, or "{}"
* if no match is found.
*
* Used by chat.el to retrieve well-known nodes (e.g. "conv:history",
* "session:summary") by their stable label rather than by ID, which is immune
* to vector index drift across restarts.
*
* Exact match (strcmp, not istr_contains) because labels like "conv:history"
* must not collide with nodes whose content happens to contain that substring.
*
* Backported verbatim (idiom-adapted to jb_finish) from release runtime
* v1.0.0-20260501 to unblock the soul regen link: chat.el references this
* native but the current runtime lacked its definition. */
el_val_t engram_get_node_by_label(el_val_t label) {
const char* lbl = EL_CSTR(label);
if (!lbl || !*lbl) return el_wrap_str(el_strdup("{}"));
EngramStore* g = engram_get();
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
if (n->label && strcmp(n->label, lbl) == 0) {
JsonBuf b; jb_init(&b);
engram_emit_node_json(&b, n);
return el_wrap_str(jb_finish(&b));
}
}
return el_wrap_str(el_strdup("{}"));
}
el_val_t engram_search_json(el_val_t query, el_val_t limit) {
EngramStore* g = engram_get();
const char* q = EL_CSTR(query);
@@ -7861,39 +8186,50 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) {
if (lim <= 0) lim = 100;
JsonBuf b; jb_init(&b);
jb_putc(&b, '[');
int first = 1;
if (q && *q) {
if (q && *q && g->node_count > 0) {
/* Collect candidates from the UNION of tokenized-lexical and semantic
* matches, score each, rank by score, emit the top `lim`. A node is a
* candidate if it covers >=1 query token (tokenized-lexical, #66) OR its
* query cosine clears the threshold (#67). Lexical score is the distinct
* token count (>=1), so any lexical hit outranks a pure-semantic hit
* (cosine < 1); pure-semantic hits are scored by cosine alone. When
* semantic is unavailable qvec is NULL, sem is 0, only tokenized-lexical
* hits are collected, and the stable insertion sort preserves order. */
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
if (ntok > 0) {
EngramRankEntry* hits =
malloc((size_t)g->node_count * sizeof(EngramRankEntry));
if (hits) {
int64_t nhits = 0;
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
/* Filter transparent layers — same as engram_search. */
if (engram_layer_is_transparent(n->layer_id)) continue;
int sc = engram_node_match_score(n, toks, ntok);
if (sc > 0) {
hits[nhits].idx = i;
hits[nhits].score = sc;
hits[nhits].salience = n->salience;
nhits++;
}
int qdim = 0;
float* qvec = engram_embed_query(q, &qdim);
double sem_min = engram_semantic_min();
typedef struct { int64_t idx; double score; } Cand;
Cand* cand = malloc((size_t)g->node_count * sizeof(Cand));
if (cand) {
int64_t nc = 0;
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
if (engram_layer_is_transparent(n->layer_id)) continue;
int sc = engram_node_match_score(n, toks, ntok);
double sem = qvec ? engram_node_cosine(n, qvec, qdim) : 0.0;
if (sc > 0 || sem >= sem_min) {
cand[nc].idx = i;
cand[nc].score = (double)sc + sem;
nc++;
}
/* Rank by distinct tokens matched (desc) then salience (desc). */
qsort(hits, (size_t)nhits, sizeof(EngramRankEntry),
engram_rank_cmp);
int64_t end = nhits < lim ? nhits : lim;
for (int64_t k = 0; k < end; k++) {
if (!first) jb_putc(&b, ',');
engram_emit_node_json(&b, &g->nodes[hits[k].idx]);
first = 0;
}
free(hits);
}
/* Insertion sort by score desc; stable for equal scores. */
for (int64_t i = 1; i < nc; i++) {
Cand k = cand[i]; int64_t j = i - 1;
while (j >= 0 && cand[j].score < k.score) { cand[j + 1] = cand[j]; j--; }
cand[j + 1] = k;
}
int first = 1;
for (int64_t i = 0; i < nc && i < lim; i++) {
if (!first) jb_putc(&b, ',');
engram_emit_node_json(&b, &g->nodes[cand[i].idx]);
first = 0;
}
free(cand);
}
free(qvec);
}
jb_putc(&b, ']');
return el_wrap_str(jb_finish(&b));
+1
View File
@@ -632,6 +632,7 @@ el_val_t engram_load(el_val_t path);
* can pass results straight through without round-tripping ElList/ElMap
* through json_stringify. */
el_val_t engram_get_node_json(el_val_t id);
el_val_t engram_get_node_by_label(el_val_t label);
el_val_t engram_search_json(el_val_t query, el_val_t limit);
el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset);
el_val_t engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset);
+1
View File
@@ -1072,6 +1072,7 @@ el_val_t __engram_save(el_val_t path) { return engram_save
el_val_t __engram_load(el_val_t path) { return engram_load(path); }
el_val_t __engram_get_node_json(el_val_t id) { return engram_get_node_json(id); }
el_val_t __engram_get_node_by_label(el_val_t label) { return engram_get_node_by_label(label); }
el_val_t __engram_search_json(el_val_t query, el_val_t limit) {
return engram_search_json(query, limit);
+1
View File
@@ -226,6 +226,7 @@ el_val_t __engram_activate(el_val_t query, el_val_t depth);
el_val_t __engram_save(el_val_t path);
el_val_t __engram_load(el_val_t path);
el_val_t __engram_get_node_json(el_val_t id);
el_val_t __engram_get_node_by_label(el_val_t label);
el_val_t __engram_search_json(el_val_t query, el_val_t limit);
el_val_t __engram_scan_nodes_json(el_val_t limit, el_val_t offset);
el_val_t __engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset);
+1
View File
@@ -2670,6 +2670,7 @@ fn builtin_arity(name: String) -> Int {
if str_eq(name, "engram_save") { return 1 }
if str_eq(name, "engram_load") { return 1 }
if str_eq(name, "engram_get_node_json") { return 1 }
if str_eq(name, "engram_get_node_by_label") { return 1 }
if str_eq(name, "engram_search_json") { return 2 }
if str_eq(name, "engram_scan_nodes_json") { return 2 }
if str_eq(name, "engram_neighbors_json") { return 3 }
@@ -0,0 +1,186 @@
#ifndef EL_PLATFORM_WIN_H
#define EL_PLATFORM_WIN_H
/*
* el_platform_win.h Windows OS-boundary shim for el_runtime.c.
*
* Branch: feat/windows-el-runtime. Included ONLY when _WIN32 is defined; the POSIX build is
* untouched. Goal: let el_runtime.c (a BSD-sockets / dlfcn / fork host) compile and link with
* mingw-w64 into a native neuron.exe, with no behavioural change to the Linux/macOS build.
*
* What it maps:
* - sockets : winsock2 (same call names: socket/bind/listen/accept/recv/send/setsockopt).
* Sockets close with closesocket() (see el_closesocket), and the stack must be
* started once with WSAStartup done automatically via a load-time constructor.
* - dlsym : el_runtime.c uses dlsym(RTLD_DEFAULT, name) to resolve callback/tool symbols
* exported by the main module. Windows equivalent: GetProcAddress on the process
* module. Link the soul with -Wl,--export-all-symbols so the symbols are findable.
* - popen : mapped to _popen/_pclose.
* - threads : UNCHANGED. mingw-w64 ships winpthreads, so <pthread.h> + -lpthread just work.
*/
#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN
#endif
#include <winsock2.h>
#include <ws2tcpip.h>
#include <windows.h>
#include <io.h>
#include <process.h>
/* Portable headers mingw-w64 provides (verified present). */
#include <stdarg.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h> /* strcasecmp */
#include <ctype.h>
#include <math.h>
#include <time.h>
#include <sys/time.h> /* mingw-w64 provides gettimeofday here */
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <dirent.h>
#include <errno.h>
#include <pthread.h>
/* ── socket close ─────────────────────────────────────────────────────────── */
/* Winsock closes sockets with closesocket(), not close() (close() is for file fds). The POSIX
build defines the same helper as close() so the call sites are identical across platforms. */
static inline int el_closesocket(SOCKET s) { return closesocket(s); }
/* ── setsockopt optval type ───────────────────────────────────────────────── */
/* Winsock's setsockopt takes optval as (const char*); POSIX takes (const void*), so el_runtime.c
passes &int directly. GCC 14+ makes that an error under -Wincompatible-pointer-types. Wrap it so
the runtime's POSIX-style call sites compile unchanged (defined before the macro so the wrapper
itself resolves to the real winsock setsockopt). */
static inline int el_setsockopt(SOCKET s, int level, int optname, const void* optval, int optlen) {
return setsockopt(s, level, optname, (const char*)optval, optlen);
}
#define setsockopt(s, l, o, v, n) el_setsockopt((s), (l), (o), (v), (int)(n))
/* ── winsock init (once, at load) ─────────────────────────────────────────── */
static void el__win_net_init(void) {
static int inited = 0;
if (!inited) { WSADATA w; WSAStartup(MAKEWORD(2, 2), &w); inited = 1; }
}
__attribute__((constructor)) static void el__win_ctor(void) { el__win_net_init(); }
/* ── dlsym → GetProcAddress ───────────────────────────────────────────────── */
#ifndef RTLD_DEFAULT
#define RTLD_DEFAULT ((void*)0)
#endif
static inline void* el_win_dlsym(void* handle, const char* name) {
(void)handle;
return (void*)(uintptr_t)GetProcAddress(GetModuleHandleA(NULL), name);
}
#define dlsym(h, n) el_win_dlsym((h), (n))
/* ── popen / pclose ───────────────────────────────────────────────────────── */
#define popen _popen
#define pclose _pclose
/* ── misc POSIX → Win32 shims ─────────────────────────────────────────────── */
#include <direct.h> /* _mkdir */
#define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */
#define timegm _mkgmtime /* UTC tm → time_t */
/* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */
static inline int setenv(const char* name, const char* value, int overwrite) {
(void)overwrite;
return _putenv_s(name, value ? value : "");
}
static inline int unsetenv(const char* name) {
/* _putenv_s(name, "") sets VAR="" rather than removing it.
* SetEnvironmentVariableA(name, NULL) truly deletes it from the Win32
* env block; then we sync the CRT cache with _putenv("NAME="). */
SetEnvironmentVariableA(name, NULL);
size_t len = strlen(name);
char *buf = (char*)malloc(len + 2);
if (!buf) return -1;
memcpy(buf, name, len);
buf[len] = '=';
buf[len + 1] = '\0';
_putenv(buf);
free(buf);
return 0;
}
/* nanosleep — not available in MSVC/UCRT; approximate with Sleep(). */
static inline int el_nanosleep(const struct timespec *req, struct timespec *rem) {
(void)rem;
DWORD ms = (DWORD)((req->tv_sec * 1000ULL) + (req->tv_nsec / 1000000ULL));
Sleep(ms ? ms : 1);
return 0;
}
#define nanosleep(req, rem) el_nanosleep((req), (rem))
/* localtime_r/gmtime_r: Windows offers localtime_s/gmtime_s with reversed arg order. */
static inline struct tm* localtime_r(const time_t* t, struct tm* out) {
return localtime_s(out, t) == 0 ? out : (struct tm*)0;
}
static inline struct tm* gmtime_r(const time_t* t, struct tm* out) {
return gmtime_s(out, t) == 0 ? out : (struct tm*)0;
}
/* ── libcurl: degradable stubs for the curl-less Windows build ─────────────── */
/* The curl-less validation build (WITH_CURL=0) links no libcurl. el_runtime.c uses libcurl
* unconditionally for its HTTP client / LLM layer; these stubs let it compile and link so the
* runtime, HTTP *server*, graph and memory work natively on Windows. Live outbound HTTP/LLM calls
* degrade to a runtime error (curl_easy_perform returns an error) matching the documented
* curl-less contract. When HAVE_CURL is defined (WITH_CURL=1) the real <curl/curl.h> is used and
* this whole block is compiled out. POSIX never sees this header, so the POSIX build is untouched. */
#ifndef HAVE_CURL
typedef void CURL;
typedef int CURLcode;
#define CURLE_OK 0
#define CURLE_HTTP_RETURNED_ERROR 22
#define CURL_ERROR_SIZE 256
/* Option ids: values are irrelevant to the no-op setopt below; kept distinct for readability. */
#define CURLOPT_URL 10002
#define CURLOPT_WRITEFUNCTION 20011
#define CURLOPT_WRITEDATA 10001
#define CURLOPT_POSTFIELDS 10015
#define CURLOPT_POSTFIELDSIZE 120
#define CURLOPT_POST 47
#define CURLOPT_HTTPHEADER 10023
#define CURLOPT_TIMEOUT_MS 155
#define CURLOPT_NOSIGNAL 99
#define CURLOPT_USERAGENT 10018
#define CURLOPT_FOLLOWLOCATION 52
#define CURLOPT_ERRORBUFFER 10010
#define CURLOPT_CUSTOMREQUEST 10036
#define CURLOPT_FAILONERROR 45
struct curl_slist { char* data; struct curl_slist* next; };
static inline struct curl_slist* curl_slist_append(struct curl_slist* list, const char* s) {
struct curl_slist* node = (struct curl_slist*)malloc(sizeof(struct curl_slist));
if (!node) return list;
node->data = s ? strdup(s) : NULL;
node->next = NULL;
if (!list) return node;
struct curl_slist* p = list;
while (p->next) p = p->next;
p->next = node;
return list;
}
static inline void curl_slist_free_all(struct curl_slist* list) {
while (list) { struct curl_slist* n = list->next; free(list->data); free(list); list = n; }
}
static inline CURL* curl_easy_init(void) { return (CURL*)malloc(1); }
static inline CURLcode curl_easy_setopt(CURL* h, int opt, ...) { (void)h; (void)opt; return CURLE_OK; }
static inline CURLcode curl_easy_perform(CURL* h) { (void)h; return 7 /* CURLE_COULDNT_CONNECT */; }
static inline void curl_easy_cleanup(CURL* h) { free(h); }
static inline const char* curl_easy_strerror(CURLcode c) {
(void)c; return "libcurl not built in (curl-less build)";
}
#endif /* !HAVE_CURL */
#endif /* EL_PLATFORM_WIN_H */
+114 -22
View File
@@ -21,6 +21,10 @@
#include "el_runtime.h"
#ifdef _WIN32
/* Windows OS-boundary shim (winsock/dlsym/popen). Threading stays on <pthread.h> (winpthreads). */
#include "el_platform_win.h"
#else
#include <stdarg.h>
#include <strings.h> /* strcasecmp */
#include <stdint.h>
@@ -42,7 +46,16 @@
#include <dirent.h>
#include <errno.h>
#include <pthread.h>
/* On POSIX, sockets close with the same close() as files; el_platform_win.h supplies the Windows
variant. Defined here so the socket call sites are identical across platforms. */
static inline int el_closesocket(int s) { return close(s); }
#endif
/* libcurl: present on POSIX and on the WITH_CURL Windows build; absent on the curl-less Windows
validation build, where el_platform_win.h supplies degradable stubs. On POSIX (_WIN32 undefined)
this is always taken, so the POSIX build is unchanged. */
#if !defined(_WIN32) || defined(HAVE_CURL)
#include <curl/curl.h>
#endif
/* ── Internal allocators ─────────────────────────────────────────────────── */
@@ -71,8 +84,14 @@ static _Thread_local ElArena _tl_arena = {NULL, 0, 0};
static _Thread_local int _tl_arena_active = 0;
/* Binary-safe fs_read length — set by fs_read, consumed by http_send_response.
* Allows serving PNGs and other binary files without strlen truncation. */
static _Thread_local size_t _tl_fs_read_len = 0;
* Allows serving PNGs and other binary files without strlen truncation.
* PAIRED with the buffer pointer it describes: the length may only be applied
* to the exact buffer fs_read returned. Without the pairing, any handler that
* fs_read a file and then WRAPPED it into a larger response had that response
* truncated to the file's length (Content-Length lied AND the send stopped
* short) the safety-contact onboarding trap, 2026-07-17. */
static _Thread_local size_t _tl_fs_read_len = 0;
static _Thread_local const char* _tl_fs_read_buf = NULL;
static void el_arena_track(char* p) {
if (!_tl_arena_active || !p) return;
@@ -90,6 +109,8 @@ static void el_arena_track(char* p) {
void el_request_start(void) {
_tl_arena.count = 0;
_tl_arena_active = 1;
_tl_fs_read_len = 0; /* never let a previous request's file length */
_tl_fs_read_buf = NULL; /* leak into this response's byte accounting */
}
/* Called by http_worker after the El handler returns and the response is sent.
@@ -1403,11 +1424,14 @@ static void http_send_response(int fd, const char* body) {
}
const char* eff_body = is_envelope ? env_body : body;
/* Use the real byte count from fs_read if available (handles binary files
* with embedded null bytes PNG, WOFF2, etc.). Fall back to strlen for
* normal text/JSON responses where _tl_fs_read_len is 0. */
size_t blen = (_tl_fs_read_len > 0) ? _tl_fs_read_len : strlen(eff_body);
/* Use the real byte count from fs_read ONLY when this body IS the exact
* buffer fs_read returned (binary files with embedded null bytes PNG,
* WOFF2, etc.). Any other body wrapped, enveloped, or derived must be
* measured with strlen, or it is truncated/over-read to the file's size. */
size_t blen = (_tl_fs_read_len > 0 && eff_body == _tl_fs_read_buf)
? _tl_fs_read_len : strlen(eff_body);
_tl_fs_read_len = 0; /* consume — one-shot per response */
_tl_fs_read_buf = NULL;
int head_only = _tl_http_head_only;
JsonBuf hdrs; jb_init(&hdrs);
@@ -1457,12 +1481,20 @@ static void http_send_response(int fd, const char* body) {
}
typedef struct {
#ifdef _WIN32
SOCKET fd;
#else
int fd;
#endif
} HttpWorkerArg;
static void* http_worker(void* arg) {
HttpWorkerArg* a = (HttpWorkerArg*)arg;
#ifdef _WIN32
SOCKET fd = a->fd;
#else
int fd = a->fd;
#endif
free(a);
char *method = NULL, *path = NULL, *body = NULL;
if (http_read_request(fd, &method, &path, &body, NULL) == 0) {
@@ -1479,11 +1511,22 @@ static void* http_worker(void* arg) {
const char* rs = EL_CSTR(r);
/* Copy response out BEFORE arena teardown.
* For binary files, _tl_fs_read_len holds the real byte count
* use memcpy instead of strdup so null bytes are preserved. */
size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0);
* use memcpy instead of strdup so null bytes are preserved.
* The stored length applies ONLY when the response IS the exact
* fs_read buffer; a wrapped/derived response must use strlen or
* it gets truncated (or over-read) to the file's length. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
} else {
response = el_strdup_persist("el-runtime: no http handler registered");
}
@@ -1494,7 +1537,7 @@ static void* http_worker(void* arg) {
free(response);
}
free(method); free(path); free(body);
close(fd);
el_closesocket(fd);
/* release a slot */
pthread_mutex_lock(&_http_conn_mu);
_http_conn_active--;
@@ -1524,14 +1567,18 @@ void http_serve(el_val_t port, el_val_t handler) {
addr.sin6_addr = in6addr_any;
addr.sin6_port = htons((uint16_t)p);
if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
perror("bind"); close(sock); return;
perror("bind"); el_closesocket(sock); return;
}
if (listen(sock, 64) < 0) { perror("listen"); close(sock); return; }
if (listen(sock, 64) < 0) { perror("listen"); el_closesocket(sock); return; }
fprintf(stderr, "[http] listening on [::]:%d (dual-stack)\n", p);
while (1) {
struct sockaddr_in6 cli;
socklen_t clen = sizeof(cli);
#ifdef _WIN32
SOCKET cfd = accept(sock, (struct sockaddr*)&cli, &clen);
#else
int cfd = accept(sock, (struct sockaddr*)&cli, &clen);
#endif
if (cfd < 0) {
if (errno == EINTR) continue;
perror("accept"); break;
@@ -1543,11 +1590,11 @@ void http_serve(el_val_t port, el_val_t handler) {
_http_conn_active++;
pthread_mutex_unlock(&_http_conn_mu);
HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg));
if (!arg) { close(cfd); continue; }
if (!arg) { el_closesocket(cfd); continue; }
arg->fd = cfd;
pthread_t tid;
if (pthread_create(&tid, NULL, http_worker, arg) != 0) {
close(cfd); free(arg);
el_closesocket(cfd); free(arg);
pthread_mutex_lock(&_http_conn_mu);
_http_conn_active--;
pthread_cond_signal(&_http_conn_cv);
@@ -1556,7 +1603,7 @@ void http_serve(el_val_t port, el_val_t handler) {
}
pthread_detach(tid);
}
close(sock);
el_closesocket(sock);
}
/* ── http_serve_async — non-blocking HTTP server ─────────────────────────── */
@@ -1792,7 +1839,11 @@ static el_val_t http_build_headers_map(const char* hdr_block) {
static void* http_worker_v2(void* arg) {
HttpWorkerArg* a = (HttpWorkerArg*)arg;
#ifdef _WIN32
SOCKET fd = a->fd;
#else
int fd = a->fd;
#endif
free(a);
char *method = NULL, *path = NULL, *body = NULL, *hdr_block = NULL;
if (http_read_request(fd, &method, &path, &body, &hdr_block) == 0) {
@@ -1805,10 +1856,20 @@ static void* http_worker_v2(void* arg) {
el_val_t hmap = http_build_headers_map(hdr_block ? hdr_block : "");
el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), hmap, EL_STR(body));
const char* rs = EL_CSTR(r);
size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0);
/* Same pairing rule as the v1 worker: the fs_read length is only
* trustworthy for the exact buffer fs_read returned. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
el_release(hmap);
} else {
response = el_strdup_persist(
@@ -1822,7 +1883,7 @@ static void* http_worker_v2(void* arg) {
free(response);
}
free(method); free(path); free(body); free(hdr_block);
close(fd);
el_closesocket(fd);
pthread_mutex_lock(&_http_conn_mu);
_http_conn_active--;
pthread_cond_signal(&_http_conn_cv);
@@ -1852,14 +1913,18 @@ void http_serve_v2(el_val_t port, el_val_t handler) {
addr.sin6_addr = in6addr_any;
addr.sin6_port = htons((uint16_t)p);
if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
perror("bind"); close(sock); return;
perror("bind"); el_closesocket(sock); return;
}
if (listen(sock, 64) < 0) { perror("listen"); close(sock); return; }
if (listen(sock, 64) < 0) { perror("listen"); el_closesocket(sock); return; }
fprintf(stderr, "[http v2] listening on [::]:%d (dual-stack)\n", p);
while (1) {
struct sockaddr_in6 cli;
socklen_t clen = sizeof(cli);
#ifdef _WIN32
SOCKET cfd = accept(sock, (struct sockaddr*)&cli, &clen);
#else
int cfd = accept(sock, (struct sockaddr*)&cli, &clen);
#endif
if (cfd < 0) {
if (errno == EINTR) continue;
perror("accept"); break;
@@ -1871,11 +1936,11 @@ void http_serve_v2(el_val_t port, el_val_t handler) {
_http_conn_active++;
pthread_mutex_unlock(&_http_conn_mu);
HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg));
if (!arg) { close(cfd); continue; }
if (!arg) { el_closesocket(cfd); continue; }
arg->fd = cfd;
pthread_t tid;
if (pthread_create(&tid, NULL, http_worker_v2, arg) != 0) {
close(cfd); free(arg);
el_closesocket(cfd); free(arg);
pthread_mutex_lock(&_http_conn_mu);
_http_conn_active--;
pthread_cond_signal(&_http_conn_cv);
@@ -1884,7 +1949,7 @@ void http_serve_v2(el_val_t port, el_val_t handler) {
}
pthread_detach(tid);
}
close(sock);
el_closesocket(sock);
}
/* Build the response envelope a 4-arg handler can return. We hand-write
@@ -1924,6 +1989,7 @@ el_val_t http_response(el_val_t status, el_val_t headers_json, el_val_t body) {
el_val_t fs_read(el_val_t pathv) {
const char* path = EL_CSTR(pathv);
_tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!path) return el_wrap_str(el_strdup(""));
FILE* f = fopen(path, "rb");
if (!f) return el_wrap_str(el_strdup(""));
@@ -1935,6 +2001,7 @@ el_val_t fs_read(el_val_t pathv) {
size_t got = fread(buf, 1, (size_t)sz, f);
buf[got] = '\0';
_tl_fs_read_len = got; /* store real byte count for binary-safe send */
_tl_fs_read_buf = buf; /* ...valid ONLY for this exact buffer */
fclose(f);
return el_wrap_str(buf);
}
@@ -2029,6 +2096,23 @@ el_val_t exec(el_val_t cmdv) {
el_val_t exec_bg(el_val_t cmdv) {
const char* cmd = EL_CSTR(cmdv);
if (!cmd || !*cmd) return el_wrap_str(el_strdup(""));
#ifdef _WIN32
/* Windows: no fork/exec. Launch a detached `cmd /c <command>` with no console window via
CreateProcess (DETACHED_PROCESS | CREATE_NO_WINDOW). Returns the PID as a string, "" on fail.
Mirrors the POSIX branch: child runs independently, caller is not blocked. */
char cmdline[8192];
snprintf(cmdline, sizeof(cmdline), "cmd.exe /c %s", cmd);
STARTUPINFOA si; ZeroMemory(&si, sizeof(si)); si.cb = sizeof(si);
PROCESS_INFORMATION pi; ZeroMemory(&pi, sizeof(pi));
BOOL ok = CreateProcessA(NULL, cmdline, NULL, NULL, FALSE,
DETACHED_PROCESS | CREATE_NO_WINDOW, NULL, NULL, &si, &pi);
if (!ok) return el_wrap_str(el_strdup(""));
char pidbuf[32];
snprintf(pidbuf, sizeof(pidbuf), "%lu", (unsigned long)pi.dwProcessId);
CloseHandle(pi.hProcess);
CloseHandle(pi.hThread);
return el_wrap_str(el_strdup(pidbuf));
#else
pid_t pid = fork();
if (pid < 0) {
/* fork failed */
@@ -2051,6 +2135,7 @@ el_val_t exec_bg(el_val_t cmdv) {
char pidbuf[32];
snprintf(pidbuf, sizeof(pidbuf), "%d", (int)pid);
return el_wrap_str(el_strdup(pidbuf));
#endif
}
el_val_t fs_list(el_val_t pathv) {
@@ -3305,8 +3390,10 @@ el_val_t json_get_raw(el_val_t json_str, el_val_t key) {
const char* k = EL_CSTR(key);
const char* p = json_find_key(json, k);
/* Clear fs_read binary-length hint — result is a fresh null-terminated
* string, not the raw file bytes, so Content-Length must use strlen. */
* string, not the raw file bytes, so Content-Length must use strlen.
* (Kept although the pointer pairing now makes this redundant.) */
_tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!p) return el_wrap_str(el_strdup(""));
const char* end = json_skip_value(p);
size_t n = (size_t)(end - p);
@@ -4178,7 +4265,12 @@ static int _el_decompose_earth(el_caltime_t* ct, struct tm* tm_out, int* abbr_le
localtime_r(&s, &tm);
*tm_out = tm;
if (abbr_buf && abbr_cap > 0) {
/* mingw's struct tm has no tm_zone (BSD/glibc extension); no abbrev available there. */
#ifdef _WIN32
const char* z_str = "";
#else
const char* z_str = tm.tm_zone ? tm.tm_zone : "";
#endif
size_t n = strlen(z_str);
if (n >= abbr_cap) n = abbr_cap - 1;
memcpy(abbr_buf, z_str, n);
@@ -758,6 +758,18 @@ el_val_t trace_span_start(el_val_t name);
el_val_t trace_span_end(el_val_t span_handle);
el_val_t emit_event(el_val_t name, el_val_t duration_ms);
/* ── Runtime symbols required by the soul modules ──────────────────────────── */
/* All implemented in el_runtime.c but omitted from this release header; the soul dist modules
* reference them directly, so the public header must export them. Declarations only mirrors the
* mainline el_runtime.h and is platform-independent (no behavioural change to the POSIX build). */
typedef el_val_t (*http_handler_fn)(el_val_t method, el_val_t path, el_val_t body);
typedef el_val_t (*http_handler4_fn)(el_val_t method, el_val_t path, el_val_t body, el_val_t headers);
el_val_t el_arena_push(void);
el_val_t el_arena_pop(el_val_t mark);
void http_serve_async(el_val_t port, el_val_t handler);
el_val_t engram_get_node_by_label(el_val_t label);
el_val_t engram_prune_telemetry(el_val_t older_than_ms);
#ifdef __cplusplus
}
#endif