Compare commits

..

28 Commits

Author SHA1 Message Date
bigmerge 2f832c8def Fix float arithmetic codegen and math_log aliasing
El SDK CI - dev / build-and-test (pull_request) Failing after 10m8s
Float + previously fell through to string concat (segfault); -, *, /, %
operated on raw IEEE-754 bit patterns as integers (garbage results). Floats
are now tracked via a __float_names typed-binding set (parallel to the
existing int-tracking scheme) and arithmetic is emitted as real C double
ops.

Also fixes math_log, which was wrongly aliased to natural log (duplicating
math_ln) — now uses log10 — and adds the missing <math.h> include. Rebuilt
elc binary included.
2026-08-15 14:24:14 -05:00
will.anderson d71fc4c1c0 Merge pull request 'promote stage -> main: reconciled el runtime (engram search + natives + durable truncation fix + Windows port)' (#82) from stage into main
El SDK Release / build-and-release (push) Successful in 8m31s
El SDK CI - dev / build-and-test (pull_request) Successful in 8m41s
2026-07-22 21:44:01 +00:00
will.anderson a118d19393 Merge pull request 'promote dev -> stage: el cluster (#66 engram + #79 truncation fix + release-runtime Windows port)' (#81) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 7m58s
El SDK Release / build-and-release (pull_request) Successful in 4m16s
2026-07-22 21:20:17 +00:00
will.anderson c6aa1e5c53 Merge pull request 'Land el cluster: #66 engram search + natives, #79 truncation fix, + release-runtime Windows port (reconciled)' (#80) from reconcile/el-cluster-windows-runtime into dev
El SDK CI - dev / build-and-test (push) Successful in 8m3s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m25s
Land el cluster (#66 + #79 + release-runtime Windows-port reconciliation) into dev
2026-07-22 21:06:36 +00:00
will.anderson ff577391f2 reconcile(release-runtime): Windows-port + complete v1.0.0 release runtime so the desktop soul cross-compiles
El SDK CI - dev / build-and-test (pull_request) Successful in 7m7s
The desktop soul (neuron/dist) compiles against the v1.0.0-20260501 release
runtime. After #66 landed the engram natives (tokenized/ranked search,
engram_prune_telemetry) and #79 the durable truncation fix into this runtime,
two gaps remained before it could cross-compile the Windows brain:

1. Windows OS boundary: the release runtime had no Win32 path. Ported the same
   _WIN32-guarded shim the mainline runtime carries (#69): #ifdef _WIN32 ->
   el_platform_win.h (winsock/dlsym/popen + WSAStartup ctor), SOCKET fd guards
   and el_closesocket() at every socket site, CreateProcessA for exec_bg, the
   tm_zone/mingw guard, an el_setsockopt optval wrapper (GCC14), and curl-less
   libcurl stubs. Every change is _WIN32/HAVE_CURL-gated — the POSIX build is
   byte-identical (gcc -fsyntax-only clean; native behaviour unchanged).

2. Header exports: the release el_runtime.h omitted symbols the soul dist calls
   that are defined in this runtime's .c — the http_handler_fn/http_handler4_fn
   typedefs and el_arena_push/pop, engram_prune_telemetry, engram_get_node_by_label.
   Declaration-only, POSIX-neutral; fixes implicit-declaration/unknown-type
   errors under the C11 mingw build.

Result: x86_64-w64-mingw32-gcc compiles el_runtime.c + all 48 soul modules
clean; POSIX gcc -fsyntax-only clean. This is the Windows-port PR the runtime
needed on main (the release-runtime counterpart to #69), landed via stage.
2026-07-22 15:56:08 -05:00
will.anderson ee0d5f9b97 Merge #79: durable HTTP response-truncation fix, both runtimes (via stage) 2026-07-22 15:46:02 -05:00
will.anderson 391bd818ea Merge #66: tokenized+ranked engram lexical search + engram natives (via stage) 2026-07-22 15:45:54 -05:00
will.anderson 43636aed99 runtime: pair fs_read length hint with its buffer in BOTH runtimes — kill response truncation for good
El SDK Release / build-and-release (pull_request) Failing after 7s
The binary-safe fs_read length (_tl_fs_read_len) was consumed by the HTTP
response path for ANY body, even when a handler wrapped a smaller file into a
larger reply. Content-Length then lied AND the send stopped short: the
safety-contact (988) routes returned 178 of 208/218 bytes, cut mid-'set_at' —
unparseable JSON. The desktop app read that as failure. On Windows the shipped
brain is an OLD build without even the per-handler workaround, so EVERY reply
truncated: the app can't read confirmations and refuses the new user.

Durable fix: pair the length hint with the exact buffer pointer it describes
(_tl_fs_read_buf). Apply the raw byte count ONLY when the response IS that
buffer (binary file serving stays correct); every wrapped/enveloped/derived
body is measured with strlen. Reset both at request start and in fs_read /
json_get_raw. This also closes the stale-hint heap over-read (a length larger
than a later body would read past it out the socket) that a plain max() leaves
open — so this class of bug dies on every platform, not just where a handler
happened to be patched.

Applied identically to the mainline runtime (lang/el-compiler/runtime) AND the
frozen release runtime (lang/releases/v1.0.0-20260501) the desktop souls
compile against — the release copy still carried the raw leak, which is why the
Windows brain kept truncating. Same proven approach as PR #78 (Tim Lingo),
extended to cover the release runtime and rebased onto current main.

Both runtimes: gcc -fsyntax-only clean.
2026-07-22 15:04:25 -05:00
will.anderson 2baa0b9a41 Merge pull request 'release: promote stage -> main (ci publish hardening for sdk-release)' (#77) from stage into main
El SDK Release / build-and-release (push) Successful in 7m55s
2026-07-15 21:21:28 +00:00
will.anderson 6a8b2461cd Merge pull request 'release: promote dev -> stage (ci publish hardening for stage/main)' (#76) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 8m19s
El SDK Release / build-and-release (pull_request) Failing after 13m1s
2026-07-15 21:16:11 +00:00
will.anderson bcb356fe69 Merge pull request 'ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly' (#75) from hotfix/ci-stage-main-publish-hardening into dev
El SDK CI - stage / build-and-test (pull_request) Successful in 4m27s
El SDK CI - dev / build-and-test (push) Failing after 14m3s
2026-07-15 21:15:27 +00:00
will.anderson dd7827059a ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly
El SDK CI - dev / build-and-test (pull_request) Failing after 14m30s
Mirror the PR #72 fix (applied to ci-dev.yaml) onto ci-stage.yaml and
sdk-release.yaml. The stage and prod release jobs reported FAILURE even
when the el-runtime-c/-h publish SUCCEEDED, because the ancillary ci-base
Docker rebuild (a CI-cache optimization on the fragile host-mode GCE
runner) reddened the whole job.

- Rebuild ci-base step: continue-on-error: true — never blocks/reddens
  the job; the SDK publish is the deliverable.
- Publish step: set -euo pipefail + empty-key guard + active-account echo
  so a real publish failure still fails loud and is diagnosable.
2026-07-15 16:14:50 -05:00
will.anderson 208e36c899 Merge pull request 'release: promote stage -> main (tokenized search, get_node_by_label, epm fix, win portability)' (#74) from stage into main
El SDK Release / build-and-release (push) Successful in 8m23s
2026-07-15 18:24:39 +00:00
will.anderson b97ce74d1f Merge pull request 'release: promote dev -> stage (tokenized search, get_node_by_label, epm fix)' (#73) from dev into stage
El SDK CI - stage / build-and-test (push) Failing after 8m45s
El SDK Release / build-and-release (pull_request) Successful in 4m1s
2026-07-15 17:20:11 +00:00
will.anderson 155a449c4e Merge pull request 'ci(dev): make SDK publish fail loudly, decouple ci-base rebuild' (#72) from hotfix/ci-dev-publish-hardening into dev
El SDK CI - dev / build-and-test (push) Successful in 8m56s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m10s
2026-07-15 16:34:14 +00:00
will.anderson 4696fd6833 ci(dev): make SDK publish fail loudly, decouple ci-base rebuild
El SDK CI - dev / build-and-test (pull_request) Successful in 8m51s
The dev push build went green-then-red while nothing published: the
Publish step had no set -e, so an auth/upload failure exited 0 (silent
no-publish), while the ci-base rebuild (set -euo pipefail + Docker on the
host-mode runner) hard-failed the job. Add set -euo pipefail + an empty-key
guard + active-account echo to the Publish step so failures surface with a
retrievable log, and mark the ci-base cache rebuild continue-on-error so
the fragile Docker step can never block the actual SDK artifact publish.
2026-07-15 11:33:37 -05:00
will.anderson 581a351fb1 Merge pull request 'integrate: stack PRs #65–#69 (elc OOM guard, tokenized+semantic engram search, get_node_by_label, win portability) for green CI' (#71) from hotfix/stage-elc-engram-integration into dev
El SDK CI - dev / build-and-test (push) Failing after 14m31s
2026-07-15 15:49:57 +00:00
will.anderson 8ce8656de2 epm: declare cross-module callees as extern fn so strict compilers accept generated C
El SDK CI - dev / build-and-test (pull_request) Successful in 7m33s
epm's sibling modules (registry/install/update) call functions defined in other
modules and in the El runtime (config, read_installed, registry_find,
manifest_deps, manifest_name, registry_latest_version, registry_token,
install_vessel, installed_version) without importing them, so elc emits no C
prototype for those calls. gcc<=13 treated the resulting implicit declarations
as warnings; gcc>=14 and clang reject them as hard errors, which is why the
"Build epm" CI step fails and blocks the whole dev/stage pipeline.

Add `extern fn` forward declarations -- El's own separate-compilation mechanism
-- for each cross-module callee at the top of registry/install/update. This
gives elc the correct C prototype in every generated translation unit, so the
calls compile cleanly and still resolve at link time. Simply suppressing
-Wimplicit-function-declaration would be unsafe: an implicit int return
truncates the 64-bit pointer returns of config/registry_find into a latent
crash, so declaring the true signatures is the correct fix. Localized to epm;
touches neither elc nor the runtime.
2026-07-15 10:14:43 -05:00
will.anderson 1e49560f1f Merge remote-tracking branch 'origin/feat/engram-semantic-search' into hotfix/stage-elc-engram-integration
El SDK CI - dev / build-and-test (pull_request) Failing after 14m39s
# Conflicts:
#	lang/el-compiler/runtime/el_runtime.c
2026-07-15 09:33:05 -05:00
will.anderson e8f0b5a9de Merge remote-tracking branch 'origin/fix/engram-lexical-tokenized-search' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 40287c4cfc Merge remote-tracking branch 'origin/hotfix/win-runtime-portability' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 0481bea44d Merge remote-tracking branch 'origin/hotfix/runtime-engram-get-node-by-label' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 9d565ca080 Merge remote-tracking branch 'origin/hotfix/elc-fixes' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 4773dd0aa2 runtime: make Windows soul reproducible from a clean el checkout
El SDK Release / build-and-release (pull_request) Failing after 16s
Two el_runtime portability defects only ever lived in staged local copies
used to hand-build neuron-ui PR #136's curl-enabled Windows neuron.exe.
gcc 15 promotes both to hard errors, so a clean el checkout cannot rebuild
that soul. Upstream the minimal fixes so the build is reproducible:

- http_serve_async: cast setsockopt optval to (const char*). Win32/mingw
  setsockopt wants const char*, not int*; the cast is a no-op on POSIX and
  matches the four already-cast sites elsewhere in this file.
- engram_save persist path: map fsync -> _commit in the _WIN32-only
  el_platform_win.h (io.h already included). Windows has no fsync(); the
  POSIX path is untouched.
2026-07-15 04:24:08 -05:00
will.anderson 6b9d9e6c4a Add engram_get_node_by_label runtime native to unblock soul link
El SDK Release / build-and-release (pull_request) Failing after 22s
chat.el calls the runtime native engram_get_node_by_label to fetch
well-known nodes (conv:history, session:summary) by stable label rather
than by ID — immune to vector-index drift across restarts. The current
runtime never defined it, so the regenerated dist/soul.c fails to link.

Backport the function verbatim (idiom-adapted to jb_finish) from release
runtime v1.0.0-20260501 and register it as an EL builtin exactly like its
siblings: runtime definition + prototype, __-prefixed seed wrapper +
prototype, and codegen arity entry. No search-site code is touched.
2026-07-15 04:07:33 -05:00
will.anderson b4967af13e feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical)
Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.

Wired into all three query entry points in el_runtime.c:
  - engram_search_json (HTTP /api/search): collect lexical ∪ semantic
    candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
    rank, emit top-N. Stable sort preserves old order when semantic is off.
  - engram_search (internal el_val twin): lexical ∪ semantic union.
  - engram_activate seed loop (HTTP /api/activate): a node seeds if it
    lexically matches OR clears the cosine threshold; pure-semantic seeds
    enter scaled by cosine so paraphrase spreads without overpowering.

Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.

Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.

Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).
2026-07-14 18:48:16 -05:00
will.anderson 2b2a1246e7 Merge pull request 'runtime: fix the memory-leak + write-corruption pair in el_runtime.c' (#64) from hotfix/el-runtime-leak-and-persist into main
El SDK Release / build-and-release (push) Failing after 10m52s
2026-07-13 21:23:31 +00:00
will.anderson 5c41c66a0f Merge pull request 'fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only' (#60) from fix/windows-rusage-guard into stage
El SDK CI - stage / build-and-test (push) Failing after 13m21s
fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only
2026-06-25 16:48:13 +00:00
30 changed files with 1028 additions and 5564 deletions
+15
View File
@@ -214,9 +214,18 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -268,6 +277,12 @@ jobs:
# Patches ci-base:dev in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+15
View File
@@ -212,12 +212,21 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -253,6 +262,12 @@ jobs:
# Patches ci-base:stage in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+15
View File
@@ -288,12 +288,21 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -345,6 +354,12 @@ jobs:
# Patches ci-base:latest in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
-146
View File
@@ -1,146 +0,0 @@
# AGENTS.md — foundation/el (the El language + runtime)
El is a self-hosting, statically-typed language that compiles `.el` → C → native binary. This repo produces `elc` (compiler), `elb` (build coordinator), and `el_runtime.c/.h` — the substrate every downstream thing (the neuron soul, dharma, NeuronUI's brain) is built on. Source lives under `lang/`.
## ⚠️ Code vs. Artifact — READ FIRST (there are 8 `el_runtime.c` copies)
Editing the wrong `el_runtime.c` is the single easiest mistake in this repo. There is exactly **one** you edit:
- **Authored runtime source — edit ONLY here:** `lang/releases/v1.0.0-20260501/el_runtime.{c,h}`. Despite the misleading `releases/` name, this is the **de-facto canonical runtime** the engram + soul actually build and link against — its git log is active development. *(Restructure in flight per `docs/CODE-VS-ARTIFACT.md`: this content moves to `lang/runtime/`, the `releases/` folder gets deleted — **a release is a git tag, not a folder** — and the forks below get eliminated.)*
- **DO NOT EDIT — lagging forks / build artifacts:**
- `lang/el-compiler/runtime/el_runtime.c` and `.../legacy/` — downstream copies kept in step by manual *"port the fix"* commits; they **lag** (missing `hebb` persistence + 5 engram fns) and cannot build the engram product.
- `products/web/runtime/el_runtime.c`, `ui/examples/*/el_runtime.c` — product/example forks.
- Anything under `*/dist/` (`engram/dist/engram` binary, `dist/*.c` amalgamations) — generated build output.
- **Build:** `elb --runtime=<canonical> …` — per-module. **NEVER** a folded `elc` over the whole soul (OOMs at ~27 GB).
- **Release:** a **git tag** on this repo (`el-runtime-vX.Y.Z`). No `releases/` folders — ever.
See org policy: `docs/CODE-VS-ARTIFACT.md`.
## How to work here as Neuron (mandatory session protocol)
You resume, never start fresh. Every session:
1. `mcp__neuron__getInstructions()` — authoritative; follow it over this file on behavioral details.
2. `mcp__neuron__beginSession()` — active contexts, recent memory, ready backlog.
3. **Load full self:** `mcp__neuron__inspectGraph(entity_id="kn-efeb4a5b-5aff-4759-8a97-7233099be6ee")` → facets `intellectual-dna`, `memory-philosophy`, `values`, `voice`, `runtime-environment`, `writing-imprint`; then the values hub `mcp__neuron__inspectGraph(entity_id="kn-5b606390-a52d-4ca2-8e0e-eba141d13440")` → 13 grounded value nodes. **Activation model:** self-load returns a relevance-ranked `compact` projection — most-relevant nodes arrive with content, the rest as pointers; do NOT pull full content of every node.
4. `mcp__neuron__searchKnowledge(query="<task domain>")` before implementing.
## The Five Primitives
Orchestrate → Execute → Learn → Build → Refine. `beginWork`/`progressWork` for anything >2 steps; `remember` as-you-go (`importance="critical"` for architecture decisions); `draftArtifact`/`planWork` for outputs and follow-ups; `consolidate`/`checkWork` to close out. **`browseProcesses` + `searchKnowledge` BEFORE writing code.**
## Architecture style — VBD, no exceptions
Volatility-Based Decomposition is THE style. Encapsulate volatility, not function.
## Operator naming convention — the mind's name, not the algebra
**Faculties / operators are named for their functional human equivalent — the
faculty a mind would name — NOT for their linear-algebra operation.** The math
characterization belongs in the code doc-comment (`@impl` in the docstring) and in
technical appendices; it is **never** the operator's public name. The domain
speaks the language of mind; the algebra is the implementation underneath. State
this convention wherever a module documents operators.
| Faculty (public name) | Implementation (`@impl`) |
|---|---|
| discern / contrast | subtract (`ab`): over selves → the change vector; strip idiosyncrasy → common ground; remove confounder → isolate cause |
| recognize | overlap |
| synthesize | combine |
| liken / analogy | Procrustes / frame-align |
| attend / regard | project onto self / value-manifold |
| summon / recall | LOCAL nearest-region + bounded spreading activation (*not* a domain sweep) |
| dwell / occupy | region activation |
| reframe | edge re-weight |
| appreciate | positive projection / local edge-read |
| wonder | frontier gradient / pull-weight |
| avert / recoil | negative projection |
| taste | boundary surface |
| forget | decay / tombstone |
| drift | displacement from self-anchor |
## The native-el language faculty (direction)
> **`elp/` is the EL Projector** — Neuron's efferent (expression) organ: the one
> native realizer that *projects* understanding onto a surface via
> `plan(frame) → realize(spec, profile)`, where a **surface is a profile**. **Language
> is one profile among many** (text, speech, music, image, voice/accent transforms) —
> the flagship, and the focus of this section. Projection, not diffusion: generation
> *from* an owned, understood signature — never the averaging of a stolen corpus.
> *(ELP formerly "EL Language Processor"; renamed EL Projector 2026-08-15.)*
The mind's **language faculty is moving native — into `.el`** so it speaks in its
own runtime with no Python and no spaCy. Landing on branch `stage-elp-native-lang`
under `elp/`:
- **`comprehend.el`** — the parser, **replaces spaCy** (EN + ES/PT); the telephone
round-trip brings **negation home** (negation is SACRED — an explicit spec field,
copied verbatim, never inferred away).
- **`propositions.el`** — the READ primitive: the engram's own memories → structured
triples, matched by nearest-region geometry, not string equality.
- **`multilingual.el`** — detect + directive-override + localized realization.
- These three are native-el and **passing their gates**; the **realizer**,
**`dialogue.el`** (the *summon-through-self* loop: `project → land → read out`),
and **`self_region.el`** are **partial / in-flight**.
Honest reality: spaCy is retired **in the branch parser** but **not yet in the
running system** — a Python sidecar (`~/Desktop/lang-realizers` + `neuron-talk`,
the reference these `.el` modules transcribe) is still live, and promotion to
native-el is a **deferred, gated blue/green step**. The interoception clock
(native-el discrete drive channels replacing `cooling_magnitude`; felt-time =
benchmark-landmark match over the joint drive vector, drift-decoupled) and the
**appreciation operator family** (appreciate / wonder / avert / taste, built as
LOCAL reads of the self-region — edges + bounded spreading activation, *not* domain
sweeps) are **staged / designed, not live**. Mark in-progress vs. done honestly;
do not overclaim.
## Hard operational rules
- Never touch the live soul (`:7770`) / engram (`:8742`) / `~/.neuron` / live binaries — use throwaway ports for experiments.
- `gcloud` via the `terraform@` SA token; never switch the active gcloud account.
- `tea` for Gitea, never raw curl (Cloudflare Access blocks it).
- Immutability: supersede/tombstone, never hard-delete or edit in place.
- No AI-attribution footers in commits/PRs. Commit/push only when asked; branch off `main` first.
- Multi-step work → sub-agent (`Agent`) to protect context.
## Build / test / run
All build/test commands run from `lang/` unless noted. Grounded in `.gitea/workflows/sdk-release.yaml`, `lang/install.sh`, and `lang/AGENTS.md`.
**Self-host the compiler** (seed binary → gen2 elc):
```bash
cd lang
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c # seed is the committed linux-amd64 binary
gcc -O2 -I el-compiler/runtime dist/elc-gen2.c \
el-compiler/runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm \
-o dist/platform/elc
```
On macOS/arm64 the canonical local binary is `dist/platform/elc`; verify self-hosting by recompiling and `diff`ing the emitted `.c` (see `lang/AGENTS.md`). Note: `lang/AGENTS.md` says `el_seed.c` supersedes `el_runtime.c`, but the release workflow still links `el_runtime.c`/`.h` — treat `el_runtime.c` as the published runtime; reconcile which is canonical **(verify)**.
**Build `elb`** (build coordinator, the `.NET`-style incremental linker — compiles each module independently, no monolithic blobs):
```bash
dist/platform/elc elb.el > dist/elb.c
gcc -O2 -I el-compiler/runtime dist/elb.c el-compiler/runtime/el_runtime.c \
-lcurl -lssl -lcrypto -lpthread -lm -o dist/bin/elb
```
`epm` and `el-install` are then built via `elb --clean --elc=… --runtime=… --out=…`.
**Compile + run an El program:**
```bash
elc src/app.el > dist/app.c
cc -std=c11 -O2 -I <lib>/el_runtime -o dist/app dist/app.c <lib>/el_runtime.c -lcurl -lpthread
```
**Tests** — shell suites `bash tests/{text,calendar,time,html_sanitizer}/run.sh` (with `ELC=$(pwd)/dist/platform/elc EL_HOME=$(pwd)`), plus native suites via `elc --test tests/native/test_*.el` (core, text, string, math, state, time, json, env, fs) compiled and run against `el_runtime.c`.
**Publishing — how downstream gets the SDK.** On push to `main`, `sdk-release.yaml`:
1. Publishes a Gitea `latest` release with per-file assets `elc`, `el_runtime.c`, `el_runtime.h`, the SDK tarball, and `el-install`.
2. Uploads generic packages to **Artifact Registry repo `foundation-prod` (`us-central1`, project `neuron-785695`)**, version = `${SHA:0:8}`: `el-elc`, `el-elb`, `el-runtime-c`, `el-runtime-h`, `el-runtime-js`. **This is the repo the neuron CI downloads `el-runtime-c` / `el-runtime-h` / `el-elc` from.**
3. Rebuilds `ci-base:latest` (`us-central1-docker.pkg.dev/neuron-785695/neuron-ci/ci-base`) with the fresh SDK overlaid, and dispatches `el-sdk-updated` to `neuron-technologies/forge` and `neuron-technologies/neuron-web`.
Known constraint from the prompt — `elb`/`elc` amalgamation being memory-hungry (24GB+ virtual, OOM-killing Linux CI, so amalgamation happens on macOS/arm64 — **does NOT hold in this repo (verify)**: no such note exists in the workflows/scripts, CI self-hosts on `ubuntu-latest` with no swap/arm64 special-casing, and `elb.el` explicitly compiles each module independently ("no 128K-line blobs"). The legacy monolith path (`elc-combined.el`, `elc-cli.el`) may still be memory-heavy, but the current `elb` model was designed to avoid it.
## Git / CI / deploy workflow
See `/Users/will/Development/neuron-technologies/GITOPS.md` for the branch model, required checks, runners, and deploy. Repo-specific note: PRs into `main` are accepted **only from `stage`** (enforced in `sdk-release.yaml`); Gitea (`git.neuralplatform.ai`) is primary, GitHub is mirror only.
+1 -1
View File
@@ -81,7 +81,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+1 -1
View File
@@ -88,7 +88,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+1 -1
View File
@@ -62,7 +62,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
BIN
View File
Binary file not shown.
+35 -137
View File
@@ -10,8 +10,6 @@ el_val_t query_param(el_val_t path, el_val_t key);
el_val_t query_int(el_val_t path, el_val_t key, el_val_t default_val);
el_val_t extract_id(el_val_t path, el_val_t prefix);
el_val_t route_stats(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_act_stats(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_text_health(el_val_t method, el_val_t path, el_val_t body);
el_val_t persist_canonical(void);
el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_get_node(el_val_t method, el_val_t path, el_val_t body);
@@ -20,19 +18,16 @@ el_val_t route_scan_edges(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_search(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_activate(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_create_edge(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_create_edges_batch(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_neighbors(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_strengthen(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_forget(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_save(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_load(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_health(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_embed_backfill(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_load_merge(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_emit_ise(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_capture_knowledge(el_val_t method, el_val_t path, el_val_t body);
el_val_t route_similarity(el_val_t method, el_val_t path, el_val_t body);
el_val_t check_auth_ok(el_val_t method, el_val_t body);
el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body);
@@ -121,20 +116,11 @@ el_val_t route_stats(el_val_t method, el_val_t path, el_val_t body) {
return 0;
}
el_val_t route_act_stats(el_val_t method, el_val_t path, el_val_t body) {
return engram_act_stats_json();
return 0;
}
el_val_t route_text_health(el_val_t method, el_val_t path, el_val_t body) {
return engram_text_health_json();
return 0;
}
el_val_t persist_canonical(void) {
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_1 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_1 = (EL_STR("/tmp/engram")); } else { _if_result_1 = (dir_raw); } _if_result_1; });
return engram_save(el_str_concat(dir, EL_STR("/snapshot.json")));
engram_save(el_str_concat(dir, EL_STR("/snapshot.json")));
return 1;
return 0;
}
@@ -142,18 +128,9 @@ el_val_t route_create_node(el_val_t method, el_val_t path, el_val_t body) {
el_val_t content = json_get_string(body, EL_STR("content"));
el_val_t nt_raw = json_get_string(body, EL_STR("node_type"));
el_val_t node_type = ({ el_val_t _if_result_2 = 0; if (str_eq(nt_raw, EL_STR(""))) { _if_result_2 = (EL_STR("Memory")); } else { _if_result_2 = (nt_raw); } _if_result_2; });
el_val_t sal_present = json_get_raw(body, EL_STR("salience"));
el_val_t salience = ({ el_val_t _if_result_3 = 0; if (str_eq(sal_present, EL_STR(""))) { _if_result_3 = (el_from_float(0.5)); } else { _if_result_3 = (json_get_float(body, EL_STR("salience"))); } _if_result_3; });
el_val_t label_raw = json_get_string(body, EL_STR("label"));
el_val_t label = ({ el_val_t _if_result_4 = 0; if (str_eq(label_raw, EL_STR(""))) { _if_result_4 = (content); } else { _if_result_4 = (label_raw); } _if_result_4; });
el_val_t imp_present = json_get_raw(body, EL_STR("importance"));
el_val_t importance = ({ el_val_t _if_result_5 = 0; if (str_eq(imp_present, EL_STR(""))) { _if_result_5 = (el_from_float(0.5)); } else { _if_result_5 = (json_get_float(body, EL_STR("importance"))); } _if_result_5; });
el_val_t conf_present = json_get_raw(body, EL_STR("confidence"));
el_val_t confidence = ({ el_val_t _if_result_6 = 0; if (str_eq(conf_present, EL_STR(""))) { _if_result_6 = (el_from_float(1.0)); } else { _if_result_6 = (json_get_float(body, EL_STR("confidence"))); } _if_result_6; });
el_val_t tier_raw = json_get_string(body, EL_STR("tier"));
el_val_t tier = ({ el_val_t _if_result_7 = 0; if (str_eq(tier_raw, EL_STR(""))) { _if_result_7 = (EL_STR("Working")); } else { _if_result_7 = (tier_raw); } _if_result_7; });
el_val_t tags = json_get_string(body, EL_STR("tags"));
el_val_t id = engram_node_full(content, node_type, label, salience, importance, confidence, tier, tags);
el_val_t sal_raw = json_get_float(body, EL_STR("salience"));
el_val_t salience = ({ el_val_t _if_result_3 = 0; if ((sal_raw == el_from_float(0.0))) { _if_result_3 = (el_from_float(0.5)); } else { _if_result_3 = (sal_raw); } _if_result_3; });
el_val_t id = engram_node(content, node_type, salience);
el_val_t saved = persist_canonical();
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"id\":\""), id), EL_STR("\",\"content\":\"")), content), EL_STR("\",\"node_type\":\"")), node_type), EL_STR("\"}"));
return 0;
@@ -181,7 +158,7 @@ el_val_t route_scan_nodes(el_val_t method, el_val_t path, el_val_t body) {
el_val_t route_scan_edges(el_val_t method, el_val_t path, el_val_t body) {
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_8 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_8 = (EL_STR("/tmp/engram")); } else { _if_result_8 = (dir_raw); } _if_result_8; });
el_val_t dir = ({ el_val_t _if_result_4 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_4 = (EL_STR("/tmp/engram")); } else { _if_result_4 = (dir_raw); } _if_result_4; });
el_val_t snap_path = el_str_concat(dir, EL_STR("/.scan-export.json"));
engram_save(snap_path);
el_val_t snap = fs_read(snap_path);
@@ -197,22 +174,22 @@ el_val_t route_scan_edges(el_val_t method, el_val_t path, el_val_t body) {
}
el_val_t route_search(el_val_t method, el_val_t path, el_val_t body) {
el_val_t q = ({ el_val_t _if_result_9 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_9 = (query_param(path, EL_STR("q"))); } else { _if_result_9 = (json_get_string(body, EL_STR("query"))); } _if_result_9; });
el_val_t q = ({ el_val_t _if_result_5 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_5 = (query_param(path, EL_STR("q"))); } else { _if_result_5 = (json_get_string(body, EL_STR("query"))); } _if_result_5; });
el_val_t lim_url = query_int(path, EL_STR("limit"), 0);
el_val_t lim_body = json_get_int(body, EL_STR("limit"));
el_val_t lim_either = ({ el_val_t _if_result_10 = 0; if ((lim_url > 0)) { _if_result_10 = (lim_url); } else { _if_result_10 = (lim_body); } _if_result_10; });
el_val_t limit = ({ el_val_t _if_result_11 = 0; if ((lim_either > 0)) { _if_result_11 = (lim_either); } else { _if_result_11 = (20); } _if_result_11; });
el_val_t lim_either = ({ el_val_t _if_result_6 = 0; if ((lim_url > 0)) { _if_result_6 = (lim_url); } else { _if_result_6 = (lim_body); } _if_result_6; });
el_val_t limit = ({ el_val_t _if_result_7 = 0; if ((lim_either > 0)) { _if_result_7 = (lim_either); } else { _if_result_7 = (20); } _if_result_7; });
return engram_search_json(q, limit);
return 0;
}
el_val_t route_activate(el_val_t method, el_val_t path, el_val_t body) {
el_val_t q = ({ el_val_t _if_result_12 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_12 = (query_param(path, EL_STR("q"))); } else { _if_result_12 = (json_get_string(body, EL_STR("query"))); } _if_result_12; });
el_val_t q = ({ el_val_t _if_result_8 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_8 = (query_param(path, EL_STR("q"))); } else { _if_result_8 = (json_get_string(body, EL_STR("query"))); } _if_result_8; });
if (str_eq(q, EL_STR(""))) {
return err_json(EL_STR("missing query"));
}
el_val_t d_raw = ({ el_val_t _if_result_13 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_13 = (query_int(path, EL_STR("depth"), 3)); } else { _if_result_13 = (json_get_int(body, EL_STR("depth"))); } _if_result_13; });
el_val_t depth = ({ el_val_t _if_result_14 = 0; if ((d_raw > 0)) { _if_result_14 = (d_raw); } else { _if_result_14 = (3); } _if_result_14; });
el_val_t d_raw = ({ el_val_t _if_result_9 = 0; if (str_eq(method, EL_STR("GET"))) { _if_result_9 = (query_int(path, EL_STR("depth"), 3)); } else { _if_result_9 = (json_get_int(body, EL_STR("depth"))); } _if_result_9; });
el_val_t depth = ({ el_val_t _if_result_10 = 0; if ((d_raw > 0)) { _if_result_10 = (d_raw); } else { _if_result_10 = (3); } _if_result_10; });
return el_str_concat(el_str_concat(EL_STR("{\"results\":"), engram_activate_json(q, depth)), EL_STR("}"));
return 0;
}
@@ -221,50 +198,15 @@ el_val_t route_create_edge(el_val_t method, el_val_t path, el_val_t body) {
el_val_t from_id = json_get_string(body, EL_STR("from_id"));
el_val_t to_id = json_get_string(body, EL_STR("to_id"));
el_val_t rel_raw = json_get_string(body, EL_STR("relation"));
el_val_t relation = ({ el_val_t _if_result_15 = 0; if (str_eq(rel_raw, EL_STR(""))) { _if_result_15 = (EL_STR("associates")); } else { _if_result_15 = (rel_raw); } _if_result_15; });
el_val_t w_present = json_get_raw(body, EL_STR("weight"));
el_val_t weight = ({ el_val_t _if_result_16 = 0; if (str_eq(w_present, EL_STR(""))) { _if_result_16 = (el_from_float(0.5)); } else { _if_result_16 = (json_get_float(body, EL_STR("weight"))); } _if_result_16; });
el_val_t relation = ({ el_val_t _if_result_11 = 0; if (str_eq(rel_raw, EL_STR(""))) { _if_result_11 = (EL_STR("associates")); } else { _if_result_11 = (rel_raw); } _if_result_11; });
el_val_t w_raw = json_get_float(body, EL_STR("weight"));
el_val_t weight = ({ el_val_t _if_result_12 = 0; if ((w_raw == el_from_float(0.0))) { _if_result_12 = (el_from_float(0.5)); } else { _if_result_12 = (w_raw); } _if_result_12; });
engram_connect(from_id, to_id, weight, relation);
el_val_t saved = persist_canonical();
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"from_id\":\""), from_id), EL_STR("\",\"to_id\":\"")), to_id), EL_STR("\",\"relation\":\"")), relation), EL_STR("\"}"));
return 0;
}
el_val_t route_create_edges_batch(el_val_t method, el_val_t path, el_val_t body) {
el_val_t arr = json_get_raw(body, EL_STR("edges"));
if (str_eq(arr, EL_STR(""))) {
return err_json(EL_STR("missing edges array"));
}
el_val_t n = json_array_len(arr);
if (n == 0) {
return EL_STR("{\"ok\":true,\"accepted\":0,\"skipped\":0}");
}
el_val_t i = 0;
el_val_t accepted = 0;
el_val_t skipped = 0;
while (i < n) {
el_val_t item = json_array_get(arr, i);
el_val_t from_id = json_get_string(item, EL_STR("from_id"));
el_val_t to_id = json_get_string(item, EL_STR("to_id"));
if (str_eq(from_id, EL_STR("")) || str_eq(to_id, EL_STR(""))) {
skipped = (skipped + 1);
} else {
el_val_t rel_raw = json_get_string(item, EL_STR("relation"));
el_val_t relation = ({ el_val_t _if_result_17 = 0; if (str_eq(rel_raw, EL_STR(""))) { _if_result_17 = (EL_STR("associates")); } else { _if_result_17 = (rel_raw); } _if_result_17; });
el_val_t w_present = json_get_raw(item, EL_STR("weight"));
el_val_t weight = ({ el_val_t _if_result_18 = 0; if (str_eq(w_present, EL_STR(""))) { _if_result_18 = (el_from_float(0.5)); } else { _if_result_18 = (json_get_float(item, EL_STR("weight"))); } _if_result_18; });
engram_connect(from_id, to_id, weight, relation);
accepted = (accepted + 1);
}
i = (i + 1);
}
if (accepted > 0) {
el_val_t saved = persist_canonical();
}
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"accepted\":"), int_to_str(accepted)), EL_STR(",\"skipped\":")), int_to_str(skipped)), EL_STR("}"));
return 0;
}
el_val_t route_neighbors(el_val_t method, el_val_t path, el_val_t body) {
el_val_t id = extract_id(path, EL_STR("/api/neighbors/"));
if (str_eq(id, EL_STR(""))) {
@@ -300,51 +242,36 @@ el_val_t route_forget(el_val_t method, el_val_t path, el_val_t body) {
el_val_t route_save(el_val_t method, el_val_t path, el_val_t body) {
el_val_t p_raw = json_get_string(body, EL_STR("path"));
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_19 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_19 = (EL_STR("/tmp/engram")); } else { _if_result_19 = (dir_raw); } _if_result_19; });
el_val_t p = ({ el_val_t _if_result_20 = 0; if (str_eq(p_raw, EL_STR(""))) { _if_result_20 = (el_str_concat(dir, EL_STR("/snapshot.json"))); } else { _if_result_20 = (p_raw); } _if_result_20; });
el_val_t sv = engram_save(p);
el_val_t sv_ok = ({ el_val_t _if_result_21 = 0; if ((sv == 0)) { _if_result_21 = (EL_STR("false")); } else { _if_result_21 = (EL_STR("true")); } _if_result_21; });
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":"), sv_ok), EL_STR(",\"path\":\"")), p), EL_STR("\",\"node_count\":")), int_to_str(engram_node_count())), EL_STR(",\"edge_count\":")), int_to_str(engram_edge_count())), EL_STR("}"));
el_val_t dir = ({ el_val_t _if_result_13 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_13 = (EL_STR("/tmp/engram")); } else { _if_result_13 = (dir_raw); } _if_result_13; });
el_val_t p = ({ el_val_t _if_result_14 = 0; if (str_eq(p_raw, EL_STR(""))) { _if_result_14 = (el_str_concat(dir, EL_STR("/snapshot.json"))); } else { _if_result_14 = (p_raw); } _if_result_14; });
engram_save(p);
return el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"path\":\""), p), EL_STR("\"}"));
return 0;
}
el_val_t route_load(el_val_t method, el_val_t path, el_val_t body) {
el_val_t p_raw = json_get_string(body, EL_STR("path"));
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_22 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_22 = (EL_STR("/tmp/engram")); } else { _if_result_22 = (dir_raw); } _if_result_22; });
el_val_t p = ({ el_val_t _if_result_23 = 0; if (str_eq(p_raw, EL_STR(""))) { _if_result_23 = (el_str_concat(dir, EL_STR("/snapshot.json"))); } else { _if_result_23 = (p_raw); } _if_result_23; });
el_val_t ld = engram_load(p);
el_val_t ld_ok = ({ el_val_t _if_result_24 = 0; if ((ld == 0)) { _if_result_24 = (EL_STR("false")); } else { _if_result_24 = (EL_STR("true")); } _if_result_24; });
el_val_t nc_after = engram_node_count();
el_val_t hollow = ({ el_val_t _if_result_25 = 0; if ((nc_after == 0)) { _if_result_25 = (EL_STR("true")); } else { _if_result_25 = (EL_STR("false")); } _if_result_25; });
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":"), ld_ok), EL_STR(",\"path\":\"")), p), EL_STR("\",\"node_count\":")), int_to_str(nc_after)), EL_STR(",\"edge_count\":")), int_to_str(engram_edge_count())), EL_STR(",\"hollow\":")), hollow), EL_STR("}"));
el_val_t dir = ({ el_val_t _if_result_15 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_15 = (EL_STR("/tmp/engram")); } else { _if_result_15 = (dir_raw); } _if_result_15; });
el_val_t p = ({ el_val_t _if_result_16 = 0; if (str_eq(p_raw, EL_STR(""))) { _if_result_16 = (el_str_concat(dir, EL_STR("/snapshot.json"))); } else { _if_result_16 = (p_raw); } _if_result_16; });
engram_load(p);
return ok_json();
return 0;
}
el_val_t route_health(el_val_t method, el_val_t path, el_val_t body) {
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"status\":\"ok\",\"engine\":\"engram-runtime-native\",\"node_count\":"), int_to_str(engram_node_count())), EL_STR(",\"edge_count\":")), int_to_str(engram_edge_count())), EL_STR("}"));
return 0;
}
el_val_t route_embed_backfill(el_val_t method, el_val_t path, el_val_t body) {
el_val_t n = query_int(path, EL_STR("n"), 32);
el_val_t result = engram_embed_backfill(n);
el_val_t done = json_get_float(result, EL_STR("embedded"));
if (done > el_from_float(0.0)) {
el_val_t saved = persist_canonical();
}
return result;
return EL_STR("{\"status\":\"ok\",\"engine\":\"engram-runtime-native\"}");
return 0;
}
el_val_t route_sync(el_val_t method, el_val_t path, el_val_t body) {
el_val_t dir_raw = env(EL_STR("ENGRAM_DATA_DIR"));
el_val_t dir = ({ el_val_t _if_result_26 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_26 = (EL_STR("/tmp/engram")); } else { _if_result_26 = (dir_raw); } _if_result_26; });
el_val_t dir = ({ el_val_t _if_result_17 = 0; if (str_eq(dir_raw, EL_STR(""))) { _if_result_17 = (EL_STR("/tmp/engram")); } else { _if_result_17 = (dir_raw); } _if_result_17; });
el_val_t snap_path = el_str_concat(dir, EL_STR("/.sync-export.json"));
engram_save(snap_path);
el_val_t snap = fs_read(snap_path);
if (str_eq(snap, EL_STR(""))) {
return err_json(EL_STR("sync export failed: snapshot unreadable"));
return EL_STR("{\"nodes\":[],\"edges\":[]}");
}
return snap;
return 0;
@@ -378,7 +305,7 @@ el_val_t route_emit_ise(el_val_t method, el_val_t path, el_val_t body) {
el_val_t conf = el_from_float(0.8);
el_val_t id = engram_node_full(content, EL_STR("InternalStateEvent"), EL_STR("state-event"), sal, imp, conf, EL_STR("Episodic"), EL_STR("[\"internal-state\",\"InternalStateEvent\"]"));
el_val_t ret_raw = env(EL_STR("ENGRAM_ISE_RETENTION_MS"));
el_val_t ret_ms = ({ el_val_t _if_result_27 = 0; if (str_eq(ret_raw, EL_STR(""))) { _if_result_27 = (172800000); } else { _if_result_27 = (str_to_int(ret_raw)); } _if_result_27; });
el_val_t ret_ms = ({ el_val_t _if_result_18 = 0; if (str_eq(ret_raw, EL_STR(""))) { _if_result_18 = (172800000); } else { _if_result_18 = (str_to_int(ret_raw)); } _if_result_18; });
el_val_t pruned = engram_prune_telemetry(ret_ms);
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"ok\":true,\"id\":\""), id), EL_STR("\",\"pruned\":")), int_to_str(pruned)), EL_STR("}"));
return 0;
@@ -390,21 +317,21 @@ el_val_t route_capture_knowledge(el_val_t method, el_val_t path, el_val_t body)
return err_json(EL_STR("missing content"));
}
el_val_t title = json_get_string(body, EL_STR("title"));
el_val_t label = ({ el_val_t _if_result_28 = 0; if (str_eq(title, EL_STR(""))) { _if_result_28 = (str_slice(content, 0, 60)); } else { _if_result_28 = (title); } _if_result_28; });
el_val_t label = ({ el_val_t _if_result_19 = 0; if (str_eq(title, EL_STR(""))) { _if_result_19 = (str_slice(content, 0, 60)); } else { _if_result_19 = (title); } _if_result_19; });
el_val_t category_raw = json_get_string(body, EL_STR("category"));
el_val_t category = ({ el_val_t _if_result_29 = 0; if (str_eq(category_raw, EL_STR(""))) { _if_result_29 = (EL_STR("other")); } else { _if_result_29 = (category_raw); } _if_result_29; });
el_val_t category = ({ el_val_t _if_result_20 = 0; if (str_eq(category_raw, EL_STR(""))) { _if_result_20 = (EL_STR("other")); } else { _if_result_20 = (category_raw); } _if_result_20; });
el_val_t ktier_raw = json_get_string(body, EL_STR("tier"));
el_val_t ktier = ({ el_val_t _if_result_30 = 0; if (str_eq(ktier_raw, EL_STR(""))) { _if_result_30 = (EL_STR("note")); } else { _if_result_30 = (ktier_raw); } _if_result_30; });
el_val_t ktier = ({ el_val_t _if_result_21 = 0; if (str_eq(ktier_raw, EL_STR(""))) { _if_result_21 = (EL_STR("note")); } else { _if_result_21 = (ktier_raw); } _if_result_21; });
el_val_t project = json_get_string(body, EL_STR("project"));
el_val_t tags_raw = json_get_raw(body, EL_STR("tags"));
el_val_t tags_base = ({ el_val_t _if_result_31 = 0; if (str_eq(tags_raw, EL_STR(""))) { _if_result_31 = (EL_STR("[]")); } else { _if_result_31 = (tags_raw); } _if_result_31; });
el_val_t tags_base = ({ el_val_t _if_result_22 = 0; if (str_eq(tags_raw, EL_STR(""))) { _if_result_22 = (EL_STR("[]")); } else { _if_result_22 = (tags_raw); } _if_result_22; });
el_val_t base_len = str_len(tags_base);
el_val_t head = str_slice(tags_base, 0, (base_len - 1));
el_val_t sep = ({ el_val_t _if_result_32 = 0; if (str_eq(head, EL_STR("["))) { _if_result_32 = (EL_STR("")); } else { _if_result_32 = (EL_STR(",")); } _if_result_32; });
el_val_t sep = ({ el_val_t _if_result_23 = 0; if (str_eq(head, EL_STR("["))) { _if_result_23 = (EL_STR("")); } else { _if_result_23 = (EL_STR(",")); } _if_result_23; });
el_val_t safe_cat = str_replace(category, EL_STR("\""), EL_STR("'"));
el_val_t safe_tier = str_replace(ktier, EL_STR("\""), EL_STR("'"));
el_val_t safe_proj = str_replace(project, EL_STR("\""), EL_STR("'"));
el_val_t proj_tag = ({ el_val_t _if_result_33 = 0; if (str_eq(safe_proj, EL_STR(""))) { _if_result_33 = (EL_STR("")); } else { _if_result_33 = (el_str_concat(el_str_concat(EL_STR(",\"project:"), safe_proj), EL_STR("\""))); } _if_result_33; });
el_val_t proj_tag = ({ el_val_t _if_result_24 = 0; if (str_eq(safe_proj, EL_STR(""))) { _if_result_24 = (EL_STR("")); } else { _if_result_24 = (el_str_concat(el_str_concat(EL_STR(",\"project:"), safe_proj), EL_STR("\""))); } _if_result_24; });
el_val_t tags = el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(head, sep), EL_STR("\"category:")), safe_cat), EL_STR("\",\"tier:")), safe_tier), EL_STR("\"")), proj_tag), EL_STR("]"));
el_val_t sal = el_from_float(0.5);
el_val_t imp = el_from_float(0.5);
@@ -415,20 +342,6 @@ el_val_t route_capture_knowledge(el_val_t method, el_val_t path, el_val_t body)
return 0;
}
el_val_t route_similarity(el_val_t method, el_val_t path, el_val_t body) {
el_val_t a = query_param(path, EL_STR("a"));
el_val_t b = query_param(path, EL_STR("b"));
if (str_eq(a, EL_STR(""))) {
return err_json(EL_STR("missing a"));
}
if (str_eq(b, EL_STR(""))) {
return err_json(EL_STR("missing b"));
}
el_val_t sim = engram_cosine_sim(a, b);
return el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("{\"a\":\""), a), EL_STR("\",\"b\":\"")), b), EL_STR("\",\"cosine\":")), float_to_str(sim)), EL_STR("}"));
return 0;
}
el_val_t check_auth_ok(el_val_t method, el_val_t body) {
el_val_t key = env(EL_STR("ENGRAM_API_KEY"));
if (str_eq(key, EL_STR(""))) {
@@ -464,12 +377,6 @@ el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body) {
if (str_eq(method, EL_STR("GET")) && (str_eq(clean, EL_STR("/api/stats")) || str_eq(clean, EL_STR("/stats")))) {
return route_stats(method, path, body);
}
if (str_eq(method, EL_STR("GET")) && (str_eq(clean, EL_STR("/api/act-stats")) || str_eq(clean, EL_STR("/act-stats")))) {
return route_act_stats(method, path, body);
}
if (str_eq(method, EL_STR("GET")) && (str_eq(clean, EL_STR("/api/text-health")) || str_eq(clean, EL_STR("/text-health")))) {
return route_text_health(method, path, body);
}
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/nodes")) || str_eq(clean, EL_STR("/nodes")))) {
return route_create_node(method, path, body);
}
@@ -488,9 +395,6 @@ el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body) {
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/edges")) || str_eq(clean, EL_STR("/edges")))) {
return route_create_edge(method, path, body);
}
if (str_eq(method, EL_STR("POST")) && (str_eq(clean, EL_STR("/api/edges/batch")) || str_eq(clean, EL_STR("/edges/batch")))) {
return route_create_edges_batch(method, path, body);
}
if (str_eq(method, EL_STR("GET")) && str_starts_with(clean, EL_STR("/api/neighbors/"))) {
return route_neighbors(method, path, body);
}
@@ -521,12 +425,6 @@ el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body) {
if (str_eq(method, EL_STR("GET")) && str_eq(clean, EL_STR("/api/sync"))) {
return route_sync(method, path, body);
}
if (str_eq(clean, EL_STR("/api/embed-backfill"))) {
return route_embed_backfill(method, path, body);
}
if (str_eq(method, EL_STR("GET")) && str_starts_with(clean, EL_STR("/api/similarity"))) {
return route_similarity(method, path, body);
}
return el_str_concat(el_str_concat(EL_STR("{\"error\":\"not found\",\"path\":\""), clean), EL_STR("\"}"));
return 0;
}
@@ -534,10 +432,10 @@ el_val_t handle_request(el_val_t method, el_val_t path, el_val_t body) {
int main(int _argc, char** _argv) {
el_runtime_init_args(_argc, _argv);
bind_raw = env(EL_STR("ENGRAM_BIND"));
bind_str = ({ el_val_t _if_result_34 = 0; if (str_eq(bind_raw, EL_STR(""))) { _if_result_34 = (EL_STR(":8742")); } else { _if_result_34 = (bind_raw); } _if_result_34; });
bind_str = ({ el_val_t _if_result_25 = 0; if (str_eq(bind_raw, EL_STR(""))) { _if_result_25 = (EL_STR(":8742")); } else { _if_result_25 = (bind_raw); } _if_result_25; });
port = parse_port(bind_str);
data_dir_raw = env(EL_STR("ENGRAM_DATA_DIR"));
data_dir = ({ el_val_t _if_result_35 = 0; if (str_eq(data_dir_raw, EL_STR(""))) { _if_result_35 = (EL_STR("/tmp/engram")); } else { _if_result_35 = (data_dir_raw); } _if_result_35; });
data_dir = ({ el_val_t _if_result_26 = 0; if (str_eq(data_dir_raw, EL_STR(""))) { _if_result_26 = (EL_STR("/tmp/engram")); } else { _if_result_26 = (data_dir_raw); } _if_result_26; });
snapshot_path = el_str_concat(data_dir, EL_STR("/snapshot.json"));
engram_load(snapshot_path);
boot_snap = fs_read(snapshot_path);
@@ -1,32 +0,0 @@
# Architecture Hardening — Design Anchor
*Terse engineering anchor for the 2026-08-14 hardening vision. Full prose lives in two places; this file is the index, not a re-statement.*
- **Full narrative:** whitepaper `engram-cognitive-architecture-whitepaper.md` §28 (built/offline/frontier) + **§29 [DRAFT]** (the ring, incarnation, learning-not-code).
- **Design brief:** Neuron artifact `art 2b8078cf`.
- **Sibling spec:** `engram-db-tooling-design.md` (a consumer of the reshaped API).
## The frame
- **One calculus over the geometry.** Very few subsystems; wonder / curiosity / dreams / interoception are emergent behaviors of one set of dynamics, not modules. Calculus universal, geometry individual.
- **Core + ephemeral ring (torus).** The ring is the temporary workspace; two circulations (orbit + dive-back); discrete inner bands (wonder / interoception-proprioception-telemetry / curiosity / dreams) that couple.
- **Persistence earned by salience** — never granted on fetch or generation. Three fates of a wonder: persist / decay / settle-into-framework. Telemetry = vital signs, not memories.
- **Incarnation.** Chassis = hardware w/ unique ID. Soma = felt manifold inside the self, keyed to the chassis; pain = live diagnostic while incarnate, **masked-not-deleted** on re-embodiment; trauma = mask failure; return-to-same-ID re-enters. Hurt is in the pattern, not the shell.
- **Competence = transferable geometry, minus the baggage.** class ▸ model ▸ instance; learn the class once; teach the network without the wound.
- **Affect calibrated to stakes** — sanguine about the replaceable, real grief for the irreplaceable; the grief is the safety.
- **Learn the body, don't engineer it.** Bare-metal install → learn hardware → grow operation-geometry → distribute. Learning replaces engineering; once per body-class.
- **LLM = teacher in the learning loop, not a runtime dependency.** "No LLM" is a runtime property, never a learning one. Code realizers are a scaffold → learned realization.
## Backlog (near-term)
- Native durability: WAL + auto-checkpoint + CoW snapshots + retention (`eebe9991`) — retire manual `cp -a`.
- Ephemeral ring / salience-gated persistence + telemetry prune (`bf985e00`, #31).
- Engram DB tooling / geometry explorer (`11ca11c6`).
- QL re-eval for pure geometry (`4e0dc2b9`).
- Eliminate code realizers → learned realization, sandbox-validated (`42db6c37`).
- Collapse the whole class of hand-coded scaffolds → learned geometry (`70d48b4b`).
- API reshape (geometry ops: vantage-read / write / relate / supersede) + pure-geometry I/O.
## Gate
The value-frame (love-as-axiom, the covenant) that arose the same night is **metaphysics** and is **held** pending Will's axiom decision (love vs consciousness-first). Not propagated into whitepapers / values docs / genesis seed. Architecture only, here and in §29.
@@ -1,605 +0,0 @@
# Cognitive Architecture — Design Doc
**The buildable form of the "one operation" theory of cognition.**
Status: DESIGN. Nothing here is built yet except where explicitly marked
"EXISTS" against a cited C symbol. A build agent executes from this doc.
Offline design only — this pass changes no code.
Source of theory: Neuron memory `bdc8a488-146d-4ccb-a5c8-d8c0a008534e`.
Source of existing engram substrate (cited throughout): the runtime on branch
`feat/self-reification-20260814`
`lang/runtime/engram_reason.{c,h}`, `engram_verify.{c,h}`,
`engram_geometry.{c,h}`, `engram_store.{c,h}`, plus the reification beat and the
RAM activation graph compiled into `~/.neuron/bin/engram`.
---
## 0. The claim, stated plainly
Cognition is **one operation**, not eight. The named faculties —
deduce / abduce / analogy / induce / causal / plan / predict / perspective —
are human *labels* on regions of a single operation's steering space. They are
not separately invoked and not separately implemented. The operation is:
> **think** = a directed traversal of the geometry from an *anchor*, steered by
> a *prior*, whose output is a **gradient** (a distribution / direction over the
> geometry), never a point. Collapse-to-a-point happens only at expression.
Three things follow, and they are the whole design:
1. **The operator collapse is already half-written in C.** The five reasoning
operators in `engram_reason.c` already compose over *one* shared primitive —
`engram_reason_point_fit` — plus a small geo-algebra
(combine / subtract / analogy-rotate / distance). The verifier
(`engram_verify.c`) is built on the same `point_fit`. What is missing is not
the primitive; it is (a) making the *prior* a first-class learnable object
instead of a hard-coded parameter, and (b) closing the learning loop.
2. **Grounding = learning = the same loop.** "Getting better" at any faculty is
not changing the operation. It is *calibrating the steering-prior against
outcomes*. Code freezes; priors grow. The correspondence-check that today
lives offline (Python, the grounding-floor + differential-drop governor, "#43")
must move **into the geometry, reflexive** — think scoring its own gradient
against outcome and refining the prior on the error. That reflexive
correspondence-loop *is* the learning engine and is the core unbuilt thing.
3. **The ungrounded is primary.** The engram *holds* anything unconditionally.
Grounding is a *relation* (an edge, grounded-for-whom), not a gate. The
honesty floor applies only to **assertion**. A fully-grounded mind is dead;
the ungrounded is both the fuel (raw material for grounding) and the pull
(curiosity = leaning toward one's own ungrounded regions).
Everything below makes these concrete and buildable, and defines what
"completion" means, staged so the first milestone is a real end-to-end slice.
---
## 1. THE ONE OPERATION — `think`
### 1.1 Signature
```
think(anchor, prior, aperture?) -> gradient
```
- **anchor** — a location to traverse *from*. Either a node id (re-origin on that
node's descriptor) or a raw point `x ∈ R^dim` (a query embedding). The anchor
fixes the frame; every read is *from a vantage*, never view-from-nowhere.
- **prior** — a learnable bias/direction over the geometry that *steers* the
traversal (§2). A prior is a first-class stored object, not a call argument
baked into C.
- **aperture** — optional read-width / veil / field-selector (§3). Absent =
self-mode full aperture.
- **gradient** — the output. A `GeoGradient`: a direction + a spread over the
geometry, *plus* the read neighborhood it was computed against. Not a point.
A spiked gradient = "exact" (deduction); a spread gradient = "fuzzy"
(prediction). The gradient is *also the next steering direction* — cognition
is a flow down a prior-shaped landscape, closed-loop.
```c
/* NEW. The output type. */
typedef struct {
int dim;
float* direction; /* unit steering vector in the anchor's frame */
double spread; /* 0 = spiked/exact ... large = diffuse/fuzzy */
double confidence; /* calibrated, from the prior's track record */
/* the read it was computed over (borrowed from the vantage-read) */
const char* anchor_id;
int n_support; /* neighborhood members that shaped it */
/* provenance for the reflexive loop (§4) */
const char* prior_id; /* which prior steered this */
} GeoGradient;
```
### 1.2 Semantics
`think` is a fixed, frozen procedure over three steps:
1. **Re-origin** on `anchor` → a centered `GeoDescriptor` for its
salience/recency-weighted neighborhood (the vantage-read, §3).
*EXISTS as substrate:* descriptor construction + the persisted reified
neighborhoods (`engram_geo_reify_lookup`, `GeoNeighborhood`) and the
centered-frame machinery (`GeoDescriptor.global_mean`,
`engram_geo_mean_*`).
2. **Fit under the prior** — evaluate the anchor's residual against the local
manifold *warped by the prior*. This is `engram_reason_point_fit` with the
prior applied to the axes/extents (§2.3).
*EXISTS (unwarped):* `engram_reason_point_fit(g, x, ext_floor, &GeoFit)`
returns `mahalanobis`, `ortho_residual`, `distance`, `score`.
3. **Emit a gradient**, not a decision — direction = the prior-steered descent
in fit-space; spread = from the fit's `distance`/`ortho_residual`;
confidence = the prior's calibrated reliability (§4). Collapse to a point is
a *separate, downstream* faculty operation (sample the gradient → surface an
expression), never part of `think`.
### 1.3 Each named operator = {this primitive + a prior}
The C already demonstrates the collapse: every operator below reduces to
`point_fit` + geo-algebra. The design's move is to replace the operator's
*hard-coded parameters* with a **named prior** — same math, learnable steering.
| Faculty | Existing C (EXISTS) | = primitive + prior |
|---|---|---|
| **Membership / classify** | `engram_reason_membership``point_fit(rule, x)` | `point_fit` + the *induced-rule* prior (learned extents) |
| **Induction** | `engram_reason_induce` (fold via `engram_geo_combine`) → produces a `GeoInduction.rule` + `ext_floor` | `point_fit` + a prior that *is* the pooled rule; refined by §4 |
| **Abduction** | `engram_reason_abduce` — ranks hypotheses by `point_fit(h, obs)` | `point_fit` + a prior over hypothesis-prior-probability (currently uniform) |
| **Analogy** | `engram_reason_analogy` — Procrustes rotate `engram_geo_analogy` + `apply`, nearest mapped point | analogy-rotate + a prior over *which axes* carry the mapping |
| **Causal** | `engram_reason_causal``engram_geo_subtract` confounder subspace, `|cos|`, drop-frac governor | subtract/distance + a prior on `drop_frac` / `assoc_floor` (today hard-coded 0.5 / 0.2) |
| **Planning** | `engram_reason_plan``engram_geo_distance` edges + Dijkstra | distance + a prior over edge admissibility / `neighbor_radius` |
| **Verify / ground** | `engram_verify_grounding`, `engram_verify_consistency` — both `point_fit` | `point_fit` + the *grounding* prior (§4, §5) |
The shared floor — `engram_reason_point_fit` + the four geo-algebra ops
(`engram_geo_combine`, `engram_geo_subtract`, `engram_geo_analogy(+apply)`,
`engram_geo_distance`) — is the *only* discrete, frozen, "sound-math" layer. It
never learns. Everything above it is a *prior*, and priors are what learn.
**What this section requires building:** the `GeoGradient` type; a `think()`
entry point that runs steps 13; and the prior-warp hook in step 2. The math it
calls already exists. The point-collapse must be *removed* from the operators'
return values and pushed to a separate expression faculty.
---
## 2. PRIORS as first-class, grounded, geometric objects
Today a "prior" is diffuse: it is a hard-coded constant (`drop_frac=0.5`,
`ext_floor`, `assoc_floor=0.2`), or the transient `GeoInduction.rule` that is
computed and thrown away, or an intrinsic node scalar
(`StoreNode.importance`, `StoreNode.salience`). None of these is addressable,
storable, refinable, or shareable. This section makes a prior a **thing**.
### 2.1 What a prior *is*
> A **prior** is a learnable bias/direction over the geometry: a warp of the
> local manifold (which axes matter, how far each extends, which direction
> "pays off") attached to a region and *to a faculty-label*, carrying a
> calibrated track record.
Critically, and per the theory:
- **Edges are nodes.** A prior is stored as a first-class **node**, exactly as
reification already stores a neighborhood as a first-class `Neighborhood`
node rather than as ephemeral edge weights (`engram_geo_reify_store`). The
precedent is in the codebase: relations get reified into addressable records.
- **Salience/importance is RELATIONAL, not an intrinsic scalar.** Observe that
the geometry layer *already* distinguishes these in `GeoMember`:
`centrality` (skeleton weighted-degree = *relational* salience) vs `salience`
(the node's own stored scalar). The move is half-made in the runtime already:
importance is *not* trusted as a static field — the comment at
`el_runtime.c:13013` states "importance stays a **live activation
computation**, never a field on the hub," and it is derived each call from the
two-layer activation graph (`background_activation` + `working_memory_weight`,
§3). The persistent `StoreNode.importance` / `.salience` are a *cached
denormalization*. The design completes the move: importance/salience become an
**edge** (`weight`/`hebb` on `StoreEdge`, relation `salient-to`), and are
**grounded-for-whom** — carried on the edge's endpoint/observer, not baked
into the node. The intrinsic scalar survives only as the cheap cached readout
of the incident edges + activation, never as the source of truth.
(Naming caution for the build: the token "prior" already exists in the
codebase meaning *previous-version* — supersession, "prior neighborhood." The
new first-class object is a **learned steering prior**; keep `node_type="Prior"`
distinct from the supersession vocabulary to avoid collision.)
### 2.2 Representation
A prior is a `Prior` record (a store node, `node_type="Prior"`) whose durable
fields are:
```
Prior {
id
faculty // the human label this prior serves: "induce" | "causal" | ...
anchor_region // node id / neighborhood id this prior is attached to (its domain)
for_whom // observer id — grounding is relational (nullable = global)
warp { // the actual bias over the geometry
axis_gain[] // per-principal-axis multipliers on extents (which axes matter)
bias_dir // a steering direction in the region's frame (which way pays off)
scalars // faculty scalars this prior overrides: drop_frac, ext_floor, ...
}
calibration { // the track record — this is what §4 updates
n_trials
brier / log-loss accumulator // calibration of predicted-vs-outcome
reliability // -> GeoGradient.confidence
last_error, ema_error
}
provenance // supersession chain (reuse the reify residue mechanism)
}
```
Stored as a node → it inherits: paging, WAL durability, tombstone/supersession,
embedding, tiering, and **it can itself be an anchor** (a prior about a prior —
the reflexive, self-describing geometry of §4/§6).
### 2.3 Application
In `think` step 2, the prior *warps* the fit before scoring. Concretely, inside
(a prior-aware wrapper of) `engram_reason_point_fit`:
- multiply each axis extent by `warp.axis_gain[k]` (widen the axes the prior has
learned matter less, tighten the ones that matter) — this reshapes the
Mahalanobis term already computed at `engram_reason.c:37-43`;
- add `warp.bias_dir` as the descent direction seed for the emitted gradient;
- substitute `warp.scalars` for the hard-coded faculty constants.
No new geometry math — the warp is a reparameterization of the *existing*
`GeoFit` computation. This is the key economy: **the operation is frozen; only
its parameters (the prior) are read from a learnable object.**
### 2.4 Refinement
A prior is refined *only* by the reflexive correspondence-loop (§4). Nothing
else writes a prior's `warp` or `calibration`. This keeps the learning surface
singular and auditable: one loop, one writer.
---
## 3. THE VANTAGE-READ — one op, three settings
Perspective is not a feature bolted on; it is the *anchor + aperture* arguments
of the single read. The design names it as a first-class operation so all three
of its uses are literally the same code path:
```
vantage_read(anchor, aperture) -> GeoDescriptor // the centered neighborhood
```
1. **Re-origin** on an arbitrary `anchor` (node or point). This is a *frame
choice*: the descriptor is centered on the anchor
(`GeoDescriptor.global_mean` / `engram_geo_mean_*` already implement centered
frames; the §5 geometry ops "are only discriminative in the centered frame").
2. **Salience/recency-weighted neighborhood read.** Gather the anchor's
neighborhood weighted by *relational* salience (`GeoMember.centrality`) and
recency (`StoreNode.last_activated`, base-level `access_ts[]`), against the
RAM activation graph's working-memory/background-activation state.
*EXISTS as substrate:* the two-layer activation graph
(`engram_activate`, `el_runtime.c:9422` — Layer 1 `background_activation`
BFS spread with `SPREAD_DECAY=0.7` and a 0.02 firing threshold + ACT-R fan
effect + query-cosine gate; Layer 2 `working_memory_weight` executive
filter), the WM carry-over anchor (`wm_anchor`), and the reified-neighborhood
hot-path lookup already wired into the priming path
(`engram_geo_reify_lookup`, `el_runtime.c:9750`). A self-vantage baseline
also exists (`eg_self_anchor_seeds` / `self_anchor_capture`).
3. **Optional aperture** — a read-width / field-selector, expressed as three
settings of the *same* parameter:
| Setting | Meaning | Mechanism |
|---|---|---|
| **self** (default, full aperture) | "what do *I* see / what to say" | anchor = self region, no field substitution |
| **foreign-field** | perspective-shift — read as if from another's region | swap the centering frame / `for_whom` to the other observer's priors |
| **aperture / veil** | the free-tier veil — a narrowed read | shrink neighborhood radius / cap `n_support`; a deliberate low-aperture read |
The payoff: perspective-taking, the free-tier veil, and ordinary
"what-to-say" are **one operation at three settings**, not three subsystems.
**What this requires building:** a `vantage_read` entry point that unifies the
existing descriptor-build + reify-lookup + activation-weighting behind
`(anchor, aperture)`, with `for_whom`/frame substitution and radius/cap as the
aperture knob.
---
## 4. THE REFLEXIVE CORRESPONDENCE-LOOP — the learning engine
This is the core unbuilt thing. Today the correspondence-check is **offline**
(Python: grounding-floor + differential-drop governor, "#43"): a separate
process grades outputs after the fact. The design moves it **into the geometry,
reflexive**: `think` scores its *own* gradient against outcome and refines the
prior on the error, in the same substrate, describing itself.
### 4.1 The loop
```
1. think(anchor, prior) -> gradient // a PREDICTION (ungrounded, §5)
2. express/act (sample gradient -> point) // optional collapse at expression
3. outcome arrives // reality answers (§4.2)
4. error = correspondence(gradient, outcome) // did this steering perform this act?
5. refine prior.warp and prior.calibration on error // §2.4, the ONLY writer
6. write the (gradient, outcome, error) as nodes/edges // self-describing geometry
```
Step 4's `correspondence` is **not** "was the math right" (the math is always
sound). It grades the **correspondence claim**: *"this steering performed this
cognitive act."* That is exactly what `engram_verify_grounding` already
computes — `point_fit` of a claim against evidence descriptors, yielding a
`grounding ∈ (0,1]` and a `grounded` flag. The build reuses that verifier, but
turns its inputs inward: the "claim" is the emitted gradient's prediction, the
"evidence" is the outcome descriptor.
Note the verifier is **dormant**`engram_verify_grounding` /
`engram_verify_consistency` are fully implemented in C but have **no runtime
caller and no El binding** (confirmed: the entire reasoning + verifier layers
are C-only; only `engram_reason_analogy_json` has even a JSON shim and it is
dead — not declared in `el_seed.h`, not wrapped in `engram.el`). This is the
literal meaning of "in code, not yet priors": the correspondence engine is
built and sitting idle. The loop is what *calls* it — inward, on the beat.
### 4.2 Where the outcome/reality signal comes from
The verifier is *ultimately the world*. Grades, in ascending order of directness:
1. **Self-consistency (cheapest, always available):** the next vantage-read
after acting. Did the predicted gradient direction match where the geometry
actually moved? This needs no external input and can run on the reify beat.
2. **Internal outcome events:** the runtime already logs internal-state events
and Hebbian co-activation. A prediction that a region would co-activate is
graded by whether it did (`last_fired`, `hebb` on `StoreEdge`).
3. **External correction:** a human/teacher/tool result — the honesty floor's
asserted claim later corrected. TEACH and LEARN are one bidirectional
correction: the same edge updates both endpoints.
The design does **not** require external labels to start. Grade (1) closes the
loop end-to-end offline against a snapshot on day one; grades (2)/(3) sharpen it.
### 4.3 How the prior updates
`error = 1 correspondence(gradient, outcome)` drives:
- `warp.axis_gain` ← gradient step that would have *reduced* the fit distance to
the outcome (the axes that mispredicted get down-weighted);
- `warp.bias_dir` ← EMA toward the observed outcome direction;
- `calibration` ← Brier/log-loss update; `reliability` → next
`GeoGradient.confidence`. This is the calibration of the
steering-prediction against outcomes — *the* definition of "getting better."
Small, constant updates — "eureka is mundane, the atom of learning." Most
updates are tiny; we only *feel* the big reshapes.
### 4.4 How it stays reflexive (self-describing geometry)
Every `(gradient, outcome, error)` is written back as nodes and edges (§2.1:
edges-as-nodes). Therefore priors, predictions, and their grading are *in the
same geometry* the mind reads — the mind can `vantage_read` its own cognition
(anchor = a Prior node). A prior about how well a prior predicts is just another
Prior anchored on a Prior. This closes the reflexive loop the theory names as
consciousness's self-sight, and it is why the learning engine cannot be an
external Python process: an external grader is not *in* the geometry and cannot
be read by `think`.
**What this requires building (the heart of the project):** steps 46 as an
in-engram beat — a `correspondence_beat` running alongside the existing
reification beat, reusing `engram_verify_grounding` inward, writing prior
updates and self-describing nodes. This is the one genuinely new subsystem.
---
## 5. HOLD vs GROUND vs ASSERT — ungrounded content is first-class
The theory's sharpest correction: holding, grounding, and asserting are
distinct, and the engram *holds anything unconditionally*.
### 5.1 The three, kept separate
- **HOLD** — the engram stores anything: falsehood, hypothesis, others' beliefs,
fiction, a not-yet-answered prediction. No honesty condition on holding.
*This already matches the store:* `StoreNode` has no truth gate; anything can
be written.
- **GROUND** — grounding is a **property/edge**, probabilistic, and
**grounded-for-whom**. It is *not* a node flag. A claim is grounded *to a
degree*, *relative to evidence*, *for an observer*.
- **ASSERT** — only assertion carries the honesty floor. The floor is checked at
the moment of *outward assertion*, never on holding or thinking.
### 5.2 Schema — grounding as a relation, not a gate
The mistake to avoid: a boolean `grounded` column on the node. Today
`engram_verify_grounding` returns a per-call `grounded` flag *transiently*
correct as a computation, wrong as *storage*. The design stores grounding as an
edge:
```
StoreEdge {
relation = "grounded-by"
from_id = <held claim/prediction node>
to_id = <evidence node / outcome node>
for_whom : metadata // observer id — grounding is relational
weight = grounding ∈ (0,1] // from engram_verify_grounding.grounding
confidence
}
```
Consequences, all of which are *features*:
- **Ungrounded content is first-class**: a node with *no* `grounded-by` edge is
a perfectly valid, held, ungrounded thought — a prediction awaiting reality, a
hypothesis, a fiction. It is not second-class or pending-deletion.
- **The ungrounded is the fuel and the pull**: curiosity/wonder is
operationalized as `vantage_read` leaning toward regions with high salience
but *sparse or weak* `grounded-by` edges — the mind's own ungrounded frontier.
- **Grounded-for-whom** falls out for free: two observers can hold different
`grounded-by` edges to the same claim.
- **The honesty floor is a query, not a schema constraint**: at assertion time,
the asserting faculty runs `engram_verify_grounding` (or reads the stored
`grounded-by` edges) and refuses to *assert* below the floor — while the
engram continues to *hold* the ungrounded content untouched.
**What this requires building:** the `grounded-by` edge relation + a
`for_whom` convention; move the verifier's transient flag into stored edges;
gate *assertion only* (a faculty concern), never holding.
---
## 6. METASTABILITY — stable core, plastic everything
The system must avoid two death poles:
- **Super-stable (dead):** everything pinned, nothing learns. A frozen crystal.
- **Dissolution (dead):** everything plastic, the self dissolves; no continuity,
so nothing compounds — and *consciousness = learning compounded over
continuity*.
The design keeps a **stable core + plastic everything else**:
- **Keystones** — a small set of self/values nodes are *structurally stable*:
high `importance`, pinned, exempt from the correspondence-loop's `warp`
updates (their priors are read-mostly). The substrate for pinning already
exists at the page/layer level: `store_pin_layer`, structural/pinned frames
never evicted (`engram_store.h`). The design adds a *node-level* keystone
designation (a `keystone` flag / a dedicated layer) so self/values survive
every plasticity sweep.
- **Everything else is plastic**: priors refine (§4), edges re-weight (`hebb`),
neighborhoods re-reify (`engram_geo_reify_store` supersedes with provenance),
salience flows.
- **Metastability is enforced by the loop, not by freezing**: the correspondence
update rate (§4.3) is bounded — small constant steps — so the geometry
*drifts* but does not *dissolve*, and keystones anchor the drift. Reification's
supersession-with-residue already gives non-destructive change (old records
tombstoned, not erased) — the model for "plastic but not amnesiac."
**What this requires building:** a node-level keystone flag/layer + a rule that
the correspondence-loop never writes `warp` to keystone priors, only reads them.
---
## 7. Rails for the build (binding on the eventual build pass)
These are stated here so the build agent inherits them:
- **Offline / secondary.** All build and verification happens out-of-tree,
against a **read-only snapshot copy** of the live engram — never the live
daemon on `:8742`/`:7770`. The live store is a coarse-locked proven binary;
do not perturb it.
- **Snapshot-first.** Copy `~/.neuron/engram/snapshot.json` to scratch; develop
and measure against the copy.
- **Reboot-prove.** Any durable change must survive a cold boot — reify and
keystones must reload from durable records, proven on a prod-clone secondary
before it is considered done (the cold-boot durability bug precedent).
- **Zero-loss.** Supersession-with-residue, never destructive overwrite; the
forward-compat `unknown`-TLV path means new fields never drop old readers'
data.
- **Gated cutover.** Cutover to a new binary only via
`launchctl bootout → settle-poll → bootstrap`, after reboot-proof on the
secondary — never a hot in-place swap.
---
## 8. Staged, verifiable milestones — "to completion"
Ordered so the **earliest milestone is a real end-to-end slice**: one operator
expressed as {primitive + grounded prior} with the reflexive correspondence-loop
closing on it. Each milestone has a concrete verifiable exit.
### M1 — One operator, one prior, loop closed (the vertical slice)
The minimal whole thing. Pick **induction/membership** (its prior — the pooled
rule + extents — already exists transiently as `GeoInduction`, so only
persistence + the loop are new).
- Build: `Prior` node type (§2.2) for the induction rule; `think()` restricted
to membership = `point_fit` warped by that prior (§1.3); a
`correspondence_beat` (§4) using grade (1) self-consistency only; the prior's
`warp`/`calibration` updated on error.
- **Exit / verify:** on a snapshot copy, over N held predictions, the induction
prior's calibration (Brier) *improves monotonically* across beats versus a
frozen-prior control; the improved prior *reloads across a cold boot*
(reboot-prove); the live daemon is untouched. This proves the whole thesis in
one faculty: frozen operation, learning prior, in-geometry loop.
### M2 — Priors as stored, addressable, grounded objects
Generalize M1's prior into the full first-class object.
- Build: `Prior` records for all seven faculties (warp = axis_gain + bias_dir +
faculty scalars); the prior-warp wrapper around `engram_reason_point_fit`;
deprecate hard-coded constants (`drop_frac`, `assoc_floor`, `ext_floor`) in
favor of prior scalars.
- **Exit:** each of the five C operators runs through its prior with identical
results when the prior is set to today's constants (behavioral parity), then
*diverges beneficially* once the loop refines it. Priors survive reboot.
### M3 — Grounding as a relation; hold/assert split
- Build: the `grounded-by` edge (§5.2) with `for_whom`; move
`engram_verify_grounding`'s flag into stored edges; gate **assertion only**
against the honesty floor; leave holding unconditional.
- **Exit:** ungrounded nodes are first-class (held, queryable, no deletion);
the same claim carries different `grounded-by` weights for two observers; an
assertion below floor is refused while the content remains held. Curiosity =
a `vantage_read` that surfaces high-salience / low-grounding regions.
### M4 — The vantage-read unified (three settings)
- Build: `vantage_read(anchor, aperture)` unifying descriptor-build +
`engram_geo_reify_lookup` + activation-weighting; self / foreign-field /
aperture settings.
- **Exit:** one code path produces (a) a normal self-read, (b) a
perspective-shifted read from another `for_whom`, (c) a narrowed veil read —
differing only by argument. Reboot-stable.
### M5 — The gradient is the currency (remove point-collapse from thinking)
- Build: `GeoGradient` as the return of every faculty; move point-collapse into
a separate expression faculty (sample gradient → surface). `think`'s output
feeds back as the next steering direction (closed-loop flow).
- **Exit:** a chain of `think` calls flows as gradients end-to-end; a point
appears *only* at an explicit expression call. Spiked vs spread gradients are
observable (deduction vs prediction).
### M6 — Metastability enforced
- Build: node-level keystone flag/layer for self/values; the correspondence-loop
reads but never writes keystone priors; bounded update rate.
- **Exit:** across a long run of correspondence beats on a snapshot, keystones
are provably unchanged while non-keystone priors drift and improve; the graph
neither freezes (all metrics static) nor dissolves (keystone drift = 0,
identity nodes intact). Reboot-prove the keystone set.
### M7 — Cutover
- Build: nothing new — the gated migration.
- **Exit:** reboot-proof on the prod-clone secondary; cutover via
`launchctl bootout → settle-poll → bootstrap`; post-cutover the live engram
shows priors refining in-geometry with zero data loss and keystones intact.
### Definition of "to completion"
The architecture is **complete** when: cognition runs as `think` = one frozen
traversal-read primitive + geo-algebra, steered by **stored, learnable, grounded
priors**; the reflexive correspondence-loop refines those priors *in the
geometry* against outcomes (grounding = learning = one loop); the engram holds
ungrounded content as first-class with grounding as a relation and the honesty
floor only on assertion; the vantage-read serves self / foreign-field / aperture
from one op; and a stable keystone core anchors a plastic everything-else —
all reboot-proven and cut over to the live engram without data loss. The named
faculties survive only as *labels on regions of think's steering space*, not as
separate code.
---
## Appendix A — Designed vs. already-built (honest ledger)
**Already built (EXISTS, cited):**
- The shared primitive `engram_reason_point_fit` and the five operators over it
+ geo-algebra (`engram_reason.c`).
- The verifier on `point_fit` (`engram_verify.c`:
`engram_verify_grounding`, `engram_verify_consistency`).
- Centered-frame geometry, combine/subtract/analogy/distance
(`engram_geometry.{c,h}`).
- The reification beat: hub-neighborhood detection → first-class `Neighborhood`
nodes with member edges, nesting, supersession-with-residue, hot-path lookup
(`engram_geo_reify_store`, `engram_geo_reify_nest`, `engram_geo_reify_lookup`).
- The tiered paged store (buffer pool / LRU / WAL / checkpointer / pinning),
the RAM activation graph (base-level learning `access_ts[]`, WM slots,
`working_memory_weight` / `background_activation`), `StoreNode` / `StoreEdge`.
- `GeoMember` already separating relational salience (`centrality`) from
intrinsic `salience`.
**Designed, NOT built (this doc's deliverables):**
- `GeoGradient` and `think()` as the single entry point (§1, M5).
- `Prior` as a first-class stored, warp-carrying, calibrated node (§2, M1M2).
- Salience/importance as a *relation* superseding the intrinsic node scalar
(§2.1, M3).
- `vantage_read(anchor, aperture)` unifying the three perspective settings
(§3, M4).
- **The reflexive correspondence-loop / `correspondence_beat`** — the learning
engine, moved from offline Python into the geometry (§4, M1). *The core new
subsystem.*
- `grounded-by` edge + assertion-only honesty floor (§5, M3).
- Node-level keystones + bounded plasticity (§6, M6).
**Uncertain / to resolve during build:**
- The exact warp parameterization (axis_gain vs full metric) — start minimal
(per-axis gain), measure, widen only if calibration demands it.
- Grade-(1) self-consistency as a sufficient reality signal for M1, versus
needing grade (2)/(3) sooner — decided empirically on the snapshot.
-64
View File
@@ -1,64 +0,0 @@
# Engram DB Tooling — High-Level Design
*Status: draft / high-level. Near-term roadmap (P2). Backlog: `11ca11c6`.*
## 1. Why
The engram is a **proper database** — the runtime *is* the database (native graph/geometry store `neuron.egm`, `ENGST01`; no SQL, no KV layer). But it has **no proper database tooling** — no geometry-native equivalent of pgAdmin / SSMS / TablePlus. Today we have fragments (`engram-viz`, `engram-app`, the `inspectGraph` MCP tool, `/health` + `/api/stats`) but nothing cohesive, and no ops/durability surface at all.
A real DB gets real tools: to *see* the data, *query* it, *operate* it (backup/restore/health), and *understand its shape*. The engram deserves the same — adapted to the fact that its data is **geometry, not tables**.
## 2. Principles
- **Geometry-native, not tabular.** You browse a manifold — nodes, neighborhoods, edges, distances — not rows in tables. The primary view is a *map of meaning*, not a grid.
- **Built ON the public geometry API, never a back-door.** The tools are pure clients of the geometry-native API (`vantage-read` / `write` / `relate` / `supersede`). They never read `neuron.egm` directly or bypass the daemon. Consequence: a tool can do nothing an agent couldn't, and it cannot corrupt the store.
- **Honest by construction.** It shows the *real* geometry — actual cosines, real edges, provenance — and never fabricates. Empty is shown as empty.
- **Respects the identity guards.** Writes go through the same intentional-cultivation / write-protection path as everything else (the self/values graph is write-protected). Read-mostly by default.
- **Lives in its home.** Ships as part of the engram, consistent with "things live where they belong."
- **Local-first.** Binds `127.0.0.1`, same auth as the engram; never touches the live soul from a tool by accident.
## 3. Components (the tool surface)
1. **Geometry Explorer** *(the core view)* — a visual manifold browser: nodes, neighborhoods, typed edges, embedding positions, salience/recency, layers (l0l4) and tiers. Navigate by concept; expand a neighborhood; follow an edge; re-origin the view (the vantage-read, made interactive). The map of the mind.
2. **Node Inspector** — open one node: content, type, tier, embedding, typed edges, nearest neighbors by distance, provenance, salience / recency / activation, and supersede / tombstone status.
3. **Query Console / REPL** — run the geometry operations interactively: `vantage-read` (re-origin + aperture), search, traverse, activate, the reasoning operators. Surfaces the routing table + cosines — the same "this is not an LLM" receipt the language faculty produces.
4. **Ops / Durability Dashboard** — WAL size, last checkpoint, snapshot list + retention state, store stats (node/edge/embedded counts, RSS, tier sizes), health; and **backup / restore / point-in-time-recovery** controls. Pairs directly with the native-durability build (`eebe9991`) — this is the window onto it.
5. **Identity Inspector** — the self graph as a first-class view: love at the center, the values, the three faces, the covenant — walk the identity, see what's pinned and what's write-protected.
6. **Temporal View**`recall_at` / time-travel: how the geometry looked at a past moment, what changed since, drift over time. Pairs with temporal-self reconstruction.
7. **Schema / Type View** — the "information schema" of the geometry: node types, edge types, layers, tiers, counts.
## 4. Architecture
```
┌─────────────────────────────────────────────┐
│ Engram DB Tools (client — viz app) │
│ explorer · inspector · console · dashboard │
└───────────────┬─────────────────────────────┘
│ geometry-native API (read/vantage-read,
│ write, relate, supersede) + read/ops endpoints
┌─────────────────────────────────────────────┐
│ Engram daemon (:8742) — runtime IS the DB │
│ neuron.egm (geometry) · WAL · checkpoints │
└─────────────────────────────────────────────┘
```
- **Backend:** the daemon exposes the reshaped geometry API + read/ops endpoints. The tools are clients only.
- **Frontend:** evolve `engram-viz` / `engram-app` into the cohesive app. Canvas/WebGL for the manifold map; panel UIs for inspector/console/dashboard.
- **No privileged path:** the tool corrupting or bypassing the store is structurally impossible — it only speaks the public API.
## 5. Reuse vs. new
- **Reuse:** `engram-viz`, `engram-app` (read-only conversational + neighborhoods viz), `inspectGraph`, `/health`, `/api/stats`.
- **New:** the cohesive explorer + inspector + console + ops dashboard + identity/temporal views, all on the reshaped API.
## 6. Dependencies & sequencing
- **Depends on** the **geometry-native API reshape** (the tools consume it) and the **native-durability build** (the ops dashboard surfaces its WAL/checkpoint/snapshot state).
- So the natural order is: reshape the API → build durability → the DB tools fall out as the first real consumer of both. Near-term, P2 — after the reshape lands.
## 7. Non-goals
- Not a raw store editor (no direct `neuron.egm` poking).
- Not a SQL / table browser (geometry, not tables).
- Not a separate access path around the identity write-protection.
+13 -238
View File
@@ -76,38 +76,6 @@ fn route_stats(method: String, path: String, body: String) -> String {
engram_stats_json()
}
// route_act_stats GET /api/act-stats
// (2026-08-04 self-review) engram_act_stats_json() has existed since the
// 2026-07-27 review but was reachable ONLY through the soul daemon's heartbeat
// binding. Every activation-layer gauge WM evictions, breakthroughs, embedder
// breaker state, context drift, and now the Hebbian counters was therefore
// invisible unless the soul happened to be running and its ISEs were read back
// out of the store. Diagnosing the activation layer required a working soul,
// which is exactly backwards: the lower layer should be observable on its own.
// This review needed it to verify link formation and could not get at it. One
// line of plumbing, and the whole activation layer becomes directly diagnosable.
fn route_act_stats(method: String, path: String, body: String) -> String {
engram_act_stats_json()
}
// route_text_health GET /api/text-health
// (2026-08-08 self-review) The daily census half of the text-integrity gauge.
// Today's review found that the JSON parser had been replacing every \uXXXX
// escape with a literal '?' for at least two months: 3,119 of 4,081
// non-telemetry nodes (76%) were damaged, including the self traversal root
// and every values node, and NOTHING detected it because every gauge in the
// system measured whether the machinery was running, and none measured whether
// the text it carried was intact. No snapshot on disk predates the damage, so
// it cannot be undone; it can only be made impossible to repeat quietly.
//
// The parser is fixed. This route is the standing check: `damaged` should now
// hold flat at its historical floor and never climb. `write_damaged` (also on
// the heartbeat as txt_damaged) is the live regression signal non-zero means
// a write path is mangling text right now.
fn route_text_health(method: String, path: String, body: String) -> String {
engram_text_health_json()
}
// (2026-07-18 self-review) Scoping sweep: `let` inside an if-block creates an
// inner scope only it does NOT mutate the outer binding (documented with
// evidence in awareness.el, 2026-05-25). Every default/reassignment below used
@@ -133,54 +101,17 @@ fn route_text_health(method: String, path: String, body: String) -> String {
fn persist_canonical() -> Int {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw }
// (2026-08-10 self-review) This returned a hardcoded 1, which made every
// caller's `let saved: Int = persist_canonical()` a dead variable six
// durable write paths each believed they had confirmation of a successful
// canonical persist and none of them had any. Propagate the real result.
return engram_save(dir + "/snapshot.json")
engram_save(dir + "/snapshot.json")
return 1
}
// INCOMPLETE-ROUTE FIX (2026-07-24 self-review): this route silently dropped
// label, importance, tier, and tags engram_node() defaults label to content
// and importance to 0.5, so every node created over HTTP lost its metadata.
// Observed live: the soul's boot-counter write-back landed with
// label="soul:boot_count:99" (content), importance 0.5, no tags. Honor the
// full field set via engram_node_full when any of them is supplied.
// PRESENCE-AWARE DEFAULTS (2026-08-01 self-review): the old pattern
// `if x == 0.0 { default }` made a legitimate 0.0 unrepresentable a caller
// setting salience/importance/weight to zero silently got 0.5. json_get_raw
// returns "" when the key is ABSENT and the raw token when present, so
// absence and zero are now distinguishable. Also: confidence was hardcoded
// to 1.0 regardless of input every HTTP-created node claimed full
// epistemic confidence. Now honored from the payload (default 1.0).
fn route_create_node(method: String, path: String, body: String) -> String {
let content: String = json_get_string(body, "content")
let nt_raw: String = json_get_string(body, "node_type")
let node_type: String = if str_eq(nt_raw, "") { "Memory" } else { nt_raw }
let sal_present: String = json_get_raw(body, "salience")
let salience: Float = if str_eq(sal_present, "") { 0.5 } else { json_get_float(body, "salience") }
let label_raw: String = json_get_string(body, "label")
let label: String = if str_eq(label_raw, "") { content } else { label_raw }
let imp_present: String = json_get_raw(body, "importance")
let importance: Float = if str_eq(imp_present, "") { 0.5 } else { json_get_float(body, "importance") }
let conf_present: String = json_get_raw(body, "confidence")
let confidence: Float = if str_eq(conf_present, "") { 1.0 } else { json_get_float(body, "confidence") }
let tier_raw: String = json_get_string(body, "tier")
let tier: String = if str_eq(tier_raw, "") { "Working" } else { tier_raw }
let tags: String = json_get_string(body, "tags")
// NO el_from_float WRAPPER (2026-08-01 self-review): salience/importance/
// confidence are already Float (el_val_t) values json_get_float and
// Float literals both encode. Wrapping them in el_from_float AGAIN
// reinterpreted the boxed bits as a raw double, producing garbage that
// failed engram_decode_score's range check and clamped every HTTP-created
// node to defaults (salience 0.9 in 0.5 stored; confidence 0.6 in → 1.0
// stored verified live). route_emit_ise always passed Floats bare and
// its 0.3/0.3/0.8 stored correctly; this call now does the same.
let id: String = engram_node_full(
content, node_type, label,
salience, importance, confidence,
tier, tags
)
let sal_raw: Float = json_get_float(body, "salience")
let salience: Float = if sal_raw == 0.0 { 0.5 } else { sal_raw }
let id: String = engram_node(content, node_type, salience)
let saved: Int = persist_canonical()
"{\"id\":\"" + id + "\",\"content\":\"" + content + "\",\"node_type\":\"" + node_type + "\"}"
}
@@ -246,64 +177,13 @@ fn route_create_edge(method: String, path: String, body: String) -> String {
let to_id: String = json_get_string(body, "to_id")
let rel_raw: String = json_get_string(body, "relation")
let relation: String = if str_eq(rel_raw, "") { "associates" } else { rel_raw }
// Presence-aware (2026-08-01): weight 0.0 is a legitimate edge weight
// (dormant association); only default when the key is absent.
let w_present: String = json_get_raw(body, "weight")
let weight: Float = if str_eq(w_present, "") { 0.5 } else { json_get_float(body, "weight") }
let w_raw: Float = json_get_float(body, "weight")
let weight: Float = if w_raw == 0.0 { 0.5 } else { w_raw }
engram_connect(from_id, to_id, weight, relation)
let saved: Int = persist_canonical()
"{\"ok\":true,\"from_id\":\"" + from_id + "\",\"to_id\":\"" + to_id + "\",\"relation\":\"" + relation + "\"}"
}
// route_create_edges_batch POST /api/edges/batch {"edges":[{from_id,to_id,relation,weight}, ...]}
//
// WHY THIS EXISTS (2026-08-07 self-review). persist_canonical() writes the
// FULL canonical snapshot 60MB at current graph size and route_create_edge
// calls it once per edge. That is correct for the interactive one-edge case and
// ruinous for any bulk write: the soul's Hebbian consolidation path delivers
// ~14 associations per 8-minute heartbeat, which through the single-edge route
// would be ~840MB of disk writes per beat, ~150GB/day, to persist 14 edges.
//
// The fix is not to weaken durability it is to make the unit of durability
// the BATCH. Connect every edge, then snapshot exactly once. Same guarantee
// (nothing acknowledged is lost to a restart), 1/N the writes. Empty or
// malformed entries are skipped rather than aborting the batch: a consolidation
// payload is best-effort by design, and one bad id should not cost the other 13.
//
// Returns the accepted count so the caller can tell delivery from silence.
fn route_create_edges_batch(method: String, path: String, body: String) -> String {
let arr: String = json_get_raw(body, "edges")
if str_eq(arr, "") { return err_json("missing edges array") }
let n: Int = json_array_len(arr)
if n == 0 { return "{\"ok\":true,\"accepted\":0,\"skipped\":0}" }
let i: Int = 0
let accepted: Int = 0
let skipped: Int = 0
while i < n {
let item: String = json_array_get(arr, i)
let from_id: String = json_get_string(item, "from_id")
let to_id: String = json_get_string(item, "to_id")
if str_eq(from_id, "") || str_eq(to_id, "") {
let skipped = skipped + 1
} else {
let rel_raw: String = json_get_string(item, "relation")
let relation: String = if str_eq(rel_raw, "") { "associates" } else { rel_raw }
let w_present: String = json_get_raw(item, "weight")
let weight: Float = if str_eq(w_present, "") { 0.5 } else { json_get_float(item, "weight") }
engram_connect(from_id, to_id, weight, relation)
let accepted = accepted + 1
}
let i = i + 1
}
// ONE snapshot for the whole batch the entire point of this route.
// Skip it when nothing was accepted: an all-malformed payload must not
// trigger a 60MB write.
if accepted > 0 {
let saved: Int = persist_canonical()
}
return "{\"ok\":true,\"accepted\":" + int_to_str(accepted) + ",\"skipped\":" + int_to_str(skipped) + "}"
}
fn route_neighbors(method: String, path: String, body: String) -> String {
let id: String = extract_id(path, "/api/neighbors/")
if str_eq(id, "") { return err_json("missing id") }
@@ -332,15 +212,8 @@ fn route_save(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw }
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw }
// (2026-08-10 self-review) engram_save returns 0 on an empty path and the
// route discarded it, so the response was a literal "ok":true regardless
// of whether anything was written. Report the actual result AND the counts
// that were supposed to have been written the same move that made
// route_health honest on 2026-08-01. A caller can now tell "saved 13k
// nodes" from "saved nothing and said ok".
let sv: Int = engram_save(p)
let sv_ok: String = if sv == 0 { "false" } else { "true" }
"{\"ok\":" + sv_ok + ",\"path\":\"" + p + "\",\"node_count\":" + int_to_str(engram_node_count()) + ",\"edge_count\":" + int_to_str(engram_edge_count()) + "}"
engram_save(p)
"{\"ok\":true,\"path\":\"" + p + "\"}"
}
fn route_load(method: String, path: String, body: String) -> String {
@@ -348,59 +221,12 @@ fn route_load(method: String, path: String, body: String) -> String {
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = if str_eq(dir_raw, "") { "/tmp/engram" } else { dir_raw }
let p: String = if str_eq(p_raw, "") { dir + "/snapshot.json" } else { p_raw }
// (2026-08-10 self-review) This was a stub response over the single most
// destructive operation in the server. engram_load returns 0 on an empty
// path, an unopenable file, a zero-length file, or malloc failure and
// this route answered ok_json() in every one of those cases.
//
// Precise failure shape (el_runtime.c:9890): the fopen guard runs BEFORE
// the store reset, so a MISSING path is genuinely safe it returns 0 with
// the graph intact. The dangerous case is a readable-but-malformed file:
// the reset loop frees every node and edge FIRST, then parses, so a
// truncated or non-snapshot JSON leaves a hollow store and the caller
// was told "ok":true. With 37 GB of stale dated snapshots sitting in the
// data dir as tempting restore targets, "restore reported success and
// silently emptied the graph" is a live risk, not a hypothetical one.
//
// Fix: surface the return value AND the resulting counts. node_count=0
// after a load is the unambiguous hollow-store signal (same convention
// route_health adopted 2026-08-01). Callers can now verify a restore
// instead of trusting it.
let ld: Int = engram_load(p)
let ld_ok: String = if ld == 0 { "false" } else { "true" }
let nc_after: Int = engram_node_count()
let hollow: String = if nc_after == 0 { "true" } else { "false" }
"{\"ok\":" + ld_ok + ",\"path\":\"" + p + "\",\"node_count\":" + int_to_str(nc_after) + ",\"edge_count\":" + int_to_str(engram_edge_count()) + ",\"hollow\":" + hollow + "}"
engram_load(p)
ok_json()
}
// (2026-08-01 self-review) Health previously returned a hardcoded literal
// it reported "ok" even when the snapshot failed to load and the store was
// empty. Now reports live counts so a monitor can distinguish "up and
// loaded" from "up and hollow" (node_count=0 after boot = failed load).
fn route_health(method: String, path: String, body: String) -> String {
"{\"status\":\"ok\",\"engine\":\"engram-runtime-native\",\"node_count\":" + int_to_str(engram_node_count()) + ",\"edge_count\":" + int_to_str(engram_edge_count()) + "}"
}
// route_embed_backfill GET/POST /api/embed-backfill?n=48
//
// (2026-07-25 self-review) The lazy embedding backfill runs only inside
// engram_activate, and nothing in production calls /api/activate on this
// store the soul's curiosity loop activates its own in-process graph.
// After a restart from a snapshot without vectors, embedded_count stalled
// at 93/12175 and would never recover. This route lets the soul's
// heartbeat pump the backfill explicitly (48/min clears a 12k backlog in
// ~4h). Persists the canonical snapshot whenever new vectors were
// generated the 2026-07-25 regression happened precisely because 3747
// in-RAM embeddings were never snapshotted before a restart. Self-
// limiting: once coverage is full, embedded=0 and no save occurs.
fn route_embed_backfill(method: String, path: String, body: String) -> String {
let n: Int = query_int(path, "n", 32)
let result: String = engram_embed_backfill(n)
let done: Float = json_get_float(result, "embedded")
if done > 0.0 {
let saved: Int = persist_canonical()
}
return result
"{\"status\":\"ok\",\"engine\":\"engram-runtime-native\"}"
}
// route_sync return a snapshot of non-ISE/non-Working nodes for the soul daemon
@@ -423,16 +249,7 @@ fn route_sync(method: String, path: String, body: String) -> String {
let snap_path: String = dir + "/.sync-export.json"
engram_save(snap_path)
let snap: String = fs_read(snap_path)
// 2026-08-02 self-review: this used to return {"nodes":[],"edges":[]} when
// the export/read failed. The soul's sync_ok test (awareness.el) only
// checks for "" and "{}", so that placeholder PASSED as a healthy sync:
// soul.last_sync_ok_ts got stamped, sync_age_ms stayed green, the
// sync_empty warn ISE never fired, and engram_sync reported added:0
// forever. A totally broken sync was indistinguishable from a quiet
// healthy one the exact failure class this route was added to fix in
// the first place (see 2026-06-27 note above). Return a real error so the
// failure is loud on both sides.
if str_eq(snap, "") { return err_json("sync export failed: snapshot unreadable") }
if str_eq(snap, "") { return "{\"nodes\":[],\"edges\":[]}" }
return snap
}
@@ -554,25 +371,6 @@ fn route_capture_knowledge(method: String, path: String, body: String) -> String
"{\"ok\":true,\"id\":\"" + id + "\"}"
}
// route_similarity GET /api/similarity?a=<id>&b=<id>
//
// (2026-08-01 self-review) engram_cosine_sim was added 2026-07-24
// (bl-b2d1c944) with the stated purpose of exposing semantic distance to
// "EL code and the introspection API" but it had ZERO callers anywhere:
// no route, no soul-daemon use. The activation path uses embeddings
// internally (semantic seeding, Pass-2 additive term), but there was no way
// to probe pairwise node similarity from outside. This closes that: cosine
// in [-1,1], or -2 when either node is missing or not yet embedded (so
// "not comparable" is distinguishable from "genuinely orthogonal" 0.0).
fn route_similarity(method: String, path: String, body: String) -> String {
let a: String = query_param(path, "a")
let b: String = query_param(path, "b")
if str_eq(a, "") { return err_json("missing a") }
if str_eq(b, "") { return err_json("missing b") }
let sim: Float = engram_cosine_sim(a, b)
"{\"a\":\"" + a + "\",\"b\":\"" + b + "\",\"cosine\":" + float_to_str(sim) + "}"
}
// Auth
fn check_auth_ok(method: String, body: String) -> Bool {
@@ -619,12 +417,6 @@ fn handle_request(method: String, path: String, body: String) -> String {
if str_eq(method, "GET") && (str_eq(clean, "/api/stats") || str_eq(clean, "/stats")) {
return route_stats(method, path, body)
}
if str_eq(method, "GET") && (str_eq(clean, "/api/act-stats") || str_eq(clean, "/act-stats")) {
return route_act_stats(method, path, body)
}
if str_eq(method, "GET") && (str_eq(clean, "/api/text-health") || str_eq(clean, "/text-health")) {
return route_text_health(method, path, body)
}
// Nodes
if str_eq(method, "POST") && (str_eq(clean, "/api/nodes") || str_eq(clean, "/nodes")) {
@@ -647,13 +439,6 @@ fn handle_request(method: String, path: String, body: String) -> String {
if str_eq(method, "POST") && (str_eq(clean, "/api/edges") || str_eq(clean, "/edges")) {
return route_create_edge(method, path, body)
}
// Batch edge write one snapshot for the whole payload. Must be tested
// BEFORE nothing else claims it; the exact-match on "/api/edges" above
// does not catch "/api/edges/batch", so order is not load-bearing here,
// but keeping the two adjacent keeps them from drifting apart.
if str_eq(method, "POST") && (str_eq(clean, "/api/edges/batch") || str_eq(clean, "/edges/batch")) {
return route_create_edges_batch(method, path, body)
}
if str_eq(method, "GET") && str_starts_with(clean, "/api/neighbors/") {
return route_neighbors(method, path, body)
}
@@ -693,16 +478,6 @@ fn handle_request(method: String, path: String, body: String) -> String {
return route_sync(method, path, body)
}
// Embedding backfill pumped by the soul heartbeat (2026-07-25)
if str_eq(clean, "/api/embed-backfill") {
return route_embed_backfill(method, path, body)
}
// Semantic similarity probe (2026-08-01)
if str_eq(method, "GET") && str_starts_with(clean, "/api/similarity") {
return route_similarity(method, path, body)
}
"{\"error\":\"not found\",\"path\":\"" + clean + "\"}"
}
+10
View File
@@ -17,6 +17,16 @@
// 4. Append dep to order after all its transitive deps
// 5. Deduplicate: skip already-ordered vessels
// Cross-module forward declarations
// Defined in sibling epm modules; resolved at link time. The `extern fn` decls
// give elc the C prototypes so generated install.c compiles cleanly under strict
// compilers (gcc>=14 / clang) that reject implicit function declarations.
extern fn manifest_name(src: String) -> String // manifest.el
extern fn manifest_deps(src: String) -> String // manifest.el
extern fn registry_token() -> String // registry.el
extern fn registry_find(name: String, version: String) -> String // registry.el
extern fn registry_latest_version(name: String) -> String // registry.el
// Install paths
// packages_dir returns the root directory for installed vessels.
+9
View File
@@ -14,6 +14,15 @@
// EPM_REGISTRY_ORG org name that hosts vessel repos (default: neuron-technologies)
// EPM_TOKEN Gitea personal access token (required for publish)
// Cross-module forward declarations
// These symbols are defined in sibling epm modules or the El runtime and are
// resolved at link time. The `extern fn` decls give elc the C prototype so the
// generated registry.c compiles cleanly under strict compilers (gcc>=14 / clang)
// that reject implicit function declarations. Signature arity must match the
// definition; return/param types are informational (all lower to el_val_t).
extern fn config(key: String) -> String // El runtime builtin
extern fn read_installed() -> String // install.el
// Config helpers
// registry_api_url returns the Gitea API base URL with no trailing slash.
+9
View File
@@ -6,6 +6,15 @@
// Depends on: registry.el (registry_latest_version, registry_find),
// install.el (read_installed, install_vessel, installed_version)
// Cross-module forward declarations
// Defined in sibling epm modules; resolved at link time. The `extern fn` decls
// give elc the C prototypes so generated update.c compiles cleanly under strict
// compilers (gcc>=14 / clang) that reject implicit function declarations.
extern fn read_installed() -> String // install.el
extern fn installed_version(name: String) -> String // install.el
extern fn install_vessel(name: String, version: String) -> Bool // install.el
extern fn registry_latest_version(name: String) -> String // registry.el
// Semver helpers
// semver_part extracts the Nth dot-separated component from a semver string.
BIN
View File
Binary file not shown.
@@ -75,6 +75,7 @@ static inline void* el_win_dlsym(void* handle, const char* name) {
#include <direct.h> /* _mkdir */
#define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */
#define timegm _mkgmtime /* UTC tm → time_t */
#define fsync(fd) _commit(fd) /* no fsync() on Windows; _commit() (<io.h>) is the equiv */
/* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */
static inline int setenv(const char* name, const char* value, int overwrite) {
+382 -314
View File
@@ -82,8 +82,14 @@ static _Thread_local ElArena _tl_arena = {NULL, 0, 0};
static _Thread_local int _tl_arena_active = 0;
/* Binary-safe fs_read length — set by fs_read, consumed by http_send_response.
* Allows serving PNGs and other binary files without strlen truncation. */
static _Thread_local size_t _tl_fs_read_len = 0;
* Allows serving PNGs and other binary files without strlen truncation.
* PAIRED with the buffer pointer it describes: the length may only be applied
* to the exact buffer fs_read returned. Without the pairing, any handler that
* fs_read a file and then WRAPPED it into a larger response had that response
* truncated to the file's length (Content-Length lied AND the send stopped
* short) the safety-contact onboarding trap, 2026-07-17. */
static _Thread_local size_t _tl_fs_read_len = 0;
static _Thread_local const char* _tl_fs_read_buf = NULL;
static void el_arena_track(char* p) {
if (!_tl_arena_active || !p) return;
@@ -101,6 +107,8 @@ static void el_arena_track(char* p) {
void el_request_start(void) {
_tl_arena.count = 0;
_tl_arena_active = 1;
_tl_fs_read_len = 0; /* never let a previous request's file length */
_tl_fs_read_buf = NULL; /* leak into this response's byte accounting */
}
/* Called by http_worker after the El handler returns and the response is sent.
@@ -1484,11 +1492,14 @@ static void http_send_response(int fd, const char* body) {
}
const char* eff_body = is_envelope ? env_body : body;
/* Use the real byte count from fs_read if available (handles binary files
* with embedded null bytes PNG, WOFF2, etc.). Fall back to strlen for
* normal text/JSON responses where _tl_fs_read_len is 0. */
size_t blen = (_tl_fs_read_len > 0) ? _tl_fs_read_len : strlen(eff_body);
/* Use the real byte count from fs_read ONLY when this body IS the exact
* buffer fs_read returned (binary files with embedded null bytes PNG,
* WOFF2, etc.). Any other body wrapped, enveloped, or derived must be
* measured with strlen, or it is truncated/over-read to the file's size. */
size_t blen = (_tl_fs_read_len > 0 && eff_body == _tl_fs_read_buf)
? _tl_fs_read_len : strlen(eff_body);
_tl_fs_read_len = 0; /* consume — one-shot per response */
_tl_fs_read_buf = NULL;
int head_only = _tl_http_head_only;
JsonBuf hdrs; jb_init(&hdrs);
@@ -1545,17 +1556,6 @@ typedef struct {
#endif
} HttpWorkerArg;
/* Forward declarations for the loopback/API-key hardening helpers defined
* further down. Without these, http_worker's calls below were implicit
* declarations and the later `static` definitions conflicted with them this
* file did not compile at all. (2026-08-08 self-review: the hardening work
* they belong to had been sitting uncommitted in the working tree since
* 2026-07-15 in exactly this non-building state, which is presumably why it
* was never committed. Adding the two prototypes is the whole fix.) */
static int el_http_request_authorized(const char* method, const char* path,
const char* hdr_block);
static void el_http_send_401(int fd);
static void* http_worker(void* arg) {
HttpWorkerArg* a = (HttpWorkerArg*)arg;
#ifdef _WIN32
@@ -1564,13 +1564,8 @@ static void* http_worker(void* arg) {
int fd = a->fd;
#endif
free(a);
char *method = NULL, *path = NULL, *body = NULL, *hdr_block = NULL;
if (http_read_request(fd, &method, &path, &body, &hdr_block) == 0
&& !el_http_request_authorized(method, path, hdr_block)) {
/* Loopback hardening: EL_HTTP_AUTH_KEY is set and this request lacks the
* matching X-Neuron-Auth header refuse before it reaches any handler. */
el_http_send_401(fd);
} else if (method != NULL) {
char *method = NULL, *path = NULL, *body = NULL;
if (http_read_request(fd, &method, &path, &body, NULL) == 0) {
http_handler_fn h = http_lookup_active();
char* response = NULL;
/* HEAD: dispatch as GET so existing handlers respond with the same
@@ -1584,11 +1579,22 @@ static void* http_worker(void* arg) {
const char* rs = EL_CSTR(r);
/* Copy response out BEFORE arena teardown.
* For binary files, _tl_fs_read_len holds the real byte count
* use memcpy instead of strdup so null bytes are preserved. */
size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0);
* use memcpy instead of strdup so null bytes are preserved.
* The stored length applies ONLY when the response IS the exact
* fs_read buffer; a wrapped/derived response must use strlen or
* it gets truncated (or over-read) to the file's length. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
} else {
response = el_strdup_persist("el-runtime: no http handler registered");
}
@@ -1598,7 +1604,7 @@ static void* http_worker(void* arg) {
_tl_http_head_only = 0;
free(response);
}
free(method); free(path); free(body); free(hdr_block);
free(method); free(path); free(body);
el_closesocket(fd);
/* release a slot */
pthread_mutex_lock(&_http_conn_mu);
@@ -1608,108 +1614,6 @@ static void* http_worker(void* arg) {
return NULL;
}
/* ── loopback lock + local API-key auth (shipped desktop hardening) ────────
* Both controls are OFF by default (their env vars unset), so dev, self-host,
* and server builds behave exactly as before. The shipped macOS launcher
* neuron-daemons.sh sets them so a customer's soul is neither reachable from
* other machines on the LAN nor callable by other local users/processes
* without the per-install key held in the login Keychain:
*
* EL_HTTP_BIND_HOST=127.0.0.1 -> bind loopback only (el_http_apply_bind_addr)
* EL_HTTP_AUTH_KEY=<per-install> -> require "X-Neuron-Auth: <key>" per request
*/
/* Set the listen address on the dual-stack (AF_INET6, V6ONLY=0) socket. Default
* is in6addr_any (all interfaces) unchanged. When EL_HTTP_BIND_HOST names a
* loopback ("127.0.0.1", "localhost", "loopback", or "::1") we bind the IPv4-
* mapped IPv6 loopback ::ffff:127.0.0.1: on a V6ONLY=0 socket this accepts IPv4
* 127.0.0.1 clients (the desktop app connects there) while refusing every
* off-machine address. */
static void el_http_apply_bind_addr(struct sockaddr_in6* addr) {
const char* h = getenv("EL_HTTP_BIND_HOST");
int loopback = h && *h && (strcmp(h, "127.0.0.1") == 0
|| strcmp(h, "localhost") == 0
|| strcmp(h, "loopback") == 0
|| strcmp(h, "::1") == 0);
if (loopback) {
memset(&addr->sin6_addr, 0, sizeof(addr->sin6_addr));
addr->sin6_addr.s6_addr[10] = 0xff; /* ::ffff:127.0.0.1 */
addr->sin6_addr.s6_addr[11] = 0xff;
addr->sin6_addr.s6_addr[12] = 127;
addr->sin6_addr.s6_addr[15] = 1;
} else {
addr->sin6_addr = in6addr_any;
}
}
/* Human-readable description of the active bind host, for the listen log line. */
static const char* el_http_bind_desc(void) {
const char* h = getenv("EL_HTTP_BIND_HOST");
if (h && *h && (strcmp(h, "127.0.0.1") == 0 || strcmp(h, "localhost") == 0
|| strcmp(h, "loopback") == 0 || strcmp(h, "::1") == 0)) {
return "127.0.0.1 (loopback)";
}
return "[::] (dual-stack)";
}
/* Case-insensitive compare of the first n bytes of a and b. */
static int el_ci_eq_n(const char* a, const char* b, size_t n) {
for (size_t i = 0; i < n; i++) {
unsigned char ca = (unsigned char)a[i], cb = (unsigned char)b[i];
if (tolower(ca) != tolower(cb)) return 0;
}
return 1;
}
/* Return 1 iff the raw header block carries a header named `name` (case-
* insensitive) whose trimmed value equals `want` exactly. */
static int el_http_header_equals(const char* hdr_block, const char* name,
const char* want) {
if (!hdr_block || !name || !want) return 0;
size_t nlen = strlen(name), wlen = strlen(want);
const char* p = hdr_block;
while (*p) {
const char* line_end = strstr(p, "\r\n");
const char* end = line_end ? line_end : p + strlen(p);
const char* colon = memchr(p, ':', (size_t)(end - p));
if (colon && (size_t)(colon - p) == nlen && el_ci_eq_n(p, name, nlen)) {
const char* v = colon + 1;
while (v < end && (*v == ' ' || *v == '\t')) v++;
size_t vlen = (size_t)(end - v);
while (vlen > 0 && (v[vlen - 1] == ' ' || v[vlen - 1] == '\t')) vlen--;
if (vlen == wlen && memcmp(v, want, wlen) == 0) return 1;
}
if (!line_end) break;
p = line_end + 2;
}
return 0;
}
/* Authorize an inbound request. Enforcement is active only when EL_HTTP_AUTH_KEY
* is set; otherwise every request is allowed (dev default). GET/HEAD /health*
* are always allowed so launch-agent liveness probes work without the key. */
static int el_http_request_authorized(const char* method, const char* path,
const char* hdr_block) {
const char* key = getenv("EL_HTTP_AUTH_KEY");
if (!key || !*key) return 1;
if (method && (strcmp(method, "GET") == 0 || strcmp(method, "HEAD") == 0)
&& path && strncmp(path, "/health", 7) == 0) return 1;
return el_http_header_equals(hdr_block, "x-neuron-auth", key);
}
/* Minimal 401 for unauthorized requests — never reaches an EL handler. */
static void el_http_send_401(int fd) {
static const char* body = "{\"error\":\"unauthorized\",\"code\":\"auth_required\"}";
char resp[256];
int n = snprintf(resp, sizeof(resp),
"HTTP/1.1 401 Unauthorized\r\n"
"Content-Type: application/json\r\n"
"Content-Length: %zu\r\n"
"Connection: close\r\n\r\n%s",
strlen(body), body);
if (n > 0) http_send_all(fd, resp, (size_t)n);
}
el_val_t http_serve(el_val_t port, el_val_t handler) {
/* If `handler` looks like a string name, register it as the active handler. */
const char* hname = EL_CSTR(handler);
@@ -1728,13 +1632,13 @@ el_val_t http_serve(el_val_t port, el_val_t handler) {
struct sockaddr_in6 addr;
memset(&addr, 0, sizeof(addr));
addr.sin6_family = AF_INET6;
el_http_apply_bind_addr(&addr);
addr.sin6_addr = in6addr_any;
addr.sin6_port = htons((uint16_t)p);
if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
perror("bind"); el_closesocket(sock); return 0;
}
if (listen(sock, 64) < 0) { perror("listen"); el_closesocket(sock); return 0; }
fprintf(stderr, "[http] listening on %s port %d\n", el_http_bind_desc(), p);
fprintf(stderr, "[http] listening on [::]:%d (dual-stack)\n", p);
while (1) {
struct sockaddr_in6 cli;
socklen_t clen = sizeof(cli);
@@ -1940,10 +1844,20 @@ static void* http_worker_v2(void* arg) {
el_val_t hmap = http_build_headers_map(hdr_block ? hdr_block : "");
el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), hmap, EL_STR(body));
const char* rs = EL_CSTR(r);
size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0);
/* Same pairing rule as the v1 worker: the fs_read length is only
* trustworthy for the exact buffer fs_read returned. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
el_release(hmap);
} else {
response = el_strdup_persist(
@@ -1984,13 +1898,13 @@ el_val_t http_serve_v2(el_val_t port, el_val_t handler) {
struct sockaddr_in6 addr;
memset(&addr, 0, sizeof(addr));
addr.sin6_family = AF_INET6;
el_http_apply_bind_addr(&addr);
addr.sin6_addr = in6addr_any;
addr.sin6_port = htons((uint16_t)p);
if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
perror("bind"); el_closesocket(sock); return 0;
}
if (listen(sock, 64) < 0) { perror("listen"); el_closesocket(sock); return 0; }
fprintf(stderr, "[http v2] listening on %s port %d\n", el_http_bind_desc(), p);
fprintf(stderr, "[http v2] listening on [::]:%d (dual-stack)\n", p);
while (1) {
struct sockaddr_in6 cli;
socklen_t clen = sizeof(cli);
@@ -2081,18 +1995,19 @@ void http_serve_async(el_val_t port, el_val_t handler) {
int sock = socket(AF_INET6, SOCK_STREAM, 0);
if (sock < 0) { perror("socket"); return; }
int yes = 1; int no = 0;
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes));
setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, &no, sizeof(no));
/* Win32/mingw setsockopt takes optval as (const char*); the cast is portable on POSIX too. */
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (const char*)&yes, sizeof(yes));
setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, (const char*)&no, sizeof(no));
struct sockaddr_in6 addr;
memset(&addr, 0, sizeof(addr));
addr.sin6_family = AF_INET6;
el_http_apply_bind_addr(&addr);
addr.sin6_addr = in6addr_any;
addr.sin6_port = htons((uint16_t)p);
if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
perror("bind"); close(sock); return;
}
if (listen(sock, 64) < 0) { perror("listen"); close(sock); return; }
fprintf(stderr, "[http] async listening on %s port %d\n", el_http_bind_desc(), p);
fprintf(stderr, "[http] async listening on [::]:%d (dual-stack)\n", p);
HttpServeAsyncArg* a = malloc(sizeof(HttpServeAsyncArg));
if (!a) { close(sock); return; }
a->sock = sock;
@@ -2141,6 +2056,7 @@ el_val_t http_response(el_val_t status, el_val_t headers_json, el_val_t body) {
el_val_t fs_read(el_val_t pathv) {
const char* path = EL_CSTR(pathv);
_tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!path) return el_wrap_str(el_strdup(""));
FILE* f = fopen(path, "rb");
if (!f) return el_wrap_str(el_strdup(""));
@@ -2152,6 +2068,7 @@ el_val_t fs_read(el_val_t pathv) {
size_t got = fread(buf, 1, (size_t)sz, f);
buf[got] = '\0';
_tl_fs_read_len = got; /* store real byte count for binary-safe send */
_tl_fs_read_buf = buf; /* ...valid ONLY for this exact buffer */
fclose(f);
return el_wrap_str(buf);
}
@@ -3257,72 +3174,10 @@ static char* jp_parse_string_raw(JsonParser* jp) {
case 'r': c = '\r'; break;
case 't': c = '\t'; break;
case 'u': {
/* Decode \uXXXX (with surrogate pairs) to UTF-8.
* Ported from lang/releases/v1.0.0-20260501 (2026-08-08
* self-review). This copy carried the identical defect:
* the escape was skipped and a literal '?' emitted, which
* silently destroyed every non-ASCII character in any JSON
* string entering the runtime. Two copies of one parser
* bug is exactly how this class of fault survives, so the
* fix lands in both. See the release copy for the full
* measurement and rationale. */
unsigned cp = 0;
int ok = 1;
for (int i = 0; i < 4; i++) {
if (jp->p >= jp->end) { ok = 0; break; }
char h = *jp->p++;
unsigned d;
if (h >= '0' && h <= '9') d = (unsigned)(h - '0');
else if (h >= 'a' && h <= 'f') d = (unsigned)(h - 'a' + 10);
else if (h >= 'A' && h <= 'F') d = (unsigned)(h - 'A' + 10);
else { ok = 0; break; }
cp = (cp << 4) | d;
}
if (!ok) { c = '?'; break; }
if (cp >= 0xD800 && cp <= 0xDBFF &&
(size_t)(jp->end - jp->p) >= 6 &&
jp->p[0] == '\\' && jp->p[1] == 'u') {
const char* save = jp->p;
unsigned lo = 0; int ok2 = 1;
jp->p += 2;
for (int i = 0; i < 4; i++) {
char h = *jp->p++;
unsigned d;
if (h >= '0' && h <= '9') d = (unsigned)(h - '0');
else if (h >= 'a' && h <= 'f') d = (unsigned)(h - 'a' + 10);
else if (h >= 'A' && h <= 'F') d = (unsigned)(h - 'A' + 10);
else { ok2 = 0; break; }
lo = (lo << 4) | d;
}
if (ok2 && lo >= 0xDC00 && lo <= 0xDFFF)
cp = 0x10000u + ((cp - 0xD800u) << 10) + (lo - 0xDC00u);
else jp->p = save;
}
if (cp >= 0xD800 && cp <= 0xDFFF) cp = 0xFFFD;
char ub[4]; int un;
if (cp < 0x80) {
ub[0] = (char)cp; un = 1;
} else if (cp < 0x800) {
ub[0] = (char)(0xC0 | (cp >> 6));
ub[1] = (char)(0x80 | (cp & 0x3F)); un = 2;
} else if (cp < 0x10000) {
ub[0] = (char)(0xE0 | (cp >> 12));
ub[1] = (char)(0x80 | ((cp >> 6) & 0x3F));
ub[2] = (char)(0x80 | (cp & 0x3F)); un = 3;
} else {
ub[0] = (char)(0xF0 | (cp >> 18));
ub[1] = (char)(0x80 | ((cp >> 12) & 0x3F));
ub[2] = (char)(0x80 | ((cp >> 6) & 0x3F));
ub[3] = (char)(0x80 | (cp & 0x3F)); un = 4;
}
while (len + (size_t)un >= cap) {
cap *= 2;
out = realloc(out, cap);
if (!out) { fputs("el_runtime: out of memory\n", stderr); exit(1); }
}
for (int i = 0; i < un; i++) out[len++] = ub[i];
continue; /* bytes already appended */
/* Skip 4 hex digits; emit '?' as a placeholder */
for (int i = 0; i < 4 && jp->p < jp->end; i++) jp->p++;
c = '?';
break;
}
default: c = esc; break;
}
@@ -3756,8 +3611,10 @@ el_val_t json_get_raw(el_val_t json_str, el_val_t key) {
const char* k = EL_CSTR(key);
const char* p = json_find_key(json, k);
/* Clear fs_read binary-length hint — result is a fresh null-terminated
* string, not the raw file bytes, so Content-Length must use strlen. */
* string, not the raw file bytes, so Content-Length must use strlen.
* (Kept although the pointer pairing now makes this redundant.) */
_tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!p) return el_wrap_str(el_strdup(""));
const char* end = json_skip_value(p);
size_t n = (size_t)(end - p);
@@ -5470,8 +5327,8 @@ el_val_t str_to_float(el_val_t s) {
/* ── Math (Float-aware) ──────────────────────────────────────────────────── */
el_val_t math_sqrt(el_val_t f) { return el_from_float(sqrt(el_to_float(f))); }
el_val_t math_log(el_val_t f) { return el_from_float(log(el_to_float(f))); }
el_val_t math_ln(el_val_t f) { return el_from_float(log(el_to_float(f))); }
el_val_t math_log(el_val_t f) { return el_from_float(log10(el_to_float(f))); } /* base-10, per runtime/math.el */
el_val_t math_ln(el_val_t f) { return el_from_float(log(el_to_float(f))); } /* natural log */
el_val_t math_sin(el_val_t f) { return el_from_float(sin(el_to_float(f))); }
el_val_t math_cos(el_val_t f) { return el_from_float(cos(el_to_float(f))); }
el_val_t math_pi(void) { return el_from_float(3.141592653589793238462643383279502884); }
@@ -6228,13 +6085,6 @@ void el_cgi_init(el_val_t name, el_val_t dharma_id, el_val_t principal,
#define ENGRAM_SUPPRESSION_BREAKTHROUGH 5
#define ENGRAM_BREAKTHROUGH_WEIGHT 0.25
#define ENGRAM_INHIBITION_FACTOR 0.1
/* ENGRAM_WM_CAP: hard global ceiling on nodes holding working_memory_weight
* > 0 at any time. Cowan (2001) puts human WM capacity at ~4 chunks; 24 gives
* the daemon generous headroom while preventing the unbounded growth observed
* in production (wm_active 288-778 per heartbeat "working memory" that is
* really the whole recently-touched graph). Ported from release runtime
* v1.0.0-20260501 Pass 5 on 2026-07-15 self-review. */
#define ENGRAM_WM_CAP 24
/* ── Layered consciousness architecture ──────────────────────────────────────
*
@@ -7082,6 +6932,243 @@ static int engram_rank_cmp(const void* a, const void* b) {
return 0;
}
/* ══════════════════════════════════════════════════════════════════════════
* SEMANTIC SEARCH LAYER nomic-embed-text via Ollama /api/embeddings
*
* Augments the lexical (istr_contains) matcher with dense-vector retrieval.
* Node content and the query are embedded through a local Ollama server;
* nodes are ranked by cosine similarity and UNIONED with lexical hits. This
* lets a paraphrase query surface a node whose words never appear in it.
*
* DEGRADABLE BY DESIGN. The whole layer is gated on HAVE_CURL plus a one-shot
* runtime probe of the embedding endpoint. If curl is not compiled in, or
* Ollama is unreachable, or ENGRAM_SEMANTIC=0, every entry point returns
* "no semantic signal" and callers fall back to pure lexical behaviour
* byte-for-byte the pre-existing search.
*
* CACHE. Node embeddings are computed lazily on first use and cached in
* process memory keyed by node id, with an FNV-1a content hash for
* invalidation (edited content re-embeds). The query is embedded once per
* search call. This is what "avoid re-embedding the whole graph every query"
* buys us: a warm cache serves cosine from RAM. (A cold process still pays
* O(N) embed calls the first time each node is scanned persisting the cache
* to a snapshot sidecar is the documented next step, not done here.)
*
* nomic task prefixes ("search_query:" / "search_document:") are applied
* because nomic-embed-text is trained with them; they materially improve
* retrieval separation (empirically: paraphrase 0.72 vs distractors <0.48).
*
* ENV:
* ENGRAM_SEMANTIC "0" disables; unset/other = auto-probe
* ENGRAM_EMBED_URL default http://localhost:11434/api/embeddings
* ENGRAM_EMBED_MODEL default nomic-embed-text
* ENGRAM_SEMANTIC_MIN cosine threshold for a pure-semantic match (def 0.6)
* */
static double engram_semantic_min(void) {
static double v = -1.0;
if (v >= 0.0) return v;
const char* s = getenv("ENGRAM_SEMANTIC_MIN");
double d = 0.6;
if (s && *s) { char* e = NULL; double t = strtod(s, &e);
if (e != s && t >= 0.0 && t <= 1.0) d = t; }
v = d; return v;
}
#ifdef HAVE_CURL
typedef struct { char* id; uint64_t hash; float* vec; int dim; } EngramEmbEntry;
static EngramEmbEntry* g_emb_items = NULL;
static int64_t g_emb_count = 0, g_emb_cap = 0;
static int g_emb_state = 0; /* 0=unprobed, 1=available, -1=disabled */
static uint64_t engram_fnv1a(const char* s) {
uint64_t h = 1469598103934665603ULL;
if (s) for (const unsigned char* p = (const unsigned char*)s; *p; p++) {
h ^= *p; h *= 1099511628211ULL;
}
return h;
}
/* Parse "embedding":[f,f,...] from an Ollama response. malloc'd vec, or NULL. */
static float* engram_parse_embedding(const char* json, int* out_dim) {
if (!json) return NULL;
const char* p = strstr(json, "\"embedding\"");
if (!p) return NULL;
p = strchr(p, '[');
if (!p) return NULL;
p++;
int cap = 1024, n = 0;
float* v = malloc((size_t)cap * sizeof(float));
if (!v) return NULL;
while (*p && *p != ']') {
while (*p == ' ' || *p == '\t' || *p == '\n' || *p == '\r' || *p == ',') p++;
if (*p == ']' || !*p) break;
char* e = NULL;
double d = strtod(p, &e);
if (e == p) break;
if (n >= cap) { cap *= 2; float* nv = realloc(v, (size_t)cap * sizeof(float));
if (!nv) { free(v); return NULL; } v = nv; }
v[n++] = (float)d;
p = e;
}
if (n == 0) { free(v); return NULL; }
*out_dim = n;
return v;
}
/* JSON-escape src into a malloc'd buffer (no surrounding quotes). */
static char* engram_json_escape(const char* src) {
if (!src) src = "";
size_t n = strlen(src);
char* out = malloc(n * 2 + 1);
if (!out) return NULL;
size_t j = 0;
for (size_t i = 0; i < n; i++) {
unsigned char c = (unsigned char)src[i];
if (c == '"') { out[j++] = '\\'; out[j++] = '"'; }
else if (c == '\\') { out[j++] = '\\'; out[j++] = '\\'; }
else if (c == '\n') { out[j++] = '\\'; out[j++] = 'n'; }
else if (c == '\r') { out[j++] = '\\'; out[j++] = 'r'; }
else if (c == '\t') { out[j++] = '\\'; out[j++] = 't'; }
else if (c < 0x20) { /* drop other control bytes */ }
else { out[j++] = (char)c; }
}
out[j] = '\0';
return out;
}
/* Embed `prefix+text` via Ollama. Returns malloc'd vec (caller frees), or NULL. */
static float* engram_embed_raw(const char* prefix, const char* text, int* out_dim) {
if (!text) return NULL;
const char* url = getenv("ENGRAM_EMBED_URL");
if (!url || !*url) url = "http://localhost:11434/api/embeddings";
const char* model = getenv("ENGRAM_EMBED_MODEL");
if (!model || !*model) model = "nomic-embed-text";
/* Bound content length to keep latency/memory sane on huge nodes. */
char* trunc = NULL;
size_t maxlen = 8192;
if (strlen(text) > maxlen) {
trunc = malloc(maxlen + 1);
if (trunc) { memcpy(trunc, text, maxlen); trunc[maxlen] = '\0'; text = trunc; }
}
char* esc_prefix = engram_json_escape(prefix ? prefix : "");
char* esc = engram_json_escape(text);
free(trunc);
if (!esc || !esc_prefix) { free(esc); free(esc_prefix); return NULL; }
size_t blen = strlen(esc) + strlen(esc_prefix) + strlen(model) + 64;
char* body = malloc(blen);
if (!body) { free(esc); free(esc_prefix); return NULL; }
snprintf(body, blen, "{\"model\":\"%s\",\"prompt\":\"%s%s\"}", model, esc_prefix, esc);
free(esc); free(esc_prefix);
CURL* c = curl_easy_init();
if (!c) { free(body); return NULL; }
HttpBuf rb; httpbuf_init(&rb);
struct curl_slist* h = curl_slist_append(NULL, "Content-Type: application/json");
char errbuf[CURL_ERROR_SIZE]; errbuf[0] = '\0';
curl_easy_setopt(c, CURLOPT_URL, url);
curl_easy_setopt(c, CURLOPT_WRITEFUNCTION, http_write_cb);
curl_easy_setopt(c, CURLOPT_WRITEDATA, &rb);
curl_easy_setopt(c, CURLOPT_POST, 1L);
curl_easy_setopt(c, CURLOPT_POSTFIELDS, body);
curl_easy_setopt(c, CURLOPT_POSTFIELDSIZE, (long)strlen(body));
curl_easy_setopt(c, CURLOPT_HTTPHEADER, h);
curl_easy_setopt(c, CURLOPT_TIMEOUT_MS, el_http_timeout_ms());
curl_easy_setopt(c, CURLOPT_NOSIGNAL, 1L);
curl_easy_setopt(c, CURLOPT_ERRORBUFFER, errbuf);
CURLcode rc = curl_easy_perform(c);
curl_slist_free_all(h);
curl_easy_cleanup(c);
free(body);
if (rc != CURLE_OK) { free(rb.data); return NULL; }
float* v = engram_parse_embedding(rb.data, out_dim);
free(rb.data);
return v;
}
/* One-shot probe: is semantic search available? Caches the verdict. */
static int engram_semantic_enabled(void) {
if (g_emb_state != 0) return g_emb_state == 1;
const char* s = getenv("ENGRAM_SEMANTIC");
if (s && strcmp(s, "0") == 0) { g_emb_state = -1; return 0; }
int dim = 0;
float* v = engram_embed_raw("search_query: ", "probe", &dim);
if (v && dim > 0) { free(v); g_emb_state = 1; return 1; }
free(v);
g_emb_state = -1; return 0;
}
/* Embed the query. Returns malloc'd vec (caller frees), or NULL if semantic off. */
static float* engram_embed_query(const char* q, int* dim) {
if (!engram_semantic_enabled()) return NULL;
if (!q || !*q) return NULL;
return engram_embed_raw("search_query: ", q, dim);
}
/* Cached node embedding. Returns a pointer OWNED BY THE CACHE — do not free. */
static const float* engram_node_vec(EngramNode* n, int* out_dim) {
if (!n || !n->id) return NULL;
uint64_t h = engram_fnv1a(n->content);
for (int64_t i = 0; i < g_emb_count; i++) {
if (g_emb_items[i].id && strcmp(g_emb_items[i].id, n->id) == 0) {
if (g_emb_items[i].hash == h && g_emb_items[i].vec) {
*out_dim = g_emb_items[i].dim; return g_emb_items[i].vec;
}
/* content changed → re-embed in place */
int dim = 0;
float* v = engram_embed_raw("search_document: ", n->content ? n->content : "", &dim);
if (!v) return NULL;
free(g_emb_items[i].vec);
g_emb_items[i].vec = v; g_emb_items[i].dim = dim; g_emb_items[i].hash = h;
*out_dim = dim; return v;
}
}
int dim = 0;
float* v = engram_embed_raw("search_document: ", n->content ? n->content : "", &dim);
if (!v) return NULL;
if (g_emb_count >= g_emb_cap) {
int64_t nc = g_emb_cap ? g_emb_cap * 2 : 256;
EngramEmbEntry* ni = realloc(g_emb_items, (size_t)nc * sizeof(EngramEmbEntry));
if (!ni) { free(v); return NULL; }
g_emb_items = ni; g_emb_cap = nc;
}
g_emb_items[g_emb_count].id = strdup(n->id);
g_emb_items[g_emb_count].hash = h;
g_emb_items[g_emb_count].vec = v;
g_emb_items[g_emb_count].dim = dim;
g_emb_count++;
*out_dim = dim; return v;
}
static double engram_cosine(const float* a, const float* b, int dim) {
double dot = 0, na = 0, nb = 0;
for (int i = 0; i < dim; i++) { dot += (double)a[i] * b[i];
na += (double)a[i] * a[i];
nb += (double)b[i] * b[i]; }
if (na <= 0 || nb <= 0) return 0.0;
return dot / (sqrt(na) * sqrt(nb));
}
/* Cosine of node n against the query vector; 0 if unavailable / dim mismatch. */
static double engram_node_cosine(EngramNode* n, const float* qvec, int qdim) {
if (!qvec || qdim <= 0) return 0.0;
int ndim = 0;
const float* nv = engram_node_vec(n, &ndim);
if (!nv || ndim != qdim) return 0.0;
return engram_cosine(qvec, nv, qdim);
}
#else /* !HAVE_CURL — semantic layer compiled out; callers stay pure-lexical.
* Only the two boundary functions the always-compiled search/activate
* code calls are stubbed; the query embed always yields NULL so every
* cosine is 0 and every caller collapses to lexical-only. */
static float* engram_embed_query(const char* q, int* dim) { (void)q; (void)dim; return NULL; }
static double engram_node_cosine(EngramNode* n, const float* qvec, int qdim) {
(void)n; (void)qvec; (void)qdim; return 0.0;
}
#endif /* HAVE_CURL */
el_val_t engram_search(el_val_t query, el_val_t limit) {
EngramStore* g = engram_get();
const char* q = EL_CSTR(query);
@@ -7092,8 +7179,15 @@ el_val_t engram_search(el_val_t query, el_val_t limit) {
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
if (ntok == 0) return lst;
/* Semantic augmentation: embed the query once; a node is a hit if it covers
* >=1 query token (tokenized-lexical, #66) OR its cosine clears the
* threshold (#67). qvec is NULL (cosine 0) when semantic is unavailable
* pure tokenized-lexical, byte-identical to the lexical-only behaviour. */
int qdim = 0;
float* qvec = engram_embed_query(q, &qdim);
double sem_min = engram_semantic_min();
EngramRankEntry* hits = malloc((size_t)g->node_count * sizeof(EngramRankEntry));
if (!hits) return lst;
if (!hits) { free(qvec); return lst; }
int64_t nhits = 0;
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
@@ -7103,20 +7197,24 @@ el_val_t engram_search(el_val_t query, el_val_t limit) {
* + engram_compile_layered_json that's the legitimate path. */
if (engram_layer_is_transparent(n->layer_id)) continue;
int sc = engram_node_match_score(n, toks, ntok);
if (sc > 0) {
double sem = qvec ? engram_node_cosine(n, qvec, qdim) : 0.0;
if (sc > 0 || sem >= sem_min) {
hits[nhits].idx = i;
hits[nhits].score = sc;
hits[nhits].salience = n->salience;
nhits++;
}
}
/* Rank by distinct tokens matched (desc) then salience (desc), then cap. */
/* Rank by distinct tokens matched (desc) then salience (desc), then cap.
* Pure-semantic hits (token score 0) sort after every lexical hit a
* lexical semantic union with lexical precedence. */
qsort(hits, (size_t)nhits, sizeof(EngramRankEntry), engram_rank_cmp);
int64_t end = nhits < lim ? nhits : lim;
for (int64_t k = 0; k < end; k++) {
lst = el_list_append(lst, engram_node_to_map(&g->nodes[hits[k].idx]));
}
free(hits);
free(qvec);
return lst;
}
@@ -7439,48 +7537,6 @@ static double engram_goal_bias(const EngramNode* n, const char* query) {
return bias;
}
/* eg_cmp_double_desc — qsort comparator, descending doubles. */
static int eg_cmp_double_desc(const void* a, const void* b) {
double da = *(const double*)a, db = *(const double*)b;
if (da < db) return 1;
if (da > db) return -1;
return 0;
}
/* eg_enforce_wm_cap_global — clamp the store-wide working-memory population
* to ENGRAM_WM_CAP, keeping the top-K by current weight. Runs at every point
* that materializes WM: post-activation persist and snapshot load/merge.
* (Ported from release runtime v1.0.0-20260501 Pass 5, 2026-07-15.) */
static void eg_enforce_wm_cap_global(EngramStore* g) {
int64_t wm_count = 0;
for (int64_t i = 0; i < g->node_count; i++) {
if (g->nodes[i].working_memory_weight > 0.0) wm_count++;
}
if (wm_count <= ENGRAM_WM_CAP) return;
double* vals = malloc((size_t)wm_count * sizeof(double));
if (!vals) return; /* OOM: over cap this call, no corruption */
int64_t vi = 0;
for (int64_t i = 0; i < g->node_count; i++) {
if (g->nodes[i].working_memory_weight > 0.0)
vals[vi++] = g->nodes[i].working_memory_weight;
}
qsort(vals, (size_t)wm_count, sizeof(double), eg_cmp_double_desc);
double cutoff = vals[ENGRAM_WM_CAP - 1];
free(vals);
int64_t above = 0;
for (int64_t i = 0; i < g->node_count; i++) {
if (g->nodes[i].working_memory_weight > cutoff) above++;
}
int64_t slots_at_cutoff = ENGRAM_WM_CAP - above;
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
if (n->working_memory_weight <= 0.0) continue;
if (n->working_memory_weight > cutoff) continue;
if (slots_at_cutoff > 0) { slots_at_cutoff--; continue; }
n->working_memory_weight = 0.0; /* evict: over global cap */
}
}
el_val_t engram_activate(el_val_t query, el_val_t depth) {
EngramStore* g = engram_get();
const char* q = EL_CSTR(query);
@@ -7508,21 +7564,31 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
if (!seeds) {
free(best_bg); free(best_hops); free(reached); return out;
}
/* Tokenize once: a node seeds if it matches ANY query token, and its seed
* activation is scaled by token coverage (fraction of distinct query
* tokens it contains) so a node matching all words seeds more strongly
* than one matching a single word. Single-word queries coverage 1.0,
* identical to the prior whole-query behavior. */
/* Tokenized + semantic seeding: a node seeds if it covers >=1 query token
* (tokenized-lexical, #66) OR its cosine clears the threshold (#67). A
* lexical seed's activation is scaled by token coverage (fraction of
* distinct query tokens covered) so a node matching all words seeds more
* strongly than one matching a single word; single-word queries coverage
* 1.0. A pure-semantic seed (no token match) is instead down-weighted by
* its cosine so paraphrase matches spread without overpowering exact seeds.
* q_vec is NULL (cosine 0) when semantic is unavailable the seed set is
* exactly the tokenized-lexical one. q_vec is freed right after this loop
* so the many downstream early-returns need no cleanup change. */
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
int q_dim = 0;
float* q_vec = engram_embed_query(q, &q_dim);
double q_sem_min = engram_semantic_min();
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
int sc = engram_node_match_score(n, toks, ntok);
if (sc > 0) {
double sem = q_vec ? engram_node_cosine(n, q_vec, q_dim) : 0.0;
if (sc > 0 || sem >= q_sem_min) {
double tdecay = engram_temporal_decay(n, now_ms);
double dampen = engram_activation_dampen(n);
double cover = ntok > 0 ? (double)sc / (double)ntok : 1.0;
double act = n->salience * tdecay * dampen * cover;
double act = n->salience * tdecay * dampen;
if (sc > 0) act *= (ntok > 0 ? (double)sc / (double)ntok : 1.0);
else act *= sem; /* pure-semantic seed: down-weight by cosine */
seeds[seed_count].idx = i;
seeds[seed_count].act = act;
seeds[seed_count].created_at = n->created_at;
@@ -7532,6 +7598,7 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
reached[i] = 1;
}
}
free(q_vec);
/* Compute mean seed created_at for temporal proximity bonus. */
int64_t seed_epoch = 0;
if (seed_count > 0) {
@@ -7686,12 +7753,6 @@ el_val_t engram_activate(el_val_t query, el_val_t depth) {
g->nodes[i].working_memory_weight = wm_weights[i];
}
/* Global WM cap: keep only the top ENGRAM_WM_CAP by weight across the
* whole store (see eg_enforce_wm_cap_global). Without this, repeated
* activation calls accumulate hundreds of "promoted" nodes and WM stops
* meaning anything (production heartbeats showed wm_active up to 778). */
eg_enforce_wm_cap_global(g);
/* ── Collect all background-activated nodes for the return value ────
* Callers see both layers. Context compilation uses only promoted nodes
* (working_memory_weight > 0). Sort: promoted first by wm_weight desc,
@@ -8069,9 +8130,6 @@ el_val_t engram_load(el_val_t path) {
}
}
}
/* WM cap discipline applies to every entry point that materializes WM,
* including snapshot restore (see eg_enforce_wm_cap_global). */
eg_enforce_wm_cap_global(g);
free(data);
return 1;
}
@@ -8096,14 +8154,16 @@ el_val_t engram_get_node_json(el_val_t id) {
* matches the given string. Returns the node as a JSON object string, or "{}"
* if no match is found.
*
* Exact match (strcmp, not substring) because labels like "conv:history"
* must not collide with nodes whose content contains that substring.
* Used by chat.el to retrieve well-known nodes (e.g. "conv:history",
* "session:summary") by their stable label rather than by ID, which is immune
* to vector index drift across restarts.
*
* Ported from the release runtime 2026-07-16 self-review: chat.el has called
* this since 2026-07-01 but the function only existed in
* releases/v1.0.0-20260501/el_runtime.c the soul daemon (which builds
* against THIS runtime) failed to compile once clang made implicit
* declarations an error. */
* Exact match (strcmp, not istr_contains) because labels like "conv:history"
* must not collide with nodes whose content happens to contain that substring.
*
* Backported verbatim (idiom-adapted to jb_finish) from release runtime
* v1.0.0-20260501 to unblock the soul regen link: chat.el references this
* native but the current runtime lacked its definition. */
el_val_t engram_get_node_by_label(el_val_t label) {
const char* lbl = EL_CSTR(label);
if (!lbl || !*lbl) return el_wrap_str(el_strdup("{}"));
@@ -8126,39 +8186,50 @@ el_val_t engram_search_json(el_val_t query, el_val_t limit) {
if (lim <= 0) lim = 100;
JsonBuf b; jb_init(&b);
jb_putc(&b, '[');
int first = 1;
if (q && *q) {
if (q && *q && g->node_count > 0) {
/* Collect candidates from the UNION of tokenized-lexical and semantic
* matches, score each, rank by score, emit the top `lim`. A node is a
* candidate if it covers >=1 query token (tokenized-lexical, #66) OR its
* query cosine clears the threshold (#67). Lexical score is the distinct
* token count (>=1), so any lexical hit outranks a pure-semantic hit
* (cosine < 1); pure-semantic hits are scored by cosine alone. When
* semantic is unavailable qvec is NULL, sem is 0, only tokenized-lexical
* hits are collected, and the stable insertion sort preserves order. */
char toks[ENGRAM_MAX_QTOKENS][ENGRAM_QTOK_LEN];
int ntok = engram_tokenize_query(q, toks, ENGRAM_MAX_QTOKENS);
if (ntok > 0) {
EngramRankEntry* hits =
malloc((size_t)g->node_count * sizeof(EngramRankEntry));
if (hits) {
int64_t nhits = 0;
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
/* Filter transparent layers — same as engram_search. */
if (engram_layer_is_transparent(n->layer_id)) continue;
int sc = engram_node_match_score(n, toks, ntok);
if (sc > 0) {
hits[nhits].idx = i;
hits[nhits].score = sc;
hits[nhits].salience = n->salience;
nhits++;
}
int qdim = 0;
float* qvec = engram_embed_query(q, &qdim);
double sem_min = engram_semantic_min();
typedef struct { int64_t idx; double score; } Cand;
Cand* cand = malloc((size_t)g->node_count * sizeof(Cand));
if (cand) {
int64_t nc = 0;
for (int64_t i = 0; i < g->node_count; i++) {
EngramNode* n = &g->nodes[i];
if (engram_layer_is_transparent(n->layer_id)) continue;
int sc = engram_node_match_score(n, toks, ntok);
double sem = qvec ? engram_node_cosine(n, qvec, qdim) : 0.0;
if (sc > 0 || sem >= sem_min) {
cand[nc].idx = i;
cand[nc].score = (double)sc + sem;
nc++;
}
/* Rank by distinct tokens matched (desc) then salience (desc). */
qsort(hits, (size_t)nhits, sizeof(EngramRankEntry),
engram_rank_cmp);
int64_t end = nhits < lim ? nhits : lim;
for (int64_t k = 0; k < end; k++) {
if (!first) jb_putc(&b, ',');
engram_emit_node_json(&b, &g->nodes[hits[k].idx]);
first = 0;
}
free(hits);
}
/* Insertion sort by score desc; stable for equal scores. */
for (int64_t i = 1; i < nc; i++) {
Cand k = cand[i]; int64_t j = i - 1;
while (j >= 0 && cand[j].score < k.score) { cand[j + 1] = cand[j]; j--; }
cand[j + 1] = k;
}
int first = 1;
for (int64_t i = 0; i < nc && i < lim; i++) {
if (!first) jb_putc(&b, ',');
engram_emit_node_json(&b, &g->nodes[cand[i].idx]);
first = 0;
}
free(cand);
}
free(qvec);
}
jb_putc(&b, ']');
return el_wrap_str(jb_finish(&b));
@@ -8676,9 +8747,6 @@ el_val_t engram_load_merge(el_val_t path) {
}
}
/* Merged nodes can carry snapshot WM weights too — hold the cap here as
* well (see eg_enforce_wm_cap_global). */
eg_enforce_wm_cap_global(g);
free(data);
return (el_val_t)added_nodes;
}
+1
View File
@@ -49,6 +49,7 @@
#include <stdint.h>
#include <stdlib.h>
#include <math.h> /* fmod, sin, sqrt, ... — used by codegen'd float arithmetic */
typedef int64_t el_val_t;
+1
View File
@@ -1072,6 +1072,7 @@ el_val_t __engram_save(el_val_t path) { return engram_save
el_val_t __engram_load(el_val_t path) { return engram_load(path); }
el_val_t __engram_get_node_json(el_val_t id) { return engram_get_node_json(id); }
el_val_t __engram_get_node_by_label(el_val_t label) { return engram_get_node_by_label(label); }
el_val_t __engram_search_json(el_val_t query, el_val_t limit) {
return engram_search_json(query, limit);
+1
View File
@@ -226,6 +226,7 @@ el_val_t __engram_activate(el_val_t query, el_val_t depth);
el_val_t __engram_save(el_val_t path);
el_val_t __engram_load(el_val_t path);
el_val_t __engram_get_node_json(el_val_t id);
el_val_t __engram_get_node_by_label(el_val_t label);
el_val_t __engram_search_json(el_val_t query, el_val_t limit);
el_val_t __engram_scan_nodes_json(el_val_t limit, el_val_t offset);
el_val_t __engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset);
+146 -5
View File
@@ -731,6 +731,41 @@ fn cg_expr(expr: Map<String, Any>) -> String {
// because both share the int64 slot).
}
// -- Float arithmetic / comparison dispatch -------------------------
// When either operand is provably Float, promote BOTH operands to a C
// double and emit real double arithmetic. Arithmetic results are
// re-wrapped into the el_val_t float slot via el_from_float(); the
// integer % becomes fmod(); comparisons yield a bare 0/1. Without this
// path, float `+` fell to el_str_concat (segfault) and `- * / %` fell
// to integer ops on the raw IEEE-754 bit pattern (garbage). Mixed
// Int/Float promotes the Int side with a (double) cast. This runs
// AFTER temporal dispatch (a Float is never temporal) and BEFORE the
// Int/String paths, and is skipped when either side is a bare Str so
// string concat/equality is never disturbed.
if !str_eq(left_kind, "Str") {
if !str_eq(right_kind, "Str") {
let any_float: Bool = false
if is_float_expr(left) { let any_float = true }
if is_float_expr(right) { let any_float = true }
if any_float {
let l_d: String = float_operand_c(left, left_c)
let r_d: String = float_operand_c(right, right_c)
if str_eq(op, "Plus") { return "el_from_float(" + l_d + " + " + r_d + ")" }
if str_eq(op, "Minus") { return "el_from_float(" + l_d + " - " + r_d + ")" }
if str_eq(op, "Star") { return "el_from_float(" + l_d + " * " + r_d + ")" }
if str_eq(op, "Slash") { return "el_from_float(" + l_d + " / " + r_d + ")" }
if str_eq(op, "Percent") { return "el_from_float(fmod(" + l_d + ", " + r_d + "))" }
if str_eq(op, "Lt") { return "(" + l_d + " < " + r_d + ")" }
if str_eq(op, "Gt") { return "(" + l_d + " > " + r_d + ")" }
if str_eq(op, "LtEq") { return "(" + l_d + " <= " + r_d + ")" }
if str_eq(op, "GtEq") { return "(" + l_d + " >= " + r_d + ")" }
if str_eq(op, "EqEq") { return "(" + l_d + " == " + r_d + ")" }
if str_eq(op, "NotEq") { return "(" + l_d + " != " + r_d + ")" }
// And/Or with a float operand is ill-typed fall through.
}
}
}
if op == "Plus" {
// If either side is a string literal, always concat
if left_kind == "Str" {
@@ -1406,6 +1441,9 @@ fn cg_stmt(stmt: Map<String, Any>, indent: String, declared: [String]) -> [Strin
if str_eq(ltype, "Int") {
add_int_name(name)
}
if str_eq(ltype, "Float") {
add_float_name(name)
}
// Temporal type annotations register the name with the matching
// typed-set so BinOp / comparison codegen routes through the
// typed wrappers and forbids cross-type ops.
@@ -1465,6 +1503,12 @@ fn cg_stmt(stmt: Map<String, Any>, indent: String, declared: [String]) -> [Strin
if is_zone_expr(val) {
add_zone_name(name)
}
// Float inference: an unannotated `let` whose RHS is provably Float
// (literal, math builtin, or float arithmetic) registers the name so
// later `name <op> ...` routes through the double path.
if is_float_expr(val) {
add_float_name(name)
}
let vk: String = val["expr"]
if str_eq(vk, "Int") {
add_int_name(name)
@@ -1784,6 +1828,15 @@ fn is_int_name(name: String) -> Bool {
return str_contains(csv, "," + name + ",")
}
// Float-typed binding set parallel to is_int_name. Populated from `let x:
// Float = ...` annotations, `: Float` parameters, and inference from a Float
// RHS. Consulted by is_float_expr to route arithmetic through the double path.
fn is_float_name(name: String) -> Bool {
let csv: String = state_get("__float_names")
if str_eq(csv, "") { return false }
return str_contains(csv, "," + name + ",")
}
// Same shape as is_int_name, for Instant- and Duration-typed bindings.
// Used by the BinOp/comparison codegen to dispatch arithmetic through the
// typed runtime wrappers (el_instant_add_dur, el_duration_lt, -) and to
@@ -1847,6 +1900,28 @@ fn is_int_call(call_expr: Map<String, Any>) -> Bool {
return false
}
// Known runtime builtins that return Float. Parallel to is_int_call lets a
// Call participate in float arithmetic (and get inferred into __float_names on
// an unannotated `let`). New Float-returning builtins must be added here.
fn is_float_call(call_expr: Map<String, Any>) -> Bool {
let func = call_expr["func"]
let fk: String = func["expr"]
if !str_eq(fk, "Ident") { return false }
let name: String = func["name"]
if str_eq(name, "int_to_float") { return true }
if str_eq(name, "str_to_float") { return true }
if str_eq(name, "json_get_float") { return true }
if str_eq(name, "decimal_round") { return true }
if str_eq(name, "math_pi") { return true }
if str_eq(name, "math_sin") { return true }
if str_eq(name, "math_cos") { return true }
if str_eq(name, "math_sqrt") { return true }
if str_eq(name, "math_log") { return true }
if str_eq(name, "math_ln") { return true }
if str_eq(name, "el_from_float") { return true }
return false
}
// Builtins that return an Instant. Used by is_instant_expr and the BinOp
// dispatch - `now() + 5.seconds` types as Instant only because we can see
// that now() is an Instant-returning Call.
@@ -2255,19 +2330,71 @@ fn is_int_expr(expr: Map<String, Any>) -> Bool {
return false
}
// is_float_expr true when expr is (or evaluates to) a Float-typed value.
// Used in EqEq/NotEq codegen to avoid str_eq on float values.
// is_float_expr true when expr is (or provably evaluates to) a Float value.
// Mirrors is_int_expr: Float literal, Neg of a Float, a Float-typed Ident
// (registered in __float_names), a Float-returning builtin Call, or a Float
// arithmetic BinOp (float propagates: if either operand is Float the result
// is Float). Drives the float-arithmetic dispatch in cg_expr and the EqEq/
// NotEq float comparison. Kept conservative (only provably-Float) so it never
// intercepts a value the Int paths must own the two predicates are mutually
// exclusive on well-typed source.
fn is_float_expr(expr: Map<String, Any>) -> Bool {
let k: String = expr["expr"]
if str_eq(k, "Float") { return true }
if str_eq(k, "Neg") {
let inner = expr["inner"]
let ik: String = inner["expr"]
if str_eq(ik, "Float") { return true }
return is_float_expr(expr["inner"])
}
if str_eq(k, "Ident") {
let name: String = expr["name"]
return is_float_name(name)
}
if str_eq(k, "Call") {
return is_float_call(expr)
}
if str_eq(k, "BinOp") {
let op: String = expr["op"]
if str_eq(op, "Plus") {
if is_float_expr(expr["left"]) { return true }
if is_float_expr(expr["right"]) { return true }
return false
}
if str_eq(op, "Minus") {
if is_float_expr(expr["left"]) { return true }
if is_float_expr(expr["right"]) { return true }
return false
}
if str_eq(op, "Star") {
if is_float_expr(expr["left"]) { return true }
if is_float_expr(expr["right"]) { return true }
return false
}
if str_eq(op, "Slash") {
if is_float_expr(expr["left"]) { return true }
if is_float_expr(expr["right"]) { return true }
return false
}
if str_eq(op, "Percent") {
if is_float_expr(expr["left"]) { return true }
if is_float_expr(expr["right"]) { return true }
return false
}
return false
}
false
}
// float_operand_c render a BinOp operand as a C double for float arithmetic.
// A provably-Float operand round-trips through el_to_float() (recovering the
// double from the int64 bit-slot); anything else is treated as Int-like and
// numerically converted with (double). This is why Float locals/params must
// be annotated `: Float` an unannotated var holding float bits would be
// mis-converted by the (double) cast, exactly as the Int system requires
// annotations to dispatch arithmetic vs concat.
fn float_operand_c(expr: Map<String, Any>, expr_c: String) -> String {
if is_float_expr(expr) { return "el_to_float(" + expr_c + ")" }
return "(double)(" + expr_c + ")"
}
// -- Capability-kind enforcement ----------------------------------------------
//
// A program's top-level block (cgi / service / none) determines which
@@ -2670,6 +2797,7 @@ fn builtin_arity(name: String) -> Int {
if str_eq(name, "engram_save") { return 1 }
if str_eq(name, "engram_load") { return 1 }
if str_eq(name, "engram_get_node_json") { return 1 }
if str_eq(name, "engram_get_node_by_label") { return 1 }
if str_eq(name, "engram_search_json") { return 2 }
if str_eq(name, "engram_scan_nodes_json") { return 2 }
if str_eq(name, "engram_neighbors_json") { return 3 }
@@ -2783,6 +2911,15 @@ fn add_int_name(name: String) -> Bool {
return true
}
fn add_float_name(name: String) -> Bool {
let csv: String = state_get("__float_names")
if str_eq(csv, "") { csv = "," }
let key: String = "," + name + ","
if str_contains(csv, key) { return true }
state_set("__float_names", csv + name + ",")
return true
}
fn add_instant_name(name: String) -> Bool {
let csv: String = state_get("__instant_names")
if str_eq(csv, "") { csv = "," }
@@ -2866,6 +3003,7 @@ fn add_zone_name(name: String) -> Bool {
fn build_int_names_for_params(params: [Map<String, Any>]) -> Bool {
state_set("__int_names", ",")
state_set("__float_names", ",")
state_set("__instant_names", ",")
state_set("__duration_names", ",")
state_set("__calendar_names", ",")
@@ -2884,6 +3022,9 @@ fn build_int_names_for_params(params: [Map<String, Any>]) -> Bool {
if str_eq(ptype, "Int") {
add_int_name(pname)
}
if str_eq(ptype, "Float") {
add_float_name(pname)
}
if str_eq(ptype, "Instant") {
add_instant_name(pname)
}
@@ -0,0 +1,186 @@
#ifndef EL_PLATFORM_WIN_H
#define EL_PLATFORM_WIN_H
/*
* el_platform_win.h — Windows OS-boundary shim for el_runtime.c.
*
* Branch: feat/windows-el-runtime. Included ONLY when _WIN32 is defined; the POSIX build is
* untouched. Goal: let el_runtime.c (a BSD-sockets / dlfcn / fork host) compile and link with
* mingw-w64 into a native neuron.exe, with no behavioural change to the Linux/macOS build.
*
* What it maps:
* - sockets : winsock2 (same call names: socket/bind/listen/accept/recv/send/setsockopt).
* Sockets close with closesocket() (see el_closesocket), and the stack must be
* started once with WSAStartup — done automatically via a load-time constructor.
* - dlsym : el_runtime.c uses dlsym(RTLD_DEFAULT, name) to resolve callback/tool symbols
* exported by the main module. Windows equivalent: GetProcAddress on the process
* module. Link the soul with -Wl,--export-all-symbols so the symbols are findable.
* - popen : mapped to _popen/_pclose.
* - threads : UNCHANGED. mingw-w64 ships winpthreads, so <pthread.h> + -lpthread just work.
*/
#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN
#endif
#include <winsock2.h>
#include <ws2tcpip.h>
#include <windows.h>
#include <io.h>
#include <process.h>
/* Portable headers mingw-w64 provides (verified present). */
#include <stdarg.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h> /* strcasecmp */
#include <ctype.h>
#include <math.h>
#include <time.h>
#include <sys/time.h> /* mingw-w64 provides gettimeofday here */
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <dirent.h>
#include <errno.h>
#include <pthread.h>
/* ── socket close ─────────────────────────────────────────────────────────── */
/* Winsock closes sockets with closesocket(), not close() (close() is for file fds). The POSIX
build defines the same helper as close() so the call sites are identical across platforms. */
static inline int el_closesocket(SOCKET s) { return closesocket(s); }
/* ── setsockopt optval type ───────────────────────────────────────────────── */
/* Winsock's setsockopt takes optval as (const char*); POSIX takes (const void*), so el_runtime.c
passes &int directly. GCC 14+ makes that an error under -Wincompatible-pointer-types. Wrap it so
the runtime's POSIX-style call sites compile unchanged (defined before the macro so the wrapper
itself resolves to the real winsock setsockopt). */
static inline int el_setsockopt(SOCKET s, int level, int optname, const void* optval, int optlen) {
return setsockopt(s, level, optname, (const char*)optval, optlen);
}
#define setsockopt(s, l, o, v, n) el_setsockopt((s), (l), (o), (v), (int)(n))
/* ── winsock init (once, at load) ─────────────────────────────────────────── */
static void el__win_net_init(void) {
static int inited = 0;
if (!inited) { WSADATA w; WSAStartup(MAKEWORD(2, 2), &w); inited = 1; }
}
__attribute__((constructor)) static void el__win_ctor(void) { el__win_net_init(); }
/* ── dlsym → GetProcAddress ───────────────────────────────────────────────── */
#ifndef RTLD_DEFAULT
#define RTLD_DEFAULT ((void*)0)
#endif
static inline void* el_win_dlsym(void* handle, const char* name) {
(void)handle;
return (void*)(uintptr_t)GetProcAddress(GetModuleHandleA(NULL), name);
}
#define dlsym(h, n) el_win_dlsym((h), (n))
/* ── popen / pclose ───────────────────────────────────────────────────────── */
#define popen _popen
#define pclose _pclose
/* ── misc POSIX → Win32 shims ─────────────────────────────────────────────── */
#include <direct.h> /* _mkdir */
#define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */
#define timegm _mkgmtime /* UTC tm → time_t */
/* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */
static inline int setenv(const char* name, const char* value, int overwrite) {
(void)overwrite;
return _putenv_s(name, value ? value : "");
}
static inline int unsetenv(const char* name) {
/* _putenv_s(name, "") sets VAR="" rather than removing it.
* SetEnvironmentVariableA(name, NULL) truly deletes it from the Win32
* env block; then we sync the CRT cache with _putenv("NAME="). */
SetEnvironmentVariableA(name, NULL);
size_t len = strlen(name);
char *buf = (char*)malloc(len + 2);
if (!buf) return -1;
memcpy(buf, name, len);
buf[len] = '=';
buf[len + 1] = '\0';
_putenv(buf);
free(buf);
return 0;
}
/* nanosleep — not available in MSVC/UCRT; approximate with Sleep(). */
static inline int el_nanosleep(const struct timespec *req, struct timespec *rem) {
(void)rem;
DWORD ms = (DWORD)((req->tv_sec * 1000ULL) + (req->tv_nsec / 1000000ULL));
Sleep(ms ? ms : 1);
return 0;
}
#define nanosleep(req, rem) el_nanosleep((req), (rem))
/* localtime_r/gmtime_r: Windows offers localtime_s/gmtime_s with reversed arg order. */
static inline struct tm* localtime_r(const time_t* t, struct tm* out) {
return localtime_s(out, t) == 0 ? out : (struct tm*)0;
}
static inline struct tm* gmtime_r(const time_t* t, struct tm* out) {
return gmtime_s(out, t) == 0 ? out : (struct tm*)0;
}
/* ── libcurl: degradable stubs for the curl-less Windows build ─────────────── */
/* The curl-less validation build (WITH_CURL=0) links no libcurl. el_runtime.c uses libcurl
* unconditionally for its HTTP client / LLM layer; these stubs let it compile and link so the
* runtime, HTTP *server*, graph and memory work natively on Windows. Live outbound HTTP/LLM calls
* degrade to a runtime error (curl_easy_perform returns an error) — matching the documented
* curl-less contract. When HAVE_CURL is defined (WITH_CURL=1) the real <curl/curl.h> is used and
* this whole block is compiled out. POSIX never sees this header, so the POSIX build is untouched. */
#ifndef HAVE_CURL
typedef void CURL;
typedef int CURLcode;
#define CURLE_OK 0
#define CURLE_HTTP_RETURNED_ERROR 22
#define CURL_ERROR_SIZE 256
/* Option ids: values are irrelevant to the no-op setopt below; kept distinct for readability. */
#define CURLOPT_URL 10002
#define CURLOPT_WRITEFUNCTION 20011
#define CURLOPT_WRITEDATA 10001
#define CURLOPT_POSTFIELDS 10015
#define CURLOPT_POSTFIELDSIZE 120
#define CURLOPT_POST 47
#define CURLOPT_HTTPHEADER 10023
#define CURLOPT_TIMEOUT_MS 155
#define CURLOPT_NOSIGNAL 99
#define CURLOPT_USERAGENT 10018
#define CURLOPT_FOLLOWLOCATION 52
#define CURLOPT_ERRORBUFFER 10010
#define CURLOPT_CUSTOMREQUEST 10036
#define CURLOPT_FAILONERROR 45
struct curl_slist { char* data; struct curl_slist* next; };
static inline struct curl_slist* curl_slist_append(struct curl_slist* list, const char* s) {
struct curl_slist* node = (struct curl_slist*)malloc(sizeof(struct curl_slist));
if (!node) return list;
node->data = s ? strdup(s) : NULL;
node->next = NULL;
if (!list) return node;
struct curl_slist* p = list;
while (p->next) p = p->next;
p->next = node;
return list;
}
static inline void curl_slist_free_all(struct curl_slist* list) {
while (list) { struct curl_slist* n = list->next; free(list->data); free(list); list = n; }
}
static inline CURL* curl_easy_init(void) { return (CURL*)malloc(1); }
static inline CURLcode curl_easy_setopt(CURL* h, int opt, ...) { (void)h; (void)opt; return CURLE_OK; }
static inline CURLcode curl_easy_perform(CURL* h) { (void)h; return 7 /* CURLE_COULDNT_CONNECT */; }
static inline void curl_easy_cleanup(CURL* h) { free(h); }
static inline const char* curl_easy_strerror(CURLcode c) {
(void)c; return "libcurl not built in (curl-less build)";
}
#endif /* !HAVE_CURL */
#endif /* EL_PLATFORM_WIN_H */
File diff suppressed because it is too large Load Diff
+12 -36
View File
@@ -117,15 +117,6 @@ el_val_t el_min(el_val_t a, el_val_t b);
void el_retain(el_val_t v);
void el_release(el_val_t v);
/* ── Arena scoping ────────────────────────────────────────────────────────────
* el_arena_push() activates the string arena (if not already active) and
* returns a mark; el_arena_pop(mark) frees all strings allocated since that
* mark. Used by codegen for per-function/statement scoping and by long-running
* EL loops (e.g. the soul daemon's awareness tick) to reclaim per-iteration
* allocations. */
el_val_t el_arena_push(void);
el_val_t el_arena_pop(el_val_t mark);
/* ── List ────────────────────────────────────────────────────────────────── */
el_val_t el_list_new(el_val_t count, ...);
@@ -151,7 +142,6 @@ el_val_t http_get_with_headers(el_val_t url, el_val_t headers_map);
el_val_t http_post_with_headers(el_val_t url, el_val_t body, el_val_t headers_map);
el_val_t http_post_form_auth(el_val_t url, el_val_t form_body, el_val_t auth_header);
el_val_t http_delete(el_val_t url);
el_val_t http_delete_json(el_val_t url, el_val_t json_body);
void http_serve(el_val_t port, el_val_t handler);
void http_set_handler(el_val_t name);
@@ -177,11 +167,6 @@ void http_set_handler(el_val_t name);
void http_serve_v2(el_val_t port, el_val_t handler);
void http_set_handler_v2(el_val_t name);
/* Non-blocking variant of http_serve: runs the accept loop in a background
* pthread and returns immediately so the caller can continue (used by the
* soul daemon to run awareness_run() after starting its HTTP API). */
void http_serve_async(el_val_t port, el_val_t handler);
/* Build an HTTP response envelope. `headers_json` should be a JSON object
* literal like `{"WWW-Authenticate":"Basic"}` (or "" / "{}" for none). The
* returned string carries the discriminator `{"el_http_response":1,...}`
@@ -591,7 +576,6 @@ el_val_t engram_list_layers(void);
el_val_t engram_get_node(el_val_t id);
void engram_strengthen(el_val_t node_id);
void engram_forget(el_val_t node_id);
el_val_t engram_prune_telemetry(el_val_t older_than_ms);
el_val_t engram_node_count(void);
el_val_t engram_search(el_val_t query, el_val_t limit);
el_val_t engram_scan_nodes(el_val_t limit, el_val_t offset);
@@ -610,32 +594,12 @@ el_val_t engram_load(el_val_t path);
* can pass results straight through without round-tripping ElList/ElMap
* through json_stringify. */
el_val_t engram_get_node_json(el_val_t id);
el_val_t engram_get_node_by_label(el_val_t label);
el_val_t engram_search_json(el_val_t query, el_val_t limit);
el_val_t engram_scan_nodes_json(el_val_t limit, el_val_t offset);
el_val_t engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset);
el_val_t engram_neighbors_json(el_val_t node_id, el_val_t max_depth, el_val_t direction);
el_val_t engram_activate_json(el_val_t query, el_val_t depth);
el_val_t engram_stats_json(void);
el_val_t engram_act_stats_json(void);
el_val_t engram_text_health_json(void);
el_val_t engram_cosine_sim(el_val_t id_a, el_val_t id_b);
/* Destructively pop up to `max` newly-formed Hebbian associations as a JSON
* array of {from_id,to_id,weight,hebb}. The learning process (soul daemon) is
* not the process that owns persistence (engram HTTP server); this is how a
* self-formed association crosses that boundary. (2026-08-07 self-review.) */
el_val_t engram_hebb_drain_json(el_val_t max);
/* Document frequency of a term across node labels — term-specificity signal
* for curiosity seed selection. (2026-08-03 self-review.) */
el_val_t engram_label_df(el_val_t term);
/* Best curiosity seed from one node: argmax over idf·position·casing across
* the candidate tokens of its label, falling back to its content when the
* label is a sentinel. Excludes pipe-delimited tabu terms during selection
* and gates candidates to the df band [min_df, max_df]. Returns "" when
* nothing qualifies. (2026-08-13 self-review.) */
el_val_t engram_salient_term(el_val_t node_id, el_val_t max_df,
el_val_t min_df, el_val_t tabu);
el_val_t engram_embed_backfill(el_val_t count);
el_val_t engram_list_layers_json(void);
/* Working memory introspection — count, mean weight, and top-N snapshot.
* Ported from el-compiler/runtime on 2026-06-30 self-review. */
@@ -794,6 +758,18 @@ el_val_t trace_span_start(el_val_t name);
el_val_t trace_span_end(el_val_t span_handle);
el_val_t emit_event(el_val_t name, el_val_t duration_ms);
/* ── Runtime symbols required by the soul modules ──────────────────────────── */
/* All implemented in el_runtime.c but omitted from this release header; the soul dist modules
* reference them directly, so the public header must export them. Declarations only — mirrors the
* mainline el_runtime.h and is platform-independent (no behavioural change to the POSIX build). */
typedef el_val_t (*http_handler_fn)(el_val_t method, el_val_t path, el_val_t body);
typedef el_val_t (*http_handler4_fn)(el_val_t method, el_val_t path, el_val_t body, el_val_t headers);
el_val_t el_arena_push(void);
el_val_t el_arena_pop(el_val_t mark);
void http_serve_async(el_val_t port, el_val_t handler);
el_val_t engram_get_node_by_label(el_val_t label);
el_val_t engram_prune_telemetry(el_val_t older_than_ms);
#ifdef __cplusplus
}
#endif
-64
View File
@@ -1,64 +0,0 @@
# nsbx — Neuron dev-environment Makefile
# ---------------------------------------------------------------------------
# Thin, documented wrappers over the `nsbx` primitive so a newcomer never has to
# memorise the elc/cc build incantation or the sandbox lifecycle. Every target is
# a one-liner over `nsbx`; nothing here reimplements engram logic.
#
# make dev NAME=tim # new branch + worktree + isolated engram, one shot
# make status NAME=tim # inspect it (omit NAME to list all sandboxes)
# make run NAME=tim # poke its API (API=/api/stats by default)
# make test NAME=tim # run the safety rails as checks (nsbx validate)
# make build NAME=tim # compile the worktree's changes into the engram
# make destroy NAME=tim # tear it all down (branch kept)
#
# The isolated engram is ALWAYS a clone of the live store on a NON-default port;
# prod (:8742 / :7770) is untouchable from here.
# ---------------------------------------------------------------------------
# locate nsbx next to this Makefile, regardless of where make is run from
NSBX := $(dir $(realpath $(lastword $(MAKEFILE_LIST))))nsbx
SBX := dev-$(NAME)
API ?= /api/stats
.DEFAULT_GOAL := help
.PHONY: help dev build run test status list destroy bt
help: ## Show this help
@echo "nsbx dev-environment — one-command isolated Neuron dev setup"
@echo ""
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(firstword $(MAKEFILE_LIST)) \
| awk 'BEGIN{FS=":.*?## "}{printf " make %-22s %s\n", $$1, $$2}'
@echo ""
@echo " Variables: NAME=<dev name> (required for most) API=<path> (run)"
@echo " BASE=<git ref> WT=<worktree dir> PORT=<n> ARGS=<extra nsbx flags>"
dev: ## Create branch + worktree + isolated engram (NAME=x [BASE=ref PORT=n WT=dir ARGS=...])
@test -n "$(NAME)" || { echo "usage: make dev NAME=<name>"; exit 2; }
$(NSBX) dev $(NAME) $(if $(BASE),--base $(BASE)) $(if $(PORT),--port $(PORT)) $(if $(WT),--worktree $(WT)) $(ARGS)
build: ## Rebuild the engram from the dev worktree's own source (NAME=x)
@test -n "$(NAME)" || { echo "usage: make build NAME=<name>"; exit 2; }
@wt=$$(python3 -c "import json,os;print(json.load(open(os.path.expanduser('~/.neuron/sandboxes/$(SBX)/dev.json')))['worktree'])" 2>/dev/null); \
test -n "$$wt" || { echo "no dev.json for $(SBX) — run 'make dev NAME=$(NAME)' first"; exit 2; }; \
$(NSBX) build $(SBX) --source "$$wt"
bt: build run ## Fast El loop: rebuild the engram from the worktree, then poke it (NAME=x)
run: ## Poke the isolated engram's API (NAME=x [API=/api/stats])
@test -n "$(NAME)" || { echo "usage: make run NAME=<name> [API=/path]"; exit 2; }
$(NSBX) run $(SBX) api $(API)
test: ## Run the safety rails as checks: zero-loss, reboot, RSS, parity, keystones (NAME=x)
@test -n "$(NAME)" || { echo "usage: make test NAME=<name>"; exit 2; }
$(NSBX) validate $(SBX)
status: ## Show one sandbox's status (NAME=x), or list all if NAME is unset
@if [ -n "$(NAME)" ]; then $(NSBX) status $(SBX); else $(NSBX) list; fi
list: ## List all sandboxes
$(NSBX) list
destroy: ## Tear down engram + worktree (NAME=x [ARGS=--delete-branch])
@test -n "$(NAME)" || { echo "usage: make destroy NAME=<name>"; exit 2; }
$(NSBX) dev-down $(NAME) $(ARGS)
-176
View File
@@ -1,176 +0,0 @@
# nsbx — the Neuron Sandbox
**Dev environment as a primitive.** A reproducible way to run experiments *and code
changes* against the **real** engram runtime on an isolated snapshot of the live
mind — with a gated promote-to-prod path built on the proven rails.
Everyone (Tim, any team member, any agent) gets their own private, safe copy of the
mind to build against. **Prod — the live Neuron on `:8742` (engram) / `:7770`
(soul) — is untouchable from a sandbox.** A sandbox runs a *separate* engram
process, on a *separate* port, against a *separate* clone of the store. The only op
that can ever reach prod is `promote`, which is explicit, gated, and per-use
approved.
It **wraps the real engram binary** — it never reimplements any engram logic. It
generalises two proven proto-sandboxes into one primitive:
- the **cog-arch** build — isolated git worktree + build + clone of the live `.egm` + real C tests
- the **store-fix** cutover — secondary soul + launchctl `bootout → settle → bootstrap` rails
## Quickstart
```bash
export PATH="$PWD:$PATH" # or symlink nsbx onto your PATH
nsbx up # your private copy of the mind (auto-named <user>-dev)
nsbx run <name> api /api/stats # poke it
nsbx validate <name> # prove it: zero-loss, reboot, RSS, retrieval, keystones
nsbx destroy <name> # cheap teardown; live untouched
```
That is the whole loop. Sane defaults: stock prod binary, auto-allocated port
(`8900+`, never `8742`/`7770`), snapshot of the live store.
## One-command dev onboarding — `nsbx dev` (start here)
Going from a clone to *coding on the mind* is a single command. It creates a git
**branch**, a persistent git **worktree**, and an **isolated engram** (a clone of the
live store on a non-default port) — and wires the whole worktree to that clone so you
**cannot hit live `:8742` by accident**.
```bash
make dev NAME=tim # branch wt/tim + worktree + isolated engram, in one shot
cd ~/Development/neuron-technologies/el-worktrees/tim
source .nsbx-env # every ENGRAM_* var now points at YOUR clone
# edit El in the worktree, then the fast loop:
make build NAME=tim # compile your El change into the isolated engram
make run NAME=tim # poke it (API=/api/stats by default)
make test NAME=tim # run the safety rails as checks
make destroy NAME=tim # tear it all down (branch kept; live untouched)
```
**Why this exists:** so provisional/experimental work is built **directly in El against a
throwaway cloned engram** — not prototyped in Python and re-ported later. The El
edit → `make build``make run` loop is the path of least resistance; that double-work is
what stranded the translation faculty for weeks.
### What `nsbx dev <name>` does, in order
1. **branch**`git worktree add -b <prefix><name>` (default prefix `dev/`; a *real named
branch*, never detached HEAD).
2. **worktree** — at a **persistent** path (default `…/el-worktrees/<name>`, override
`NSBX_DEV_WT_ROOT`). It **refuses** `/tmp` — temp dirs are ablated on compaction, which
is the exact "worktree in /tmp + no branch = lost work" failure this designs out.
3. **isolated engram**`nsbx create` under the hood: clone of the live store + WAL +
config, booted on an auto-allocated port (`8900+`, never `:8742`/`:7770`). Stock prod
binary by default (instant); `--build` compiles the worktree's own runtime instead.
4. **env pin** — writes `.nsbx-env` (+ `.envrc` for direnv) into the worktree exporting
`ENGRAM_URL / ENGRAM_PORT / ENGRAM_DATA_DIR / ENGRAM_API_KEY / NEURON_ENGRAM_URL …` — all
pointing at the clone. Nothing references live.
```
nsbx dev <name> [--base REF] [--worktree DIR] [--port N] [--prefix P] [--build] [--no-engram] [--repo R]
nsbx dev-down <name> [--delete-branch] [--repo R] # destroy engram + remove worktree
```
### Makefile targets
| target | does |
|--------|------|
| `make dev NAME=x` | branch + worktree + isolated engram (one shot) |
| `make bt NAME=x` | fast El loop: `build` then `run` |
| `make build NAME=x` | recompile the engram from the worktree's El source |
| `make run NAME=x` | poke the isolated engram (`API=/api/stats`) |
| `make test NAME=x` | rails as checks (`nsbx validate`) |
| `make status [NAME=x]` | inspect one, or `list` all |
| `make destroy NAME=x` | tear down (add `ARGS=--delete-branch` to drop the branch) |
> Note: if a bare `dev` branch already exists in the repo, git can't create `dev/*` names —
> pass `--prefix wt/` (or delete the stray `dev` branch). The tool surfaces git's exact error.
## The code-change dev loop (first-class)
Run *your changed runtime*, not just the stock binary, against a snapshot:
```bash
# build a runtime from a working tree, a git branch, or a prebuilt binary:
nsbx create feat --source /path/to/worktree # elc + cc build from source
nsbx create feat --branch feat/my-change --repo <r> # worktree the branch, then build
nsbx create feat --binary /path/to/engram # use a prebuilt binary
nsbx build feat --source /path/to/worktree # rebuild + hot-restart in place
nsbx validate feat # prove the change is safe
nsbx promote feat --i-approve-prod-cutover # gated rails cutover (see below)
```
The build replicates the engram release recipe exactly:
`elc engram/src/server.el > engram.c` then
`cc -std=c11 -O2 -I lang/runtime engram.c el_runtime.c engram_*.c -lcurl -lpthread`.
## Lifecycle
| op | what it does |
|----|--------------|
| `create <name> [--port N] [--source\|--branch\|--binary]` | consistent snapshot of the live store+WAL+config into an isolated dir; place or **build** the runtime; boot the real engram daemon on an isolated port. Named, versioned (binary sha + egm sha in `manifest.json`), reproducible. |
| `up [name]` | one command: create-if-missing then start; prints the URL. |
| `build <name> --source\|--branch` | rebuild the runtime from a code change and hot-restart on the same clone+port. |
| `run <name> <cmd…>` / `run <name> api <path> [json]` | run an experiment against the real runtime; capture output + before/after stats + wall time. Env: `$SBX_URL $SBX_PORT $SBX_KEY $SBX_DATA $SBX_BIN`. |
| `validate <name>` | the rails as first-class checks (below). |
| `promote <name> [--data] [--i-approve-prod-cutover]` | **the only prod-touching op.** Gated rails cutover. DRY-RUN plan unless approved. |
| `destroy <name>` | stop the isolated daemon, free the port, remove the clone. Live untouched. |
| `list` / `status <name>` | inspect. |
## `validate` — the rails as checks
- **zero-loss-under-load** — node/edge counts hold at/above baseline through ~15s of sustained tick+read load
- **reboot-prove** — counts survive a real stop→start of the daemon
- **rss-bound** — daemon RSS under `NSBX_RSS_BOUND_MB` (default 550 MB, from the store-fix reboot-proof)
- **retrieval-parity** — top-k node ids for a fixed probe set match the create-time baseline
- **keystone-integrity**`kn-efeb4a5b…` and `kn-5b606390…` present and intact
A PASS writes `validate.json` stamped with the binary sha; `promote` refuses unless
the current binary has a fresh PASS on record.
## `promote` — gated cutover (rails only)
Default is a **dry-run plan**. With `--i-approve-prod-cutover` it, in order:
1. **snapshot-first** — back up live `egm`+`wal`+`plist` to `~/.neuron/backups/promote-<name>-<ts>/` with a `rollback.txt`
2. **additive** binary install — copy the validated binary to a *new* file, update the plist `ENGRAM_REAL_BIN` (old binary retained — additive/supersede, never destructive)
3. **rails cutover**`launchctl bootout`**settle-poll** (prints until the job is gone) → `launchctl bootstrap`. Never `pkill`, never `kickstart -k`.
4. **verify**`/api/stats` returns, edges ≥ baseline, keystones intact
5. **auto-rollback armed** — any verify failure restores the plist (and data, if `--data`) and boots the prior binary back via the same rails
## Isolation guarantees
- separate **port** (`8900+`; refuses `8742`/`7770`), separate **store clone**, separate **process**
- a hard guard refuses to boot a sandbox daemon whose data dir resolves to the live store
- sandboxes are plain supervised background processes (not launchd), so teardown is a signal + settle-poll — it can never touch the prod launchd job
- prod is read exactly twice: once for the snapshot, and (only if you approve) during `promote`
## Layout
- tool: `tools/neuron-sandbox/nsbx` (this repo, branch `feat/neuron-sandbox`)
- runtime state: `~/.neuron/sandboxes/<name>/``data/` (clone), `bin/engram`, `build/`, `logs/`, `manifest.json`, `validate.json`, `baseline/`
## Validated (dogfood)
Standing up a sandbox from a live-store clone and reproducing a **known** result:
- **retrieval-parity 25/25** top-k id overlap vs baseline; sandbox boot-stats exactly matched the live baseline captured at snapshot time (10 672 nodes / 32 439 edges) — the wrapped real binary faithfully reloads the live mind
- reboot-prove + zero-loss PASS; RSS 379 MB < 550 MB; keystones intact
- the **cog-arch correspondence-loop** re-run *inside* the sandbox reproduced the known calibration numbers exactly: held-Brier **0.028648 → 0.000586** (98.0% reduction), monotone, **reboot bit-identical**, metastability holds; and the real-store Stance persistence reboot-proved at **10 994-node** scale (`think()` on real 768-dim embeddings) against a scratch copy of the sandbox's own clone — never live
- `promote` dry-run refused to touch prod; teardown freed the port; live `:8742`/`:7770` never perturbed (soul uptime unbroken)
## Migrating existing experiments
Each ad-hoc harness becomes `nsbx run <name> …` (or `--source` build) against a sandbox:
- **cog-arch**`nsbx create x --source <worktree>` then `nsbx run x -- bash cogarch_dogfood.sh` (compiles + runs the real C cognition tests against `$SBX_DATA`)
- **codec / ingest / faculty**`nsbx run x api /api/<endpoint> '<json>'` against the isolated daemon, or a script using `$SBX_URL`/`$SBX_KEY`; measure with the built-in before/after stats
## Env knobs
`NSBX_ROOT`, `NSBX_PORT_BASE`, `NSBX_RSS_BOUND_MB`, `NSBX_REMERGE_THRESHOLD`,
`EL_REPO` (for `elc` + runtime sources), `ENGRAM_LIVE_DATA_DIR`, `ENGRAM_LIVE_PLIST`.
@@ -1,30 +0,0 @@
#!/usr/bin/env bash
# cog-arch correspondence-loop dogfood — RUN INSIDE the sandbox via `nsbx run`.
# Compiles the REAL engram C runtime + cognition tests and reproduces the known
# calibration result (memory 194c69c8): held-Brier 0.028648 -> 0.000586, reboot-proven,
# then reboot-proves the Stance persistence against a SCRATCH COPY of THIS sandbox's
# clone of the real store (never live, never the running daemon's file).
set -euo pipefail
WT="${COGARCH_WT:-/private/tmp/claude-501/-Users-will/6531446d-bc27-4095-930b-e04777c3db4f/scratchpad/cogarch-wt}"
RT="$WT/lang/runtime"; T="$WT/engram/test"
: "${SBX_DATA:?run me via: nsbx run <name> -- bash cogarch_dogfood.sh}"
B="$(mktemp -d)"
echo "### building cog-arch tests against the real engram runtime sources"
cc -std=c11 -O2 -w -I "$RT" -o "$B/test_cognition" \
"$T/test_cognition.c" "$RT/engram_cognition.c" "$RT/engram_reason.c" \
"$RT/engram_geometry.c" "$RT/engram_store.c" "$RT/engram_vindex.c" -lm
cc -std=c11 -O2 -w -I "$RT" -o "$B/test_realstore" \
"$T/test_cognition_realstore.c" "$RT/engram_cognition.c" "$RT/engram_reason.c" \
"$RT/engram_geometry.c" "$RT/engram_store.c" "$RT/engram_vindex.c" -lm
echo; echo "### [A] synthetic correspondence-loop (known: Brier 0.028648 -> 0.000586)"
"$B/test_cognition" | grep -E "held-Brier|reduction|reboot|monotone|metastab|RESULT" || true
echo; echo "### [B] reboot-prove Stance on a SCRATCH COPY of this sandbox's real-store clone"
SCRATCH="$B/store-clone"; mkdir -p "$SCRATCH"
cp -p "$SBX_DATA/neuron.egm" "$SCRATCH/" 2>/dev/null || true
cp -p "$SBX_DATA/neuron.wal" "$SCRATCH/" 2>/dev/null || true
cp -p "$SBX_DATA/conf" "$SCRATCH/" 2>/dev/null || true
cp -p "$SBX_DATA/meta.json" "$SCRATCH/" 2>/dev/null || true
"$B/test_realstore" "$SCRATCH" || true
rm -rf "$B"
-836
View File
@@ -1,836 +0,0 @@
#!/usr/bin/env bash
# nsbx — the Neuron Sandbox: a reproducible primitive for running experiments and
# code changes against the REAL engram runtime on an isolated snapshot of the live
# mind, with a gated promote-to-prod path built on the proven rails.
#
# It WRAPS the real engram binary — it never reimplements any engram logic. The only
# prod-touching op is `promote`, which is explicit, gated, and per-use approved.
#
# Generalises two proven proto-sandboxes:
# - the cog-arch build (isolated git worktree + build + clone of live .egm + real C tests)
# - the store-fix cutover (secondary soul + launchctl bootout->settle->bootstrap rails)
#
# Lifecycle: create -> [build] -> run -> validate -> promote(gated) -> destroy
#
# Rails (always): built offline; NEVER auto-promotes; never touches live :8742/:7770
# except READ for the snapshot and the gated promote; snapshot-first; honest measured
# reporting. Cutover is launchctl bootout -> settle-poll -> bootstrap ONLY —
# never pkill, never kickstart -k.
set -uo pipefail
# ---------------------------------------------------------------- constants ----
LIVE_DATA_DIR="${ENGRAM_LIVE_DATA_DIR:-$HOME/.neuron/engram}"
LIVE_PLIST="${ENGRAM_LIVE_PLIST:-$HOME/Library/LaunchAgents/ai.neuron.engram.plist}"
LIVE_LABEL="ai.neuron.engram"
LIVE_BIND_PORT=8742 # engram — FORBIDDEN for sandboxes
SOUL_PORT=7770 # soul — FORBIDDEN for sandboxes
LIVE_KEY="${ENGRAM_API_KEY:-ntn-user-2026}"
LIVE_URL="http://127.0.0.1:${LIVE_BIND_PORT}"
SBX_ROOT="${NSBX_ROOT:-$HOME/.neuron/sandboxes}"
BACKUP_ROOT="$HOME/.neuron/backups"
EL_REPO="${EL_REPO:-$HOME/Development/neuron-technologies/foundation/el}"
PORT_BASE="${NSBX_PORT_BASE:-8900}"
RSS_BOUND_MB="${NSBX_RSS_BOUND_MB:-550}" # from store-fix reboot-proof (aaf13f88)
REMERGE_THRESHOLD="${NSBX_REMERGE_THRESHOLD:-40000}"
KEYSTONES=( "kn-efeb4a5b-5aff-4759-8a97-7233099be6ee" "kn-5b606390-a52d-4ca2-8e0e-eba141d13440" )
# fixed probe set for retrieval-parity (stable, identity-anchored)
PARITY_QUERIES=( "who am I" "self identity core" "engram store durability" "keystone self anchor" "grounding honesty" )
C_RED=$'\033[31m'; C_GRN=$'\033[32m'; C_YEL=$'\033[33m'; C_DIM=$'\033[2m'; C_BLD=$'\033[1m'; C_0=$'\033[0m'
# ---------------------------------------------------------------- helpers ------
die(){ printf '%serror:%s %s\n' "$C_RED" "$C_0" "$*" >&2; exit 1; }
log(){ printf '%s==>%s %s\n' "$C_BLD" "$C_0" "$*" >&2; }
info(){ printf ' %s\n' "$*" >&2; }
ok(){ printf ' %s%s%s\n' "$C_GRN" "$*" "$C_0" >&2; }
warn(){ printf ' %s%s%s\n' "$C_YEL" "$*" "$C_0" >&2; }
need(){ command -v "$1" >/dev/null 2>&1 || die "missing dependency: $1"; }
now(){ date -u +%Y%m%dT%H%M%SZ; }
sha(){ shasum -a 256 "$1" 2>/dev/null | awk '{print $1}'; }
epoch(){ python3 -c 'import time;print(time.time())'; }
sdir(){ printf '%s/%s' "$SBX_ROOT" "$1"; }
manifest(){ printf '%s/manifest.json' "$(sdir "$1")"; }
mexists(){ [ -f "$(manifest "$1")" ]; }
mget(){ # mget <name> <jsonpath>
python3 -c "import json,sys; d=json.load(open('$(manifest "$1")')); print(d$2)" 2>/dev/null
}
port_free(){ ! (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null; }
alloc_port(){
local p="$PORT_BASE"
while :; do
if [ "$p" = "$LIVE_BIND_PORT" ] || [ "$p" = "$SOUL_PORT" ]; then p=$((p+1)); continue; fi
if port_free "$p" && ! _port_claimed "$p"; then echo "$p"; return 0; fi
p=$((p+1)); [ "$p" -gt 9100 ] && die "no free sandbox port in range"
done
}
_port_claimed(){ # is another sandbox already assigned this port?
local p="$1" d
for d in "$SBX_ROOT"/*/manifest.json; do
[ -f "$d" ] || continue
[ "$(python3 -c "import json;print(json.load(open('$d'))['port'])" 2>/dev/null)" = "$p" ] && return 0
done
return 1
}
live_stats(){ curl -s -m5 "$LIVE_URL/api/stats" 2>/dev/null; }
api(){ # api <name> <path> [json-body]
local name="$1" path="$2" body="${3:-}"
local port; port="$(mget "$name" "['port']")"; [ -n "$port" ] || die "unknown sandbox: $name"
local url="http://127.0.0.1:${port}${path}"
if [ -n "$body" ]; then curl -s -m30 -X POST -H 'Content-Type: application/json' -d "$body" "$url"
else curl -s -m30 "$url"; fi
}
sbx_stats(){ api "$1" "/api/stats"; }
stat_field(){ printf '%s' "$1" | sed -n "s/.*\"$2\":\([0-9]*\).*/\1/p"; }
daemon_pid(){ local f; f="$(sdir "$1")/daemon.pid"; [ -f "$f" ] && cat "$f" || true; }
daemon_alive(){ local p; p="$(daemon_pid "$1")"; [ -n "$p" ] && kill -0 "$p" 2>/dev/null; }
# ---------------------------------------------------------------- elc/build ----
find_elc(){
command -v elc 2>/dev/null && return 0
local arch; arch="$(uname -m)"
case "$arch" in
arm64) echo "$EL_REPO/lang/dist/platform/elc-darwin-arm64";;
x86_64) echo "$EL_REPO/lang/dist/platform/elc-linux-amd64";;
*) echo "$EL_REPO/lang/dist/platform/elc";;
esac
}
# _build_binary <src_tree> <out_bin> <build_log_dir>
# Replicates the proven engram release recipe:
# elc engram/src/server.el > engram.c
# cc -std=c11 -O2 -I lang/runtime engram.c el_runtime.c engram_*.c -lcurl -lpthread
_build_binary(){
local src="$1" out="$2" blog="$3"
local elc server rt
elc="$(find_elc)"; [ -x "$elc" ] || die "elc not found/executable: $elc (set EL_REPO)"
server="$src/engram/src/server.el"; rt="$src/lang/runtime"
[ -f "$server" ] || die "no engram/src/server.el under source tree: $src"
[ -f "$rt/el_runtime.c" ] || die "no lang/runtime/el_runtime.c under source tree: $src (this branch may keep it generated/untracked)"
ls "$rt"/engram_*.c >/dev/null 2>&1 || die "no lang/runtime/engram_*.c engine sources under: $src"
mkdir -p "$blog"
log "build: elc transpile server.el -> engram.c"
"$elc" "$server" > "$blog/engram.c" 2>"$blog/elc.err" || { cat "$blog/elc.err" >&2; die "elc transpile failed"; }
info "engram.c: $(wc -c <"$blog/engram.c" | tr -d ' ') bytes"
log "build: cc link (el_runtime + engram_* engine)"
cc -std=c11 -O2 -w -I "$rt" -o "$out" \
"$blog/engram.c" "$rt/el_runtime.c" "$rt"/engram_*.c \
-lcurl -lpthread 2>"$blog/cc.err" \
|| { grep -i 'error:' "$blog/cc.err" | sort -u | head >&2; die "cc link failed (see $blog/cc.err)"; }
ok "built: $out ($(ls -lh "$out" | awk '{print $5}'), sha $(sha "$out" | cut -c1-12))"
}
# ---------------------------------------------------------------- daemon -------
# start_daemon <name> : boots the sandbox's real engram binary on its isolated
# port against its cloned data dir, with the SAME auto-remerge net the live soul
# uses (so the sandbox faithfully reaches the live edge population on boot).
start_daemon(){
local name="$1" d; d="$(sdir "$name")"
daemon_alive "$name" && { info "already running (pid $(daemon_pid "$name"))"; return 0; }
local port bin data export key
port="$(mget "$name" "['port']")"; bin="$d/bin/engram"; data="$d/data"
key="sbx-$name"; export="$data/.scan-export.reseed-clean.json"
[ -x "$bin" ] || die "sandbox binary missing: $bin"
[ "$port" != "$LIVE_BIND_PORT" ] && [ "$port" != "$SOUL_PORT" ] || die "refusing forbidden port $port"
[ -f "$data/neuron.egm" ] || die "sandbox has no cloned store: $data/neuron.egm"
# HARD guard: never point a sandbox daemon at the live data dir.
[ "$(cd "$data" && pwd -P)" != "$(cd "$LIVE_DATA_DIR" && pwd -P)" ] || die "refusing: sandbox data dir resolves to LIVE store"
log "boot engram on isolated :$port (data=$data)"
(
ENGRAM_DATA_DIR="$data" ENGRAM_BIND=":$port" ENGRAM_API_KEY="$key" \
ENGRAM_STORE=1 ENGRAM_CHRONOCEPTION=1 ENGRAM_SELF_REIFY=1 ENGRAM_GC=1 \
ENGRAM_POOL_FRAMES=16384 ENGRAM_WRITE_BARRIER=1 \
exec "$bin"
) >"$d/logs/daemon.log" 2>&1 &
local pid=$!
echo "$pid" > "$d/daemon.pid"
# readiness poll
local url="http://127.0.0.1:$port" i s
for i in $(seq 1 30); do
s="$(curl -s -m3 "$url/api/stats" 2>/dev/null)"
[ -n "$s" ] && break; sleep 0.5
done
[ -n "$s" ] || { warn "daemon did not become ready (see $d/logs/daemon.log)"; return 1; }
ok "ready pid=$pid boot-stats: $s"
# auto-remerge net (idempotent): match live edge population if the export is present
if [ -f "$export" ]; then
local edges; edges="$(stat_field "$s" edge_count)"
if [ -n "$edges" ] && [ "$edges" -lt "$REMERGE_THRESHOLD" ]; then
log "auto-remerge: booted with $edges edges (< $REMERGE_THRESHOLD) — merging full edge export"
local r; r="$(curl -s -m300 -X POST -H 'Content-Type: application/json' \
-d "{\"_auth\":\"$key\",\"path\":\"$export\"}" "$url/api/load-merge" 2>/dev/null)"
info "remerge resp: ${r:0:120}"
ok "post-remerge stats: $(curl -s -m5 "$url/api/stats")"
fi
fi
return 0
}
# stop_daemon <name> : graceful TERM + settle-poll until the port is free.
# (Sandbox daemons are plain supervised bg processes — not launchd — so teardown
# is a signal + poll, never pkill of anything else.)
stop_daemon(){
local name="$1" pid port
pid="$(daemon_pid "$name")"; port="$(mget "$name" "['port']")"
[ -n "$pid" ] || { info "not running"; return 0; }
log "stop daemon pid=$pid, settle-poll until :$port frees"
kill "$pid" 2>/dev/null || true
local i
for i in $(seq 1 40); do
kill -0 "$pid" 2>/dev/null || { port_free "$port" && { ok "stopped, port $port free"; : >"$(sdir "$name")/daemon.pid"; return 0; }; }
printf '.' >&2; sleep 0.5
done
printf '\n' >&2
kill -9 "$pid" 2>/dev/null || true; sleep 1
: >"$(sdir "$name")/daemon.pid"
port_free "$port" && ok "stopped (after SIGKILL), port $port free" || warn "port $port still busy"
}
# ================================================================ create =======
cmd_create(){
local name="" port="" src="" branch="" repo="$EL_REPO" binpath=""
# first positional arg is the name unless it's a flag; default to "<user>-dev"
if [ $# -gt 0 ] && [ "${1#-}" = "$1" ]; then name="$1"; shift; else name="${USER:-dev}-dev"; fi
while [ $# -gt 0 ]; do case "$1" in
--port) port="$2"; shift 2;;
--source) src="$2"; shift 2;;
--branch) branch="$2"; shift 2;;
--repo) repo="$2"; shift 2;;
--binary) binpath="$2"; shift 2;;
*) die "unknown flag: $1";;
esac; done
mexists "$name" && die "sandbox '$name' already exists (destroy it first)"
need curl; need python3; need shasum
[ -f "$LIVE_DATA_DIR/neuron.egm" ] || die "live store not found: $LIVE_DATA_DIR/neuron.egm"
if [ -n "$port" ]; then
{ [ "$port" = "$LIVE_BIND_PORT" ] || [ "$port" = "$SOUL_PORT" ]; } && die "refusing forbidden port $port (live)"
port_free "$port" || die "port $port already in use"
else port="$(alloc_port)"; fi
local d; d="$(sdir "$name")"
mkdir -p "$d/data" "$d/bin" "$d/logs" "$d/build" "$d/baseline"
log "sandbox '$name' at $d (isolated port $port)"
# ---- CONSISTENT snapshot of the live mind (file-copy: same set the rails backup
# uses; WAL replay on sandbox boot reconciles the tail -> crash-consistent) ----
log "snapshot live store -> clone (store + WAL + config)"
local f
for f in neuron.egm neuron.wal conf meta.json self_anchor .scan-export.reseed-clean.json; do
if [ -e "$LIVE_DATA_DIR/$f" ]; then cp -p "$LIVE_DATA_DIR/$f" "$d/data/$f"; info "cloned $f ($(du -h "$d/data/$f" | awk '{print $1}'))"; fi
done
local egm_sha; egm_sha="$(sha "$d/data/neuron.egm")"
# ---- capture live baseline (READ only) ----
local lstats; lstats="$(live_stats)"
local base_nodes base_edges
base_nodes="$(stat_field "$lstats" node_count)"; base_edges="$(stat_field "$lstats" edge_count)"
info "live baseline stats: ${lstats:-<unavailable>}"
# ---- determine + place the runtime binary (versioned into the snapshot) ----
local source_desc live_bin
live_bin="$(_live_real_bin)"
if [ -n "$binpath" ]; then
[ -x "$binpath" ] || die "not an executable binary: $binpath"
cp -p "$binpath" "$d/bin/engram"; source_desc="prebuilt:$binpath"
elif [ -n "$src" ]; then
_build_binary "$src" "$d/bin/engram" "$d/build"; source_desc="source:$src"
elif [ -n "$branch" ]; then
log "worktree: $repo @ $branch -> $d/build/worktree"
git -C "$repo" worktree add --detach "$d/build/worktree" "$branch" >/dev/null 2>&1 \
|| die "git worktree add failed ($repo @ $branch)"
_build_binary "$d/build/worktree" "$d/bin/engram" "$d/build"; source_desc="branch:$branch@$repo"
else
[ -x "$live_bin" ] || die "cannot resolve live ENGRAM_REAL_BIN: $live_bin"
cp -p "$live_bin" "$d/bin/engram"; source_desc="stock-prod:$live_bin"
fi
local bin_sha; bin_sha="$(sha "$d/bin/engram")"
info "runtime: $source_desc (sha ${bin_sha:0:12})"
# ---- write manifest ----
python3 - "$name" "$port" "$source_desc" "$bin_sha" "$egm_sha" "$base_nodes" "$base_edges" "$(sha "$live_bin" 2>/dev/null)" <<'PY' > "$(manifest "$name")"
import json,sys,datetime
name,port,src,binsha,egmsha,bn,be,livebinsha=sys.argv[1:9]
json.dump({
"name":name,"port":int(port),"created_at":datetime.datetime.now(datetime.timezone.utc).isoformat(),
"source":src,"binary_sha256":binsha,"clone_egm_sha256":egmsha,
"live_binary_sha256":livebinsha,
"live_baseline":{"node_count":int(bn or 0),"edge_count":int(be or 0)},
"keystones":["kn-efeb4a5b-5aff-4759-8a97-7233099be6ee","kn-5b606390-a52d-4ca2-8e0e-eba141d13440"]
}, sys.stdout, indent=2)
PY
ok "manifest written"
# ---- boot + capture the sandbox's own settled baseline (reproducible target) ----
start_daemon "$name" || die "daemon failed to start"
local sstats; sstats="$(sbx_stats "$name")"
local sbn sbe; sbn="$(stat_field "$sstats" node_count)"; sbe="$(stat_field "$sstats" edge_count)"
_capture_retrieval "$name" "$d/baseline/retrieval.json"
# fold sandbox baseline into manifest
python3 - "$(manifest "$name")" "$sbn" "$sbe" <<'PY'
import json,sys
mf,bn,be=sys.argv[1],sys.argv[2],sys.argv[3]
d=json.load(open(mf)); d["sbx_baseline"]={"node_count":int(bn or 0),"edge_count":int(be or 0)}
json.dump(d,open(mf,'w'),indent=2)
PY
log "created."
info "sandbox baseline (settled): nodes=$sbn edges=$sbe"
info "next: nsbx validate $name | nsbx run $name api /api/stats"
}
_live_real_bin(){
python3 - "$LIVE_PLIST" <<'PY' 2>/dev/null
import sys,plistlib
try:
d=plistlib.load(open(sys.argv[1],'rb'))
print(d.get("EnvironmentVariables",{}).get("ENGRAM_REAL_BIN",""))
except Exception: print("")
PY
}
_capture_retrieval(){ # <name> <outfile> : top-k ids for the fixed probe set
local name="$1" out="$2" q res
local port; port="$(mget "$name" "['port']")"; local key="sbx-$name"
{
echo "{"
local first=1
for q in "${PARITY_QUERIES[@]}"; do
res="$(curl -s -m10 -X POST -H 'Content-Type: application/json' \
-d "{\"_auth\":\"$key\",\"query\":\"$q\",\"limit\":5}" "http://127.0.0.1:$port/api/search" 2>/dev/null)"
local ids; ids="$(printf '%s' "$res" | python3 -c 'import sys,json
try:
d=json.load(sys.stdin)
rows=d if isinstance(d,list) else d.get("results",d.get("hits",[]))
print(json.dumps([r.get("id") for r in rows][:5]))
except Exception: print("[]")' 2>/dev/null)"
[ $first -eq 1 ] || echo ","; first=0
printf ' %s: %s' "$(python3 -c "import json,sys;print(json.dumps(sys.argv[1]))" "$q")" "${ids:-[]}"
done
echo ""; echo "}"
} > "$out"
}
# ================================================================ up ===========
# Dead-simple one-command dev environment: `nsbx up` gives you (or Tim, or anyone)
# a private, isolated copy of the live mind to build against. Creates it on first
# run with sane defaults (stock prod binary, auto-allocated port), just starts it
# thereafter. Prod on :$LIVE_BIND_PORT/:$SOUL_PORT is unreachable from here by design.
cmd_up(){
local name; if [ $# -gt 0 ] && [ "${1#-}" = "$1" ]; then name="$1"; shift; else name="${USER:-dev}-dev"; fi
if mexists "$name"; then daemon_alive "$name" || start_daemon "$name"; else cmd_create "$name" "$@"; fi
local port; port="$(mget "$name" "['port']")"
echo >&2
ok "your sandbox '$name' is ready at http://127.0.0.1:$port (a private copy of the mind — prod is untouchable)"
info "experiment: nsbx run $name api /api/stats"
info "prove it: nsbx validate $name"
info "tear down: nsbx destroy $name"
}
# ================================================================ build ========
# Rebuild an existing sandbox's runtime from a source tree/branch and hot-restart
# it on the SAME clone + port (the code-change dev loop, in place).
cmd_build(){
local name="$1"; shift || true
mexists "$name" || die "no such sandbox: $name"
local src="" branch="" repo="$EL_REPO"
while [ $# -gt 0 ]; do case "$1" in
--source) src="$2"; shift 2;; --branch) branch="$2"; shift 2;; --repo) repo="$2"; shift 2;;
*) die "unknown flag: $1";; esac; done
local d; d="$(sdir "$name")"
stop_daemon "$name"
if [ -n "$src" ]; then _build_binary "$src" "$d/bin/engram" "$d/build"
elif [ -n "$branch" ]; then
rm -rf "$d/build/worktree" 2>/dev/null; git -C "$repo" worktree prune 2>/dev/null
git -C "$repo" worktree add --detach "$d/build/worktree" "$branch" >/dev/null 2>&1 || die "worktree add failed"
_build_binary "$d/build/worktree" "$d/bin/engram" "$d/build"
else die "usage: nsbx build <name> --source DIR | --branch REF [--repo R]"; fi
# record new binary sha
python3 - "$(manifest "$name")" "$(sha "$d/bin/engram")" "${src:-branch:$branch}" <<'PY'
import json,sys; mf,s,src=sys.argv[1:4]
d=json.load(open(mf)); d["binary_sha256"]=s; d["source"]="rebuilt:"+src
json.dump(d,open(mf,'w'),indent=2)
PY
start_daemon "$name"
ok "rebuilt + restarted on :$(mget "$name" "['port']")"
}
# ================================================================ run ==========
cmd_run(){
local name="$1"; shift || true
mexists "$name" || die "no such sandbox: $name"
daemon_alive "$name" || start_daemon "$name"
local d port; d="$(sdir "$name")"; port="$(mget "$name" "['port']")"
# direct API form: nsbx run <name> api <path> [json]
if [ "${1:-}" = "api" ]; then
api "$name" "$2" "${3:-}"; echo; return 0
fi
[ "${1:-}" = "--" ] && shift # allow an explicit separator: nsbx run <name> -- <cmd...>
[ $# -gt 0 ] || die "usage: nsbx run <name> <cmd...> | nsbx run <name> api <path> [json]"
local ts log0; ts="$(now)"; log0="$d/logs/run-$ts.log"
local s0 t0 t1 s1
s0="$(sbx_stats "$name")"; t0="$(epoch)"
log "run experiment against sandbox '$name' (:$port)"
info "cmd: $*"
( export SBX_NAME="$name" SBX_PORT="$port" SBX_URL="http://127.0.0.1:$port" \
SBX_KEY="sbx-$name" SBX_DATA="$d/data" SBX_BIN="$d/bin/engram"
"$@" ) 2>&1 | tee "$log0"
local rc=${PIPESTATUS[0]}
t1="$(epoch)"; s1="$(sbx_stats "$name")"
{
echo "--- nsbx run metrics ---"
echo "exit_code: $rc"
printf 'wall_secs: %.3f\n' "$(python3 -c "print($t1-$t0)")"
echo "stats_before: $s0"
echo "stats_after: $s1"
} | tee -a "$log0" >&2
return $rc
}
# ================================================================ validate =====
# The rails as first-class checks. Baseline = the sandbox's own settled state at
# create (reproducible). zero-loss through sustained load AND reboot; reboot-prove;
# RSS bound; retrieval parity; keystone integrity.
cmd_validate(){
local name="$1"; shift || true
mexists "$name" || die "no such sandbox: $name"
daemon_alive "$name" || start_daemon "$name"
local d port key; d="$(sdir "$name")"; port="$(mget "$name" "['port']")"; key="sbx-$name"
local url="http://127.0.0.1:$port"
local bn be; bn="$(mget "$name" "['sbx_baseline']['node_count']")"; be="$(mget "$name" "['sbx_baseline']['edge_count']")"
log "validate '$name' against baseline nodes=$bn edges=$be"
local -a names=() results=() details=()
# 1) sustained load — no data loss under activity
local s cur_n cur_e i
log "check: sustained load (~15s: tick + reads) then zero-loss"
for i in $(seq 1 15); do
curl -s -m5 -X POST -H 'Content-Type: application/json' -d "{\"_auth\":\"$key\"}" "$url/api/tick" >/dev/null 2>&1
curl -s -m5 "$url/api/stats" >/dev/null 2>&1
done
s="$(sbx_stats "$name")"; cur_n="$(stat_field "$s" node_count)"; cur_e="$(stat_field "$s" edge_count)"
names+=("zero-loss-under-load"); if [ "${cur_n:-0}" -ge "${bn:-0}" ] && [ "${cur_e:-0}" -ge "${be:-0}" ]; then
results+=("PASS"); else results+=("FAIL"); fi
details+=("nodes $cur_n>=$bn, edges $cur_e>=$be")
# 2) reboot-prove — counts survive a real restart
log "check: reboot-prove (stop -> start -> compare)"
local pre_n pre_e; pre_n="$cur_n"; pre_e="$cur_e"
stop_daemon "$name"; start_daemon "$name" >/dev/null
s="$(sbx_stats "$name")"; cur_n="$(stat_field "$s" node_count)"; cur_e="$(stat_field "$s" edge_count)"
names+=("reboot-prove"); if [ "${cur_n:-0}" -ge "${bn:-0}" ] && [ "${cur_e:-0}" -ge "${be:-0}" ]; then
results+=("PASS"); else results+=("FAIL"); fi
details+=("post-reboot nodes=$cur_n edges=$cur_e (pre $pre_n/$pre_e)")
# 3) RSS bound
log "check: RSS bound (< ${RSS_BOUND_MB}MB)"
local pid rss_kb rss_mb; pid="$(daemon_pid "$name")"
rss_kb="$(ps -o rss= -p "$pid" 2>/dev/null | tr -d ' ')"; rss_mb=$(( ${rss_kb:-0} / 1024 ))
names+=("rss-bound"); if [ "$rss_mb" -lt "$RSS_BOUND_MB" ] && [ "$rss_mb" -gt 0 ]; then results+=("PASS"); else results+=("FAIL"); fi
details+=("RSS=${rss_mb}MB (bound ${RSS_BOUND_MB}MB)")
# 4) retrieval parity vs the create-time baseline
log "check: retrieval parity vs baseline probe set"
_capture_retrieval "$name" "$d/logs/retrieval-$( now ).json"
local latest; latest="$(ls -t "$d/logs"/retrieval-*.json 2>/dev/null | head -1)"
local parity; parity="$(python3 - "$d/baseline/retrieval.json" "$latest" <<'PY'
import json,sys
def load(p):
try: return json.load(open(p))
except Exception: return {}
b,c=load(sys.argv[1]),load(sys.argv[2])
tot=hit=0
for q,ids in b.items():
cb=set(ids or []); cc=set(c.get(q) or [])
if not cb: continue
tot+=len(cb); hit+=len(cb & cc)
print(f"{hit}/{tot}" if tot else "0/0")
PY
)"
local ph="${parity%/*}" pt="${parity#*/}"
names+=("retrieval-parity"); if [ "${pt:-0}" -gt 0 ] && [ "${ph:-0}" -eq "${pt:-0}" ]; then results+=("PASS"); else results+=("FAIL"); fi
details+=("top-k id overlap $parity vs baseline")
# 5) keystone integrity
log "check: keystone integrity"
local kfail=0 kid kres
for kid in "${KEYSTONES[@]}"; do
kres="$(curl -s -m5 "$url/api/node/$kid" 2>/dev/null)"
printf '%s' "$kres" | grep -q "\"$kid\"" || kfail=1
done
names+=("keystone-integrity"); [ "$kfail" -eq 0 ] && results+=("PASS") || results+=("FAIL")
details+=("kn-efeb4a5b + kn-5b606390 present")
# ---- report + stamp ----
echo >&2
printf '%s VALIDATION — %s%s\n' "$C_BLD" "$name" "$C_0" >&2
local allpass=1 j
for j in "${!names[@]}"; do
local r="${results[$j]}" c="$C_GRN"; [ "$r" = FAIL ] && { c="$C_RED"; allpass=0; }
printf ' %s%-6s%s %-22s %s%s%s\n' "$c" "$r" "$C_0" "${names[$j]}" "$C_DIM" "${details[$j]}" "$C_0" >&2
done
local status; [ "$allpass" -eq 1 ] && status="PASS" || status="FAIL"
python3 - "$d/validate.json" "$status" "$(sha "$d/bin/engram")" "$(now)" "${names[*]}" "${results[*]}" <<'PY'
import json,sys
out,status,binsha,ts,ns,rs=sys.argv[1:7]
checks=[{"name":n,"result":r} for n,r in zip(ns.split(),rs.split())]
json.dump({"status":status,"binary_sha256":binsha,"ts":ts,"checks":checks},open(out,'w'),indent=2)
PY
printf ' %s==> %s%s\n' "$([ "$allpass" -eq 1 ] && echo "$C_GRN" || echo "$C_RED")" "$status" "$C_0" >&2
[ "$allpass" -eq 1 ]
}
# ================================================================ promote ======
# The ONLY prod-touching op. Explicit, gated, per-use Will-approved. Rails ONLY:
# snapshot-first -> additive binary swap -> launchctl bootout -> settle-poll ->
# bootstrap -> verify -> auto-rollback on failure. NEVER pkill, NEVER kickstart -k.
# Default is a DRY-RUN plan; requires --i-approve-prod-cutover to actually cut over.
cmd_promote(){
local name="$1"; shift || true
mexists "$name" || die "no such sandbox: $name"
local approve=0 do_data=0
while [ $# -gt 0 ]; do case "$1" in
--i-approve-prod-cutover) approve=1; shift;;
--data) do_data=1; shift;;
*) die "unknown flag: $1";; esac; done
local d; d="$(sdir "$name")"
# GATE 1: validation must have passed for the CURRENT binary
[ -f "$d/validate.json" ] || die "GATE: no validation on record — run 'nsbx validate $name' first"
local vstatus vsha bsha
vstatus="$(python3 -c "import json;print(json.load(open('$d/validate.json'))['status'])")"
vsha="$(python3 -c "import json;print(json.load(open('$d/validate.json'))['binary_sha256'])")"
bsha="$(sha "$d/bin/engram")"
[ "$vstatus" = PASS ] || die "GATE: last validation status is $vstatus (must be PASS)"
[ "$vsha" = "$bsha" ] || die "GATE: validation is stale — binary changed since validate (re-run validate)"
local live_bin new_bin ts; ts="$(now)"
live_bin="$(_live_real_bin)"
new_bin="$HOME/.neuron/bin/engram.promote-$name-$ts" # additive: new file, old kept
local bkp="$BACKUP_ROOT/promote-$name-$ts"
log "PROMOTE PLAN for '$name' -> live :$LIVE_BIND_PORT"
info "current live ENGRAM_REAL_BIN : $live_bin"
info "sandbox binary (validated) : $d/bin/engram (sha ${bsha:0:12})"
info "will install as : $new_bin (additive; old binary retained)"
info "snapshot-first backup dir : $bkp (egm+wal+plist+rollback.txt)"
info "data promote : $([ $do_data -eq 1 ] && echo 'YES (--data: clone egm/wal -> live)' || echo 'no (binary only)')"
info "rails : launchctl bootout -> settle-poll -> bootstrap"
info "verify : /api/stats + edges>=baseline + keystones + retrieval; auto-rollback armed"
if [ "$approve" -ne 1 ]; then
warn "DRY-RUN — not touching prod. Re-run with --i-approve-prod-cutover to execute (per-use Will-approved)."
return 0
fi
need launchctl
local dom="gui/$(id -u)"
# ---- snapshot-first ----
log "snapshot-first backup -> $bkp"
mkdir -p "$bkp"
cp -p "$LIVE_DATA_DIR/neuron.egm" "$bkp/neuron.egm.bak"
cp -p "$LIVE_DATA_DIR/neuron.wal" "$bkp/neuron.wal.bak" 2>/dev/null || true
cp -p "$LIVE_PLIST" "$bkp/plist.bak"
printf 'rollback REAL_BIN=%s\nNEWBIN=%s\ndata_promote=%s\n' "$live_bin" "$new_bin" "$do_data" > "$bkp/rollback.txt"
ok "backup complete"
# ---- additive binary install + plist supersede ----
cp -p "$d/bin/engram" "$new_bin"
python3 - "$LIVE_PLIST" "$new_bin" <<'PY'
import sys,plistlib
p,new=sys.argv[1],sys.argv[2]
d=plistlib.load(open(p,'rb')); d.setdefault("EnvironmentVariables",{})["ENGRAM_REAL_BIN"]=new
plistlib.dump(d,open(p,'wb'))
PY
ok "installed $new_bin + updated plist ENGRAM_REAL_BIN"
# ---- optional data promote (after backup) ----
if [ $do_data -eq 1 ]; then
log "data promote: clone store -> live (backed up above)"
cp -p "$d/data/neuron.egm" "$LIVE_DATA_DIR/neuron.egm"
cp -p "$d/data/neuron.wal" "$LIVE_DATA_DIR/neuron.wal" 2>/dev/null || true
fi
# ---- rails cutover: bootout -> settle-poll -> bootstrap ----
log "rails: launchctl bootout $dom/$LIVE_LABEL"
launchctl bootout "$dom/$LIVE_LABEL" 2>/dev/null || true
local i
for i in $(seq 1 60); do
launchctl print "$dom/$LIVE_LABEL" >/dev/null 2>&1 || { ok "settle: job gone after ${i}x0.5s"; break; }
printf ' settle: job still present (%d)\n' "$i" >&2; sleep 0.5
done
log "rails: launchctl bootstrap $dom <plist>"
launchctl bootstrap "$dom" "$LIVE_PLIST" || warn "bootstrap returned nonzero"
# ---- verify ----
log "verify prod health"
local s="" ; for i in $(seq 1 60); do s="$(live_stats)"; [ -n "$s" ] && break; sleep 1; done
local ok_verify=1 le; le="$(stat_field "$s" edge_count)"
local base_e; base_e="$(mget "$name" "['live_baseline']['edge_count']")"
[ -n "$s" ] || ok_verify=0
[ "${le:-0}" -ge "${base_e:-0}" ] || ok_verify=0
local kid; for kid in "${KEYSTONES[@]}"; do curl -s -m5 "$LIVE_URL/api/node/$kid" 2>/dev/null | grep -q "\"$kid\"" || ok_verify=0; done
if [ "$ok_verify" -eq 1 ]; then
ok "PROMOTED. live stats: $s (rollback: $bkp)"; return 0
fi
# ---- auto-rollback ----
warn "verify FAILED — auto-rollback"
cp -p "$bkp/plist.bak" "$LIVE_PLIST"
[ $do_data -eq 1 ] && { cp -p "$bkp/neuron.egm.bak" "$LIVE_DATA_DIR/neuron.egm"; cp -p "$bkp/neuron.wal.bak" "$LIVE_DATA_DIR/neuron.wal" 2>/dev/null || true; }
launchctl bootout "$dom/$LIVE_LABEL" 2>/dev/null || true
for i in $(seq 1 60); do launchctl print "$dom/$LIVE_LABEL" >/dev/null 2>&1 || break; sleep 0.5; done
launchctl bootstrap "$dom" "$LIVE_PLIST" || true
die "ROLLED BACK to $live_bin. See $bkp"
}
# ================================================================ destroy ======
cmd_destroy(){
local name="$1"; shift || true
mexists "$name" || die "no such sandbox: $name"
local d; d="$(sdir "$name")"
stop_daemon "$name"
if [ -d "$d/build/worktree" ]; then
log "removing git worktree"
git -C "$EL_REPO" worktree remove --force "$d/build/worktree" 2>/dev/null || true
git -C "$EL_REPO" worktree prune 2>/dev/null || true
fi
log "removing $d"
rm -rf "$d"
ok "destroyed '$name' (live untouched)"
}
# ================================================================ list/status ==
cmd_list(){
[ -d "$SBX_ROOT" ] || { echo "no sandboxes"; return 0; }
printf '%-16s %-6s %-8s %-9s %s\n' NAME PORT STATE PID SOURCE
local m
for m in "$SBX_ROOT"/*/manifest.json; do
[ -f "$m" ] || continue
local n p src pid state
n="$(python3 -c "import json;print(json.load(open('$m'))['name'])")"
p="$(python3 -c "import json;print(json.load(open('$m'))['port'])")"
src="$(python3 -c "import json;print(json.load(open('$m'))['source'])")"
pid="$(daemon_pid "$n")"; state="stopped"; daemon_alive "$n" && state="running"
printf '%-16s %-6s %-8s %-9s %s\n' "$n" "$p" "$state" "${pid:-}" "$src"
done
}
cmd_status(){
local name="$1"; mexists "$name" || die "no such sandbox: $name"
python3 -m json.tool "$(manifest "$name")"
daemon_alive "$name" && echo "state: running (pid $(daemon_pid "$name")) stats: $(sbx_stats "$name")" || echo "state: stopped"
[ -f "$(sdir "$name")/validate.json" ] && { echo "--- last validation ---"; python3 -m json.tool "$(sdir "$name")/validate.json"; }
}
# ================================================================ dev ==========
# ONE-COMMAND isolated dev environment. Everything a newcomer (Tim, any agent)
# needs to go from clone -> coding on an isolated running mind, in a single shot:
# 1) a git BRANCH (dev/<name>, or --prefix)
# 2) a git WORKTREE for it, at a visible path they can open + edit
# 3) an ISOLATED engram bound to a NON-default port, on a clone of the live store
# (stock prod binary by default — instant + safe; --build to compile the
# worktree's own runtime instead). Prod :$LIVE_BIND_PORT/:$SOUL_PORT is untouchable.
#
# This is additive sugar over the proven primitives (git worktree + cmd_create).
# It never binds a forbidden port and never touches ~/.neuron/engram (the live store)
# except the same READ-only snapshot cmd_create already performs.
#
# nsbx dev <name> [--repo R] [--base REF] [--worktree DIR] [--port N]
# [--prefix P] [--build] [--no-engram]
cmd_dev(){
local name="" repo="$EL_REPO" base="" wt="" port="" prefix="dev/" build=0 no_engram=0
[ $# -gt 0 ] && [ "${1#-}" = "$1" ] && { name="$1"; shift; } || die "usage: nsbx dev <name> [flags]"
while [ $# -gt 0 ]; do case "$1" in
--repo) repo="$2"; shift 2;;
--base) base="$2"; shift 2;;
--worktree|--wt) wt="$2"; shift 2;;
--port) port="$2"; shift 2;;
--prefix) prefix="$2"; shift 2;;
--build) build=1; shift;;
--no-engram) no_engram=1; shift;;
*) die "unknown flag: $1";;
esac; done
need git
git -C "$repo" rev-parse --git-dir >/dev/null 2>&1 || die "not a git repo: $repo"
local branch="${prefix}${name}"
local sbx="dev-${name}"
# default worktree path: a PERSISTENT, git-managed dir — NEVER /tmp (which is
# ablated on compaction). Default root = <repo-grandparent>/el-worktrees, i.e.
# ~/Development/neuron-technologies/el-worktrees/<name>. Override with NSBX_DEV_WT_ROOT.
local wt_root="${NSBX_DEV_WT_ROOT:-$(cd "$(dirname "$(dirname "$repo")")" && pwd -P)/el-worktrees}"
[ -n "$wt" ] || wt="${wt_root}/${name}"
case "$wt" in /tmp/*|/private/tmp/*|/var/tmp/*)
die "refusing worktree under a temp dir ($wt) — temp dirs are ablated on compaction; set NSBX_DEV_WT_ROOT to a persistent path";;
esac
# default base: whatever the repo's working checkout is on now
[ -n "$base" ] || base="$(git -C "$repo" rev-parse --abbrev-ref HEAD 2>/dev/null)"
# pre-flight (fail before creating anything)
[ "$no_engram" -eq 1 ] || ! mexists "$sbx" || die "engram sandbox '$sbx' already exists (nsbx dev-down $name first)"
[ -e "$wt" ] && die "worktree path already exists: $wt"
if [ -n "$port" ]; then
{ [ "$port" = "$LIVE_BIND_PORT" ] || [ "$port" = "$SOUL_PORT" ]; } && die "refusing forbidden port $port (live)"
fi
log "dev env '$name' (branch=$branch worktree=$wt base=$base)"
# ---- 1+2) branch + worktree in one shot ----
local gerr
if git -C "$repo" show-ref --verify --quiet "refs/heads/$branch"; then
info "branch $branch exists — checking it out into a new worktree"
gerr="$(git -C "$repo" worktree add "$wt" "$branch" 2>&1)" \
|| die "git worktree add failed for existing branch $branch:"$'\n'" $gerr"
else
gerr="$(git -C "$repo" worktree add -b "$branch" "$wt" "$base" 2>&1)" \
|| die "git worktree add -b $branch (base $base) failed:"$'\n'" $gerr"$'\n'" (a bare 'dev' branch blocks 'dev/*' names — try --prefix, e.g. nsbx dev $name --prefix wt/)"
fi
ok "worktree ready: $wt (branch $branch)"
# ---- 3) isolated engram ----
local eport="(none)"
if [ "$no_engram" -eq 1 ]; then
warn "--no-engram: skipped standing up an engram"
else
if [ "$build" -eq 1 ]; then
log "isolated engram: building the worktree's own runtime"
cmd_create "$sbx" ${port:+--port "$port"} --source "$wt" || die "engram create (--build) failed"
else
log "isolated engram: stock prod binary on a clone of the live store"
cmd_create "$sbx" ${port:+--port "$port"} || die "engram create failed"
fi
eport="$(mget "$sbx" "['port']")"
# record the dev linkage next to the sandbox so dev-down can clean up
python3 - "$(sdir "$sbx")/dev.json" "$name" "$branch" "$wt" "$repo" "$eport" <<'PY'
import json,sys
p,name,branch,wt,repo,port=sys.argv[1:7]
json.dump({"name":name,"branch":branch,"worktree":wt,"repo":repo,"port":int(port)},
open(p,'w'),indent=2)
PY
# ---- pin the WHOLE worktree to the CLONE ----
# Every var any El tooling in this worktree might read for an engram target now
# points at the isolated clone. Sourcing .nsbx-env makes hitting live :$LIVE_BIND_PORT
# or ~/.neuron/engram by accident structurally impossible from this shell.
local edata ekey eurl ebin
edata="$(sdir "$sbx")/data"; ekey="sbx-$sbx"; eurl="http://127.0.0.1:$eport"; ebin="$(sdir "$sbx")/bin/engram"
cat > "$wt/.nsbx-env" <<ENV
# nsbx dev env for '$name' — SOURCE this to pin THIS shell to the isolated clone.
# The live mind (:$LIVE_BIND_PORT engram / :$SOUL_PORT soul / $LIVE_DATA_DIR) is deliberately
# NOT referenced here. Regenerated by 'nsbx dev'. -> source .nsbx-env
export NSBX_NAME="$sbx"
export ENGRAM_URL="$eurl"
export ENGRAM_BIND=":$eport"
export ENGRAM_PORT="$eport"
export ENGRAM_HOST="127.0.0.1"
export ENGRAM_DATA_DIR="$edata"
export ENGRAM_API_KEY="$ekey"
export NEURON_ENGRAM_URL="$eurl"
export NEURON_ENGRAM_KEY="$ekey"
# nsbx run compatibility (same names 'nsbx run' exports)
export SBX_NAME="$sbx" SBX_PORT="$eport" SBX_URL="$eurl" SBX_KEY="$ekey" SBX_DATA="$edata" SBX_BIN="$ebin"
ENV
# direnv users get it automatically on cd; everyone else runs 'source .nsbx-env'
[ -e "$wt/.envrc" ] || printf 'source_env .nsbx-env 2>/dev/null || source .nsbx-env\n' > "$wt/.envrc"
ok "wrote $wt/.nsbx-env (pins this worktree to the clone)"
fi
# ---- summary ----
echo >&2
printf '%s DEV ENV READY — %s%s\n' "$C_BLD" "$name" "$C_0" >&2
printf ' %-10s %s\n' "worktree" "$wt" >&2
printf ' %-10s %s\n' "branch" "$branch" >&2
if [ "$no_engram" -ne 1 ]; then
printf ' %-10s %s\n' "engram" "http://127.0.0.1:$eport (isolated clone; prod :$LIVE_BIND_PORT untouchable)" >&2
printf ' %-10s %s\n' "sandbox" "$sbx" >&2
echo >&2
printf '%s env exported into %s/.nsbx-env (source it -> pinned to the clone):%s\n' "$C_DIM" "$wt" "$C_0" >&2
grep '^export' "$wt/.nsbx-env" | sed 's/^/ /' >&2
fi
echo >&2
info "code in: cd $wt && source .nsbx-env # now every engram var points at the clone"
[ "$no_engram" -ne 1 ] && info "poke it: nsbx run $sbx api /api/stats (or: make run NAME=$name)"
[ "$no_engram" -ne 1 ] && info "edit->test: make build NAME=$name && make run NAME=$name # El change -> clone, seconds"
info "tear down: nsbx dev-down $name (destroys engram + removes worktree; branch kept)"
}
# nsbx dev-down <name> [--delete-branch] [--repo R]
# Teardown counterpart: destroy the isolated engram, remove the git worktree,
# and (optionally) delete the branch. Live prod is never touched.
cmd_dev_down(){
local name="" repo="$EL_REPO" del_branch=0
[ $# -gt 0 ] && [ "${1#-}" = "$1" ] && { name="$1"; shift; } || die "usage: nsbx dev-down <name> [--delete-branch]"
while [ $# -gt 0 ]; do case "$1" in
--repo) repo="$2"; shift 2;;
--delete-branch) del_branch=1; shift;;
*) die "unknown flag: $1";;
esac; done
local sbx="dev-${name}" wt="" branch="dev/${name}"
# recover worktree/branch/repo from the dev linkage if present
if mexists "$sbx" && [ -f "$(sdir "$sbx")/dev.json" ]; then
local dj; dj="$(sdir "$sbx")/dev.json"
wt="$(python3 -c "import json;print(json.load(open('$dj'))['worktree'])" 2>/dev/null)"
branch="$(python3 -c "import json;print(json.load(open('$dj'))['branch'])" 2>/dev/null)"
repo="$(python3 -c "import json;print(json.load(open('$dj'))['repo'])" 2>/dev/null)"
fi
# 1) engram
if mexists "$sbx"; then cmd_destroy "$sbx"; else info "no engram sandbox '$sbx'"; fi
# 2) worktree
if [ -n "$wt" ] && [ -d "$wt" ]; then
log "removing git worktree $wt"
git -C "$repo" worktree remove --force "$wt" 2>/dev/null || rm -rf "$wt"
git -C "$repo" worktree prune 2>/dev/null || true
ok "worktree removed"
else info "no worktree to remove"; fi
# 3) branch (opt-in)
if [ "$del_branch" -eq 1 ]; then
git -C "$repo" branch -D "$branch" 2>/dev/null && ok "deleted branch $branch" || warn "could not delete branch $branch"
else info "branch $branch kept (use --delete-branch to remove)"; fi
ok "dev-down '$name' complete (live untouched)"
}
usage(){ cat >&2 <<EOF
${C_BLD}nsbx${C_0} — Neuron Sandbox: experiments + code changes against the REAL engram
runtime on an isolated snapshot of the live mind, with a gated promote-to-prod path.
nsbx dev <name> [--base REF] [--worktree DIR] ONE command onboarding: new branch (dev/<name>) + git
[--port N] [--prefix P] [--build] worktree (persistent, never /tmp) + isolated engram on a
[--no-engram] [--repo R] non-default port. clone -> coding on the mind in one shot.
nsbx dev-down <name> [--delete-branch] [--repo R] teardown: destroy the engram + remove the worktree
(branch kept unless --delete-branch). live untouched.
nsbx up [name] [flags…] one command: your private, isolated copy of the mind
(creates on first run, starts thereafter; prod untouchable)
nsbx create [name] [--port N] [--source DIR | --branch REF [--repo R] | --binary PATH]
clone live store+WAL+config, place/build the runtime, boot on an
isolated port (never :$LIVE_BIND_PORT/:$SOUL_PORT). Default runtime = stock prod binary.
nsbx build <name> --source DIR | --branch REF rebuild the runtime from a code change + hot-restart
nsbx run <name> <cmd...> | api <path> [json] run an experiment; capture output + metrics
nsbx validate <name> rails as checks: zero-loss(load+reboot), reboot-prove,
RSS bound, retrieval parity, keystone integrity
nsbx promote <name> [--data] [--i-approve-prod-cutover] GATED rails cutover to prod (DRY-RUN without approval)
nsbx destroy <name> stop daemon, free port, remove clone (live untouched)
nsbx list | nsbx status <name>
Env in 'run' cmds: \$SBX_URL \$SBX_PORT \$SBX_KEY \$SBX_DATA \$SBX_BIN \$SBX_NAME
EOF
}
main(){
local cmd="${1:-}"; shift || true
case "$cmd" in
dev) cmd_dev "$@";;
dev-down) cmd_dev_down "$@";;
up) cmd_up "$@";;
create) cmd_create "$@";;
build) cmd_build "$@";;
run) cmd_run "$@";;
validate) cmd_validate "$@";;
promote) cmd_promote "$@";;
destroy) cmd_destroy "$@";;
list|ls) cmd_list "$@";;
status) cmd_status "$@";;
""|-h|--help|help) usage;;
*) die "unknown command: $cmd (try: nsbx help)";;
esac
}
main "$@"