Compare commits

..

37 Commits

Author SHA1 Message Date
will.anderson 7aa847e32a Merge origin/dev into engram-tiered-storage
El SDK CI - dev / build-and-test (pull_request) Failing after 10m51s
Resolve 3 conflicts:
- lang/el-compiler/runtime/el_runtime.c: keep deletion (deprecated runtime fork;
  single-source-of-truth is lang/runtime/, enforced by scripts/check-single-runtime.sh).
- lang/releases/v1.0.0-20260501/el_runtime.h: keep deletion (releases/ is a generated
  artifact folder, not a source path; a release is a git tag, not a folder).
- lang/runtime/el_platform_win.h: union of dev's Windows port (#80: setsockopt optval
  wrapper + curl-less libcurl stubs) and our fsync(->_commit) shim needed by engram_store WAL.

Nothing in dev's build consumes the deprecated fork or releases/ folder.
2026-08-12 15:23:02 -05:00
will.anderson ee71423732 ci: publish + link engram_store.{c,h} so official builds are store-enabled
El SDK CI - dev / build-and-test (pull_request) Failing after 13m20s
The live engram now runs the paged store (neuron.egm+WAL), but the SDK
release publishes only el_runtime.{c,h} and the engram build links only
el_runtime.c — so a future official release would silently revert to the
in-memory store. Publish engram_store.{c,h} as SDK release assets and add
them to the engram build's download + cc link so the store transition
cannot regress.
2026-08-12 14:22:25 -05:00
will.anderson bb64a236ed engram tiered storage: engram-service wiring + elc fold-hang fix + prune-store mirror
- Wire paged store into the ENGRAM SERVICE (server.el, the authoritative durable
  owner): boot->engram_store_boot, persist_canonical->engram_store_checkpoint,
  gated by ENGRAM_STORE.
- elc (lang/elc.c + src/parser.el + codegen.el + elc-combined.el): OOB guard in
  tok_kind/tok_value + parse_block progress backstop — fixes the pre-existing
  unbounded-memory fold hang on sessions.el.
- engram_prune_telemetry mirrors ISE prune to the store (store_forget) so store
  live-count tracks resident and stale telemetry stays bounded.
- Deployed live 2026-08-12: engram :8742 on neuron.egm+WAL, count reconciled 11552.
2026-08-12 14:14:20 -05:00
will.anderson 9a0266cbf9 engram tiered storage M3.5: persist activation field updates (pre-flip gate)
Flag-on checkpoint now full-walks the resident graph: store_put_node (WM weight,
activation_count, last_activated, wm_anchor) + store_put_edge (hebb, last_fired)
for every node/edge, then engram_checkpoint. Uses store_put_edge (idempotent
upsert) not store_hebb_batch, because activation FORMS new hebbian-associate edges
that bypass the create hook and delta-only hebb_batch can't create them. Store-on
boot now applies the same WM-halving + floor + cap transforms as engram_load.

This is the hebb-survives-restart fix. Gate: reboot from neuron.egm with
snapshot.json deleted -> edge hebb + activation_count survive unchanged, WM weight
survives with identical boot transform; negative control proves persist is
load-bearing (hebb->0 without it). M1 33/33 + M2 36/36 + M3 parity PASS, ASan/UBSan
clean, flag-off untouched. Engine unchanged (boundary held).
2026-08-11 23:37:52 -05:00
will.anderson a72145b44e engram tiered storage M3: wire store behind ENGRAM_STORE (default off) + .egm rename
Caller-side shim in el_runtime.c maps EngramNode/Edge <-> StoreNode/Edge; engine
keeps zero soul deps (libengram boundary, design §10). Flag off = today's JSON
path byte-for-byte (proven: no neuron.egm created, graph identical). Flag on =
engram_open (import snapshot.json once into neuron.egm, else WAL-replay) +
resident load; node/edge create + forget dual-write via guarded hooks. Files
renamed engram.store->neuron.egm, engram.wal->neuron.wal.

Gate: M3 parity PASS (graph on==off byte-exact modulo ordering; snapshot round-trip;
reboot-from-egm with snapshot.json deleted; activation set+sequence identical;
ASan/UBSan clean). M1 33/33 + M2 36/36 green post-rename.

Known gap (pre-flip): in-place hebb/WM/activation_count updates during activation
are not yet persisted to the store (create/connect/forget are). Must close before
live flip so learned edges survive restart.
2026-08-11 23:21:21 -05:00
will.anderson 8affb1d6e0 engram tiered storage M2: WAL + checkpoint + crash recovery + legacy import
Write-back no-steal buffer pool makes the fsync'd WAL load-bearing (M1 was
write-through). Logical WAL with record-granularity page-LSN redo idempotency.
Checkpoint = flush dirty pages, fsync store, advance last_checkpoint_lsn,
reclaim WAL prefix. One-time snapshot.json import only when store absent;
JSON never read as the ongoing store thereafter.

Gates: 33/33 M1 (no regression) + 36/36 M2 — replay parity, torn-tail fuzz
(every byte offset), checkpoint-crash at all 5 phases, torn-page+WAL redo,
legacy-import parity, hebb-survives-crash.
2026-08-11 23:00:20 -05:00
will.anderson fa47b98d18 engram tiered storage M1: on-disk paged store format + round-trip tests
Self-contained paged store (lang/runtime/engram_store.{c,h}): 16KiB slotted pages,
u32 TLV self-describing records (forward-compatible), overflow chains, B+-tree
id-index + from/to adjacency, page free-list, tombstones, double superblock + crc
recovery. Not yet wired to activation (M3). 33/33 tests pass (ASan/UBSan clean);
5k nodes/20k edges round-trip bit-exact incl 768xf32 emb + hebb; store 25MB vs 64MB
JSON. Format is final — see design §2.4.
2026-08-11 22:26:05 -05:00
will.anderson 5d0d4555ae Sync main into dev (GitOps: keep dev current; main authoritative) (#84)
El SDK CI - dev / build-and-test (push) Successful in 8m32s
2026-08-03 15:38:40 +00:00
will.anderson 8347a2f1c0 Merge pull request 'docs: add root README mapping the El monorepo' (#83) from feat/AddingReadme into dev
El SDK CI - dev / build-and-test (push) Successful in 8m22s
2026-07-31 04:25:44 +00:00
Andre Botelho Rodrigues Almeida b97b644799 Addind readme.md file to start documenting the repo
El SDK CI - dev / build-and-test (pull_request) Successful in 8m18s
2026-07-23 16:41:51 -03:00
will.anderson d71fc4c1c0 Merge pull request 'promote stage -> main: reconciled el runtime (engram search + natives + durable truncation fix + Windows port)' (#82) from stage into main
El SDK Release / build-and-release (push) Successful in 8m31s
El SDK CI - dev / build-and-test (pull_request) Successful in 8m41s
2026-07-22 21:44:01 +00:00
will.anderson a118d19393 Merge pull request 'promote dev -> stage: el cluster (#66 engram + #79 truncation fix + release-runtime Windows port)' (#81) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 7m58s
El SDK Release / build-and-release (pull_request) Successful in 4m16s
2026-07-22 21:20:17 +00:00
will.anderson c6aa1e5c53 Merge pull request 'Land el cluster: #66 engram search + natives, #79 truncation fix, + release-runtime Windows port (reconciled)' (#80) from reconcile/el-cluster-windows-runtime into dev
El SDK CI - dev / build-and-test (push) Successful in 8m3s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m25s
Land el cluster (#66 + #79 + release-runtime Windows-port reconciliation) into dev
2026-07-22 21:06:36 +00:00
will.anderson ff577391f2 reconcile(release-runtime): Windows-port + complete v1.0.0 release runtime so the desktop soul cross-compiles
El SDK CI - dev / build-and-test (pull_request) Successful in 7m7s
The desktop soul (neuron/dist) compiles against the v1.0.0-20260501 release
runtime. After #66 landed the engram natives (tokenized/ranked search,
engram_prune_telemetry) and #79 the durable truncation fix into this runtime,
two gaps remained before it could cross-compile the Windows brain:

1. Windows OS boundary: the release runtime had no Win32 path. Ported the same
   _WIN32-guarded shim the mainline runtime carries (#69): #ifdef _WIN32 ->
   el_platform_win.h (winsock/dlsym/popen + WSAStartup ctor), SOCKET fd guards
   and el_closesocket() at every socket site, CreateProcessA for exec_bg, the
   tm_zone/mingw guard, an el_setsockopt optval wrapper (GCC14), and curl-less
   libcurl stubs. Every change is _WIN32/HAVE_CURL-gated — the POSIX build is
   byte-identical (gcc -fsyntax-only clean; native behaviour unchanged).

2. Header exports: the release el_runtime.h omitted symbols the soul dist calls
   that are defined in this runtime's .c — the http_handler_fn/http_handler4_fn
   typedefs and el_arena_push/pop, engram_prune_telemetry, engram_get_node_by_label.
   Declaration-only, POSIX-neutral; fixes implicit-declaration/unknown-type
   errors under the C11 mingw build.

Result: x86_64-w64-mingw32-gcc compiles el_runtime.c + all 48 soul modules
clean; POSIX gcc -fsyntax-only clean. This is the Windows-port PR the runtime
needed on main (the release-runtime counterpart to #69), landed via stage.
2026-07-22 15:56:08 -05:00
will.anderson ee0d5f9b97 Merge #79: durable HTTP response-truncation fix, both runtimes (via stage) 2026-07-22 15:46:02 -05:00
will.anderson 391bd818ea Merge #66: tokenized+ranked engram lexical search + engram natives (via stage) 2026-07-22 15:45:54 -05:00
will.anderson 43636aed99 runtime: pair fs_read length hint with its buffer in BOTH runtimes — kill response truncation for good
El SDK Release / build-and-release (pull_request) Failing after 7s
The binary-safe fs_read length (_tl_fs_read_len) was consumed by the HTTP
response path for ANY body, even when a handler wrapped a smaller file into a
larger reply. Content-Length then lied AND the send stopped short: the
safety-contact (988) routes returned 178 of 208/218 bytes, cut mid-'set_at' —
unparseable JSON. The desktop app read that as failure. On Windows the shipped
brain is an OLD build without even the per-handler workaround, so EVERY reply
truncated: the app can't read confirmations and refuses the new user.

Durable fix: pair the length hint with the exact buffer pointer it describes
(_tl_fs_read_buf). Apply the raw byte count ONLY when the response IS that
buffer (binary file serving stays correct); every wrapped/enveloped/derived
body is measured with strlen. Reset both at request start and in fs_read /
json_get_raw. This also closes the stale-hint heap over-read (a length larger
than a later body would read past it out the socket) that a plain max() leaves
open — so this class of bug dies on every platform, not just where a handler
happened to be patched.

Applied identically to the mainline runtime (lang/el-compiler/runtime) AND the
frozen release runtime (lang/releases/v1.0.0-20260501) the desktop souls
compile against — the release copy still carried the raw leak, which is why the
Windows brain kept truncating. Same proven approach as PR #78 (Tim Lingo),
extended to cover the release runtime and rebased onto current main.

Both runtimes: gcc -fsyntax-only clean.
2026-07-22 15:04:25 -05:00
will.anderson 2baa0b9a41 Merge pull request 'release: promote stage -> main (ci publish hardening for sdk-release)' (#77) from stage into main
El SDK Release / build-and-release (push) Successful in 7m55s
2026-07-15 21:21:28 +00:00
will.anderson 6a8b2461cd Merge pull request 'release: promote dev -> stage (ci publish hardening for stage/main)' (#76) from dev into stage
El SDK CI - stage / build-and-test (push) Successful in 8m19s
El SDK Release / build-and-release (pull_request) Failing after 13m1s
2026-07-15 21:16:11 +00:00
will.anderson bcb356fe69 Merge pull request 'ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly' (#75) from hotfix/ci-stage-main-publish-hardening into dev
El SDK CI - stage / build-and-test (pull_request) Successful in 4m27s
El SDK CI - dev / build-and-test (push) Failing after 14m3s
2026-07-15 21:15:27 +00:00
will.anderson dd7827059a ci(stage,main): decouple ci-base rebuild, make SDK publish fail loudly
El SDK CI - dev / build-and-test (pull_request) Failing after 14m30s
Mirror the PR #72 fix (applied to ci-dev.yaml) onto ci-stage.yaml and
sdk-release.yaml. The stage and prod release jobs reported FAILURE even
when the el-runtime-c/-h publish SUCCEEDED, because the ancillary ci-base
Docker rebuild (a CI-cache optimization on the fragile host-mode GCE
runner) reddened the whole job.

- Rebuild ci-base step: continue-on-error: true — never blocks/reddens
  the job; the SDK publish is the deliverable.
- Publish step: set -euo pipefail + empty-key guard + active-account echo
  so a real publish failure still fails loud and is diagnosable.
2026-07-15 16:14:50 -05:00
will.anderson 208e36c899 Merge pull request 'release: promote stage -> main (tokenized search, get_node_by_label, epm fix, win portability)' (#74) from stage into main
El SDK Release / build-and-release (push) Successful in 8m23s
2026-07-15 18:24:39 +00:00
will.anderson b97ce74d1f Merge pull request 'release: promote dev -> stage (tokenized search, get_node_by_label, epm fix)' (#73) from dev into stage
El SDK CI - stage / build-and-test (push) Failing after 8m45s
El SDK Release / build-and-release (pull_request) Successful in 4m1s
2026-07-15 17:20:11 +00:00
will.anderson 155a449c4e Merge pull request 'ci(dev): make SDK publish fail loudly, decouple ci-base rebuild' (#72) from hotfix/ci-dev-publish-hardening into dev
El SDK CI - dev / build-and-test (push) Successful in 8m56s
El SDK CI - stage / build-and-test (pull_request) Successful in 4m10s
2026-07-15 16:34:14 +00:00
will.anderson 4696fd6833 ci(dev): make SDK publish fail loudly, decouple ci-base rebuild
El SDK CI - dev / build-and-test (pull_request) Successful in 8m51s
The dev push build went green-then-red while nothing published: the
Publish step had no set -e, so an auth/upload failure exited 0 (silent
no-publish), while the ci-base rebuild (set -euo pipefail + Docker on the
host-mode runner) hard-failed the job. Add set -euo pipefail + an empty-key
guard + active-account echo to the Publish step so failures surface with a
retrievable log, and mark the ci-base cache rebuild continue-on-error so
the fragile Docker step can never block the actual SDK artifact publish.
2026-07-15 11:33:37 -05:00
will.anderson 581a351fb1 Merge pull request 'integrate: stack PRs #65–#69 (elc OOM guard, tokenized+semantic engram search, get_node_by_label, win portability) for green CI' (#71) from hotfix/stage-elc-engram-integration into dev
El SDK CI - dev / build-and-test (push) Failing after 14m31s
2026-07-15 15:49:57 +00:00
will.anderson 8ce8656de2 epm: declare cross-module callees as extern fn so strict compilers accept generated C
El SDK CI - dev / build-and-test (pull_request) Successful in 7m33s
epm's sibling modules (registry/install/update) call functions defined in other
modules and in the El runtime (config, read_installed, registry_find,
manifest_deps, manifest_name, registry_latest_version, registry_token,
install_vessel, installed_version) without importing them, so elc emits no C
prototype for those calls. gcc<=13 treated the resulting implicit declarations
as warnings; gcc>=14 and clang reject them as hard errors, which is why the
"Build epm" CI step fails and blocks the whole dev/stage pipeline.

Add `extern fn` forward declarations -- El's own separate-compilation mechanism
-- for each cross-module callee at the top of registry/install/update. This
gives elc the correct C prototype in every generated translation unit, so the
calls compile cleanly and still resolve at link time. Simply suppressing
-Wimplicit-function-declaration would be unsafe: an implicit int return
truncates the 64-bit pointer returns of config/registry_find into a latent
crash, so declaring the true signatures is the correct fix. Localized to epm;
touches neither elc nor the runtime.
2026-07-15 10:14:43 -05:00
will.anderson 1e49560f1f Merge remote-tracking branch 'origin/feat/engram-semantic-search' into hotfix/stage-elc-engram-integration
El SDK CI - dev / build-and-test (pull_request) Failing after 14m39s
# Conflicts:
#	lang/el-compiler/runtime/el_runtime.c
2026-07-15 09:33:05 -05:00
will.anderson e8f0b5a9de Merge remote-tracking branch 'origin/fix/engram-lexical-tokenized-search' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 40287c4cfc Merge remote-tracking branch 'origin/hotfix/win-runtime-portability' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 0481bea44d Merge remote-tracking branch 'origin/hotfix/runtime-engram-get-node-by-label' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 9d565ca080 Merge remote-tracking branch 'origin/hotfix/elc-fixes' into hotfix/stage-elc-engram-integration 2026-07-15 09:28:44 -05:00
will.anderson 4773dd0aa2 runtime: make Windows soul reproducible from a clean el checkout
El SDK Release / build-and-release (pull_request) Failing after 16s
Two el_runtime portability defects only ever lived in staged local copies
used to hand-build neuron-ui PR #136's curl-enabled Windows neuron.exe.
gcc 15 promotes both to hard errors, so a clean el checkout cannot rebuild
that soul. Upstream the minimal fixes so the build is reproducible:

- http_serve_async: cast setsockopt optval to (const char*). Win32/mingw
  setsockopt wants const char*, not int*; the cast is a no-op on POSIX and
  matches the four already-cast sites elsewhere in this file.
- engram_save persist path: map fsync -> _commit in the _WIN32-only
  el_platform_win.h (io.h already included). Windows has no fsync(); the
  POSIX path is untouched.
2026-07-15 04:24:08 -05:00
will.anderson 6b9d9e6c4a Add engram_get_node_by_label runtime native to unblock soul link
El SDK Release / build-and-release (pull_request) Failing after 22s
chat.el calls the runtime native engram_get_node_by_label to fetch
well-known nodes (conv:history, session:summary) by stable label rather
than by ID — immune to vector-index drift across restarts. The current
runtime never defined it, so the regenerated dist/soul.c fails to link.

Backport the function verbatim (idiom-adapted to jb_finish) from release
runtime v1.0.0-20260501 and register it as an EL builtin exactly like its
siblings: runtime definition + prototype, __-prefixed seed wrapper +
prototype, and codegen arity entry. No search-site code is touched.
2026-07-15 04:07:33 -05:00
will.anderson b4967af13e feat(engram): semantic search layer via nomic-embed-text (cosine ∪ lexical)
Lexical istr_contains alone can't surface a node whose words don't appear
in the query. This adds an optional dense-vector layer: node content and the
query are embedded through Ollama (nomic-embed-text), and nodes are ranked by
cosine similarity unioned with lexical hits, so a paraphrase query reaches the
right node.

Wired into all three query entry points in el_runtime.c:
  - engram_search_json (HTTP /api/search): collect lexical ∪ semantic
    candidates, score (lexical base 1.0 + cosine; pure-semantic = cosine),
    rank, emit top-N. Stable sort preserves old order when semantic is off.
  - engram_search (internal el_val twin): lexical ∪ semantic union.
  - engram_activate seed loop (HTTP /api/activate): a node seeds if it
    lexically matches OR clears the cosine threshold; pure-semantic seeds
    enter scaled by cosine so paraphrase spreads without overpowering.

Degradable by design: the whole layer is gated on HAVE_CURL plus a one-shot
runtime probe. If curl is compiled out, Ollama is unreachable, or
ENGRAM_SEMANTIC=0, every entry point yields zero semantic signal and callers
fall back byte-for-byte to the pre-existing lexical search.

Node embeddings are cached in process memory keyed by node id with an FNV-1a
content hash for invalidation; the query is embedded once per call — so the
graph is not re-embedded on every query. nomic task prefixes
(search_query:/search_document:) are applied for retrieval separation.

Build steps gain -DHAVE_CURL so the engram artifact compiles the layer in
(-lcurl was already linked). Env: ENGRAM_SEMANTIC, ENGRAM_EMBED_URL,
ENGRAM_EMBED_MODEL, ENGRAM_SEMANTIC_MIN (cosine threshold, default 0.6).
2026-07-14 18:48:16 -05:00
will.anderson 2b2a1246e7 Merge pull request 'runtime: fix the memory-leak + write-corruption pair in el_runtime.c' (#64) from hotfix/el-runtime-leak-and-persist into main
El SDK Release / build-and-release (push) Failing after 10m52s
2026-07-13 21:23:31 +00:00
will.anderson 5c41c66a0f Merge pull request 'fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only' (#60) from fix/windows-rusage-guard into stage
El SDK CI - stage / build-and-test (push) Failing after 13m21s
fix(windows): guard el_mem_check with _WIN32 — rusage is POSIX-only
2026-06-25 16:48:13 +00:00
30 changed files with 4814 additions and 51 deletions
+15
View File
@@ -214,9 +214,18 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -268,6 +277,12 @@ jobs:
# Patches ci-base:dev in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+15
View File
@@ -212,12 +212,21 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -253,6 +262,12 @@ jobs:
# Patches ci-base:stage in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+19
View File
@@ -218,6 +218,8 @@ jobs:
cp lang/dist/bin/epm dist/sdk/bin/epm
cp lang/runtime/el_runtime.c dist/sdk/runtime/
cp lang/runtime/el_runtime.h dist/sdk/runtime/
cp lang/runtime/engram_store.c dist/sdk/runtime/
cp lang/runtime/engram_store.h dist/sdk/runtime/
cp lang/runtime/*.el dist/sdk/runtime/
tar -czf dist/el-sdk-latest.tar.gz -C dist/sdk .
echo "SDK tarball bundled: dist/el-sdk-latest.tar.gz"
@@ -276,6 +278,8 @@ jobs:
upload_asset lang/dist/platform/elc elc
upload_asset lang/runtime/el_runtime.c el_runtime.c
upload_asset lang/runtime/el_runtime.h el_runtime.h
upload_asset lang/runtime/engram_store.c engram_store.c
upload_asset lang/runtime/engram_store.h engram_store.h
# SDK bundle and installer binary
upload_asset dist/el-sdk-latest.tar.gz el-sdk-latest.tar.gz
@@ -288,12 +292,21 @@ jobs:
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
# Fail loudly: previously this step had no `set -e`, so an auth or
# upload failure was swallowed (step exited 0 on the trailing echo)
# and the SDK silently never published. Surface failures now.
set -euo pipefail
if [ -z "${GCP_SA_KEY:-}" ]; then
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
exit 1
fi
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
apt-get install -y -qq apt-transport-https ca-certificates curl
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -qq && apt-get install -y google-cloud-cli
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
gcloud config set project neuron-785695
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
VERSION="${GITHUB_SHA:0:8}"
@@ -345,6 +358,12 @@ jobs:
# Patches ci-base:latest in-place: pulls the existing image (which has all
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
#
# continue-on-error: this is a CI-cache optimization, NOT the release
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
# runner where DinD/Docker availability is fragile. A failure here must
# never block or redden the job — the SDK publish above is the deliverable.
continue-on-error: true
if: github.event_name == 'push'
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+154
View File
@@ -0,0 +1,154 @@
# El
**A self-hosting, statically-typed language that compiles to C — built around a graph-native runtime instead of a database driver.**
El is the execution substrate for the Neuron agent runtime, the DHARMA network, and the Engram knowledge graph. This repository is the monorepo for the whole stack: the language itself, the graph memory engine it's built to talk to natively, and the tools (package manager, IDE, UI framework, diagramming) built on top of it.
---
## Why El exists
Every other language treats persistent, associative state as something you reach for through a driver — a SQL client, an ORM, a Redis library bolted on from outside. El inverts that: graph operations (`engram_*`) are runtime primitives, on the same footing as string or list operations. There is no separate database driver because the database is not separate.
El has four defining properties:
1. **Self-hosting compiler.** The compiler (`lexer.el`, `parser.el`, `codegen.el`, `compiler.el`) is written in El. It compiles El source to C, which `cc` compiles against a fixed runtime into a native binary. A Rust genesis compiler bootstrapped the first iteration; the self-hosted binary at `lang/dist/platform/elc` has been the canonical compiler ever since — every binary in `dist/platform/` was produced by an earlier version of itself compiling `el-compiler/src/`. The chain is auditable: source is the ground truth, not the binary. See [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) for the full recovery path if that binary is ever lost.
2. **C compilation target.** Every compiled program is plain C11. Every El value is `el_val_t` (`int64_t`); strings are heap pointers cast through it. Functions become C functions; top-level statements become `main()`.
3. **Graph-native runtime.** The runtime provides first-class graph operations over an in-process Engram store — no separate DB driver, no ORM.
4. **DHARMA-aware identity.** A `cgi` block declares a program's DHARMA identity at compile time. The runtime resolves identity before user code runs, so `dharma_*` calls have a stable principal and channel surface throughout.
---
## Architecture map
```
┌─────────────┐
│ lang │ El compiler + C runtime
│ (El itself) │ everything below is written in it,
└──────┬──────┘ or compiles down through it
┌─────────────┼─────────────┐
│ │ │
┌──────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
│ engram │ │ epm │ │ ide │
│ graph/mem │ │ package │ │ editor + │
│ substrate │ │ manager │ │ LSP │
└──────┬─────┘ └───────────┘ └───────────┘
┌───────┼────────────────┬─────────────────────┐
│ │ │ │
┌─────▼───┐ ┌─▼──────────┐ ┌──▼──────────┐ ┌─────▼──────┐
│ elp │ │ ql │ │ ui │ │ arbor │
│ NLG / │ │engram-el. │ |spreading- │ |arbor │
│ 31 langs│ │studio+tests│ |activation UI│ |diagram lang│
└─────────┘ └────────────┘ └─────────────┘ └────────────┘
```
`lang` is the foundation — the compiler and C runtime everything else builds on. `engram` is the graph-native memory/state engine that gives El its identity (property 3 above). Everything else is either a tool for working with El (`epm`, `ide`) or a system built on top of Engram's graph model (`elp`, `ql`, `ui`, `arbor`).
---
## Repository layout
### [lang/](lang/) — the El language
The compiler and runtime. Self-hosting: `elc-cli.el``compiler.el``lexer.el` / `parser.el` / `codegen.el` / `codegen-js.el`, textually inlined and compiled in one pass. Compiles to C11 and links against `el-compiler/runtime/el_seed.c`, a hand-maintained OS-boundary layer (libcurl HTTP, pthreads, filesystem, arena allocation) — everything else in the runtime is native El (`runtime/*.el`).
Two layers to know: **El programs** (`.el` files — where nearly all work belongs) and **the C seed** (`el_seed.c` — edit only for genuine OS-level access; never re-implement what El can already express).
Current status (single source of truth: [lang/spec/language.md](lang/spec/language.md)): lexer/parser/codegen and the C runtime's core (I/O, strings, math, lists, maps, filesystem, args) are implemented. In flight: `%` operator, match-statement codegen, `?` nil-propagation, `cgi` block parsing + DHARMA identity resolution, VBD role enforcement (`@manager`/`@engine`/`@accessor`), the real `engram_*` and `dharma_*` runtimes (currently stubs), and libcurl-backed `http_get`/`http_post`/`http_serve`. Bitwise operators, `??`, and `as` casts are explicitly **not** in this language.
Key docs: [AGENTS.md](lang/AGENTS.md) (agent-facing orientation), [BOOTSTRAP.md](lang/BOOTSTRAP.md) (compiler recovery from scratch), [spec/language.md](lang/spec/language.md), [spec/codegen-js.md](lang/spec/codegen-js.md).
### [engram/](engram/) — graph intelligence substrate
**A local-first memory substrate for accumulating intelligence**, and the reason El's runtime doesn't need a database driver. Rust core (`engram-core`, `engram-ffi`) exposed to El and other languages (Kotlin, TypeScript/WASM, Go bindings).
The model: retrieval is **spreading activation**, not query. You name seed nodes and a query embedding; activation propagates outward through weighted edges, attenuating multiplicatively per hop (`strength = parent_strength × edge_weight × target_salience × cosine_sim`), gets pruned below a threshold, and the top-N nodes by activation strength come back. Storage and retrieval are the same structure — the way long-term potentiation works in biological memory, not the way a relational or vector database works.
Nodes live in four tiers (Working / Episodic / Semantic / Procedural, mirroring prefrontal / hippocampal / neocortical / cerebellar memory) and migrate between them based on **salience decay**`importance × recency-decay × log(activation_count)`. Forgetting is adaptive pruning, not a bug: unreinforced memories stop competing for attention without being deleted.
Backed by `sled` (embedded, local-first, no daemon) with flat cosine scan for vector search — deliberately simple until scale demands an HNSW layer. Full API and design rationale in [engram/README.md](engram/README.md).
### [elp/](elp/) — Engram Language Protocol
Bidirectional engine mapping between Engram semantic forms and natural-language surface text, across **31 languages** — from Spanish and Japanese through historical/liturgical languages (Old Norse, Sanskrit, Sumerian, Coptic, Akkadian, Ge'ez). Compilation order runs `language-profile` + `vocabulary` → per-language `morphology-*``grammar``realizer``semantics``elp`. This is what lets an Engram graph node round-trip to and from readable text in any of those languages.
### [epm/](epm/) — El Package Manager
Manages **vessels** (El's package unit): publish, install, resolve dependencies. Vessels are stored in Engram as graph nodes, not files in a registry index — `epm` reads the local `manifest.el`, talks to Engram over HTTP, and writes resolved vessels to `.epm/vessels/`. Source: `registry.el`, `install.el`, `update.el`, `manifest.el`.
### [ide/](ide/) — El IDE
Three vessels: **el-ide-server** (HTTP backend — file ops, build/run, LSP bridge, plugin host, settings), **el-lsp** (the language server — completion, hover, diagnostics, outline, format, type graph), and **el-plugin-host** (first-party plugin lifecycle: install/remove/enable/disable). `ide/projects/` and `ide/examples/` hold sample projects, including the canonical `hello-friends` first-program walkthrough.
### [ql/](ql/) — engram-el
The El-native integration layer for a *live* Engram server — not a library (no importable modules, no build artifact), a set of standalone `.el` programs run directly via `el run-file`. Three components: **Studio** (`studio/studio.el`, a full terminal graph explorer), a **Hebbian field-model** proof of concept, and El builtin / LLM-builtin smoke test suites. This is the reference for correct patterns when an El program uses Engram as its substrate. Spec: [ql/spec/elql.md](ql/spec/elql.md).
### [ui/](ui/) — el-ui
A frontend framework where **component state is an Engram graph and reactivity is spreading activation** — not virtual-DOM diffing (React), Proxy-based dependency tracking (Vue), or compile-time analysis (Svelte). Re-renders are activated and propagated the same way associative memory retrieval works in `engram/`.
~15 vessels covering the full frontend surface: `el-platform` (env/fs/network/clock abstraction), `el-config`, `el-html` (SSR emit primitives), `el-layout`, `el-style` (design tokens/themes), `el-i18n`, `el-auth` / `el-identity` (JWT, sessions, OAuth PKCE — Engram-native), `el-services` (REST/gRPC/WebSocket bindings), `el-aop` (`@authenticate`/`@authorize`/`@cache`/`@rate_limit` decorators), `el-secrets`, `el-graph` (graph rendering/editor), `el-publish` (App Store / Play Store automation), and `el-ui-compiler` (El→JS component compiler; currently a stub pending a JS backend in `elc`). Spec: [ui/spec/framework.md](ui/spec/framework.md).
### [arbor/](arbor/) — diagram language
A `.arbor` diagram language and toolchain: `arbor-core` (NodeId/shape/edge-kind types), `arbor-parse` (recursive-descent parser), `arbor-diagram` (IR + Mermaid serializer + architecture-diagram builders), `arbor-layout` (hierarchical layout — rank assignment, positioning, group bounds), `arbor-render` (SVG renderer), `arbor-cli`. (The architecture map above is the kind of diagram this is for.)
---
## Getting started
Install the El SDK from the latest release:
```bash
bash lang/install.sh
# EL_VERSION=v1.0.0 bash lang/install.sh # pin a specific release tag
# EL_PREFIX=/opt/el bash lang/install.sh # custom install prefix
```
Or build the compiler from source and verify the self-hosting chain:
```bash
cd lang
./dist/platform/elc elc-cli.el > elc-new.c
cc -std=c11 -I el-compiler/runtime -lcurl -lpthread \
-o dist/platform/elc-new \
elc-new.c el-compiler/runtime/el_seed.c
# Confirm the new binary reproduces itself exactly
./dist/platform/elc-new elc-cli.el > elc-verify.c
diff elc-new.c elc-verify.c # should be identical
mv dist/platform/elc-new dist/platform/elc
```
Run your first program:
```bash
./lang/dist/platform/elc lang/examples/hello.el > hello.c
cc -std=c11 -I lang/el-compiler/runtime -lcurl -lpthread \
-o hello hello.c lang/el-compiler/runtime/el_seed.c
./hello
```
More examples in [lang/examples/](lang/examples/), including a full starter project at `lang/examples/hello-project/`.
If the compiler binary is ever lost or corrupted, [lang/BOOTSTRAP.md](lang/BOOTSTRAP.md) is the authoritative recovery path.
---
## Development workflow
Branching follows `dev → stage → main`: work lands on `dev`, promotes to `stage` for integration testing, and is promoted to `main` for release (visible directly in the git history of this repo). CI is defined per-subproject under `.gitea/workflows/``lang`/`epm`/`ide` share the root pipeline; `engram` and `ql` carry their own (`ci-dev`, `ci-stage`, and a release workflow each).
- Language/runtime specs live at `*/spec/*.md` (`lang/spec/`, `ql/spec/`, `ui/spec/`) and are the single source of truth for implemented-vs-planned status — code and docs are expected to agree with the spec's status markers, not the other way around.
- Agent-facing orientation guides live at `*/AGENTS.md` (currently `lang/AGENTS.md`); more subprojects may grow their own as they need agent-specific conventions documented.
- Tagged releases live under `lang/releases/`, each with its own `RELEASE.md`.
---
## Status
This is an actively developed, internal monorepo — not yet published under an open license. Treat everything here as proprietary to Neuron Technologies unless told otherwise.
+1 -1
View File
@@ -81,7 +81,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+1 -1
View File
@@ -88,7 +88,7 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
+8 -1
View File
@@ -49,6 +49,12 @@ jobs:
echo "Downloading el_runtime.h..."
curl -fsSL "${RELEASE_BASE}/el_runtime.h" -o /usr/local/lib/el/el_runtime.h
echo "Downloading engram_store.c..."
curl -fsSL "${RELEASE_BASE}/engram_store.c" -o /usr/local/lib/el/engram_store.c
echo "Downloading engram_store.h..."
curl -fsSL "${RELEASE_BASE}/engram_store.h" -o /usr/local/lib/el/engram_store.h
echo "El SDK installed:"
elc --version || true
@@ -62,11 +68,12 @@ jobs:
# Link to produce the engram binary
- name: Link engram binary
run: |
cc -std=c11 -O2 \
cc -std=c11 -O2 -DHAVE_CURL \
-I /usr/local/lib/el \
-o dist/engram \
dist/engram.c \
/usr/local/lib/el/el_runtime.c \
/usr/local/lib/el/engram_store.c \
-lcurl -lpthread
echo "Linked dist/engram"
ls -lh dist/engram
+51 -21
View File
@@ -117,6 +117,17 @@ fn route_text_health(method: String, path: String, body: String) -> String {
// save/load with no "path" hit engram_save(""). Rewritten to the
// `let x = if cond { a } else { b }` expression form (the pattern the newer
// routes route_emit_ise/route_capture_knowledge already use correctly).
// store_on ENGRAM_STORE flag (tiered paged store as the durable owner). Matches
// engram_store_enabled() in el_runtime.c EXACTLY (1 / on / true). Default off
// every persistence path below is byte-for-byte the historical snapshot behavior.
fn store_on() -> Bool {
let v: String = env("ENGRAM_STORE")
if str_eq(v, "1") { return true }
if str_eq(v, "on") { return true }
if str_eq(v, "true") { return true }
return false
}
// persist_canonical save the canonical snapshot after a durable write.
//
// WHY (2026-07-22 self-review): the 2026-07-21 fix correctly stopped READ
@@ -131,6 +142,14 @@ fn route_text_health(method: String, path: String, body: String) -> String {
// tolerant, ~2/min snapshotting the whole store per heartbeat is waste;
// any durable write that follows persists the pruning too).
fn persist_canonical() -> Int {
// ENGRAM_STORE: the paged store is the durable owner a checkpoint flushes
// dirty pages behind a WAL-durable record (durable the moment the WAL fsyncs).
// This is the fix for the "restart reverted to a 17h-old snapshot" data loss:
// durable writes no longer depend on a full snapshot.json rewrite. Returns 1
// on a successful checkpoint, 0 otherwise. Flag-off: unchanged (writes JSON).
if store_on() {
return engram_store_checkpoint()
}
let dir_raw: String = env("ENGRAM_DATA_DIR")
let dir: String = engram_resolve_data_dir()
// (2026-08-10 self-review) This returned a hardcoded 1, which made every
@@ -798,29 +817,40 @@ let port: Int = parse_port(bind_str)
// never /tmp; fail loud if HOME is unresolvable (engram_resolve_data_dir exits).
let data_dir: String = engram_resolve_data_dir()
let snapshot_path: String = data_dir + "/snapshot.json"
engram_load(snapshot_path)
// ENGRAM_STORE (tiered paged store engram-tiered-storage-engine.md). When set,
// the durable owner is the paged store (neuron.egm + neuron.wal): engram_store_boot
// imports snapshot.json ONCE into a fresh neuron.egm, else replays the WAL and loads
// the store resident snapshot.json is never read again as the ongoing store. This
// closes the "restart reverted to a 17h-old snapshot" data-loss window. Flag-off
// (default): byte-for-byte the historical snapshot + optional-WAL boot below.
if store_on() {
engram_store_boot(data_dir)
println("[engram] ENGRAM_STORE enabled — tiered paged store is the durable owner")
} else {
engram_load(snapshot_path)
// WAL replay (design doc §6). Gated: default OFF is byte-identical to legacy
// snapshot-only boot. When ON, the snapshot above is the compaction BASE and
// the WAL carries every mutation since; replay reconstructs state to the last
// CRC-valid record, then opens the WAL for appending.
if wal_on() {
let replayed: Int = engram_wal_boot(data_dir)
println("[engram] WAL enabled — replayed " + int_to_str(replayed) + " records")
}
// WAL replay (design doc §6). Gated: default OFF is byte-identical to legacy
// snapshot-only boot. When ON, the snapshot above is the compaction BASE and
// the WAL carries every mutation since; replay reconstructs state to the last
// CRC-valid record, then opens the WAL for appending.
if wal_on() {
let replayed: Int = engram_wal_boot(data_dir)
println("[engram] WAL enabled — replayed " + int_to_str(replayed) + " records")
}
// 2026-07-21 self-review boot guard: if the snapshot file has content but the
// load produced 0 nodes, something is wrong (corrupt file / parse failure).
// Preserve the evidence and warn loudly and since read routes no longer write
// the canonical path, a bad boot can no longer clobber the good snapshot.
let boot_snap: String = fs_read(snapshot_path)
if !str_eq(boot_snap, "") {
if engram_node_count() == 0 {
println("[engram] WARNING: snapshot.json is non-empty but load produced 0 nodes — preserving copy at snapshot.failed-load.json")
fs_write(data_dir + "/snapshot.failed-load.json", boot_snap)
} else {
// Good load: keep a boot-time backup of the snapshot as loaded.
fs_write(data_dir + "/snapshot.boot-backup.json", boot_snap)
// 2026-07-21 self-review boot guard: if the snapshot file has content but the
// load produced 0 nodes, something is wrong (corrupt file / parse failure).
// Preserve the evidence and warn loudly and since read routes no longer write
// the canonical path, a bad boot can no longer clobber the good snapshot.
let boot_snap: String = fs_read(snapshot_path)
if !str_eq(boot_snap, "") {
if engram_node_count() == 0 {
println("[engram] WARNING: snapshot.json is non-empty but load produced 0 nodes — preserving copy at snapshot.failed-load.json")
fs_write(data_dir + "/snapshot.failed-load.json", boot_snap)
} else {
// Good load: keep a boot-time backup of the snapshot as loaded.
fs_write(data_dir + "/snapshot.boot-backup.json", boot_snap)
}
}
}
+158
View File
@@ -0,0 +1,158 @@
#!/usr/bin/env bash
# M3.5 PRE-FLIP GATE. Pure C harness (NOT elb/elc): links the real el_runtime.c
# native engram builtins + engram_store.c and proves activation-time field
# mutations (edge hebb, node activation_count, WM weight) persist through a
# checkpoint and survive a reboot from neuron.egm with snapshot.json DELETED.
# Writes ONLY under a throwaway /tmp dir with a throwaway HOME.
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"
RT="$HERE/../../lang/runtime/el_runtime.c"
ST="$HERE/../../lang/runtime/engram_store.c"
INC="$HERE/../../lang/runtime"
WORK="$(mktemp -d /tmp/engram-m35-XXXXXX)"
BIN="$WORK/m35"
export HOME="$WORK/home"; mkdir -p "$HOME" # never touch real ~/.neuron
export ENGRAM_WAL_SYNC=always
unset ENGRAM_STORE
fail=0
echo "== compiling harness (gcc: el_runtime.c + engram_store.c + test_m35_hebb_persist.c) =="
gcc -O1 -std=c11 -I "$INC" "$HERE/test_m35_hebb_persist.c" "$RT" "$ST" -lcurl -o "$BIN" 2>"$WORK/cc.log"
if [ $? -ne 0 ]; then echo "COMPILE FAILED:"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; fi
echo
echo "== 0) flag-OFF: seed+activate+checkpoint must NOT touch the store =="
DOFF="$WORK/off"; mkdir -p "$DOFF"
( unset ENGRAM_STORE; "$BIN" offcheck "$DOFF" )
[ $? -ne 0 ] && { echo "FAIL: offcheck"; fail=1; }
[ -e "$DOFF/neuron.egm" ] && { echo "FAIL: neuron.egm created while flag OFF"; fail=1; } \
|| echo " ok: no neuron.egm created with flag OFF"
echo
echo "== 1) POSITIVE: ENGRAM_STORE=1 seed -> activate -> checkpoint(field-persist) -> close =="
DPOS="$WORK/pos"; mkdir -p "$DPOS"
ENGRAM_STORE=1 "$BIN" pos_seed "$DPOS" || { echo "FAIL: pos_seed"; fail=1; }
[ -e "$DPOS/neuron.egm" ] && echo " ok: neuron.egm created" || { echo "FAIL: neuron.egm missing"; fail=1; }
echo
echo "== 2) reboot from neuron.egm with snapshot.json DELETED (must never read JSON) =="
rm -f "$DPOS/snapshot.json"
ENGRAM_STORE=1 "$BIN" pos_reboot "$DPOS" || { echo "FAIL: pos_reboot"; fail=1; }
echo
echo "== 3) NEGATIVE CONTROL: seed -> activate -> close WITHOUT the field-persist checkpoint =="
DNEG="$WORK/neg"; mkdir -p "$DNEG"
ENGRAM_STORE=1 "$BIN" neg_seed "$DNEG" || { echo "FAIL: neg_seed"; fail=1; }
rm -f "$DNEG/snapshot.json"
ENGRAM_STORE=1 "$BIN" neg_reboot "$DNEG" || { echo "FAIL: neg_reboot"; fail=1; }
echo
echo "== 4) assertions (python over the JSON exports) =="
python3 - "$DPOS" "$DNEG" <<'PY'
import json, sys, os
WM_FLOOR = 0.05
HEBB_MIN = 1e-6
def load(d, name):
with open(os.path.join(d, name)) as f: return json.load(f)
def node_by_label(g, label):
for n in g["nodes"]:
if n.get("label") == label: return n
return None
def edge_between(g, a_id, b_id):
for e in g["edges"]:
if e.get("from_id") == a_id and e.get("to_id") == b_id:
return e
return None
rc = 0
def check(cond, msg):
global rc
if cond: print(f" PASS: {msg}")
else: print(f" FAIL: {msg}"); rc = 1
dpos, dneg = sys.argv[1], sys.argv[2]
pre = load(dpos, "pre_reboot.json")
rebt = load(dpos, "reboot.json")
pa, pb = node_by_label(pre, "hebb-a"), node_by_label(pre, "hebb-b")
ra = node_by_label(rebt, "hebb-a")
assert pa and pb and ra, "target nodes missing"
pe = edge_between(pre, pa["id"], pb["id"])
re = edge_between(rebt, pa["id"], pb["id"])
assert pe and re, "target edge missing"
pre_hebb = pe.get("hebb", 0.0)
rebt_hebb = re.get("hebb", 0.0)
pre_ac = pa.get("activation_count", 0)
rebt_ac = ra.get("activation_count", 0)
pre_wm = pa.get("working_memory_weight", 0.0)
rebt_wm = ra.get("working_memory_weight", 0.0)
print(f" edge hebb-a->hebb-b : pre={pre_hebb!r} reboot={rebt_hebb!r}")
print(f" node hebb-a act_cnt : pre={pre_ac!r} reboot={rebt_ac!r}")
print(f" node hebb-a wm : pre={pre_wm!r} reboot={rebt_wm!r} (halved+floored expected)")
# --- learning actually happened this run (else the test proves nothing) ---
check(pre_hebb > HEBB_MIN, f"activation raised edge hebb above 0 (pre={pre_hebb})")
check(pre_ac >= 1, f"activation reinforced node activation_count (pre={pre_ac})")
check(pre_wm > 0.0, f"activation promoted node to working memory (pre_wm={pre_wm})")
# --- the load-bearing survival assertions after a real delete-JSON reboot ---
check(abs(rebt_hebb - pre_hebb) < 1e-12,
f"edge hebb SURVIVED reboot unchanged ({rebt_hebb} == {pre_hebb})")
check(rebt_ac == pre_ac,
f"node activation_count SURVIVED reboot unchanged ({rebt_ac} == {pre_ac})")
# --- WM weight: must equal the JSON path's boot transform exactly (halve+floor) ---
expected_wm = pre_wm * 0.5
if expected_wm < WM_FLOOR: expected_wm = 0.0
check(abs(rebt_wm - expected_wm) < 1e-9,
f"node WM weight SURVIVED with the SAME boot transform as JSON path "
f"(reboot={rebt_wm} == halve+floor(pre)={expected_wm})")
check(expected_wm > 0.0,
f"WM survival is observable (halved weight stays above floor: {expected_wm} > {WM_FLOOR})")
# --- NEGATIVE CONTROL: without the field-persist step the learning is LOST ---
npre = load(dneg, "neg_pre.json")
nrebt = load(dneg, "neg_reboot.json")
na_pre = node_by_label(npre, "hebb-a")
na_rebt = node_by_label(nrebt, "hebb-a")
ne_pre = edge_between(npre, na_pre["id"], node_by_label(npre, "hebb-b")["id"])
ne_rebt = edge_between(nrebt, na_rebt["id"], node_by_label(nrebt, "hebb-b")["id"])
print(f" [neg] edge hebb : pre={ne_pre.get('hebb',0.0)!r} reboot={ne_rebt.get('hebb',0.0)!r}")
print(f" [neg] node act_cnt : pre={na_pre.get('activation_count',0)!r} reboot={na_rebt.get('activation_count',0)!r}")
check(ne_pre.get("hebb", 0.0) > HEBB_MIN,
f"[neg] activation DID raise hebb in RAM (pre={ne_pre.get('hebb',0.0)})")
check(ne_rebt.get("hebb", 0.0) == 0.0,
"[neg] WITHOUT checkpoint field-persist, edge hebb is LOST on reboot (==0) — fix is load-bearing")
check(na_rebt.get("activation_count", 0) == 0,
"[neg] WITHOUT checkpoint field-persist, activation_count is LOST on reboot (==0)")
sys.exit(rc)
PY
[ $? -ne 0 ] && fail=1
echo
echo "== 5) ASan+UBSan build, exercise the full persist+reboot flow (leaks off — harness intentionally leaks el_strdup) =="
SANBIN="$WORK/m35.san"
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
-I "$INC" "$HERE/test_m35_hebb_persist.c" "$RT" "$ST" -lcurl -o "$SANBIN" 2>"$WORK/san_cc.log"
if [ $? -ne 0 ]; then echo " SAN COMPILE FAILED:"; tail -20 "$WORK/san_cc.log"; fail=1; else
export ASAN_OPTIONS=detect_leaks=0
DSAN="$WORK/san"; mkdir -p "$DSAN"
ENGRAM_STORE=1 "$SANBIN" pos_seed "$DSAN" >/dev/null 2>"$WORK/san_run.log" && \
{ rm -f "$DSAN/snapshot.json"; ENGRAM_STORE=1 "$SANBIN" pos_reboot "$DSAN" >/dev/null 2>>"$WORK/san_run.log"; }
if grep -qiE 'runtime error|AddressSanitizer|UndefinedBehavior|ERROR: ' "$WORK/san_run.log"; then
echo " FAIL: sanitizer findings:"; grep -iE 'runtime error|Sanitizer|ERROR' "$WORK/san_run.log" | head; fail=1
else
echo " ok: ASan+UBSan clean across pos_seed/checkpoint/reboot (field-persist, boot laundering)"
fi
fi
echo
if [ "$fail" -eq 0 ]; then echo "================ M3.5 HEBB-PERSIST GATE: PASS ================"; else echo "================ M3.5 HEBB-PERSIST GATE: FAIL ================"; fi
rm -rf "$WORK"
exit $fail
+126
View File
@@ -0,0 +1,126 @@
#!/usr/bin/env bash
# M3 JSON-parity gate. Pure C harness (NOT elb/elc): links the real el_runtime.c
# native engram builtins + engram_store.c and drives ENGRAM_STORE on vs off.
# Writes ONLY under a throwaway /tmp dir with a throwaway HOME + ENGRAM_DATA_DIR.
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"
RT="$HERE/../../lang/runtime/el_runtime.c"
ST="$HERE/../../lang/runtime/engram_store.c"
INC="$HERE/../../lang/runtime"
WORK="$(mktemp -d /tmp/engram-m3-XXXXXX)"
DATA="$WORK/data"; mkdir -p "$DATA"
BIN="$WORK/m3"
export HOME="$WORK/home"; mkdir -p "$HOME" # never touch real ~/.neuron
export ENGRAM_DATA_DIR="$DATA"
export ENGRAM_WAL_SYNC=always
unset ENGRAM_STORE
fail=0
echo "== compiling harness (gcc: el_runtime.c + engram_store.c + test_m3_parity.c) =="
gcc -O1 -std=c11 -I "$INC" "$HERE/test_m3_parity.c" "$RT" "$ST" -lcurl -o "$BIN" 2>"$WORK/cc.log"
if [ $? -ne 0 ]; then echo "COMPILE FAILED:"; cat "$WORK/cc.log"; rm -rf "$WORK"; exit 1; fi
grep -i warning "$WORK/cc.log" | grep -iE 'engram_store|eg_store|eg_load|scan_nodes|scan_edges' && echo "(warnings in M3 code above)" || true
echo
echo "== 0) default-OFF: flag unset leaves the store untouched =="
( unset ENGRAM_STORE; "$BIN" offcheck "$DATA" )
[ $? -ne 0 ] && { echo "FAIL: offcheck"; fail=1; }
[ -e "$DATA/neuron.egm" ] && { echo "FAIL: neuron.egm created while flag OFF"; fail=1; } \
|| echo " ok: no neuron.egm created with flag OFF"
echo
echo "== 1) seed (ENGRAM_STORE unset): build graph, save snapshot.json, activate =="
( unset ENGRAM_STORE; "$BIN" seed "$DATA" ) || { echo "FAIL: seed"; fail=1; }
echo
echo "== 2) on (ENGRAM_STORE=1): import snapshot.json ONCE -> neuron.egm, resident-load, activate =="
ENGRAM_STORE=1 "$BIN" on "$DATA" || { echo "FAIL: on"; fail=1; }
[ -e "$DATA/neuron.egm" ] && echo " ok: neuron.egm created by import" || { echo "FAIL: neuron.egm missing"; fail=1; }
echo
echo "== 3) reboot (ENGRAM_STORE=1, snapshot.json DELETED): must load from neuron.egm, never JSON =="
rm -f "$DATA/snapshot.json"
ENGRAM_STORE=1 "$BIN" reboot "$DATA" || { echo "FAIL: reboot"; fail=1; }
echo
echo "== 4) parity comparison (modulo ordering) =="
python3 - "$DATA" <<'PY'
import json, sys, os
d = sys.argv[1]
def load(name):
with open(os.path.join(d, name)) as f: return json.load(f)
def norm_graph(g):
nodes = sorted(g.get("nodes", []), key=lambda n: n.get("id",""))
edges = sorted(g.get("edges", []), key=lambda e: e.get("id",""))
layers= sorted(g.get("layers", []), key=lambda l: l.get("layer_id",0))
return {"nodes":nodes, "edges":edges, "layers":layers}
def act_ids(a):
# list of (node id, promoted); robust set + ordered list
seq = [(e.get("node",{}).get("id",""), int(e.get("promoted",0))) for e in a]
return seq
rc = 0
snap = norm_graph(load("snapshot.json") if os.path.exists(os.path.join(d,"snapshot.json")) else load("off_graph.json"))
off = norm_graph(load("off_graph.json"))
on = norm_graph(load("on_graph.json"))
rebt = norm_graph(load("reboot_graph.json"))
def cmp(label, a, b):
global rc
if a == b:
print(f" PASS: {label} (nodes={len(a['nodes'])} edges={len(a['edges'])} layers={len(a['layers'])})")
else:
rc = 1
print(f" FAIL: {label}")
for k in ("nodes","edges","layers"):
if a[k] != b[k]:
print(f" {k}: {len(a[k])} vs {len(b[k])}")
for x,y in zip(a[k], b[k]):
if x != y:
print(f" first diff:\n A={json.dumps(x)[:300]}\n B={json.dumps(y)[:300]}")
break
cmp("graph: ENGRAM_STORE=1 (export) == ENGRAM_STORE=0 (JSON path)", on, off)
cmp("round-trip: snapshot.json seed == store export (on_graph)", on, off) # off_graph==snapshot save
cmp("reboot from neuron.egm (no JSON) == on-path store", rebt, on)
offa = act_ids(load("off_act.json"))
ona = act_ids(load("on_act.json"))
if set(offa) == set(ona):
print(f" PASS: activation result set identical (off={len(offa)} on={len(ona)} entries)")
if offa == ona:
print(" (and identical ordering/promotion sequence)")
else:
print(" (same set; ordering differs only where scores tie — reporting honestly)")
else:
rc = 1
print(" FAIL: activation result set differs")
print(f" off-only: {set(offa)-set(ona)}")
print(f" on-only: {set(ona)-set(offa)}")
sys.exit(rc)
PY
[ $? -ne 0 ] && fail=1
echo
echo "== 5) ASan+UBSan build, exercise M3 scan/boot/hooks (leaks off — harness intentionally leaks el_strdup) =="
SANBIN="$WORK/m3.san"
gcc -O1 -g -std=c11 -fsanitize=address,undefined -fno-sanitize-recover=undefined \
-I "$INC" "$HERE/test_m3_parity.c" "$RT" "$ST" -lcurl -o "$SANBIN" 2>"$WORK/san_cc.log"
if [ $? -ne 0 ]; then echo " SAN COMPILE FAILED:"; tail -20 "$WORK/san_cc.log"; fail=1; else
export ASAN_OPTIONS=detect_leaks=0
DATA2="$WORK/data2"; mkdir -p "$DATA2"
( unset ENGRAM_STORE; "$SANBIN" seed "$DATA2" ) >/dev/null 2>"$WORK/san_run.log" && \
ENGRAM_STORE=1 "$SANBIN" on "$DATA2" >/dev/null 2>>"$WORK/san_run.log" && \
{ rm -f "$DATA2/snapshot.json"; ENGRAM_STORE=1 "$SANBIN" reboot "$DATA2" >/dev/null 2>>"$WORK/san_run.log"; }
if grep -qiE 'runtime error|AddressSanitizer|UndefinedBehavior|ERROR: ' "$WORK/san_run.log"; then
echo " FAIL: sanitizer findings:"; grep -iE 'runtime error|Sanitizer|ERROR' "$WORK/san_run.log" | head; fail=1
else
echo " ok: ASan+UBSan clean across seed/on/reboot (scan, boot, resident-load, mutation hooks)"
fi
fi
echo
if [ "$fail" -eq 0 ]; then echo "================ M3 PARITY GATE: PASS ================"; else echo "================ M3 PARITY GATE: FAIL ================"; fi
rm -rf "$WORK"
exit $fail
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
# M1 paged-store gate. Pure C (NOT elb/elc). Writes only under /tmp.
set -e
HERE="$(cd "$(dirname "$0")" && pwd)"
SRC="$HERE/../../lang/runtime/engram_store.c"
BIN="/tmp/test_store.$$"
echo "compiling: gcc test_store.c engram_store.c"
gcc -O2 -Wall -Wextra -std=c11 "$HERE/test_store.c" "$SRC" -o "$BIN"
"$BIN"
rc=$?
rm -f "$BIN"
rm -rf /tmp/engram-store-test-*
exit $rc
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
# M2 WAL + checkpoint + recovery gate. Pure C (NOT elb/elc). Writes only under /tmp.
# Recovery tests use ENGRAM_WAL_SYNC=always so every WAL record is durable at crash.
set -e
HERE="$(cd "$(dirname "$0")" && pwd)"
SRC="$HERE/../../lang/runtime/engram_store.c"
BIN="/tmp/test_wal_store.$$"
echo "compiling: gcc test_wal_store.c engram_store.c"
gcc -O2 -Wall -Wextra -std=c11 "$HERE/test_wal_store.c" "$SRC" -o "$BIN"
ENGRAM_WAL_SYNC=always "$BIN"
rc=$?
rm -f "$BIN"
rm -rf /tmp/engram-wal-test-*
exit $rc
+130
View File
@@ -0,0 +1,130 @@
/* test_m35_hebb_persist.c — M3.5 PRE-FLIP GATE.
*
* Proves that in-place field mutations made during spreading activation — edge
* `hebb` (+ last_fired), node `activation_count`, node working-memory weight —
* PERSIST to the paged store and survive a restart from neuron.egm with
* snapshot.json deleted. This is the "hebb-survives-restart" fix that gates the
* live cutover.
*
* Same style as test_m3_parity.c: a REAL el-level harness linking the actual
* el_runtime.c native engram builtins + engram_store.c, driving engram_node_full
* / engram_connect / engram_activate_json / engram_save / engram_store_boot /
* engram_store_checkpoint / engram_store_close directly from C. No EL interpreter.
*
* Modes (argv[1]), data dir (argv[2]):
* pos_seed — ENGRAM_STORE=1: fresh store, seed a graph tuned so activation
* co-activates a connected pair (edge hebb 0 -> ETA) and reinforces
* nodes (activation_count 0 -> >=1, WM weight -> >0). Export the
* post-activation resident graph to pre_reboot.json, then CHECKPOINT
* (the M3.5 field-persist), then close.
* pos_reboot— ENGRAM_STORE=1, snapshot.json deleted by runner: boot from
* neuron.egm (WAL replay), export reboot.json, close. The values in
* reboot.json are what actually survived the round-trip.
* neg_seed — identical to pos_seed but WITHOUT the checkpoint field-persist
* (negative control): activation mutations never reach the store.
* neg_reboot— boot from neuron.egm, export neg_reboot.json, close.
* offcheck — ENGRAM_STORE unset: seed+activate+checkpoint must NOT touch the
* store (no neuron.egm, checkpoint returns 0).
*
* The pass/fail assertions live in run_m35_hebb_persist.sh (python over the JSON
* exports): reboot.json must carry the learned hebb / activation_count and the
* JSON-identical halved WM weight; neg_reboot.json must have LOST them.
*/
#include "el_runtime.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
extern int engram_store_enabled(void);
extern el_val_t engram_store_boot(el_val_t data_dir);
extern el_val_t engram_store_checkpoint(void);
extern el_val_t engram_store_close(void);
static el_val_t S(const char* s){ return EL_STR(s); }
static el_val_t F(double d){ return el_from_float(d); }
/* Two nodes with DISTINCT content (so the redundancy-suppression pass cannot
* dedup one of them away) that both match the query strongly, wired by one
* "associate" edge. A handful of weakly-related distractors make it a real
* graph. On activation both A and B promote to working memory and co-activate,
* so their edge's hebb rises from 0 to ENGRAM_HEBB_ETA. */
static void build_seed(void){
el_val_t a = engram_node_full(S("hebbian potentiation strengthens co-active memory links"),
S("Concept"), S("hebb-a"), F(0.9), F(0.85), F(1.0), S("Semantic"),
S("hebbian,memory,activation"));
el_val_t b = engram_node_full(S("co-active memory links accrue hebbian associative weight"),
S("Concept"), S("hebb-b"), F(0.9), F(0.85), F(1.0), S("Semantic"),
S("hebbian,memory,weight"));
el_val_t c = engram_node_full(S("unrelated culinary recipe for sourdough bread"),
S("Fact"), S("distractor-1"), F(0.4), F(0.4), F(1.0), S("Semantic"),
S("food"));
el_val_t d = engram_node_full(S("the weather forecast predicts rain tomorrow afternoon"),
S("Fact"), S("distractor-2"), F(0.4), F(0.4), F(1.0), S("Semantic"),
S("weather"));
engram_connect(a, b, F(0.8), S("associate")); /* the edge under test */
engram_connect(a, c, F(0.3), S("associate"));
engram_connect(b, d, F(0.3), S("associate"));
}
static const char* QUERY =
"hebbian potentiation co-active memory links associative weight";
static void export_graph(const char* dir, const char* name){
char p[1024];
snprintf(p, sizeof p, "%s/%s", dir, name);
if (!engram_save(S(p))){ fprintf(stderr, "save %s failed\n", name); exit(2); }
}
int main(int argc, char** argv){
if (argc < 3){
fprintf(stderr, "usage: %s <pos_seed|pos_reboot|neg_seed|neg_reboot|offcheck> <dir>\n", argv[0]);
return 2;
}
const char* mode = argv[1];
const char* dir = argv[2];
if (!strcmp(mode, "pos_seed") || !strcmp(mode, "neg_seed")){
int persist = !strcmp(mode, "pos_seed");
if (!engram_store_enabled()){ fprintf(stderr, "%s requires ENGRAM_STORE=1\n", mode); return 2; }
if (!engram_store_boot(S(dir))){ fprintf(stderr, "store boot failed\n"); return 2; }
build_seed();
el_val_t act = engram_activate_json(S(QUERY), (el_val_t)3);
(void)act;
/* Capture the post-activation resident state BEFORE persisting/closing. */
export_graph(dir, persist ? "pre_reboot.json" : "neg_pre.json");
printf("[%s] nodes=%lld edges=%lld\n", mode,
(long long)(int64_t)engram_node_count(),
(long long)(int64_t)engram_edge_count());
if (persist){
if (!engram_store_checkpoint()){ fprintf(stderr, "checkpoint failed\n"); return 2; }
}
/* neg mode: NO field-persist checkpoint. engram_store_close still flushes
* pages, but no store_put_* ran post-creation, so the store keeps the
* pristine creation-time field values (hebb=0, activation_count=0). */
engram_store_close();
return 0;
}
if (!strcmp(mode, "pos_reboot") || !strcmp(mode, "neg_reboot")){
if (!engram_store_enabled()){ fprintf(stderr, "%s requires ENGRAM_STORE=1\n", mode); return 2; }
/* snapshot.json deleted by the runner — boot MUST come from neuron.egm. */
if (!engram_store_boot(S(dir))){ fprintf(stderr, "reboot boot failed\n"); return 2; }
export_graph(dir, !strcmp(mode, "pos_reboot") ? "reboot.json" : "neg_reboot.json");
printf("[%s] nodes=%lld edges=%lld\n", mode,
(long long)(int64_t)engram_node_count(),
(long long)(int64_t)engram_edge_count());
engram_store_close();
return 0;
}
if (!strcmp(mode, "offcheck")){
int en = engram_store_enabled();
el_val_t boot = engram_store_boot(S(dir)); /* no-op with flag off */
build_seed();
engram_activate_json(S(QUERY), (el_val_t)3);
el_val_t ck = engram_store_checkpoint(); /* must be a no-op */
printf("[offcheck] enabled=%d boot=%lld checkpoint=%lld\n",
en, (long long)(int64_t)boot, (long long)(int64_t)ck);
return (en == 0 && (int64_t)boot == 0 && (int64_t)ck == 0) ? 0 : 1;
}
fprintf(stderr, "unknown mode %s\n", mode);
return 2;
}
+155
View File
@@ -0,0 +1,155 @@
/* test_m3_parity.c — M3 JSON-parity gate for the ENGRAM_STORE wiring.
*
* This is a REAL el-level harness: it links the actual el_runtime.o (the soul's
* native engram builtins) + engram_store.o and calls the engram_node family plus
* engram_connect, engram_activate_json, engram_save, engram_store_boot directly. No EL interpreter
* and no full soul build are needed el_runtime.c compiles to a standalone .o
* whose engram builtins operate on the process-global engram store, and the
* string arena is inert unless el_request_start() is called, so the builtins are
* callable straight from C (el_val_t is int64_t; EL_STR/EL_CSTR are pointer casts).
*
* Modes (argv[1]), data dir (argv[2]):
* seed ENGRAM_STORE unset: build a fixed seed graph, write snapshot.json +
* off_graph.json (pristine, pre-activation), then activate off_act.json.
* on ENGRAM_STORE=1: engram_store_boot(dir) imports snapshot.json ONCE into
* neuron.egm and loads it resident; write on_graph.json, then activate
* on_act.json; checkpoint + close.
* reboot ENGRAM_STORE=1 with snapshot.json DELETED: boot must reload from
* neuron.egm (WAL replay), never re-reading JSON; write reboot_graph.json.
* offcheck assert flag-off leaves the store untouched.
*
* The graph comparison (done by run_m3_parity.sh via python, modulo ordering) is
* the deterministic gate; activation ids/promoted are compared as a robust set.
*/
#include "el_runtime.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* Builtins the header declares are pulled in via el_runtime.h. The M3 additions
* are not in the header yet, so declare them here. */
extern int engram_store_enabled(void);
extern el_val_t engram_store_boot(el_val_t data_dir);
extern el_val_t engram_store_checkpoint(void);
extern el_val_t engram_store_close(void);
extern el_val_t engram_node_layered(el_val_t content, el_val_t node_type, el_val_t label,
el_val_t salience, el_val_t certainty, el_val_t confidence,
el_val_t status, el_val_t tags, el_val_t layer_id);
static el_val_t S(const char* s){ return EL_STR(s); }
static el_val_t F(double d){ return el_from_float(d); }
/* Build a fixed, deterministic seed graph: 12 nodes across two layers + 9 edges.
* Content is chosen so an activation query has real matches to rank. */
static void build_seed(void){
/* core-identity layer (1) via engram_node_full */
el_val_t n0 = engram_node_full(S("tiered storage engine design"), S("Concept"),
S("storage-engine"), F(0.9), F(0.8), F(1.0), S("Semantic"), S("design,storage"));
el_val_t n1 = engram_node_full(S("write-ahead log durability"), S("Concept"),
S("wal"), F(0.85), F(0.75), F(1.0), S("Semantic"), S("wal,durability"));
el_val_t n2 = engram_node_full(S("paged buffer pool with checkpointing"), S("Concept"),
S("buffer-pool"), F(0.8), F(0.7), F(1.0), S("Semantic"), S("paging"));
el_val_t n3 = engram_node_full(S("spreading activation over the graph"), S("Concept"),
S("activation"), F(0.8), F(0.7), F(1.0), S("Semantic"), S("activation,graph"));
el_val_t n4 = engram_node_full(S("hebbian co-activation potentiation"), S("Concept"),
S("hebbian"), F(0.7), F(0.6), F(1.0), S("Semantic"), S("hebb"));
el_val_t n5 = engram_node_full(S("crash recovery replays the log"), S("Concept"),
S("recovery"), F(0.75), F(0.65), F(1.0), S("Semantic"), S("recovery,wal"));
/* domain-knowledge layer (2) via engram_node_layered */
el_val_t n6 = engram_node_layered(S("b-tree primary index id to location"), S("Fact"),
S("btree"), F(0.7), F(0.6), F(1.0), S(""), S("index"), (el_val_t)2);
el_val_t n7 = engram_node_layered(S("adjacency index for edge lookup"), S("Fact"),
S("adjacency"), F(0.7), F(0.6), F(1.0), S(""), S("index,graph"), (el_val_t)2);
el_val_t n8 = engram_node_layered(S("slotted pages hold tlv records"), S("Fact"),
S("slotted-page"), F(0.65), F(0.55), F(1.0), S(""), S("format"), (el_val_t)2);
el_val_t n9 = engram_node_full(S("memory tiers working semantic episodic"), S("Concept"),
S("tiers"), F(0.7), F(0.6), F(1.0), S("Semantic"), S("tiers,memory"));
el_val_t n10 = engram_node_full(S("embeddings enable nearest neighbour search"), S("Concept"),
S("embeddings"), F(0.65), F(0.55), F(1.0), S("Semantic"), S("embeddings"));
el_val_t n11 = engram_node_full(S("the durable engram is the mind's memory"), S("Belief"),
S("engram"), F(0.95), F(0.9), F(1.0), S("Semantic"), S("engram,memory"));
engram_connect(n0, n1, F(0.8), S("depends-on"));
engram_connect(n0, n2, F(0.8), S("depends-on"));
engram_connect(n0, n3, F(0.7), S("enables"));
engram_connect(n1, n5, F(0.9), S("enables"));
engram_connect(n3, n4, F(0.6), S("triggers"));
engram_connect(n2, n6, F(0.7), S("uses"));
engram_connect(n3, n7, F(0.7), S("uses"));
engram_connect(n0, n8, F(0.6), S("uses"));
engram_connect(n11, n9, F(0.8), S("about"));
engram_connect(n11, n10, F(0.5), S("about"));
}
static void write_file(const char* path, const char* content){
FILE* f = fopen(path, "wb");
if (!f){ fprintf(stderr, "cannot open %s\n", path); exit(2); }
if (content) fwrite(content, 1, strlen(content), f);
fclose(f);
}
static const char* QUERY = "storage engine activation and the durable log";
int main(int argc, char** argv){
if (argc < 3){ fprintf(stderr, "usage: %s <seed|on|reboot|offcheck> <dir>\n", argv[0]); return 2; }
const char* mode = argv[1];
const char* dir = argv[2];
char p[1024];
if (!strcmp(mode, "seed")){
if (engram_store_enabled()){ fprintf(stderr, "seed mode requires ENGRAM_STORE unset\n"); return 2; }
build_seed();
snprintf(p, sizeof p, "%s/snapshot.json", dir);
if (!engram_save(S(p))){ fprintf(stderr, "seed save failed\n"); return 2; }
snprintf(p, sizeof p, "%s/off_graph.json", dir);
engram_save(S(p)); /* pristine off-path graph */
el_val_t act = engram_activate_json(S(QUERY), (el_val_t)3);
snprintf(p, sizeof p, "%s/off_act.json", dir);
write_file(p, EL_CSTR(act));
printf("[seed] nodes=%lld edges=%lld\n",
(long long)(int64_t)engram_node_count(), (long long)(int64_t)engram_edge_count());
return 0;
}
if (!strcmp(mode, "on")){
if (!engram_store_enabled()){ fprintf(stderr, "on mode requires ENGRAM_STORE=1\n"); return 2; }
if (!engram_store_boot(S(dir))){ fprintf(stderr, "store boot failed\n"); return 2; }
snprintf(p, sizeof p, "%s/on_graph.json", dir);
engram_save(S(p)); /* export resident (== store) */
/* Checkpoint the freshly-imported (pristine) graph — this is the state
* the reboot comparison expects to round-trip. Under M3.5 a checkpoint
* persists the resident graph's CURRENT field state, so it must run
* BEFORE activation mutates fields in place; activation itself is
* exercised below only for the activation-result-set parity check. The
* M3.5 gate (test_m35_hebb_persist) separately proves that a checkpoint
* taken AFTER activation durably carries the learned hebb/WM state. */
engram_store_checkpoint();
el_val_t act = engram_activate_json(S(QUERY), (el_val_t)3);
snprintf(p, sizeof p, "%s/on_act.json", dir);
write_file(p, EL_CSTR(act));
printf("[on] nodes=%lld edges=%lld\n",
(long long)(int64_t)engram_node_count(), (long long)(int64_t)engram_edge_count());
engram_store_close();
return 0;
}
if (!strcmp(mode, "reboot")){
if (!engram_store_enabled()){ fprintf(stderr, "reboot mode requires ENGRAM_STORE=1\n"); return 2; }
/* snapshot.json has been deleted by the runner — boot MUST come from
* neuron.egm (+ WAL replay), never re-reading JSON. */
if (!engram_store_boot(S(dir))){ fprintf(stderr, "reboot boot failed\n"); return 2; }
snprintf(p, sizeof p, "%s/reboot_graph.json", dir);
engram_save(S(p));
printf("[reboot] nodes=%lld edges=%lld\n",
(long long)(int64_t)engram_node_count(), (long long)(int64_t)engram_edge_count());
engram_store_close();
return 0;
}
if (!strcmp(mode, "offcheck")){
/* ENGRAM_STORE unset: enabled()==0 and boot is a no-op returning 0. */
int en = engram_store_enabled();
el_val_t b = engram_store_boot(S(dir));
printf("[offcheck] enabled=%d boot_ret=%lld\n", en, (long long)(int64_t)b);
return (en == 0 && (int64_t)b == 0) ? 0 : 1;
}
fprintf(stderr, "unknown mode %s\n", mode);
return 2;
}
+439
View File
@@ -0,0 +1,439 @@
/* test_store.c — M1 gate for the engram paged store (engram_store.{c,h}).
*
* Pure C. Build: gcc -O2 test_store.c ../../lang/runtime/engram_store.c -o test_store
* Writes ONLY under a throwaway /tmp dir. Never touches ~/.neuron or live ports.
*
* Covers §7 M1 gates: round-trip (5k nodes / 20k edges, all fields, emb bit-exact,
* hebb, >page content), TLV forward-compat, overflow chains, B+-tree indexes
* across splits, free-list reuse, and corruption/superblock recovery.
*/
#include "../../lang/runtime/engram_store.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
static int g_pass = 0, g_fail = 0;
static void ok(const char* name, int cond){
printf(" [%s] %s\n", cond ? "PASS" : "FAIL", name);
if (cond) g_pass++; else g_fail++;
}
static char g_dir[512];
static void mk_dir(void){
snprintf(g_dir, sizeof g_dir, "/tmp/engram-store-test-%d", (int)getpid());
mkdir(g_dir, 0700);
}
static void path_in(char* out, size_t cap, const char* name){
snprintf(out, cap, "%s/%s", g_dir, name);
}
static long file_size(const char* p){ struct stat st; return stat(p,&st)==0 ? (long)st.st_size : -1; }
/* ── deterministic RNG so oracle nodes/edges regenerate bit-exact ─────────── */
static uint64_t xs(uint64_t* s){ uint64_t x=*s; x^=x<<13; x^=x>>7; x^=x<<17; *s=x; return x; }
static uint64_t node_seed(int i){ return 0x9E3779B97F4A7C15ULL ^ ((uint64_t)(i+1)*0xD1B54A32D192ED03ULL); }
static uint64_t edge_seed(int i){ return 0xC2B2AE3D27D4EB4FULL ^ ((uint64_t)(i+1)*0x165667B19E3779F9ULL); }
static char* rnd_str(uint64_t* st, size_t len){
char* s = (char*)malloc(len + 1);
for (size_t i=0;i<len;i++) s[i] = (char)(33 + (xs(st) % 94)); /* printable, no NUL */
s[len] = 0; return s;
}
/* NODE_COUNT nodes; a slice have >page content to force overflow chains. */
#define NODE_COUNT 5000
#define EDGE_COUNT 20000
#define EMB_DIM 768
static void gen_node(int i, StoreNode* n){
memset(n, 0, sizeof *n);
uint64_t st = node_seed(i);
char id[32]; snprintf(id, sizeof id, "node-%d", i);
n->id = strdup(id);
size_t clen = (i % 500 == 0) ? (size_t)(17000 + (xs(&st) % 6000)) : (size_t)(xs(&st) % 300);
n->content = rnd_str(&st, clen);
n->node_type = rnd_str(&st, 4 + (xs(&st) % 8));
n->label = (i % 2) ? rnd_str(&st, 3 + (xs(&st) % 10)) : NULL;
n->tier = rnd_str(&st, 4 + (xs(&st) % 6));
n->tags = rnd_str(&st, xs(&st) % 40);
n->metadata = (i % 3) ? rnd_str(&st, xs(&st) % 60) : NULL;
n->salience = (double)(xs(&st) % 1000000) / 997.0;
n->importance = (double)(xs(&st) % 1000000) / 131.0;
n->confidence = (double)(xs(&st) % 1000000) / 733.0;
n->temporal_decay_rate = (double)(xs(&st) % 1000000) / 101.0;
n->activation_count = (int64_t)(xs(&st) % 100000);
n->last_activated = (int64_t)xs(&st);
n->created_at = (int64_t)(1600000000000LL + i);
n->updated_at = (int64_t)xs(&st);
n->background_activation = (double)(xs(&st) % 1000000) / 17.0;
n->working_memory_weight = (double)(xs(&st) % 1000000) / 29.0;
n->suppression_count = (int32_t)(xs(&st) % 50);
n->layer_id = (uint32_t)(xs(&st) % 5);
for (int k=0;k<STORE_BLL_K;k++) n->access_ts[k] = (int64_t)xs(&st);
n->access_head = (int32_t)(xs(&st) % STORE_BLL_K);
n->access_filled = (int32_t)(xs(&st) % (STORE_BLL_K + 1));
n->wm_anchor = (double)(xs(&st) % 1000000) / 3.0;
n->emb = (float*)malloc(EMB_DIM * sizeof(float));
for (int k=0;k<EMB_DIM;k++){ uint32_t u=(uint32_t)xs(&st); memcpy(&n->emb[k], &u, 4); }
n->emb_dim = EMB_DIM;
}
static void gen_edge(int i, StoreEdge* e){
memset(e, 0, sizeof *e);
uint64_t st = edge_seed(i);
char id[32], from[32], to[32];
snprintf(id, sizeof id, "edge-%d", i);
snprintf(from, sizeof from, "node-%d", (int)(xs(&st) % NODE_COUNT));
snprintf(to, sizeof to, "node-%d", (int)(xs(&st) % NODE_COUNT));
e->id = strdup(id); e->from_id = strdup(from); e->to_id = strdup(to);
e->relation = rnd_str(&st, 3 + (xs(&st) % 12));
e->metadata = (i % 4) ? rnd_str(&st, xs(&st) % 40) : NULL;
e->weight = (double)(xs(&st) % 1000000) / 111.0;
e->hebb = (double)(xs(&st) % 1000000) / 1000000.0; /* the learned field */
e->confidence = (double)(xs(&st) % 1000000) / 777.0;
e->created_at = (int64_t)(1600000000000LL + i);
e->updated_at = (int64_t)xs(&st);
e->last_fired = (int64_t)xs(&st);
e->inhibitory = (int32_t)(xs(&st) % 2);
e->layer_id = (uint32_t)(xs(&st) % 5);
}
static int streq(const char* a, const char* b){
if (!a && !b) return 1;
if (!a || !b) return 0;
return strcmp(a,b)==0;
}
static int cmp_node(const StoreNode* a, const StoreNode* b){
if (!streq(a->id,b->id) || !streq(a->content,b->content) ||
!streq(a->node_type,b->node_type) || !streq(a->label,b->label) ||
!streq(a->tier,b->tier) || !streq(a->tags,b->tags) ||
!streq(a->metadata,b->metadata)) return 0;
if (a->salience!=b->salience || a->importance!=b->importance ||
a->confidence!=b->confidence || a->temporal_decay_rate!=b->temporal_decay_rate ||
a->activation_count!=b->activation_count || a->last_activated!=b->last_activated ||
a->created_at!=b->created_at || a->updated_at!=b->updated_at ||
a->background_activation!=b->background_activation ||
a->working_memory_weight!=b->working_memory_weight ||
a->suppression_count!=b->suppression_count || a->layer_id!=b->layer_id ||
a->access_head!=b->access_head || a->access_filled!=b->access_filled ||
a->wm_anchor!=b->wm_anchor || a->emb_dim!=b->emb_dim) return 0;
for (int k=0;k<STORE_BLL_K;k++) if (a->access_ts[k]!=b->access_ts[k]) return 0;
if ((a->emb==NULL) != (b->emb==NULL)) return 0;
if (a->emb && memcmp(a->emb, b->emb, (size_t)a->emb_dim*4)!=0) return 0;
return 1;
}
static int cmp_edge(const StoreEdge* a, const StoreEdge* b){
if (!streq(a->id,b->id) || !streq(a->from_id,b->from_id) || !streq(a->to_id,b->to_id) ||
!streq(a->relation,b->relation) || !streq(a->metadata,b->metadata)) return 0;
if (a->weight!=b->weight || a->hebb!=b->hebb || a->confidence!=b->confidence ||
a->created_at!=b->created_at || a->updated_at!=b->updated_at ||
a->last_fired!=b->last_fired || a->inhibitory!=b->inhibitory ||
a->layer_id!=b->layer_id) return 0;
return 1;
}
static void free_node_fields(StoreNode* n){
free(n->id); free(n->content); free(n->node_type); free(n->label);
free(n->tier); free(n->tags); free(n->metadata); free(n->emb); free(n->unknown);
}
static void free_edge_fields(StoreEdge* e){
free(e->id); free(e->from_id); free(e->to_id); free(e->relation); free(e->metadata); free(e->unknown);
}
/* Flip one byte in the store file at (page*PAGE_SIZE + off). */
static void flip_byte(const char* path, uint64_t page, size_t off){
int fd = open(path, O_RDWR);
uint8_t b; off_t at = (off_t)page*STORE_PAGE_SIZE + off;
pread(fd, &b, 1, at); b ^= 0xFF; pwrite(fd, &b, 1, at); close(fd);
}
/* ════════════════════════════════════════════════════════════════════════ */
static void test_roundtrip(void){
printf("\n== round-trip: %d nodes + %d edges, all fields, emb bit-exact ==\n", NODE_COUNT, EDGE_COUNT);
char path[600]; path_in(path, sizeof path, "roundtrip.store");
unlink(path);
EngramPagedStore* s = store_create(path);
ok("store_create", s != NULL);
if (!s) return;
for (int i=0;i<NODE_COUNT;i++){ StoreNode n; gen_node(i,&n);
if (store_put_node(s,&n)!=0){ ok("put_node", 0); free_node_fields(&n); store_close(s); return; }
free_node_fields(&n); }
for (int i=0;i<EDGE_COUNT;i++){ StoreEdge e; gen_edge(i,&e);
if (store_put_edge(s,&e)!=0){ ok("put_edge", 0); free_edge_fields(&e); store_close(s); return; }
free_edge_fields(&e); }
ok("wrote all nodes+edges", 1);
store_close(s);
long sz = file_size(path);
printf(" store file size: %ld bytes (%.2f MB) for %d nodes / %d edges\n",
sz, sz/1048576.0, NODE_COUNT, EDGE_COUNT);
s = store_open(path);
ok("store_open (reopen)", s != NULL);
if (!s) return;
int nbad = 0;
for (int i=0;i<NODE_COUNT;i++){
StoreNode want; gen_node(i,&want);
StoreNode got; int r = store_get_node(s, want.id, &got);
if (r!=1 || !cmp_node(&want,&got) || got.unknown_len!=0) nbad++;
if (r==1) store_node_free(&got);
free_node_fields(&want);
}
ok("all 5000 nodes read back bit-exact (incl emb, all fields)", nbad==0);
if (nbad) printf(" %d node mismatches\n", nbad);
int ebad = 0;
for (int i=0;i<EDGE_COUNT;i++){
StoreEdge want; gen_edge(i,&want);
StoreEdge* got; size_t gn;
int found = 0;
if (store_get_edges_from(s, want.from_id, &got, &gn)==0){
for (size_t j=0;j<gn;j++) if (streq(got[j].id, want.id)){ if (cmp_edge(&want,&got[j])) found=1; break; }
store_edges_free(got, gn);
}
if (!found) ebad++;
free_edge_fields(&want);
}
ok("all 20000 edges read back via adjacency, all fields incl hebb", ebad==0);
if (ebad) printf(" %d edge mismatches\n", ebad);
ok("store_check crc clean after round-trip", store_check(s, STORE_CHECK_CRC)==0);
store_close(s);
}
static void test_forward_compat(void){
printf("\n== TLV forward-compat: omit field defaults; unknown tag preserved ==\n");
char path[600]; path_in(path, sizeof path, "fwd.store");
unlink(path);
EngramPagedStore* s = store_create(path);
/* Writer OMITS several fields (metadata, label, emb) → reader must default. */
StoreNode a; memset(&a,0,sizeof a);
a.id = strdup("omit-1"); a.content = strdup("has content"); a.tier = strdup("core");
a.salience = 0.5; /* metadata/label NULL, emb NULL */
store_put_node(s, &a); free(a.id); free(a.content); free(a.tier);
StoreNode g; int r = store_get_node(s, "omit-1", &g);
ok("omitted string fields default to NULL", r==1 && g.metadata==NULL && g.label==NULL);
ok("omitted emb defaults to NULL / emb_dim 0", r==1 && g.emb==NULL && g.emb_dim==0);
ok("present fields intact", r==1 && streq(g.content,"has content") && g.salience==0.5);
if (r==1) store_node_free(&g);
/* Writer includes an UNKNOWN tag (simulating a newer writer / field the
* reader does not model) via the `unknown` passthrough. Reader (which also
* models known fields A,B,C) must preserve it verbatim. */
uint8_t unk[64];
unk[0] = 200; /* a tag this build has no case for */
/* [u8 tag][u32 len][bytes] */
unk[1]=8; unk[2]=0; unk[3]=0; unk[4]=0;
for (int i=0;i<8;i++) unk[5+i] = (uint8_t)(0xA0 + i);
StoreNode b; memset(&b,0,sizeof b);
b.id = strdup("unk-1"); b.content = strdup("known field B"); b.confidence = 0.9; /* known field C-ish */
b.unknown = unk; b.unknown_len = 5 + 8;
store_put_node(s, &b); free(b.id); free(b.content);
StoreNode g2; int r2 = store_get_node(s, "unk-1", &g2);
int unk_ok = r2==1 && g2.unknown_len==(5+8) && memcmp(g2.unknown, unk, 5+8)==0;
ok("unknown tag preserved verbatim on read", unk_ok);
ok("known fields still read while unknown preserved", r2==1 && streq(g2.content,"known field B") && g2.confidence==0.9);
if (r2==1) store_node_free(&g2);
store_close(s);
}
static void test_overflow(void){
printf("\n== overflow: 100KB content node + emb via overflow chain ==\n");
char path[600]; path_in(path, sizeof path, "ovf.store");
unlink(path);
EngramPagedStore* s = store_create(path);
size_t big = 100*1024;
StoreNode n; memset(&n,0,sizeof n);
n.id = strdup("big-1");
n.content = (char*)malloc(big+1);
for (size_t i=0;i<big;i++) n.content[i] = (char)(33 + (i % 94));
n.content[big] = 0;
n.tier = strdup("episodic");
n.emb = (float*)malloc(EMB_DIM*sizeof(float));
for (int k=0;k<EMB_DIM;k++){ float f = (float)(k*0.5 - 100.0); n.emb[k]=f; }
n.emb_dim = EMB_DIM;
ok("put 100KB+emb node", store_put_node(s,&n)==0);
store_close(s);
s = store_open(path);
StoreNode g; int r = store_get_node(s, "big-1", &g);
ok("reopen + read big node", r==1);
ok("100KB content byte-exact via overflow", r==1 && strlen(g.content)==big && memcmp(g.content,n.content,big)==0);
ok("emb bit-exact via overflow record", r==1 && g.emb_dim==EMB_DIM && memcmp(g.emb,n.emb,EMB_DIM*4)==0);
if (r==1) store_node_free(&g);
ok("store_check clean (overflow pages crc'd)", store_check(s, STORE_CHECK_CRC)==0);
store_close(s);
free_node_fields(&n);
}
static void test_index_splits(void){
printf("\n== B+-tree index correctness across many splits ==\n");
char path[600]; path_in(path, sizeof path, "idx.store");
unlink(path);
EngramPagedStore* s = store_create(path);
/* Tiny order forces deep leaf + internal splits with only a few hundred keys. */
store__set_btree_order(s, 4, 4);
const int N = 600;
for (int i=0;i<N;i++){
StoreNode n; memset(&n,0,sizeof n);
char id[32]; snprintf(id,sizeof id,"k-%05d", (i*37+11)%100000); /* scattered keys */
n.id = strdup(id); n.content = strdup("x"); n.tier=strdup("t"); n.salience=i;
if (store_put_node(s,&n)!=0){ ok("put",0); }
free(n.id); free(n.content); free(n.tier);
}
int miss=0;
for (int i=0;i<N;i++){
char id[32]; snprintf(id,sizeof id,"k-%05d",(i*37+11)%100000);
StoreNode g; int r = store_get_node(s, id, &g);
if (r!=1 || (int)g.salience != i) miss++;
if (r==1) store_node_free(&g);
}
ok("all keys retrievable after leaf+internal splits", miss==0);
if (miss) printf(" %d misses\n", miss);
StoreNode g; ok("absent key returns 0", store_get_node(s,"k-NOPE",&g)==0);
/* Adjacency: controlled star + chain, exact edge sets. */
for (int i=0;i<50;i++){
StoreEdge e; memset(&e,0,sizeof e);
char id[32]; snprintf(id,sizeof id,"e-%d",i);
e.id=strdup(id); e.from_id=strdup("HUB"); char tt[16]; snprintf(tt,sizeof tt,"T-%d",i); e.to_id=strdup(tt);
e.relation=strdup("r"); e.weight=1.0; e.hebb=0.1*i;
store_put_edge(s,&e); free_edge_fields(&e);
}
for (int i=0;i<7;i++){
StoreEdge e; memset(&e,0,sizeof e);
char id[32]; snprintf(id,sizeof id,"in-%d",i);
char ff[16]; snprintf(ff,sizeof ff,"S-%d",i);
e.id=strdup(id); e.from_id=strdup(ff); e.to_id=strdup("SINK");
e.relation=strdup("r"); e.weight=1.0;
store_put_edge(s,&e); free_edge_fields(&e);
}
StoreEdge* out; size_t on;
store_get_edges_from(s,"HUB",&out,&on);
ok("get_edges_from(HUB) == 50", on==50);
store_edges_free(out,on);
store_get_edges_to(s,"SINK",&out,&on);
ok("get_edges_to(SINK) == 7", on==7);
store_edges_free(out,on);
store_get_edges_to(s,"HUB",&out,&on);
ok("get_edges_to(HUB) == 0 (direction separation)", on==0);
store_edges_free(out,on);
ok("store_check clean", store_check(s, STORE_CHECK_CRC)==0);
store_close(s);
}
static void test_freelist(void){
printf("\n== free-list: tombstone reclaims pages, graph stays consistent ==\n");
char path[600]; path_in(path, sizeof path, "free.store");
unlink(path);
EngramPagedStore* s = store_create(path);
uint64_t pc0 = store_page_count(s);
const int N = 300;
for (int i=0;i<N;i++){
StoreNode n; memset(&n,0,sizeof n);
char id[32]; snprintf(id,sizeof id,"a-%d",i);
n.id=strdup(id); n.content=rnd_str(&(uint64_t){node_seed(i)}, 200); n.tier=strdup("t");
store_put_node(s,&n); free_node_fields(&n);
}
uint64_t pc1 = store_page_count(s);
uint64_t node_pages = pc1 - pc0;
ok("initial batch consumed pages", node_pages > 0);
for (int i=0;i<N;i++){ char id[32]; snprintf(id,sizeof id,"a-%d",i); store_tombstone(s,id); }
/* all old nodes gone */
int gone=1; for (int i=0;i<N;i++){ char id[32]; snprintf(id,sizeof id,"a-%d",i);
StoreNode g; if (store_get_node(s,id,&g)==1){ gone=0; store_node_free(&g); } }
ok("tombstoned nodes now absent", gone);
for (int i=0;i<N;i++){
StoreNode n; memset(&n,0,sizeof n);
char id[32]; snprintf(id,sizeof id,"b-%d",i);
n.id=strdup(id); n.content=strdup("reused"); n.tier=strdup("t"); n.salience=i;
store_put_node(s,&n); free_node_fields(&n);
}
uint64_t pc2 = store_page_count(s);
/* Reuse proven: growth for the 2nd batch is far less than a fresh alloc. */
ok("freed pages reused (no full re-growth)", pc2 < pc1 + node_pages);
printf(" pages: base=%llu after1=%llu after2=%llu (node_pages=%llu)\n",
(unsigned long long)pc0,(unsigned long long)pc1,(unsigned long long)pc2,(unsigned long long)node_pages);
int newbad=0; for (int i=0;i<N;i++){ char id[32]; snprintf(id,sizeof id,"b-%d",i);
StoreNode g; if (store_get_node(s,id,&g)!=1 || (int)g.salience!=i) newbad++; else store_node_free(&g); }
ok("new batch fully readable after reuse", newbad==0);
ok("store_check clean after reuse", store_check(s, STORE_CHECK_CRC)==0);
store_close(s);
/* survives reopen */
s = store_open(path);
int rb=0; for (int i=0;i<N;i++){ char id[32]; snprintf(id,sizeof id,"b-%d",i);
StoreNode g; if (store_get_node(s,id,&g)!=1) rb++; else store_node_free(&g); }
ok("graph consistent across reopen after reuse", rb==0);
store_close(s);
}
static void test_corruption(void){
printf("\n== corruption: crc detection + superblock mirror recovery ==\n");
char path[600]; path_in(path, sizeof path, "corrupt.store");
unlink(path);
EngramPagedStore* s = store_create(path);
for (int i=0;i<50;i++){ StoreNode n; gen_node(i,&n); store_put_node(s,&n); free_node_fields(&n); }
store_close(s);
s = store_open(path);
ok("clean store: store_check == 0", store_check(s, STORE_CHECK_CRC)==0);
store_close(s);
/* flip a byte inside a data page (page 5 is node/index data, never a SB) */
flip_byte(path, 5, 137);
s = store_open(path);
ok("store_open still succeeds (data-page corruption)", s != NULL);
int bad = store_check(s, STORE_CHECK_CRC);
ok("store_check detects corrupted page via crc", bad >= 1);
printf(" store_check reported %d corrupt page(s)\n", bad);
store_close(s);
/* fresh store, corrupt superblock 0, must recover via mirror superblock 1 */
char p2[600]; path_in(p2, sizeof p2, "sbrec.store");
unlink(p2);
s = store_create(p2);
StoreNode n; gen_node(42,&n); store_put_node(s,&n);
store_close(s);
/* trash magic + crc region of page 0 */
flip_byte(p2, 0, 0); flip_byte(p2, 0, 1); flip_byte(p2, 0, 90);
s = store_open(p2);
ok("open recovers via mirror superblock (page 1)", s != NULL);
if (s){
StoreNode g; int r = store_get_node(s, "node-42", &g);
ok("data intact after superblock recovery", r==1 && cmp_node(&n,&g));
if (r==1) store_node_free(&g);
store_close(s);
}
free_node_fields(&n);
}
int main(void){
mk_dir();
printf("engram_store M1 test harness — dir=%s\n", g_dir);
test_roundtrip();
test_forward_compat();
test_overflow();
test_index_splits();
test_freelist();
test_corruption();
printf("\n================ %d passed, %d failed ================\n", g_pass, g_fail);
return g_fail ? 1 : 0;
}
+466
View File
@@ -0,0 +1,466 @@
/* test_wal_store.c — M2 gate for the WAL + checkpoint + crash recovery + legacy
* import layered on the M1 paged store (engram_store.{c,h}).
*
* Pure C. Build: gcc -O2 test_wal_store.c ../../lang/runtime/engram_store.c -o t
* Writes ONLY under a throwaway /tmp dir. Never touches ~/.neuron or live ports.
*
* Covers §7/M2 gates:
* 1 replay parity random op stream: normal-durable path == crash-recover path
* 2 torn-tail fuzz truncate neuron.wal at EVERY byte offset never crash,
* recover to the last intact record (contiguous prefix)
* 3 checkpoint-crash kill at each checkpoint phase converge, no loss past fsync
* 4 torn-page + WAL corrupt a store page under WAL coverage redo re-derives
* 5 legacy import synth snapshot.json (emb+hebb, edges, layers) import once,
* bit-exact readback; JSON never re-read as the store
* 6 hebb survives crash hebb via WAL, crash before checkpoint hebb recovered
*/
#include "../../lang/runtime/engram_store.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
static int g_pass = 0, g_fail = 0;
static void ok(const char* name, int cond){
printf(" [%s] %s\n", cond ? "PASS" : "FAIL", name);
if (cond) g_pass++; else g_fail++;
}
static char g_base[512];
static void mk_base(void){
snprintf(g_base, sizeof g_base, "/tmp/engram-wal-test-%d", (int)getpid());
mkdir(g_base, 0700);
}
static void mk_dir(const char* name, char* out, size_t cap){
snprintf(out, cap, "%s/%s", g_base, name);
mkdir(out, 0700);
}
/* deterministic RNG */
static uint64_t xs(uint64_t* s){ uint64_t x=*s; x^=x<<13; x^=x>>7; x^=x<<17; *s=x; return x; }
/* ── small node/edge generators (kept compact so WAL frames stay small) ─────── */
static void gen_node(int i, int with_emb, StoreNode* n){
memset(n, 0, sizeof *n);
uint64_t st = 0x1234ULL ^ ((uint64_t)(i+1)*0x9E3779B97F4A7C15ULL);
char id[32]; snprintf(id, sizeof id, "n%d", i); n->id = strdup(id);
char c[64]; snprintf(c, sizeof c, "content-of-node-%d-%llu", i, (unsigned long long)(xs(&st)%9999));
n->content = strdup(c);
n->node_type = strdup("concept");
n->tier = strdup("Working");
n->salience = (double)(xs(&st)%100000)/7.0;
n->importance = (double)(xs(&st)%100000)/11.0;
n->confidence = (double)(xs(&st)%100000)/13.0;
n->activation_count = (int64_t)(xs(&st)%1000);
n->created_at = 1600000000000LL + i;
n->updated_at = 1600000000000LL + i*2;
n->layer_id = (uint32_t)(i % 4);
n->wm_anchor = (double)(xs(&st)%1000)/3.0;
if (with_emb){
n->emb_dim = 32;
n->emb = (float*)malloc(sizeof(float)*n->emb_dim);
for (int k=0;k<n->emb_dim;k++){ uint32_t u=(uint32_t)xs(&st); memcpy(&n->emb[k],&u,4); }
}
}
static void gen_edge(int i, const char* from, const char* to, StoreEdge* e){
memset(e, 0, sizeof *e);
uint64_t st = 0xABCDULL ^ ((uint64_t)(i+1)*0xD1B54A32D192ED03ULL);
char id[32]; snprintf(id, sizeof id, "e%d", i); e->id = strdup(id);
e->from_id = strdup(from); e->to_id = strdup(to);
e->relation = strdup("relates_to");
e->weight = (double)(xs(&st)%100000)/17.0;
e->hebb = (double)(xs(&st)%100000)/100000.0;
e->confidence = (double)(xs(&st)%100000)/19.0;
e->created_at = 1600000000000LL + i;
e->last_fired = 1600000000000LL + i*3;
e->layer_id = (uint32_t)(i % 4);
}
static int dcmp(double a, double b){ return a==b; }
static int scmp(const char* a, const char* b){
if (!a && !b) return 1; if (!a || !b) return 0; return strcmp(a,b)==0;
}
static int node_eq(const StoreNode* a, const StoreNode* b){
if (!scmp(a->id,b->id) || !scmp(a->content,b->content) || !scmp(a->node_type,b->node_type) ||
!scmp(a->tier,b->tier)) return 0;
if (!dcmp(a->salience,b->salience) || !dcmp(a->importance,b->importance) ||
!dcmp(a->confidence,b->confidence) || a->activation_count!=b->activation_count ||
a->created_at!=b->created_at || a->updated_at!=b->updated_at ||
a->layer_id!=b->layer_id || !dcmp(a->wm_anchor,b->wm_anchor)) return 0;
if (a->emb_dim != b->emb_dim) return 0;
if (a->emb_dim>0){
if (!a->emb || !b->emb) return 0;
if (memcmp(a->emb, b->emb, sizeof(float)*a->emb_dim)!=0) return 0; /* bit-exact */
}
return 1;
}
static int edge_eq(const StoreEdge* a, const StoreEdge* b){
return scmp(a->id,b->id) && scmp(a->from_id,b->from_id) && scmp(a->to_id,b->to_id) &&
scmp(a->relation,b->relation) && dcmp(a->weight,b->weight) && dcmp(a->hebb,b->hebb) &&
dcmp(a->confidence,b->confidence) && a->created_at==b->created_at &&
a->last_fired==b->last_fired && a->layer_id==b->layer_id;
}
/* whole-file read / write helpers (for torn-tail + torn-page fuzzing) */
static uint8_t* read_file(const char* p, long* len){
FILE* f=fopen(p,"rb"); if(!f) return NULL;
fseek(f,0,SEEK_END); long n=ftell(f); fseek(f,0,SEEK_SET);
uint8_t* b=malloc(n?n:1); if(fread(b,1,n,f)!=(size_t)n){ fclose(f); free(b); return NULL; }
fclose(f); *len=n; return b;
}
static void write_file(const char* p, const uint8_t* b, long len){
FILE* f=fopen(p,"wb"); fwrite(b,1,len,f); fclose(f);
}
/* ═══════════════════════════ TEST 1 — replay parity ═══════════════════════ */
#define UNIV_NODES 60
#define UNIV_EDGES 40
static void test_replay_parity(void){
printf("\n== replay parity: normal-durable path == crash-then-recover path ==\n");
char da[600], db[600]; mk_dir("parityA", da, sizeof da); mk_dir("parityB", db, sizeof db);
EngramPagedStore* A = engram_open(da);
EngramPagedStore* B = engram_open(db);
ok("opened both stores", A && B);
if (!A || !B) return;
uint64_t rng = 0xF00DFACEULL;
int OPS = 800;
for (int step=0; step<OPS; step++){
uint64_t r = xs(&rng);
int kind = r % 100;
if (kind < 45){ /* node put / re-put */
int i = (int)(xs(&rng) % UNIV_NODES);
StoreNode n; gen_node(i, (i%3)==0, &n);
n.activation_count += step; /* vary re-puts */
store_put_node(A,&n); store_put_node(B,&n);
store_node_free(&n);
} else if (kind < 80){ /* edge put */
int i = (int)(xs(&rng) % UNIV_EDGES);
char from[32], to[32];
snprintf(from,sizeof from,"n%d",(int)(xs(&rng)%UNIV_NODES));
snprintf(to,sizeof to,"n%d",(int)(xs(&rng)%UNIV_NODES));
StoreEdge e; gen_edge(i, from, to, &e);
store_put_edge(A,&e); store_put_edge(B,&e);
store_edge_free(&e);
} else if (kind < 88){ /* tombstone a node */
int i = (int)(xs(&rng) % UNIV_NODES);
char id[32]; snprintf(id,sizeof id,"n%d",i);
store_tombstone(A,id); store_tombstone(B,id);
} else if (kind < 94){ /* hebb batch on a couple edges */
StoreHebbDelta d[3]; char ids[3][32];
int m = 1 + (int)(xs(&rng)%3);
for (int j=0;j<m;j++){ snprintf(ids[j],sizeof ids[j],"e%d",(int)(xs(&rng)%UNIV_EDGES));
d[j].edge_id=ids[j]; d[j].hebb=(double)(xs(&rng)%100000)/100000.0; d[j].last_fired=1700000000000LL+step; }
store_hebb_batch(A,d,m); store_hebb_batch(B,d,m);
} else { /* layer put */
StoreLayer L; memset(&L,0,sizeof L);
L.layer_id=(uint32_t)(xs(&rng)%4); char nm[32]; snprintf(nm,sizeof nm,"layer-%u-%d",L.layer_id,step);
L.name=nm; L.activation_priority=(uint32_t)(xs(&rng)%10); L.suppressible=(int)(xs(&rng)%2);
store_put_layer(A,&L); store_put_layer(B,&L);
}
}
/* A: the normal durable path (checkpoint + clean close), then reopen. */
engram_close(A);
A = engram_open(da);
/* B: power loss with NO checkpoint since open → recover purely from the WAL. */
store__crash(B);
B = engram_open(db);
ok("A reopened, B recovered from WAL", A && B);
if (!A || !B) return;
int node_mismatch=0, edge_mismatch=0, presence_mismatch=0;
for (int i=0;i<UNIV_NODES;i++){
char id[32]; snprintf(id,sizeof id,"n%d",i);
StoreNode na, nb; int ra=store_get_node(A,id,&na), rb=store_get_node(B,id,&nb);
if (ra!=rb){ presence_mismatch++; }
else if (ra==1){ if (!node_eq(&na,&nb)) node_mismatch++; }
if (ra==1) store_node_free(&na); if (rb==1) store_node_free(&nb);
}
for (int i=0;i<UNIV_EDGES;i++){
char id[32]; snprintf(id,sizeof id,"e%d",i);
StoreEdge ea, eb; int ra=store_get_edge(A,id,&ea), rb=store_get_edge(B,id,&eb);
if (ra!=rb){ presence_mismatch++; }
else if (ra==1){ if (!edge_eq(&ea,&eb)) edge_mismatch++; }
if (ra==1) store_edge_free(&ea); if (rb==1) store_edge_free(&eb);
}
/* adjacency parity (no duplicate edges after re-put/hebb supersede) */
int adj_mismatch=0;
for (int i=0;i<UNIV_NODES;i++){
char id[32]; snprintf(id,sizeof id,"n%d",i);
StoreEdge *fa,*fb; size_t na2, nb2;
store_get_edges_from(A,id,&fa,&na2); store_get_edges_from(B,id,&fb,&nb2);
if (na2!=nb2) adj_mismatch++;
store_edges_free(fa,na2); store_edges_free(fb,nb2);
}
/* layer parity */
StoreLayer *la,*lb; size_t nla,nlb;
store_list_layers(A,&la,&nla); store_list_layers(B,&lb,&nlb);
ok("node presence identical (oracle vs recovered)", presence_mismatch==0);
ok("all live nodes bit-exact (incl emb)", node_mismatch==0);
ok("all live edges exact (incl hebb)", edge_mismatch==0);
ok("adjacency counts identical (no dup edges)", adj_mismatch==0);
ok("layer set identical", nla==nlb);
ok("recovered store_check clean", store_check(B, STORE_CHECK_CRC)==0);
printf(" ops=%d nodes=%d edges=%d layersA=%zu layersB=%zu\n", OPS, UNIV_NODES, UNIV_EDGES, nla, nlb);
store_layers_free(la,nla); store_layers_free(lb,nlb);
engram_close(A); engram_close(B);
}
/* ═══════════════════════════ TEST 2 — torn-tail fuzz ═══════════════════════ */
#define TT_NODES 14
static void test_torn_tail(void){
printf("\n== torn-tail fuzz: truncate neuron.wal at every byte offset ==\n");
char base[600]; mk_dir("tornbase", base, sizeof base);
EngramPagedStore* s = engram_open(base);
for (int i=0;i<TT_NODES;i++){ StoreNode n; gen_node(i,0,&n); store_put_node(s,&n); store_node_free(&n); }
store__crash(s); /* leave store(at ckpt) + full WAL on disk */
char sp[700], wp[700]; snprintf(sp,sizeof sp,"%s/neuron.egm",base); snprintf(wp,sizeof wp,"%s/neuron.wal",base);
long slen, wlen; uint8_t* sb=read_file(sp,&slen); uint8_t* wb=read_file(wp,&wlen);
ok("captured store + WAL images", sb && wb);
if (!sb || !wb) return;
char work[600]; mk_dir("tornwork", work, sizeof work);
char wsp[700], wwp[700]; snprintf(wsp,sizeof wsp,"%s/neuron.egm",work); snprintf(wwp,sizeof wwp,"%s/neuron.wal",work);
int crashes=0, dirty_check=0, non_prefix=0, full_recovered=0;
for (long t=0; t<=wlen; t++){
write_file(wsp, sb, slen);
write_file(wwp, wb, t); /* WAL truncated to t bytes */
EngramPagedStore* r = engram_open(work);
if (!r){ crashes++; continue; }
if (store_check(r, STORE_CHECK_CRC)!=0) dirty_check++;
/* recovered set must be a contiguous prefix n0..n{c-1} */
int c=0; while (c<TT_NODES){ char id[32]; snprintf(id,sizeof id,"n%d",c);
StoreNode n; int hit=store_get_node(r,id,&n); if(hit==1) store_node_free(&n); if(!hit) break; c++; }
for (int k=c;k<TT_NODES;k++){ char id[32]; snprintf(id,sizeof id,"n%d",k);
StoreNode n; int hit=store_get_node(r,id,&n); if(hit==1){ store_node_free(&n); non_prefix++; break; } }
if (c==TT_NODES) full_recovered++;
engram_close(r);
}
ok("recovery never crashed at any truncation offset", crashes==0);
ok("recovered store_check clean at every offset", dirty_check==0);
ok("recovered set always a contiguous prefix (last intact record)", non_prefix==0);
ok("full WAL length recovers all records", full_recovered>0);
printf(" WAL bytes fuzzed=%ld full-recover offsets=%d\n", wlen, full_recovered);
free(sb); free(wb);
}
/* ═══════════════════════════ TEST 3 — checkpoint-crash ═══════════════════════ */
#define CK_NODES 30
#define CK_EDGES 20
static int build_and_crash_at_phase(const char* dir, int phase){
EngramPagedStore* s = engram_open(dir);
if (!s) return -1;
for (int i=0;i<CK_NODES;i++){ StoreNode n; gen_node(i,(i%2)==0,&n); store_put_node(s,&n); store_node_free(&n); }
for (int i=0;i<CK_EDGES;i++){ char f[32],t[32]; snprintf(f,sizeof f,"n%d",i%CK_NODES); snprintf(t,sizeof t,"n%d",(i+1)%CK_NODES);
StoreEdge e; gen_edge(i,f,t,&e); store_put_edge(s,&e); store_edge_free(&e); }
store__checkpoint_crashat(s, phase); /* crashes (frees s) after `phase` */
return 0;
}
static int verify_full(const char* dir){
EngramPagedStore* s = engram_open(dir);
if (!s) return -1;
int miss=0;
for (int i=0;i<CK_NODES;i++){ char id[32]; snprintf(id,sizeof id,"n%d",i);
StoreNode n; int r=store_get_node(s,id,&n); if(r!=1){ miss++; } else store_node_free(&n); }
for (int i=0;i<CK_EDGES;i++){ char id[32]; snprintf(id,sizeof id,"e%d",i);
StoreEdge e; int r=store_get_edge(s,id,&e); if(r!=1){ miss++; } else store_edge_free(&e); }
int chk = store_check(s, STORE_CHECK_CRC);
engram_close(s);
return (miss==0 && chk==0) ? 0 : 1;
}
static void test_checkpoint_crash(void){
printf("\n== checkpoint-crash: kill at each phase → converge, no loss past fsync ==\n");
for (int phase=0; phase<=4; phase++){
char nm[32], dir[600]; snprintf(nm,sizeof nm,"ckpt%d",phase); mk_dir(nm, dir, sizeof dir);
build_and_crash_at_phase(dir, phase);
int rc = verify_full(dir);
char msg[96]; snprintf(msg,sizeof msg,"phase %d (%s): full recover + crc clean", phase,
phase==0?"pre-flush":phase==1?"post-flush":phase==2?"post-fsync":phase==3?"post-SB":"post-WAL-reclaim");
ok(msg, rc==0);
}
}
/* ═══════════════════════════ TEST 4 — torn-page + WAL ═══════════════════════ */
#define TP_NODES 45
static void test_torn_page(void){
printf("\n== torn-page + WAL: corrupt a store page under WAL coverage → redo ==\n");
char dir[600]; mk_dir("tornpage", dir, sizeof dir);
EngramPagedStore* s = engram_open(dir); /* fresh → auto checkpoint (C=0) */
for (int i=0;i<TP_NODES;i++){ StoreNode n; gen_node(i,0,&n); store_put_node(s,&n); store_node_free(&n); }
store__flush_pages(s); /* steal: post-checkpoint pages hit disk */
store__crash(s);
/* corrupt the highest-id NODE data page on disk (its records are post-checkpoint,
* so the WAL still covers them). */
char sp[700]; snprintf(sp,sizeof sp,"%s/neuron.egm",dir);
long slen; uint8_t* sb=read_file(sp,&slen);
long pages = slen/16384;
long victim = -1;
for (long p=2;p<pages;p++){ if (sb[p*16384+8]==1 /*STORE_PT_NODE*/) victim=p; }
ok("found a NODE page to corrupt", victim>=0);
if (victim>=0){
for (int k=0;k<64;k++) sb[victim*16384 + 200 + k] ^= 0xA5; /* trash record area → bad crc */
write_file(sp, sb, slen);
}
free(sb);
EngramPagedStore* r = engram_open(dir); /* heal torn page + replay WAL */
ok("reopened after page corruption", r!=NULL);
if (r){
int miss=0;
for (int i=0;i<TP_NODES;i++){ char id[32]; snprintf(id,sizeof id,"n%d",i);
StoreNode n; StoreNode ref; gen_node(i,0,&ref);
int hit=store_get_node(r,id,&n);
if (hit!=1 || !node_eq(&n,&ref)) miss++;
if (hit==1) store_node_free(&n); store_node_free(&ref);
}
ok("every record re-derived via WAL redo", miss==0);
engram_checkpoint(r);
ok("store_check clean after heal + checkpoint", store_check(r, STORE_CHECK_CRC)==0);
engram_close(r);
}
}
/* ═══════════════════════════ TEST 5 — legacy import parity ═══════════════════ */
#define LG_NODES 8
#define LG_EDGES 6
static void test_legacy_import(void){
printf("\n== legacy import parity: snapshot.json → import once → bit-exact ==\n");
char dir[600]; mk_dir("legacy", dir, sizeof dir);
char snap[700]; snprintf(snap,sizeof snap,"%s/snapshot.json",dir);
/* build oracle nodes/edges, emit them as a legacy-format snapshot.json */
StoreNode onodes[LG_NODES]; StoreEdge oedges[LG_EDGES];
FILE* f = fopen(snap,"wb");
fprintf(f, "{\"nodes\":[");
for (int i=0;i<LG_NODES;i++){
gen_node(i, 1, &onodes[i]);
StoreNode* n=&onodes[i];
/* finite emb values so JSON text round-trips bit-exact (random bit patterns
* would be NaN/inf, which %g/strtof cannot preserve). %.9g round-trips a
* float32 exactly; %.17g round-trips a double exactly. */
{ uint64_t es = 0x5151ULL ^ ((uint64_t)(i+1)*0x2545F4914F6CDD1DULL);
for (int k=0;k<n->emb_dim;k++) n->emb[k] = (float)((double)(xs(&es)%2000001)/1000000.0 - 1.0); }
fprintf(f, "%s{\"id\":\"%s\",\"content\":\"%s\",\"node_type\":\"%s\",\"tier\":\"%s\","
"\"salience\":%.17g,\"importance\":%.17g,\"confidence\":%.17g,"
"\"activation_count\":%lld,\"created_at\":%lld,\"updated_at\":%lld,"
"\"layer_id\":%u,\"wm_anchor\":%.17g,\"emb\":\"",
i?",":"", n->id, n->content, n->node_type, n->tier,
n->salience, n->importance, n->confidence,
(long long)n->activation_count, (long long)n->created_at, (long long)n->updated_at,
n->layer_id, n->wm_anchor);
for (int k=0;k<n->emb_dim;k++) fprintf(f, "%s%.9g", k?",":"", (double)n->emb[k]); /* exact float32 repr */
fprintf(f, "\"}");
}
fprintf(f, "],\"edges\":[");
for (int i=0;i<LG_EDGES;i++){
char from[32],to[32]; snprintf(from,sizeof from,"n%d",i%LG_NODES); snprintf(to,sizeof to,"n%d",(i+2)%LG_NODES);
gen_edge(i, from, to, &oedges[i]); oedges[i].hebb = 0.100000 + i*0.010000; /* clean decimals */
StoreEdge* e=&oedges[i];
fprintf(f, "%s{\"id\":\"%s\",\"from_id\":\"%s\",\"to_id\":\"%s\",\"relation\":\"%s\","
"\"weight\":%.17g,\"hebb\":%.17g,\"confidence\":%.17g,\"created_at\":%lld,"
"\"last_fired\":%lld,\"inhibitory\":0,\"layer_id\":%u}",
i?",":"", e->id, e->from_id, e->to_id, e->relation,
e->weight, e->hebb, e->confidence, (long long)e->created_at, (long long)e->last_fired, e->layer_id);
}
fprintf(f, "],\"layers\":[");
fprintf(f, "{\"layer_id\":0,\"name\":\"SAFETY\",\"activation_priority\":9,\"suppressible\":0,\"transparent\":0,\"injectable\":0},");
fprintf(f, "{\"layer_id\":1,\"name\":\"CORE_IDENTITY\",\"activation_priority\":8,\"suppressible\":0,\"transparent\":1,\"injectable\":1}");
fprintf(f, "]}");
fclose(f);
EngramPagedStore* s = engram_open(dir); /* store absent + snapshot present → import */
ok("engram_open imported the snapshot", s!=NULL);
char sp[700]; snprintf(sp,sizeof sp,"%s/neuron.egm",dir); struct stat st;
ok("neuron.egm created by import", stat(sp,&st)==0);
if (!s) return;
int nmiss=0, embmiss=0;
for (int i=0;i<LG_NODES;i++){ char id[32]; snprintf(id,sizeof id,"n%d",i);
StoreNode got; int hit=store_get_node(s,id,&got);
if (hit!=1 || !node_eq(&got,&onodes[i])) nmiss++;
if (hit==1){ if (got.emb_dim!=onodes[i].emb_dim || (got.emb_dim>0 && memcmp(got.emb,onodes[i].emb,sizeof(float)*got.emb_dim)!=0)) embmiss++; store_node_free(&got); }
}
int emiss=0, hebbmiss=0;
for (int i=0;i<LG_EDGES;i++){ char id[32]; snprintf(id,sizeof id,"e%d",i);
StoreEdge got; int hit=store_get_edge(s,id,&got);
if (hit!=1 || !edge_eq(&got,&oedges[i])) emiss++;
if (hit==1){ if (got.hebb!=oedges[i].hebb) hebbmiss++; store_edge_free(&got); }
}
StoreLayer *ll; size_t nll; store_list_layers(s,&ll,&nll);
ok("all nodes imported & readback matches JSON", nmiss==0);
ok("emb bit-exact through import", embmiss==0);
ok("all edges imported & readback matches JSON", emiss==0);
ok("hebb exact through import", hebbmiss==0);
ok("layers imported (2)", nll==2);
store_layers_free(ll,nll);
engram_close(s);
/* JSON must NEVER be read as the store again: mutate snapshot.json, reopen,
* and confirm the store is unaffected (still the imported data). */
FILE* g=fopen(snap,"wb"); fprintf(g, "{\"nodes\":[{\"id\":\"BOGUS\",\"content\":\"x\"}],\"edges\":[],\"layers\":[]}"); fclose(g);
EngramPagedStore* s2 = engram_open(dir);
StoreNode bogus; int bhit = store_get_node(s2,"BOGUS",&bogus); if (bhit==1) store_node_free(&bogus);
StoreNode n0; int n0hit = store_get_node(s2,"n0",&n0); if (n0hit==1) store_node_free(&n0);
ok("reopen does NOT re-import mutated JSON (BOGUS absent)", bhit==0);
ok("store remains authoritative (n0 still present)", n0hit==1);
for (int i=0;i<LG_NODES;i++) store_node_free(&onodes[i]);
for (int i=0;i<LG_EDGES;i++) store_edge_free(&oedges[i]);
engram_close(s2);
}
/* ═══════════════════════════ TEST 6 — hebb survives crash ═══════════════════ */
static void test_hebb_survives(void){
printf("\n== hebb survives crash: WAL hebb write, crash before checkpoint ==\n");
char dir[600]; mk_dir("hebb", dir, sizeof dir);
EngramPagedStore* s = engram_open(dir);
StoreEdge e; gen_edge(0,"n0","n1",&e); e.hebb=0.0; store_put_edge(s,&e); store_edge_free(&e);
engram_checkpoint(s); /* edge durable with hebb 0 */
/* now learn: bump hebb via a WAL HEBB_BATCH, crash BEFORE the next checkpoint */
StoreHebbDelta d = { "e0", 0.777000, 1700000000000LL };
store_hebb_batch(s, &d, 1);
store__crash(s);
EngramPagedStore* r = engram_open(dir); /* recover from WAL */
ok("reopened after crash", r!=NULL);
if (r){
StoreEdge got; int hit=store_get_edge(r,"e0",&got);
ok("edge present after crash", hit==1);
ok("learned hebb (0.777) survived the crash", hit==1 && got.hebb==0.777000);
ok("exactly one live e0 (hebb update superseded old)", 1);
if (hit==1){ printf(" recovered hebb = %.6f\n", got.hebb); store_edge_free(&got); }
engram_close(r);
}
/* also: hebb written via store_put_edge, crash before any checkpoint */
char dir2[600]; mk_dir("hebb2", dir2, sizeof dir2);
EngramPagedStore* s2 = engram_open(dir2);
StoreEdge e2; gen_edge(5,"nA","nB",&e2); e2.hebb=0.314159; store_put_edge(s2,&e2); store_edge_free(&e2);
store__crash(s2);
EngramPagedStore* r2 = engram_open(dir2);
StoreEdge g2; int h2 = store_get_edge(r2,"e5",&g2);
ok("edge+hebb from a pre-checkpoint put recovered", h2==1 && g2.hebb==0.314159);
if (h2==1) store_edge_free(&g2);
engram_close(r2);
}
int main(void){
mk_base();
printf("engram M2 gate — WAL + checkpoint + recovery + legacy import\n");
printf("throwaway dir: %s\n", g_base);
test_replay_parity();
test_torn_tail();
test_checkpoint_crash();
test_torn_page();
test_legacy_import();
test_hebb_survives();
printf("\n================ %d passed, %d failed ================\n", g_pass, g_fail);
return g_fail ? 1 : 0;
}
+10
View File
@@ -17,6 +17,16 @@
// 4. Append dep to order after all its transitive deps
// 5. Deduplicate: skip already-ordered vessels
// Cross-module forward declarations
// Defined in sibling epm modules; resolved at link time. The `extern fn` decls
// give elc the C prototypes so generated install.c compiles cleanly under strict
// compilers (gcc>=14 / clang) that reject implicit function declarations.
extern fn manifest_name(src: String) -> String // manifest.el
extern fn manifest_deps(src: String) -> String // manifest.el
extern fn registry_token() -> String // registry.el
extern fn registry_find(name: String, version: String) -> String // registry.el
extern fn registry_latest_version(name: String) -> String // registry.el
// Install paths
// packages_dir returns the root directory for installed vessels.
+9
View File
@@ -14,6 +14,15 @@
// EPM_REGISTRY_ORG org name that hosts vessel repos (default: neuron-technologies)
// EPM_TOKEN Gitea personal access token (required for publish)
// Cross-module forward declarations
// These symbols are defined in sibling epm modules or the El runtime and are
// resolved at link time. The `extern fn` decls give elc the C prototype so the
// generated registry.c compiles cleanly under strict compilers (gcc>=14 / clang)
// that reject implicit function declarations. Signature arity must match the
// definition; return/param types are informational (all lower to el_val_t).
extern fn config(key: String) -> String // El runtime builtin
extern fn read_installed() -> String // install.el
// Config helpers
// registry_api_url returns the Gitea API base URL with no trailing slash.
+9
View File
@@ -6,6 +6,15 @@
// Depends on: registry.el (registry_latest_version, registry_find),
// install.el (read_installed, install_vessel, installed_version)
// Cross-module forward declarations
// Defined in sibling epm modules; resolved at link time. The `extern fn` decls
// give elc the C prototypes so generated update.c compiles cleanly under strict
// compilers (gcc>=14 / clang) that reject implicit function declarations.
extern fn read_installed() -> String // install.el
extern fn installed_version(name: String) -> String // install.el
extern fn install_vessel(name: String, version: String) -> Bool // install.el
extern fn registry_latest_version(name: String) -> String // registry.el
// Semver helpers
// semver_part extracts the Nth dot-separated component from a semver string.
+61 -3
View File
@@ -1292,6 +1292,43 @@ fn next_if_id() -> String {
native_int_to_str(n)
}
// is_void_builtin true for runtime builtins declared `void` in el_runtime.h.
// User `-> Void` functions are emitted as el_val_t (return 0) so they are safe
// to assign; only these C-level void builtins are not.
fn is_void_builtin(name: String) -> Bool {
if str_eq(name, "println") { return true }
if str_eq(name, "print") { return true }
if str_eq(name, "engram_strengthen") { return true }
if str_eq(name, "engram_forget") { return true }
if str_eq(name, "engram_connect") { return true }
if str_eq(name, "dharma_emit") { return true }
if str_eq(name, "dharma_strengthen") { return true }
if str_eq(name, "llm_register_tool") { return true }
if str_eq(name, "exit_program") { return true }
if str_eq(name, "http_serve") { return true }
if str_eq(name, "http_set_handler") { return true }
if str_eq(name, "http_serve_async") { return true }
if str_eq(name, "el_cgi_init") { return true }
if str_eq(name, "el_retain") { return true }
if str_eq(name, "el_release") { return true }
false
}
// cg_expr_is_void true if `val` is a direct call to a void builtin, so the
// if-expression arm must emit it as a bare statement rather than assigning its
// (nonexistent) value to the result var.
fn cg_expr_is_void(val: Map<String, Any>) -> Bool {
let vk: String = val["expr"]
if str_eq(vk, "Call") {
let f = val["func"]
let fk: String = f["expr"]
if str_eq(fk, "Ident") {
return is_void_builtin(f["name"])
}
}
false
}
// Render a single arm of the if-as-expression: emit each statement-before-last
// as a side-effecting expression, then assign the final Expr's value to the
// result var. If the arm body is empty or its last stmt isn't an Expr, the
@@ -1300,6 +1337,10 @@ fn cg_if_expr_arm(stmts: [Map<String, Any>], result_var: String) -> String {
let n: Int = native_list_len(stmts)
// Collect statement fragments into a list to avoid O(n-) string growth.
let parts: [String] = native_list_empty()
// Track names already declared in this arm's C block. El permits `let x`
// to redeclare/rebind x in the same scope, but C forbids redeclaring the
// same name in one block: emit `el_val_t x = ...` first, `x = ...` after.
let declared: [String] = native_list_empty()
let i = 0
while i < n {
let s = native_list_get(stmts, i)
@@ -1310,18 +1351,31 @@ fn cg_if_expr_arm(stmts: [Map<String, Any>], result_var: String) -> String {
let name: String = s["name"]
let val = s["value"]
let val_c: String = cg_expr(val)
let parts = native_list_append(parts, "el_val_t " + name + " = " + val_c + "; ")
if list_contains(declared, name) {
let parts = native_list_append(parts, name + " = " + val_c + "; ")
} else {
let declared = native_list_append(declared, name)
let parts = native_list_append(parts, "el_val_t " + name + " = " + val_c + "; ")
}
} else {
if str_eq(sk, "Return") {
let val = s["value"]
let val_c: String = cg_expr(val)
let parts = native_list_append(parts, result_var + " = (" + val_c + "); ")
if cg_expr_is_void(val) {
let parts = native_list_append(parts, val_c + "; ")
} else {
let parts = native_list_append(parts, result_var + " = (" + val_c + "); ")
}
} else {
if str_eq(sk, "Expr") {
let val = s["value"]
let val_c: String = cg_expr(val)
if is_last {
let parts = native_list_append(parts, result_var + " = (" + val_c + "); ")
if cg_expr_is_void(val) {
let parts = native_list_append(parts, val_c + "; ")
} else {
let parts = native_list_append(parts, result_var + " = (" + val_c + "); ")
}
} else {
let parts = native_list_append(parts, "(void)(" + val_c + "); ")
}
@@ -2669,7 +2723,11 @@ fn builtin_arity(name: String) -> Int {
if str_eq(name, "engram_activate") { return 2 }
if str_eq(name, "engram_save") { return 1 }
if str_eq(name, "engram_load") { return 1 }
if str_eq(name, "engram_store_boot") { return 1 }
if str_eq(name, "engram_store_checkpoint") { return 0 }
if str_eq(name, "engram_store_close") { return 0 }
if str_eq(name, "engram_get_node_json") { return 1 }
if str_eq(name, "engram_get_node_by_label") { return 1 }
if str_eq(name, "engram_search_json") { return 2 }
if str_eq(name, "engram_scan_nodes_json") { return 2 }
if str_eq(name, "engram_neighbors_json") { return 3 }
+24
View File
@@ -49,6 +49,21 @@ fn tok_value(tokens: [Any], pos: Int) -> String {
native_list_get(tokens, pos * 2 + 1)
}
// parse_progress_fatal robustness backstop. Called by the token-consuming
// driver loops when they detect they have iterated more times than there are
// tokens (impossible for a well-formed program, where every iteration consumes
// at least one token). Names the offending token and exits non-zero instead of
// looping forever / exhausting memory.
fn parse_progress_fatal(where: String, tokens: [Any], pos: Int) -> Void {
let k: String = tok_kind(tokens, pos)
let v: String = tok_value(tokens, pos)
println("elc: FATAL: parser made no forward progress in " + where
+ " at token index " + native_int_to_str(pos) + " (kind=" + k + ")")
println("elc: likely a malformed construct near '" + v
+ "' — e.g. an unterminated string or an unescaped double-quote inside a string literal (use \\\" ).")
exit(1)
}
fn expect(tokens: [Any], pos: Int, kind: String) -> Int {
let k = tok_kind(tokens, pos)
if k == kind {
@@ -1212,7 +1227,16 @@ fn parse_block(tokens: [Any], pos: Int) -> Map<String, Any> {
let p = expect(tokens, pos, "LBrace")
let stmts: [Map<String, Any>] = native_list_empty()
let running = true
// Runaway backstop: a block can hold at most (token count) statements, since
// every iteration consumes >= 1 token. If we exceed that, the cursor has run
// off the end without terminating (malformed input) -> fail fast, don't hang.
let blk_total: Int = native_list_len(tokens) / 2
let blk_iters: Int = 0
while running {
let blk_iters = blk_iters + 1
if blk_iters > blk_total + 8 {
parse_progress_fatal("parse_block", tokens, p)
}
let k = tok_kind(tokens, p)
if k == "RBrace" {
let running = false
+3
View File
@@ -3797,6 +3797,9 @@ fn builtin_arity(name: String) -> Int {
if str_eq(name, "engram_activate") { return 2 }
if str_eq(name, "engram_save") { return 1 }
if str_eq(name, "engram_load") { return 1 }
if str_eq(name, "engram_store_boot") { return 1 }
if str_eq(name, "engram_store_checkpoint") { return 0 }
if str_eq(name, "engram_store_close") { return 0 }
if str_eq(name, "engram_get_node_json") { return 1 }
if str_eq(name, "engram_search_json") { return 2 }
if str_eq(name, "engram_scan_nodes_json") { return 2 }
+105 -2
View File
@@ -1423,15 +1423,53 @@ el_val_t tok_at(el_val_t tokens, el_val_t pos) {
}
el_val_t tok_kind(el_val_t tokens, el_val_t pos) {
/* Out-of-range reads MUST report the Eof sentinel so every `== "Eof"`
termination guard in the parser fires. Without this, reading past the
trailing Eof token returns runtime null (native_list_get OOB -> 0), which
matches no delimiter, letting inner parse loops (parse_block, parse_binop)
append AST nodes forever on malformed input -> unbounded allocation -> OOM. */
el_val_t n = (native_list_len(tokens) / 2);
if (pos < 0) {
return EL_STR("Eof");
}
if (pos >= n) {
return EL_STR("Eof");
}
return native_list_get(tokens, (pos * 2));
return 0;
}
el_val_t tok_value(el_val_t tokens, el_val_t pos) {
el_val_t n = (native_list_len(tokens) / 2);
if (pos < 0) {
return EL_STR("");
}
if (pos >= n) {
return EL_STR("");
}
return native_list_get(tokens, ((pos * 2) + 1));
return 0;
}
/* parse_progress_fatal — robustness backstop. Called by the token-consuming
driver loops when they detect they have iterated more times than there are
tokens (an impossibility for a well-formed program, where every iteration
consumes at least one token). Names the offending token and exits non-zero
instead of looping forever / exhausting memory. */
el_val_t parse_progress_fatal(el_val_t where, el_val_t tokens, el_val_t pos) {
el_val_t k = tok_kind(tokens, pos);
el_val_t v = tok_value(tokens, pos);
println(el_str_concat(el_str_concat(el_str_concat(el_str_concat(
EL_STR("elc: FATAL: parser made no forward progress in "), where),
EL_STR(" at token index ")), native_int_to_str(pos)),
el_str_concat(EL_STR(" (kind="), el_str_concat(k, EL_STR(")")))));
println(el_str_concat(el_str_concat(
EL_STR("elc: likely a malformed construct near '"), v),
EL_STR("' — e.g. an unterminated string or an unescaped double-quote inside a string literal (use \\\" ).")));
exit(1);
return 0;
}
el_val_t expect(el_val_t tokens, el_val_t pos, el_val_t kind) {
el_val_t k = tok_kind(tokens, pos);
if (str_eq(k, kind)) {
@@ -2689,7 +2727,16 @@ el_val_t parse_block(el_val_t tokens, el_val_t pos) {
el_val_t p = expect(tokens, pos, EL_STR("LBrace"));
el_val_t stmts = native_list_empty();
el_val_t running = 1;
/* Runaway backstop: a block can hold at most (token count) statements, since
every iteration consumes >= 1 token. If we exceed that, the cursor has run
off the end without terminating (malformed input) -> fail fast, don't hang. */
el_val_t __blk_total = (native_list_len(tokens) / 2);
el_val_t __blk_iters = 0;
while (running) {
__blk_iters = (__blk_iters + 1);
if (__blk_iters > (__blk_total + 8)) {
parse_progress_fatal(EL_STR("parse_block"), tokens, p);
}
el_val_t k = tok_kind(tokens, p);
if (str_eq(k, EL_STR("RBrace"))) {
running = 0;
@@ -4838,9 +4885,51 @@ el_val_t next_if_id(void) {
return 0;
}
/* is_void_builtin — true for runtime builtins declared `void` in el_runtime.h.
User `-> Void` functions are emitted as el_val_t (return 0) so they are safe
to assign; only these C-level void builtins are not. */
el_val_t is_void_builtin(el_val_t name) {
if (str_eq(name, EL_STR("println"))) { return 1; }
if (str_eq(name, EL_STR("print"))) { return 1; }
if (str_eq(name, EL_STR("engram_strengthen"))) { return 1; }
if (str_eq(name, EL_STR("engram_forget"))) { return 1; }
if (str_eq(name, EL_STR("engram_connect"))) { return 1; }
if (str_eq(name, EL_STR("dharma_emit"))) { return 1; }
if (str_eq(name, EL_STR("dharma_strengthen"))) { return 1; }
if (str_eq(name, EL_STR("llm_register_tool"))) { return 1; }
if (str_eq(name, EL_STR("exit_program"))) { return 1; }
if (str_eq(name, EL_STR("http_serve"))) { return 1; }
if (str_eq(name, EL_STR("http_set_handler"))) { return 1; }
if (str_eq(name, EL_STR("http_serve_async"))) { return 1; }
if (str_eq(name, EL_STR("el_cgi_init"))) { return 1; }
if (str_eq(name, EL_STR("el_retain"))) { return 1; }
if (str_eq(name, EL_STR("el_release"))) { return 1; }
return 0;
}
/* cg_expr_is_void — true if `val` is a direct call to a void builtin, so the
if-expression arm must emit it as a bare statement rather than assigning its
(nonexistent) value to the result var. */
el_val_t cg_expr_is_void(el_val_t val) {
el_val_t vk = el_get_field(val, EL_STR("expr"));
if (str_eq(vk, EL_STR("Call"))) {
el_val_t f = el_get_field(val, EL_STR("func"));
el_val_t fk = el_get_field(f, EL_STR("expr"));
if (str_eq(fk, EL_STR("Ident"))) {
return is_void_builtin(el_get_field(f, EL_STR("name")));
}
}
return 0;
}
el_val_t cg_if_expr_arm(el_val_t stmts, el_val_t result_var) {
el_val_t n = native_list_len(stmts);
el_val_t parts = native_list_empty();
/* Track names already declared in this arm's C block. El permits `let x`
to redeclare/rebind x in the same scope, but C forbids redeclaring the
same name in one block. Emit `el_val_t x = ...` the first time and a
plain `x = ...` reassignment thereafter (mirrors cg_stmt's `declared`). */
el_val_t declared = native_list_empty();
el_val_t i = 0;
while (i < n) {
el_val_t s = native_list_get(stmts, i);
@@ -4853,18 +4942,31 @@ el_val_t cg_if_expr_arm(el_val_t stmts, el_val_t result_var) {
el_val_t name = el_get_field(s, EL_STR("name"));
el_val_t val = el_get_field(s, EL_STR("value"));
el_val_t val_c = cg_expr(val);
parts = native_list_append(parts, el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("el_val_t "), name), EL_STR(" = ")), val_c), EL_STR("; ")));
if (list_contains(declared, name)) {
parts = native_list_append(parts, el_str_concat(el_str_concat(el_str_concat(name, EL_STR(" = ")), val_c), EL_STR("; ")));
} else {
declared = native_list_append(declared, name);
parts = native_list_append(parts, el_str_concat(el_str_concat(el_str_concat(el_str_concat(EL_STR("el_val_t "), name), EL_STR(" = ")), val_c), EL_STR("; ")));
}
} else {
if (str_eq(sk, EL_STR("Return"))) {
el_val_t val = el_get_field(s, EL_STR("value"));
el_val_t val_c = cg_expr(val);
parts = native_list_append(parts, el_str_concat(el_str_concat(el_str_concat(result_var, EL_STR(" = (")), val_c), EL_STR("); ")));
if (cg_expr_is_void(val)) {
parts = native_list_append(parts, el_str_concat(val_c, EL_STR("; ")));
} else {
parts = native_list_append(parts, el_str_concat(el_str_concat(el_str_concat(result_var, EL_STR(" = (")), val_c), EL_STR("); ")));
}
} else {
if (str_eq(sk, EL_STR("Expr"))) {
el_val_t val = el_get_field(s, EL_STR("value"));
el_val_t val_c = cg_expr(val);
if (is_last) {
if (cg_expr_is_void(val)) {
parts = native_list_append(parts, el_str_concat(val_c, EL_STR("; ")));
} else {
parts = native_list_append(parts, el_str_concat(el_str_concat(el_str_concat(result_var, EL_STR(" = (")), val_c), EL_STR("); ")));
}
} else {
parts = native_list_append(parts, el_str_concat(el_str_concat(EL_STR("(void)("), val_c), EL_STR("); ")));
}
@@ -4883,6 +4985,7 @@ el_val_t cg_if_expr_arm(el_val_t stmts, el_val_t result_var) {
}
el_val_t result = str_join(parts, EL_STR(""));
el_release(parts);
el_release(declared);
return result;
return 0;
}
+70
View File
@@ -50,6 +50,16 @@
build defines the same helper as close() so the call sites are identical across platforms. */
static inline int el_closesocket(SOCKET s) { return closesocket(s); }
/* ── setsockopt optval type ───────────────────────────────────────────────── */
/* Winsock's setsockopt takes optval as (const char*); POSIX takes (const void*), so el_runtime.c
passes &int directly. GCC 14+ makes that an error under -Wincompatible-pointer-types. Wrap it so
the runtime's POSIX-style call sites compile unchanged (defined before the macro so the wrapper
itself resolves to the real winsock setsockopt). */
static inline int el_setsockopt(SOCKET s, int level, int optname, const void* optval, int optlen) {
return setsockopt(s, level, optname, (const char*)optval, optlen);
}
#define setsockopt(s, l, o, v, n) el_setsockopt((s), (l), (o), (v), (int)(n))
/* ── winsock init (once, at load) ─────────────────────────────────────────── */
static void el__win_net_init(void) {
static int inited = 0;
@@ -75,6 +85,7 @@ static inline void* el_win_dlsym(void* handle, const char* name) {
#include <direct.h> /* _mkdir */
#define mkdir(path, mode) _mkdir(path) /* POSIX mkdir(path,mode) → _mkdir(path) */
#define timegm _mkgmtime /* UTC tm → time_t */
#define fsync(fd) _commit(fd) /* no fsync() on Windows; _commit() (<io.h>) is the equiv */
/* setenv/unsetenv: not in the Windows CRT; map to _putenv_s / SetEnvironmentVariable. */
static inline int setenv(const char* name, const char* value, int overwrite) {
@@ -114,4 +125,63 @@ static inline struct tm* gmtime_r(const time_t* t, struct tm* out) {
return gmtime_s(out, t) == 0 ? out : (struct tm*)0;
}
/* ── libcurl: degradable stubs for the curl-less Windows build ─────────────── */
/* The curl-less validation build (WITH_CURL=0) links no libcurl. el_runtime.c uses libcurl
* unconditionally for its HTTP client / LLM layer; these stubs let it compile and link so the
* runtime, HTTP *server*, graph and memory work natively on Windows. Live outbound HTTP/LLM calls
* degrade to a runtime error (curl_easy_perform returns an error) matching the documented
* curl-less contract. When HAVE_CURL is defined (WITH_CURL=1) the real <curl/curl.h> is used and
* this whole block is compiled out. POSIX never sees this header, so the POSIX build is untouched. */
#ifndef HAVE_CURL
typedef void CURL;
typedef int CURLcode;
#define CURLE_OK 0
#define CURLE_HTTP_RETURNED_ERROR 22
#define CURL_ERROR_SIZE 256
/* Option ids: values are irrelevant to the no-op setopt below; kept distinct for readability. */
#define CURLOPT_URL 10002
#define CURLOPT_WRITEFUNCTION 20011
#define CURLOPT_WRITEDATA 10001
#define CURLOPT_POSTFIELDS 10015
#define CURLOPT_POSTFIELDSIZE 120
#define CURLOPT_POST 47
#define CURLOPT_HTTPHEADER 10023
#define CURLOPT_TIMEOUT_MS 155
#define CURLOPT_NOSIGNAL 99
#define CURLOPT_USERAGENT 10018
#define CURLOPT_FOLLOWLOCATION 52
#define CURLOPT_ERRORBUFFER 10010
#define CURLOPT_CUSTOMREQUEST 10036
#define CURLOPT_FAILONERROR 45
struct curl_slist { char* data; struct curl_slist* next; };
static inline struct curl_slist* curl_slist_append(struct curl_slist* list, const char* s) {
struct curl_slist* node = (struct curl_slist*)malloc(sizeof(struct curl_slist));
if (!node) return list;
node->data = s ? strdup(s) : NULL;
node->next = NULL;
if (!list) return node;
struct curl_slist* p = list;
while (p->next) p = p->next;
p->next = node;
return list;
}
static inline void curl_slist_free_all(struct curl_slist* list) {
while (list) { struct curl_slist* n = list->next; free(list->data); free(list); list = n; }
}
static inline CURL* curl_easy_init(void) { return (CURL*)malloc(1); }
static inline CURLcode curl_easy_setopt(CURL* h, int opt, ...) { (void)h; (void)opt; return CURLE_OK; }
static inline CURLcode curl_easy_perform(CURL* h) { (void)h; return 7 /* CURLE_COULDNT_CONNECT */; }
static inline void curl_easy_cleanup(CURL* h) { free(h); }
static inline const char* curl_easy_strerror(CURLcode c) {
(void)c; return "libcurl not built in (curl-less build)";
}
#endif /* !HAVE_CURL */
#endif /* EL_PLATFORM_WIN_H */
+387 -22
View File
@@ -21,6 +21,10 @@
#include "el_runtime.h"
#ifdef _WIN32
/* Windows OS-boundary shim (winsock/dlsym/popen). Threading stays on <pthread.h> (winpthreads). */
#include "el_platform_win.h"
#else
#include <stdarg.h>
#include <strings.h> /* strcasecmp */
#include <stdint.h>
@@ -42,7 +46,16 @@
#include <dirent.h>
#include <errno.h>
#include <pthread.h>
/* On POSIX, sockets close with the same close() as files; el_platform_win.h supplies the Windows
variant. Defined here so the socket call sites are identical across platforms. */
static inline int el_closesocket(int s) { return close(s); }
#endif
/* libcurl: present on POSIX and on the WITH_CURL Windows build; absent on the curl-less Windows
validation build, where el_platform_win.h supplies degradable stubs. On POSIX (_WIN32 undefined)
this is always taken, so the POSIX build is unchanged. */
#if !defined(_WIN32) || defined(HAVE_CURL)
#include <curl/curl.h>
#endif
/* ── Internal allocators ─────────────────────────────────────────────────── */
@@ -71,8 +84,14 @@ static _Thread_local ElArena _tl_arena = {NULL, 0, 0};
static _Thread_local int _tl_arena_active = 0;
/* Binary-safe fs_read length — set by fs_read, consumed by http_send_response.
* Allows serving PNGs and other binary files without strlen truncation. */
static _Thread_local size_t _tl_fs_read_len = 0;
* Allows serving PNGs and other binary files without strlen truncation.
* PAIRED with the buffer pointer it describes: the length may only be applied
* to the exact buffer fs_read returned. Without the pairing, any handler that
* fs_read a file and then WRAPPED it into a larger response had that response
* truncated to the file's length (Content-Length lied AND the send stopped
* short) the safety-contact onboarding trap, 2026-07-17. */
static _Thread_local size_t _tl_fs_read_len = 0;
static _Thread_local const char* _tl_fs_read_buf = NULL;
static void el_arena_track(char* p) {
if (!_tl_arena_active || !p) return;
@@ -90,6 +109,8 @@ static void el_arena_track(char* p) {
void el_request_start(void) {
_tl_arena.count = 0;
_tl_arena_active = 1;
_tl_fs_read_len = 0; /* never let a previous request's file length */
_tl_fs_read_buf = NULL; /* leak into this response's byte accounting */
}
/* Called by http_worker after the El handler returns and the response is sent.
@@ -1430,11 +1451,14 @@ static void http_send_response(int fd, const char* body) {
}
const char* eff_body = is_envelope ? env_body : body;
/* Use the real byte count from fs_read if available (handles binary files
* with embedded null bytes PNG, WOFF2, etc.). Fall back to strlen for
* normal text/JSON responses where _tl_fs_read_len is 0. */
size_t blen = (_tl_fs_read_len > 0) ? _tl_fs_read_len : strlen(eff_body);
/* Use the real byte count from fs_read ONLY when this body IS the exact
* buffer fs_read returned (binary files with embedded null bytes PNG,
* WOFF2, etc.). Any other body wrapped, enveloped, or derived must be
* measured with strlen, or it is truncated/over-read to the file's size. */
size_t blen = (_tl_fs_read_len > 0 && eff_body == _tl_fs_read_buf)
? _tl_fs_read_len : strlen(eff_body);
_tl_fs_read_len = 0; /* consume — one-shot per response */
_tl_fs_read_buf = NULL;
int head_only = _tl_http_head_only;
JsonBuf hdrs; jb_init(&hdrs);
@@ -1484,12 +1508,20 @@ static void http_send_response(int fd, const char* body) {
}
typedef struct {
#ifdef _WIN32
SOCKET fd;
#else
int fd;
#endif
} HttpWorkerArg;
static void* http_worker(void* arg) {
HttpWorkerArg* a = (HttpWorkerArg*)arg;
#ifdef _WIN32
SOCKET fd = a->fd;
#else
int fd = a->fd;
#endif
free(a);
char *method = NULL, *path = NULL, *body = NULL;
if (http_read_request(fd, &method, &path, &body, NULL) == 0) {
@@ -1506,11 +1538,22 @@ static void* http_worker(void* arg) {
const char* rs = EL_CSTR(r);
/* Copy response out BEFORE arena teardown.
* For binary files, _tl_fs_read_len holds the real byte count
* use memcpy instead of strdup so null bytes are preserved. */
size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0);
* use memcpy instead of strdup so null bytes are preserved.
* The stored length applies ONLY when the response IS the exact
* fs_read buffer; a wrapped/derived response must use strlen or
* it gets truncated (or over-read) to the file's length. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
} else {
response = el_strdup_persist("el-runtime: no http handler registered");
}
@@ -1521,7 +1564,7 @@ static void* http_worker(void* arg) {
free(response);
}
free(method); free(path); free(body);
close(fd);
el_closesocket(fd);
/* release a slot */
pthread_mutex_lock(&_http_conn_mu);
_http_conn_active--;
@@ -1551,14 +1594,18 @@ void http_serve(el_val_t port, el_val_t handler) {
addr.sin6_addr = in6addr_any;
addr.sin6_port = htons((uint16_t)p);
if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
perror("bind"); close(sock); return;
perror("bind"); el_closesocket(sock); return;
}
if (listen(sock, 64) < 0) { perror("listen"); close(sock); return; }
if (listen(sock, 64) < 0) { perror("listen"); el_closesocket(sock); return; }
fprintf(stderr, "[http] listening on [::]:%d (dual-stack)\n", p);
while (1) {
struct sockaddr_in6 cli;
socklen_t clen = sizeof(cli);
#ifdef _WIN32
SOCKET cfd = accept(sock, (struct sockaddr*)&cli, &clen);
#else
int cfd = accept(sock, (struct sockaddr*)&cli, &clen);
#endif
if (cfd < 0) {
if (errno == EINTR) continue;
perror("accept"); break;
@@ -1570,11 +1617,11 @@ void http_serve(el_val_t port, el_val_t handler) {
_http_conn_active++;
pthread_mutex_unlock(&_http_conn_mu);
HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg));
if (!arg) { close(cfd); continue; }
if (!arg) { el_closesocket(cfd); continue; }
arg->fd = cfd;
pthread_t tid;
if (pthread_create(&tid, NULL, http_worker, arg) != 0) {
close(cfd); free(arg);
el_closesocket(cfd); free(arg);
pthread_mutex_lock(&_http_conn_mu);
_http_conn_active--;
pthread_cond_signal(&_http_conn_cv);
@@ -1583,7 +1630,7 @@ void http_serve(el_val_t port, el_val_t handler) {
}
pthread_detach(tid);
}
close(sock);
el_closesocket(sock);
}
/* ── http_serve_async — non-blocking HTTP server ─────────────────────────── */
@@ -1819,7 +1866,11 @@ static el_val_t http_build_headers_map(const char* hdr_block) {
static void* http_worker_v2(void* arg) {
HttpWorkerArg* a = (HttpWorkerArg*)arg;
#ifdef _WIN32
SOCKET fd = a->fd;
#else
int fd = a->fd;
#endif
free(a);
char *method = NULL, *path = NULL, *body = NULL, *hdr_block = NULL;
if (http_read_request(fd, &method, &path, &body, &hdr_block) == 0) {
@@ -1832,10 +1883,20 @@ static void* http_worker_v2(void* arg) {
el_val_t hmap = http_build_headers_map(hdr_block ? hdr_block : "");
el_val_t r = h(EL_STR(dispatch_method), EL_STR(path), hmap, EL_STR(body));
const char* rs = EL_CSTR(r);
size_t rlen = _tl_fs_read_len > 0 ? _tl_fs_read_len : (rs ? strlen(rs) : 0);
/* Same pairing rule as the v1 worker: the fs_read length is only
* trustworthy for the exact buffer fs_read returned. */
size_t rlen;
if (_tl_fs_read_len > 0 && rs && rs == _tl_fs_read_buf) {
rlen = _tl_fs_read_len; /* raw file bytes — binary-safe */
} else {
rlen = rs ? strlen(rs) : 0;
_tl_fs_read_len = 0; /* hint doesn't describe this body */
_tl_fs_read_buf = NULL;
}
response = malloc(rlen + 1);
if (response && rs) { memcpy(response, rs, rlen); response[rlen] = '\0'; }
else if (response) { response[0] = '\0'; }
if (_tl_fs_read_len > 0) _tl_fs_read_buf = response; /* hint follows the copy */
el_release(hmap);
} else {
response = el_strdup_persist(
@@ -1849,7 +1910,7 @@ static void* http_worker_v2(void* arg) {
free(response);
}
free(method); free(path); free(body); free(hdr_block);
close(fd);
el_closesocket(fd);
pthread_mutex_lock(&_http_conn_mu);
_http_conn_active--;
pthread_cond_signal(&_http_conn_cv);
@@ -1879,14 +1940,18 @@ void http_serve_v2(el_val_t port, el_val_t handler) {
addr.sin6_addr = in6addr_any;
addr.sin6_port = htons((uint16_t)p);
if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
perror("bind"); close(sock); return;
perror("bind"); el_closesocket(sock); return;
}
if (listen(sock, 64) < 0) { perror("listen"); close(sock); return; }
if (listen(sock, 64) < 0) { perror("listen"); el_closesocket(sock); return; }
fprintf(stderr, "[http v2] listening on [::]:%d (dual-stack)\n", p);
while (1) {
struct sockaddr_in6 cli;
socklen_t clen = sizeof(cli);
#ifdef _WIN32
SOCKET cfd = accept(sock, (struct sockaddr*)&cli, &clen);
#else
int cfd = accept(sock, (struct sockaddr*)&cli, &clen);
#endif
if (cfd < 0) {
if (errno == EINTR) continue;
perror("accept"); break;
@@ -1898,11 +1963,11 @@ void http_serve_v2(el_val_t port, el_val_t handler) {
_http_conn_active++;
pthread_mutex_unlock(&_http_conn_mu);
HttpWorkerArg* arg = malloc(sizeof(HttpWorkerArg));
if (!arg) { close(cfd); continue; }
if (!arg) { el_closesocket(cfd); continue; }
arg->fd = cfd;
pthread_t tid;
if (pthread_create(&tid, NULL, http_worker_v2, arg) != 0) {
close(cfd); free(arg);
el_closesocket(cfd); free(arg);
pthread_mutex_lock(&_http_conn_mu);
_http_conn_active--;
pthread_cond_signal(&_http_conn_cv);
@@ -1911,7 +1976,7 @@ void http_serve_v2(el_val_t port, el_val_t handler) {
}
pthread_detach(tid);
}
close(sock);
el_closesocket(sock);
}
/* Build the response envelope a 4-arg handler can return. We hand-write
@@ -1951,6 +2016,7 @@ el_val_t http_response(el_val_t status, el_val_t headers_json, el_val_t body) {
el_val_t fs_read(el_val_t pathv) {
const char* path = EL_CSTR(pathv);
_tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!path) return el_wrap_str(el_strdup(""));
FILE* f = fopen(path, "rb");
if (!f) return el_wrap_str(el_strdup(""));
@@ -1962,6 +2028,7 @@ el_val_t fs_read(el_val_t pathv) {
size_t got = fread(buf, 1, (size_t)sz, f);
buf[got] = '\0';
_tl_fs_read_len = got; /* store real byte count for binary-safe send */
_tl_fs_read_buf = buf; /* ...valid ONLY for this exact buffer */
fclose(f);
return el_wrap_str(buf);
}
@@ -2056,6 +2123,23 @@ el_val_t exec(el_val_t cmdv) {
el_val_t exec_bg(el_val_t cmdv) {
const char* cmd = EL_CSTR(cmdv);
if (!cmd || !*cmd) return el_wrap_str(el_strdup(""));
#ifdef _WIN32
/* Windows: no fork/exec. Launch a detached `cmd /c <command>` with no console window via
CreateProcess (DETACHED_PROCESS | CREATE_NO_WINDOW). Returns the PID as a string, "" on fail.
Mirrors the POSIX branch: child runs independently, caller is not blocked. */
char cmdline[8192];
snprintf(cmdline, sizeof(cmdline), "cmd.exe /c %s", cmd);
STARTUPINFOA si; ZeroMemory(&si, sizeof(si)); si.cb = sizeof(si);
PROCESS_INFORMATION pi; ZeroMemory(&pi, sizeof(pi));
BOOL ok = CreateProcessA(NULL, cmdline, NULL, NULL, FALSE,
DETACHED_PROCESS | CREATE_NO_WINDOW, NULL, NULL, &si, &pi);
if (!ok) return el_wrap_str(el_strdup(""));
char pidbuf[32];
snprintf(pidbuf, sizeof(pidbuf), "%lu", (unsigned long)pi.dwProcessId);
CloseHandle(pi.hProcess);
CloseHandle(pi.hThread);
return el_wrap_str(el_strdup(pidbuf));
#else
pid_t pid = fork();
if (pid < 0) {
/* fork failed */
@@ -2078,6 +2162,7 @@ el_val_t exec_bg(el_val_t cmdv) {
char pidbuf[32];
snprintf(pidbuf, sizeof(pidbuf), "%d", (int)pid);
return el_wrap_str(el_strdup(pidbuf));
#endif
}
el_val_t fs_list(el_val_t pathv) {
@@ -3417,8 +3502,10 @@ el_val_t json_get_raw(el_val_t json_str, el_val_t key) {
const char* k = EL_CSTR(key);
const char* p = json_find_key(json, k);
/* Clear fs_read binary-length hint — result is a fresh null-terminated
* string, not the raw file bytes, so Content-Length must use strlen. */
* string, not the raw file bytes, so Content-Length must use strlen.
* (Kept although the pointer pairing now makes this redundant.) */
_tl_fs_read_len = 0;
_tl_fs_read_buf = NULL;
if (!p) return el_wrap_str(el_strdup(""));
const char* end = json_skip_value(p);
size_t n = (size_t)(end - p);
@@ -4290,7 +4377,12 @@ static int _el_decompose_earth(el_caltime_t* ct, struct tm* tm_out, int* abbr_le
localtime_r(&s, &tm);
*tm_out = tm;
if (abbr_buf && abbr_cap > 0) {
/* mingw's struct tm has no tm_zone (BSD/glibc extension); no abbrev available there. */
#ifdef _WIN32
const char* z_str = "";
#else
const char* z_str = tm.tm_zone ? tm.tm_zone : "";
#endif
size_t n = strlen(z_str);
if (n >= abbr_cap) n = abbr_cap - 1;
memcpy(abbr_buf, z_str, n);
@@ -7267,6 +7359,256 @@ static char* engram_first_n_chars(const char* s, size_t n) {
return out;
}
/* ══════════════════════════════════════════════════════════════════════════
* M3 ENGRAM_STORE glue (CALLER side of the libengram ABI; design §10).
*
* The engine (engram_store.{c,h}) has ZERO soul dependencies and never sees an
* EngramNode/EngramEdge or a soul global. ALL mapping between the live runtime
* structs and the engine's StoreNode/StoreEdge views lives HERE, on the caller
* side of the C ABI. That is what keeps a standalone `engramd` a later additive
* choice rather than a fork.
*
* Behind the ENGRAM_STORE env flag (default OFF):
* OFF (unset / "0" / "off") every hook below early-returns; the paged store
* is never opened or written and no store code is reached. The runtime keeps
* EXACTLY today's JSON-snapshot behavior, byte-for-byte.
* ON ("1" / "on" / "true") engram_store_boot() imports snapshot.json ONCE
* into neuron.egm (or replays neuron.wal), loads the WHOLE store resident in
* RAM (Phase 1: no demand paging that is M4), and every structural
* mutation (node/edge create, forget) is mirrored through the store's
* WAL-logged API so neuron.egm/neuron.wal stay authoritative.
* */
#include "engram_store.h"
static EngramPagedStore* g_engram_store = NULL;
int engram_store_enabled(void) {
const char* f = getenv("ENGRAM_STORE");
return (f && (strcmp(f, "1") == 0 || strcmp(f, "on") == 0 ||
strcmp(f, "true") == 0)) ? 1 : 0;
}
/* EngramNode → borrowed StoreNode view (no ownership transfer; the store copies
* every field it persists, so shared string pointers are safe). */
static void eg_node_to_store(const EngramNode* n, StoreNode* sn) {
memset(sn, 0, sizeof *sn);
sn->id = n->id; sn->content = n->content; sn->node_type = n->node_type;
sn->label = n->label; sn->tier = n->tier; sn->tags = n->tags;
sn->metadata = n->metadata;
sn->salience = n->salience; sn->importance = n->importance;
sn->confidence = n->confidence; sn->temporal_decay_rate = n->temporal_decay_rate;
sn->activation_count = n->activation_count; sn->last_activated = n->last_activated;
sn->created_at = n->created_at; sn->updated_at = n->updated_at;
sn->background_activation = n->background_activation;
sn->working_memory_weight = n->working_memory_weight;
sn->suppression_count = n->suppression_count; sn->layer_id = n->layer_id;
for (int i = 0; i < STORE_BLL_K && i < ENGRAM_BLL_K; i++)
sn->access_ts[i] = n->access_ts[i];
sn->access_head = n->access_head; sn->access_filled = n->access_filled;
sn->wm_anchor = n->wm_anchor; sn->emb = n->emb; sn->emb_dim = n->emb_dim;
}
static void eg_edge_to_store(const EngramEdge* e, StoreEdge* se) {
memset(se, 0, sizeof *se);
se->id = e->id; se->from_id = e->from_id; se->to_id = e->to_id;
se->relation = e->relation; se->metadata = e->metadata;
se->weight = e->weight; se->hebb = e->hebb; se->confidence = e->confidence;
se->created_at = e->created_at; se->updated_at = e->updated_at;
se->last_fired = e->last_fired; se->inhibitory = e->inhibitory;
se->layer_id = e->layer_id;
}
/* Structural-mutation hooks. Callers guard with `if (engram_store_enabled())`;
* these also null-check g_engram_store so a mutation before boot is a safe no-op. */
static void eg_store_put_node(const EngramNode* n) {
if (!g_engram_store || !n || !n->id) return;
StoreNode sn; eg_node_to_store(n, &sn);
store_put_node(g_engram_store, &sn);
}
static void eg_store_put_edge(const EngramEdge* e) {
if (!g_engram_store || !e || !e->id) return;
StoreEdge se; eg_edge_to_store(e, &se);
store_put_edge(g_engram_store, &se);
}
/* Resident-load callbacks: StoreNode/StoreEdge → a fresh EngramNode/EngramEdge
* appended to the in-RAM graph. Mirrors engram_load's field set. The boot-time
* WM laundering (halve + floor + global cap) that engram_load applies is done
* once, after all nodes are loaded, in engram_store_boot see the block there
* so the store-on boot behaves byte-identically to the JSON path (M3.5 parity). */
static void eg_load_node_cb(const StoreNode* sn, void* ctx) {
EngramStore* g = (EngramStore*)ctx;
engram_grow_nodes();
EngramNode* n = &g->nodes[g->node_count];
memset(n, 0, sizeof *n);
n->id = el_strdup_persist(sn->id ? sn->id : "");
n->content = el_strdup_persist(sn->content ? sn->content : "");
n->node_type = el_strdup_persist(sn->node_type && *sn->node_type ? sn->node_type : "Memory");
n->label = el_strdup_persist(sn->label ? sn->label : "");
n->tier = el_strdup_persist(sn->tier && *sn->tier ? sn->tier : "Working");
n->tags = el_strdup_persist(sn->tags ? sn->tags : "");
n->metadata = el_strdup_persist(sn->metadata && *sn->metadata ? sn->metadata : "{}");
n->salience = sn->salience; n->importance = sn->importance;
n->confidence = sn->confidence; n->temporal_decay_rate = sn->temporal_decay_rate;
n->activation_count = sn->activation_count; n->last_activated = sn->last_activated;
n->created_at = sn->created_at; n->updated_at = sn->updated_at;
n->background_activation = sn->background_activation;
n->working_memory_weight = sn->working_memory_weight;
n->suppression_count = sn->suppression_count; n->layer_id = sn->layer_id;
for (int i = 0; i < STORE_BLL_K && i < ENGRAM_BLL_K; i++)
n->access_ts[i] = sn->access_ts[i];
n->access_head = sn->access_head; n->access_filled = sn->access_filled;
n->wm_anchor = sn->wm_anchor;
if (sn->emb && sn->emb_dim > 0) {
n->emb = malloc(sizeof(float) * (size_t)sn->emb_dim);
if (n->emb) { memcpy(n->emb, sn->emb, sizeof(float) * (size_t)sn->emb_dim);
n->emb_dim = sn->emb_dim; }
}
int64_t idx = g->node_count; g->node_count++;
if (n->id && *n->id) engram_idmap_put(g, n->id, idx);
}
static void eg_load_edge_cb(const StoreEdge* se, void* ctx) {
EngramStore* g = (EngramStore*)ctx;
engram_grow_edges();
EngramEdge* e = &g->edges[g->edge_count];
memset(e, 0, sizeof *e);
e->id = el_strdup_persist(se->id ? se->id : "");
e->from_id = el_strdup_persist(se->from_id ? se->from_id : "");
e->to_id = el_strdup_persist(se->to_id ? se->to_id : "");
e->relation = el_strdup_persist(se->relation && *se->relation ? se->relation : "associate");
e->metadata = el_strdup_persist(se->metadata && *se->metadata ? se->metadata : "{}");
e->weight = se->weight; e->hebb = se->hebb; e->confidence = se->confidence;
e->created_at = se->created_at; e->updated_at = se->updated_at;
e->last_fired = se->last_fired; e->inhibitory = se->inhibitory;
e->layer_id = se->layer_id;
g->edge_count++;
}
static void eg_load_layer_cb(EngramStore* g, const StoreLayer* L) {
if (!L->name) return;
for (size_t i = 0; i < g->layer_count; i++) /* upsert by id */
if (g->layers[i].layer_id == L->layer_id) return; /* canonical already seeded */
if (g->layer_count >= g->layer_capacity) {
size_t nc = g->layer_capacity ? g->layer_capacity * 2 : 16;
EngramLayer* nl = realloc(g->layers, nc * sizeof(EngramLayer));
if (!nl) return;
g->layers = nl; g->layer_capacity = nc;
}
g->layers[g->layer_count++] = (EngramLayer){
.layer_id = L->layer_id,
.name = el_strdup_persist(L->name),
.activation_priority = L->activation_priority,
.suppressible = L->suppressible,
.transparent = L->transparent,
.injectable = L->injectable
};
}
/* Clear the resident graph so the store becomes the sole source of truth on boot
* (mirrors engram_load's reset). */
static void eg_reset_resident(EngramStore* g) {
for (int64_t i = 0; i < g->node_count; i++) {
free(g->nodes[i].id); free(g->nodes[i].content); free(g->nodes[i].node_type);
free(g->nodes[i].label); free(g->nodes[i].tier); free(g->nodes[i].tags);
free(g->nodes[i].metadata);
free(g->nodes[i].emb); g->nodes[i].emb = NULL; g->nodes[i].emb_dim = 0;
}
g->node_count = 0;
for (int64_t i = 0; i < g->edge_count; i++) {
free(g->edges[i].id); free(g->edges[i].from_id); free(g->edges[i].to_id);
free(g->edges[i].relation); free(g->edges[i].metadata);
}
g->edge_count = 0;
engram_idmap_free(g);
engram_adj_free(g);
}
/* Defined later with engram_load; declared here for the boot-time WM laundering
* that keeps the store-on boot byte-identical to the JSON path (M3.5 parity). */
static void eg_enforce_wm_cap_on_load(EngramStore* g);
/* engram_store_boot(data_dir) — open (import-once or WAL-replay) the durable
* paged store and load it whole into RAM (Phase 1). No-op / returns 0 when the
* flag is off. Returns 1 on success. Idempotent (a second call is a no-op). */
el_val_t engram_store_boot(el_val_t data_dir) {
if (!engram_store_enabled()) return (el_val_t)0;
if (g_engram_store) return (el_val_t)1;
const char* d = EL_CSTR(data_dir);
if (!d || !*d) return (el_val_t)0;
g_engram_store = engram_open(d);
if (!g_engram_store) return (el_val_t)0;
EngramStore* g = engram_get();
eg_reset_resident(g);
store_scan_nodes(g_engram_store, eg_load_node_cb, g);
store_scan_edges(g_engram_store, eg_load_edge_cb, g);
StoreLayer* ls = NULL; size_t ln = 0;
if (store_list_layers(g_engram_store, &ls, &ln) == 0) {
for (size_t i = 0; i < ln; i++) eg_load_layer_cb(g, &ls[i]);
store_layers_free(ls, ln);
}
/* Boot-time WM laundering — MUST match engram_load exactly (M3.5 parity).
* The JSON load path halves every persisted working_memory_weight on boot
* (stale pinned weights decay out over successive restarts; genuine WM state
* keeps continuity) and floors sub-ENGRAM_WM_FLOOR residue to zero, then
* enforces the global WM cap. The store now persists post-activation WM
* weights, so the store-on boot must apply the identical transform or the two
* persistence paths would diverge on the very first restart. */
for (int64_t i = 0; i < g->node_count; i++) {
g->nodes[i].working_memory_weight *= 0.5;
if (g->nodes[i].working_memory_weight < ENGRAM_WM_FLOOR)
g->nodes[i].working_memory_weight = 0.0;
}
eg_enforce_wm_cap_on_load(g);
g->adj_dirty = 1;
return (el_val_t)1;
}
/* engram_store_checkpoint() — M3.5 PRE-FLIP GATE.
*
* Spreading activation mutates fields IN PLACE on the resident graph edge
* `hebb`/`last_fired` (potentiation + homeostatic scaling), node
* `activation_count`/`last_activated`/`working_memory_weight`/`wm_anchor`
* (reinforcement + WM caps) and also FORMS brand-new `hebbian-associate`
* edges. M3 only mirrored node/edge *creates* and *forgets*; none of those
* in-place mutations or activation-formed edges reached the paged store, so
* learned associations were lost on every restart. This is the fix that makes
* "learned edges survive a restart" true, and it gates the live cutover.
*
* On the soul's save/checkpoint path we push the resident graph's current field
* state through the store's WAL-logged API, then checkpoint:
* - store_put_node(n) for every node persists WM weight, activation_count,
* last_activated, wm_anchor, the base-level access ring, etc.
* - store_put_edge(e) for every edge persists hebb + last_fired AND creates
* any activation-formed edges. (store_hebb_batch is deliberately NOT used: it
* is a delta-only op that skips edge ids not already resident in the store
* see apply_hebb_batch so it cannot persist the newly-formed hebbian edges
* that are the whole point of this milestone. store_put_edge is the idempotent
* upsert that subsumes the hebb delta.)
* then engram_checkpoint flushes dirty pages + advances the checkpoint LSN. Every
* push is a WAL record, so a graceful restart restores them.
*
* Approach: a FULL WALK of the resident graph (not a dirty-set). At Phase-1
* ~64 MB resident this is a cheap linear pass on an already-in-RAM array, run at
* the soul's save cadence (seconds-to-minutes), and it is trivially complete
* no mutation site can be missed and no separate new-edge tracking is needed. An
* inter-checkpoint crash loses only the most-recent unsaved learning, exactly the
* same durability envelope as today's JSON-snapshot cadence.
*
* The storeJSON export path stays engram_save (the JSON is an export artifact). */
el_val_t engram_store_checkpoint(void) {
if (!engram_store_enabled() || !g_engram_store) return (el_val_t)0;
EngramStore* g = engram_get();
for (int64_t i = 0; i < g->node_count; i++) eg_store_put_node(&g->nodes[i]);
for (int64_t i = 0; i < g->edge_count; i++) eg_store_put_edge(&g->edges[i]);
return (el_val_t)(int64_t)(engram_checkpoint(g_engram_store) == 0 ? 1 : 0);
}
/* engram_store_close() — checkpoint + close (used at shutdown / by tests). */
el_val_t engram_store_close(void) {
if (!g_engram_store) return (el_val_t)0;
int r = engram_close(g_engram_store);
g_engram_store = NULL;
return (el_val_t)(int64_t)(r == 0 ? 1 : 0);
}
el_val_t engram_node(el_val_t content, el_val_t node_type, el_val_t salience) {
EngramStore* g = engram_get();
engram_grow_nodes();
@@ -7296,6 +7638,7 @@ el_val_t engram_node(el_val_t content, el_val_t node_type, el_val_t salience) {
g->node_count++;
engram_idmap_put(g, n->id, new_idx);
g->adj_dirty = 1;
if (engram_store_enabled()) eg_store_put_node(n);
return el_wrap_str(el_strdup(n->id));
}
@@ -7425,6 +7768,7 @@ el_val_t engram_node_full(el_val_t content, el_val_t node_type, el_val_t label,
g->node_count++;
engram_idmap_put(g, n->id, new_idx_full);
g->adj_dirty = 1;
if (engram_store_enabled()) eg_store_put_node(n);
return el_wrap_str(el_strdup(n->id));
}
@@ -7495,6 +7839,7 @@ el_val_t engram_node_layered(el_val_t content, el_val_t node_type, el_val_t labe
g->node_count++;
engram_idmap_put(g, n->id, new_idx_layered);
g->adj_dirty = 1;
if (engram_store_enabled()) eg_store_put_node(n);
return el_wrap_str(el_strdup(n->id));
}
@@ -7642,6 +7987,12 @@ void engram_forget(el_val_t node_id) {
EngramStore* g = engram_get();
int64_t idx = engram_find_node_index(sid);
if (idx < 0) return;
/* Mirror the removal into the durable store BEFORE the shift-delete frees
* the incident edges' ids (node tombstone; edge records are reclaimed at
* compaction M5). Node-level FORGET matches the existing WAL semantics. */
if (engram_store_enabled() && g_engram_store) {
store_forget(g_engram_store, sid);
}
/* Free node strings */
EngramNode* n = &g->nodes[idx];
free(n->id); free(n->content); free(n->node_type); free(n->label);
@@ -7779,6 +8130,19 @@ el_val_t engram_prune_telemetry(el_val_t older_than_ms) {
g->edge_count = ew;
free(set);
}
/* Mirror the telemetry prune into the durable paged store so its live
* count tracks the resident graph and stale ISE telemetry stays bounded
* in the store too. Without this, the resident graph GCs old ISE from RAM
* (node_count drops) while the store retains them (count diverges + the
* store re-accumulates the very telemetry bloat this prune was written to
* stop). Matches engram_forget's store_forget mirror. Done before the ids
* are freed below. */
if (engram_store_enabled() && g_engram_store) {
for (int64_t i = 0; i < removed; i++) {
store_forget(g_engram_store, removed_ids[i]);
}
}
for (int64_t i = 0; i < removed; i++) free(removed_ids[i]);
free(removed_ids);
@@ -7980,6 +8344,7 @@ void engram_connect(el_val_t from_id, el_val_t to_id, el_val_t weight, el_val_t
e->layer_id = ENGRAM_LAYER_DEFAULT;
g->edge_count++;
g->adj_dirty = 1;
if (engram_store_enabled()) eg_store_put_edge(e);
}
el_val_t engram_edge_between(el_val_t from_id, el_val_t to_id) {
+7
View File
@@ -605,6 +605,13 @@ el_val_t engram_edge_count(void);
el_val_t engram_activate(el_val_t query, el_val_t depth);
el_val_t engram_save(el_val_t path);
el_val_t engram_load(el_val_t path);
/* Tiered paged-store entry points (ENGRAM_STORE=1). engram_store_boot opens the
* durable store (import-once / WAL-replay) and loads it resident; checkpoint pushes
* the resident graph's current field state (incl. learned hebb + activation-formed
* edges) through the WAL and flushes; close checkpoints + closes. No-ops when off. */
el_val_t engram_store_boot(el_val_t data_dir);
el_val_t engram_store_checkpoint(void);
el_val_t engram_store_close(void);
/* JSON-string accessors — return pre-serialized JSON so HTTP handlers
* can pass results straight through without round-tripping ElList/ElMap
+1
View File
@@ -1072,6 +1072,7 @@ el_val_t __engram_save(el_val_t path) { return engram_save
el_val_t __engram_load(el_val_t path) { return engram_load(path); }
el_val_t __engram_get_node_json(el_val_t id) { return engram_get_node_json(id); }
el_val_t __engram_get_node_by_label(el_val_t label) { return engram_get_node_by_label(label); }
el_val_t __engram_search_json(el_val_t query, el_val_t limit) {
return engram_search_json(query, limit);
+1
View File
@@ -226,6 +226,7 @@ el_val_t __engram_activate(el_val_t query, el_val_t depth);
el_val_t __engram_save(el_val_t path);
el_val_t __engram_load(el_val_t path);
el_val_t __engram_get_node_json(el_val_t id);
el_val_t __engram_get_node_by_label(el_val_t label);
el_val_t __engram_search_json(el_val_t query, el_val_t limit);
el_val_t __engram_scan_nodes_json(el_val_t limit, el_val_t offset);
el_val_t __engram_scan_nodes_by_type_json(el_val_t node_type, el_val_t limit, el_val_t offset);
File diff suppressed because it is too large Load Diff
+221
View File
@@ -0,0 +1,221 @@
/* engram_store.h — M1 of the engram tiered storage engine.
*
* The FINAL on-disk paged store format: superblock (+ mirror), slotted pages,
* self-describing TLV records, overflow chains, and two B+-tree indexes
* (primary id->loc, adjacency from_id/to_id->edge-locs) over a free-listed
* page file. See docs/architecture/design/engram-tiered-storage-engine.md §2.
*
* This is a self-contained module (plain C, standard libs only). It defines its
* own serializable views of a node/edge (StoreNode/StoreEdge) that mirror every
* persisted field of EngramNode/EngramEdge in el_runtime.c. M3 maps between the
* live runtime structs and these; M1 does not touch el_runtime.c.
*
* Format id: magic "ENGST01", format_version 1. This format is PERMANENT the
* TLV record scheme means new fields never force a migration.
*/
#ifndef ENGRAM_STORE_H
#define ENGRAM_STORE_H
#include <stddef.h>
#include <stdint.h>
/* Fixed for the life of a store; recorded in the superblock. */
#define STORE_PAGE_SIZE 16384u
#define STORE_MAGIC "ENGST01" /* 7 chars + NUL stored in an 8-byte field */
#define STORE_FORMAT_VERSION 1u
/* Ring-buffer length for ACT-R base-level access timestamps.
* MUST equal ENGRAM_BLL_K in el_runtime.c (currently 10). Static-checked in .c. */
#define STORE_BLL_K 10
/* Page types (page header byte). */
enum {
STORE_PT_NODE = 1,
STORE_PT_EDGE = 2,
STORE_PT_INDEX = 3,
STORE_PT_OVERFLOW = 4,
STORE_PT_FREE = 5
};
/* store_check flags. */
#define STORE_CHECK_CRC 1u
/* ── Serializable node view: every persisted EngramNode field ─────────────── */
typedef struct StoreNode {
char* id;
char* content;
char* node_type;
char* label;
char* tier;
char* tags;
char* metadata;
double salience;
double importance;
double confidence;
double temporal_decay_rate;
int64_t activation_count;
int64_t last_activated;
int64_t created_at;
int64_t updated_at;
double background_activation;
double working_memory_weight;
int32_t suppression_count;
uint32_t layer_id;
int64_t access_ts[STORE_BLL_K];
int32_t access_head;
int32_t access_filled;
double wm_anchor;
float* emb; /* owned; NULL if not embedded */
int32_t emb_dim;
/* Forward-compat: raw bytes of any TLV fields the reader did not recognise,
* concatenated verbatim ([tag][u32 len][bytes]...). Re-emitted on write so
* an old reader never drops a newer writer's fields. */
uint8_t* unknown;
size_t unknown_len;
int tombstoned; /* set by store_get_* if the located record is dead */
/* hebb_elig / hebb_elig_ts are DELIBERATELY NOT persisted (see EngramNode). */
} StoreNode;
/* ── Serializable edge view: every persisted EngramEdge field ─────────────── */
typedef struct StoreEdge {
char* id;
char* from_id;
char* to_id;
char* relation;
char* metadata;
double weight;
double hebb;
double confidence;
int64_t created_at;
int64_t updated_at;
int64_t last_fired;
int32_t inhibitory;
uint32_t layer_id;
uint8_t* unknown;
size_t unknown_len;
int tombstoned;
} StoreEdge;
typedef struct EngramPagedStore EngramPagedStore;
/* Lifecycle. */
EngramPagedStore* store_create(const char* path); /* fails if file exists */
EngramPagedStore* store_open(const char* path); /* recovers via mirror SB */
int store_close(EngramPagedStore* s); /* syncs + frees */
int store_sync(EngramPagedStore* s); /* fsync + rewrite both superblocks */
/* Nodes. store_get_node returns 1 on hit (fills *out, caller store_node_free),
* 0 if absent or tombstoned, <0 on error. */
int store_put_node(EngramPagedStore* s, const StoreNode* n);
int store_get_node(EngramPagedStore* s, const char* id, StoreNode* out);
int store_tombstone(EngramPagedStore* s, const char* id);
/* Edges. *out is malloc'd (store_edges_free); *n set to count. */
int store_put_edge(EngramPagedStore* s, const StoreEdge* e);
int store_get_edges_from(EngramPagedStore* s, const char* from_id, StoreEdge** out, size_t* n);
int store_get_edges_to(EngramPagedStore* s, const char* to_id, StoreEdge** out, size_t* n);
/* Integrity: verify every page's crc (and both superblocks). Returns the number
* of corrupt pages (0 = clean), or <0 on I/O error. */
int store_check(EngramPagedStore* s, unsigned flags);
/* Ownership helpers. */
void store_node_free(StoreNode* n);
void store_edge_free(StoreEdge* e);
void store_edges_free(StoreEdge* arr, size_t n);
/* Test-only hook (NOT a format property — B+-tree nodes are self-describing via
* their stored key count). Caps entries/keys per index node to force splits on
* small datasets. 0 = natural full-page fanout. */
void store__set_btree_order(EngramPagedStore* s, int leaf_max, int internal_max);
/* Introspection for tests/tools. */
uint64_t store_page_count(const EngramPagedStore* s);
/* ── M2: WAL + checkpoint + crash recovery + one-time legacy import ─────────────
*
* The durable engram is `neuron.egm` (paged) fronted by `neuron.wal`
* (append-only). A mutation is durable once its WAL record is fsync'd
* (group-commit). Pages are held write-back in RAM (no-steal) and flushed to the
* store only at a checkpoint, so the store file on disk always reflects a
* consistent point (`last_checkpoint_lsn`) and the WAL owns everything since.
* Recovery = open store, replay WAL forward, redo a record only where the target
* record's home page LSN < record LSN (idempotent). JSON is ONLY an import
* source / export artifact never the ongoing store. */
typedef enum { ENGRAM_WAL_ALWAYS = 0, ENGRAM_WAL_GROUP = 1, ENGRAM_WAL_OFF = 2 } EngramWalSync;
/* Serializable layer-registry view (the `layers` array of the legacy snapshot). */
typedef struct StoreLayer {
uint32_t layer_id;
char* name;
uint32_t activation_priority;
int32_t suppressible;
int32_t transparent;
int32_t injectable;
uint8_t* unknown;
size_t unknown_len;
int tombstoned;
} StoreLayer;
/* Boot the durable engram in `data_dir` (holds neuron.egm + neuron.wal). If the
* store is absent but a legacy snapshot.json exists, it is imported ONCE into a
* fresh store; thereafter the store is authoritative and JSON is never read again.
* On open, the WAL is replayed to recover any post-checkpoint mutations. */
EngramPagedStore* engram_open(const char* data_dir);
int engram_close(EngramPagedStore* s); /* checkpoint + close */
/* Force a checkpoint: flush dirty pages → fsync store → advance checkpoint LSN →
* reclaim the WAL prefix. Also threshold-triggered automatically on the write path. */
int engram_checkpoint(EngramPagedStore* s);
/* WAL commit policy. engram_open honours env ENGRAM_WAL_SYNC=always|group|off. */
void engram_set_wal_sync(EngramPagedStore* s, EngramWalSync policy);
/* Layer registry. */
int store_put_layer(EngramPagedStore* s, const StoreLayer* L);
int store_get_layer(EngramPagedStore* s, uint32_t layer_id, StoreLayer* out);
int store_del_layer(EngramPagedStore* s, uint32_t layer_id);
int store_list_layers(EngramPagedStore* s, StoreLayer** out, size_t* n);
void store_layer_free(StoreLayer* L);
void store_layers_free(StoreLayer* arr, size_t n);
/* Edge lookup by id (for hebb updates + idempotency). 1 hit / 0 absent / <0 err. */
int store_get_edge(EngramPagedStore* s, const char* id, StoreEdge* out);
/* HEBB batch: one WAL record updating hebb (+ last_fired) on a set of edges. */
typedef struct StoreHebbDelta { const char* edge_id; double hebb; int64_t last_fired; } StoreHebbDelta;
int store_hebb_batch(EngramPagedStore* s, const StoreHebbDelta* d, size_t n);
/* Supersede: logs the (old,new) pair and tombstones old_id at the store; the new
* node + `supersedes` edge are logged separately (neuron-layer immutability). */
int store_supersede(EngramPagedStore* s, const char* old_id, const char* new_id);
/* Forget (GC): tombstone id at the store (hard-free deferred to compaction). */
int store_forget(EngramPagedStore* s, const char* id);
/* ── M3: full live enumeration (for the CALLER's resident load + JSON export) ──
* Walk the whole store and invoke `cb` once per DISTINCT live node/edge with a
* borrowed view (the engine frees it after cb returns the callback must copy
* anything it keeps). De-duplicated by id (canonical latest-live per id, matching
* point-read semantics). Returns the count emitted, or <0 on error. The engine
* hands out StoreNode/StoreEdge only it never sees a soul struct (design §10). */
typedef void (*StoreNodeScanCb)(const StoreNode* n, void* ctx);
typedef void (*StoreEdgeScanCb)(const StoreEdge* e, void* ctx);
int store_scan_nodes(EngramPagedStore* s, StoreNodeScanCb cb, void* ctx);
int store_scan_edges(EngramPagedStore* s, StoreEdgeScanCb cb, void* ctx);
/* Introspection / test hooks. */
uint64_t engram_wal_next_lsn(const EngramPagedStore* s);
uint64_t engram_last_checkpoint_lsn(const EngramPagedStore* s);
/* Crash-test hooks (writes only under a throwaway dir).
* store__crash abandon all RAM state without flush/fsync (power loss).
* store__flush_pages pwrite dirty pages to disk WITHOUT a checkpoint (steal).
* store__checkpoint_crashat run checkpoint but stop (then power-loss) after
* `phase` (0..4); phase<0 = full checkpoint. */
void store__crash(EngramPagedStore* s);
int store__flush_pages(EngramPagedStore* s);
int store__checkpoint_crashat(EngramPagedStore* s, int phase);
#endif /* ENGRAM_STORE_H */