From 8c2406ff6b0eefd90efd75a6879ce471cd09f1a9 Mon Sep 17 00:00:00 2001 From: bigmerge Date: Sun, 16 Aug 2026 16:44:26 -0500 Subject: [PATCH] =?UTF-8?q?runtime:=20the=20link=20set=20is=20multi-file?= =?UTF-8?q?=20=E2=80=94=20name=20it=20once,=20ship=20all=20of=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit el_runtime.c was created 2026-05-03 as an explicitly temporary build shim. It was deleted that afternoon ("runtime is 100% native El") and restored 25 minutes later "UNTIL the compiler is updated to emit #include el_seed.h". The `until` never came. 3.5 months on it is 20,527 lines, and nothing was ever set up to notice — a file scheduled for deletion gets no owner, no budget, no boundary. What kept it growing is not inertia, it is an instruction. lang/AGENTS.md said el_runtime.c "is the authoritative single-file link target ... THIS IS WHERE A NEW C BUILTIN'S IMPLEMENTATION MUST CURRENTLY LIVE TO BE LINKABLE", and made it step 1 of the add-a-builtin recipe. That is false. Placement is a link-time concern: builtin_arity maps NAME -> ARITY INT only, the El name is emitted as the exact C symbol, and `ld` resolves it — the compiler cannot tell which .c a symbol came from. `nm lang/dist/platform/elc` on the shipped compiler already shows T _engram_geo_reify_index_new, T _vindex_insert, T _engram_think, T _engram_reason_abduce: it is linked from ten translation units today. In a repo where agents write most of the code, a false instruction in the instruction file is the forcing function. The file grew because the recipe said to grow it. The multi-file runtime is therefore already real, and the docs and the distribution never caught up — which left a live, shipped bug: * Linking el_runtime.c alone FAILS at `ld` (undefined engram_ground_json, engram_activate_inner, eg_find_relation, cog_assert_two_axis, ...) because el_runtime.c #includes six engram headers and calls into all six siblings. * sdk-release.yaml shipped el_runtime.c/.h + engram_store.c/.h and none of the other five required .c files, so downstream consumers of the el-runtime-c Artifact Registry package and of install.sh got a lib/ that cannot link. * .githooks/pre-commit linked el_runtime.c alone with stderr to /dev/null, so it reported all 13 native suites as FAILED with the real ld error invisible. * AGENTS.md's self-host recipe compiled el-compiler/runtime/el_runtime.c — a path the same file's "DO NOT EDIT" list names as a lagging fork. The root fix is to stop writing the list down eight times: * lang/runtime/SOURCES — the canonical link set, in one place, in link order. * scripts/el-runtime-sources.sh — prints it, optionally prefixed; --check fails loudly on a missing file, --headers for the shipped headers. * Every link line in AGENTS.md, lang/AGENTS.md, DESIGN.md, lang/spec/language.md, the three workflows and the pre-commit hook now reads that one list. * Adding a concern's .c is one line in SOURCES, so a new builtin no longer has to be appended to el_runtime.c just because appending was the cheaper edit. Distribution: ship the siblings rather than amalgamate. Amalgamation needs a new tool and contradicts DESIGN.md's compile-once-link-many; the siblings are already independently authored and independently tested (engram/test/*.sh link subsets directly), and engram_store.c was already shipped, so this completes a mechanism that existed rather than inventing one. Source is also a superset: a consumer that wants one file can concatenate, one that wants separate TUs cannot undo an amalgamation. el-runtime-c/-h stay for backward compatibility; el-runtime-src is added carrying the complete set plus SOURCES. lang/AGENTS.md now points new C builtins at the concern-owning .c and states plainly that the compiler cannot tell which .c a symbol came from, with the nm evidence. AGENTS.md's "reconcile which is canonical (verify)" note is resolved: neither file supersedes the other, the canonical unit is the set. Verified locally (the bar; not CI): * engram/src/server.el compiles and links against the SOURCES set. * Compile-once-link-many into libel.a links the same program. * elb builds from the corrected recipe. * Self-host fixpoint byte-identical (11,110 lines, stage2 == stage3) built with the SOURCES-driven link line. * pre-commit hook: 0 of 13 native suites passing -> 8 of 13. The 5 still-failing suites are PRE-EXISTING and untouched here: test_fs (fs_list_json undeclared), test_state (state_has, state_get_or undeclared), test_json (json_build_array/json_build_object/json_escape_string undefined), test_time (now_ns undefined), test_env (1 assertion). Builtins registered in builtin_arity with no implementation or no declaration anywhere — the same recipe defect, now visible because the linker error is no longer suppressed. Not attempted: making elc emit #include el_seed.h and dropping elb's hardcoded runtime path. That is the correct long-term fix and finishes the 2026-05-03 migration, but it touches codegen and self-hosting and belongs in its own change. --- .gitea/workflows/ci-dev.yaml | 43 +++++++++-------- .gitea/workflows/ci-stage.yaml | 27 +++++------ .gitea/workflows/sdk-release.yaml | 76 ++++++++++++++++++++++--------- .githooks/pre-commit | 36 +++++++++++++-- AGENTS.md | 32 ++++++++++--- DESIGN.md | 10 ++-- lang/AGENTS.md | 26 +++++++++-- lang/install.sh | 48 +++++++++++++++---- lang/runtime/SOURCES | 52 +++++++++++++++++++++ lang/spec/language.md | 12 ++++- scripts/el-runtime-sources.sh | 73 +++++++++++++++++++++++++++++ 11 files changed, 355 insertions(+), 80 deletions(-) create mode 100644 lang/runtime/SOURCES create mode 100755 scripts/el-runtime-sources.sh diff --git a/.gitea/workflows/ci-dev.yaml b/.gitea/workflows/ci-dev.yaml index d4194b8..a0af2c5 100644 --- a/.gitea/workflows/ci-dev.yaml +++ b/.gitea/workflows/ci-dev.yaml @@ -41,7 +41,7 @@ jobs: gcc -O2 \ -I runtime \ dist/elc-gen2.c \ - runtime/el_runtime.c \ + $(../scripts/el-runtime-sources.sh runtime) \ -lcurl -lssl -lcrypto -lpthread -lm \ -o dist/platform/elc chmod +x dist/platform/elc @@ -56,7 +56,7 @@ jobs: gcc -O2 \ -I runtime \ dist/elb.c \ - runtime/el_runtime.c \ + $(../scripts/el-runtime-sources.sh runtime) \ -lcurl -lssl -lcrypto -lpthread -lm \ -o dist/bin/elb chmod +x dist/bin/elb @@ -87,14 +87,20 @@ jobs: bash tests/html_sanitizer/run.sh # Native El test suites (elc --test, compile-link-run) - # el_runtime.c is precompiled to .o once and reused by all 8 modules. - - name: Precompile el_runtime.o + # The runtime is MULTI-FILE (see lang/runtime/SOURCES). Every .c is compiled + # once into /tmp/libel.a and reused by all 8 test modules — compile-once, + # link-many, as prescribed in DESIGN.md. Linking el_runtime.c alone fails + # at `ld`: it calls into all six engram sibling TUs. + - name: Precompile runtime into libel.a run: | set -euo pipefail RUNTIME="$(pwd)/runtime" - gcc -O2 -c -I "$RUNTIME" "$RUNTIME/el_runtime.c" \ - -o /tmp/el_runtime.o - echo "el_runtime.o compiled" + rm -rf /tmp/elrt && mkdir -p /tmp/elrt + for src in $(../scripts/el-runtime-sources.sh --check "$RUNTIME"); do + gcc -O2 -c -I "$RUNTIME" "$src" -o "/tmp/elrt/$(basename "${src%.c}").o" + done + ar rcs /tmp/libel.a /tmp/elrt/*.o + echo "libel.a built from $(ls /tmp/elrt/*.o | wc -l) translation units" - name: Run tests - native (core) run: | @@ -102,7 +108,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c /tmp/el_runtime.o \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c /tmp/libel.a \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core /tmp/el_native_core @@ -112,7 +118,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c /tmp/el_runtime.o \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c /tmp/libel.a \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text /tmp/el_native_text @@ -122,7 +128,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c /tmp/el_runtime.o \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c /tmp/libel.a \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string /tmp/el_native_string @@ -132,7 +138,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c /tmp/el_runtime.o \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c /tmp/libel.a \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math /tmp/el_native_math @@ -142,7 +148,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c /tmp/el_runtime.o \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c /tmp/libel.a \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state /tmp/el_native_state @@ -152,7 +158,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c /tmp/el_runtime.o \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c /tmp/libel.a \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time /tmp/el_native_time @@ -162,7 +168,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c /tmp/el_runtime.o \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c /tmp/libel.a \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json /tmp/el_native_json @@ -172,7 +178,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c /tmp/el_runtime.o \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c /tmp/libel.a \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env /tmp/el_native_env @@ -182,7 +188,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c /tmp/el_runtime.o \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c /tmp/libel.a \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs /tmp/el_native_fs @@ -306,8 +312,9 @@ jobs: FROM ${BASE} COPY dist/platform/elc /opt/el/dist/platform/elc COPY dist/bin/elb /opt/el/dist/bin/elb - COPY runtime/el_runtime.c /opt/el/runtime/el_runtime.c - COPY runtime/el_runtime.h /opt/el/runtime/el_runtime.h + # Whole runtime link set — el_runtime.c alone does not link (it calls + # into the six engram sibling TUs). See lang/runtime/SOURCES. + COPY runtime/ /opt/el/runtime/ COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb EOF diff --git a/.gitea/workflows/ci-stage.yaml b/.gitea/workflows/ci-stage.yaml index 3280050..ad82268 100644 --- a/.gitea/workflows/ci-stage.yaml +++ b/.gitea/workflows/ci-stage.yaml @@ -48,7 +48,7 @@ jobs: gcc -O2 \ -I runtime \ dist/elc-gen2.c \ - runtime/el_runtime.c \ + $(../scripts/el-runtime-sources.sh runtime) \ -lcurl -lssl -lcrypto -lpthread -lm \ -o dist/platform/elc chmod +x dist/platform/elc @@ -86,7 +86,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core /tmp/el_native_core @@ -96,7 +96,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text /tmp/el_native_text @@ -106,7 +106,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string /tmp/el_native_string @@ -116,7 +116,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math /tmp/el_native_math @@ -126,7 +126,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state /tmp/el_native_state @@ -136,7 +136,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time /tmp/el_native_time @@ -146,7 +146,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json /tmp/el_native_json @@ -156,7 +156,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env /tmp/el_native_env @@ -166,7 +166,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs /tmp/el_native_fs @@ -178,7 +178,7 @@ jobs: gcc -O2 \ -I runtime \ dist/elb.c \ - runtime/el_runtime.c \ + $(../scripts/el-runtime-sources.sh runtime) \ -lcurl -lssl -lcrypto -lpthread -lm \ -o dist/bin/elb chmod +x dist/bin/elb @@ -290,8 +290,9 @@ jobs: FROM ${BASE} COPY dist/platform/elc /opt/el/dist/platform/elc COPY dist/bin/elb /opt/el/dist/bin/elb - COPY runtime/el_runtime.c /opt/el/runtime/el_runtime.c - COPY runtime/el_runtime.h /opt/el/runtime/el_runtime.h + # Whole runtime link set — el_runtime.c alone does not link (it calls + # into the six engram sibling TUs). See lang/runtime/SOURCES. + COPY runtime/ /opt/el/runtime/ COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb EOF diff --git a/.gitea/workflows/sdk-release.yaml b/.gitea/workflows/sdk-release.yaml index d8eb0c7..f79347e 100644 --- a/.gitea/workflows/sdk-release.yaml +++ b/.gitea/workflows/sdk-release.yaml @@ -49,7 +49,7 @@ jobs: gcc -O2 \ -I runtime \ dist/elc-gen2.c \ - runtime/el_runtime.c \ + $(../scripts/el-runtime-sources.sh runtime) \ -lcurl -lssl -lcrypto -lpthread -lm \ -o dist/platform/elc chmod +x dist/platform/elc @@ -64,7 +64,7 @@ jobs: gcc -O2 \ -I runtime \ dist/elb.c \ - runtime/el_runtime.c \ + $(../scripts/el-runtime-sources.sh runtime) \ -lcurl -lssl -lcrypto -lpthread -lm \ -o dist/bin/elb chmod +x dist/bin/elb @@ -123,7 +123,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core /tmp/el_native_core @@ -133,7 +133,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text /tmp/el_native_text @@ -143,7 +143,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string /tmp/el_native_string @@ -153,7 +153,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math /tmp/el_native_math @@ -163,7 +163,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state /tmp/el_native_state @@ -173,7 +173,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time /tmp/el_native_time @@ -183,7 +183,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json /tmp/el_native_json @@ -193,7 +193,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env /tmp/el_native_env @@ -203,7 +203,7 @@ jobs: ELC="$(pwd)/dist/platform/elc" RUNTIME="$(pwd)/runtime" "$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c - gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c "$RUNTIME/el_runtime.c" \ + gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c $(../scripts/el-runtime-sources.sh "$RUNTIME") \ -lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs /tmp/el_native_fs @@ -216,10 +216,17 @@ jobs: cp lang/dist/platform/elc dist/sdk/bin/elc cp lang/dist/bin/elb dist/sdk/bin/elb cp lang/dist/bin/epm dist/sdk/bin/epm - cp lang/runtime/el_runtime.c dist/sdk/runtime/ - cp lang/runtime/el_runtime.h dist/sdk/runtime/ - cp lang/runtime/engram_store.c dist/sdk/runtime/ - cp lang/runtime/engram_store.h dist/sdk/runtime/ + # Ship the WHOLE runtime link set, not el_runtime.c alone. el_runtime.c + # #includes six engram headers and calls into all six sibling .c files, + # so an SDK carrying only el_runtime.c{,.h} + engram_store.c{,.h} cannot + # link — downstream `ld` fails on engram_ground_json, eg_find_relation, + # cog_assert_two_axis and friends. lang/runtime/SOURCES is the source of + # truth; --check makes a missing file fail the release loudly. + for f in $(scripts/el-runtime-sources.sh --check) \ + $(scripts/el-runtime-sources.sh --headers --check); do + cp "lang/runtime/${f}" dist/sdk/runtime/ + done + cp lang/runtime/SOURCES dist/sdk/runtime/ cp lang/runtime/*.el dist/sdk/runtime/ tar -czf dist/el-sdk-latest.tar.gz -C dist/sdk . echo "SDK tarball bundled: dist/el-sdk-latest.tar.gz" @@ -274,12 +281,16 @@ jobs: "${GITEA_API}/repos/${REPO}/releases/${RELEASE_ID}/assets" } - # Per-file assets (downstream CI needs these individually) + # Per-file assets (downstream CI needs these individually). + # lang/install.sh downloads every one of these by name — the list is + # lang/runtime/SOURCES. Shipping el_runtime.c alone produced a lib/ + # that could not link; that is the bug this loop closes. upload_asset lang/dist/platform/elc elc - upload_asset lang/runtime/el_runtime.c el_runtime.c - upload_asset lang/runtime/el_runtime.h el_runtime.h - upload_asset lang/runtime/engram_store.c engram_store.c - upload_asset lang/runtime/engram_store.h engram_store.h + for f in $(scripts/el-runtime-sources.sh --check) \ + $(scripts/el-runtime-sources.sh --headers --check); do + upload_asset "lang/runtime/${f}" "${f}" + done + upload_asset lang/runtime/SOURCES SOURCES # SDK bundle and installer binary upload_asset dist/el-sdk-latest.tar.gz el-sdk-latest.tar.gz @@ -350,6 +361,26 @@ jobs: --version="${VERSION}" \ --source=runtime/el_runtime.js + # el-runtime-src — the COMPLETE runtime link set as one tarball. + # + # The el-runtime-c / el-runtime-h packages above are single files and are + # kept for backward compatibility with consumers that already pull them, + # but they are NOT sufficient to link: el_runtime.c calls into six engram + # sibling translation units. New consumers should pull el-runtime-src and + # link everything named in its SOURCES file. + tar -czf /tmp/el-runtime-src.tar.gz \ + -C runtime SOURCES \ + $(../scripts/el-runtime-sources.sh --check) \ + $(../scripts/el-runtime-sources.sh --headers --check) + + gcloud artifacts generic upload \ + --repository=foundation-prod \ + --location=us-central1 \ + --project=neuron-785695 \ + --package=el-runtime-src \ + --version="${VERSION}" \ + --source=/tmp/el-runtime-src.tar.gz + echo "Published El SDK version=${VERSION} to foundation-prod" # Keep key alive for the ci-base rebuild step below # (deleted in that step after docker push) @@ -386,8 +417,9 @@ jobs: FROM ${BASE} COPY dist/platform/elc /opt/el/dist/platform/elc COPY dist/bin/elb /opt/el/dist/bin/elb - COPY runtime/el_runtime.c /opt/el/runtime/el_runtime.c - COPY runtime/el_runtime.h /opt/el/runtime/el_runtime.h + # Whole runtime link set — el_runtime.c alone does not link (it calls + # into the six engram sibling TUs). See lang/runtime/SOURCES. + COPY runtime/ /opt/el/runtime/ COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb EOF diff --git a/.githooks/pre-commit b/.githooks/pre-commit index bc78c05..2ce3237 100755 --- a/.githooks/pre-commit +++ b/.githooks/pre-commit @@ -12,10 +12,40 @@ ELC="$LANG_DIR/dist/platform/elc" # If elc isn't built yet, skip with a warning rather than blocking if [ ! -x "$ELC" ]; then echo "⚠ elc not found at lang/dist/platform/elc — skipping pre-commit tests" - echo " Build it first: cd lang && gcc -O2 -I runtime dist/elc-bootstrap.c runtime/el_runtime.c -lcurl -lpthread -o dist/elc-gen2 && ./dist/elc-gen2 el-compiler/src/compiler.el > /tmp/elc.c && gcc -O2 -I runtime /tmp/elc.c runtime/el_runtime.c -lcurl -lpthread -o dist/platform/elc" + echo " Build it first: see 'Rebuilding the Compiler' in lang/AGENTS.md" + echo " (link \$($ROOT/scripts/el-runtime-sources.sh $RUNTIME) — NOT el_runtime.c alone)" exit 0 fi +# The runtime is MULTI-FILE (lang/runtime/SOURCES). This hook used to link +# "$RUNTIME/el_runtime.c" alone with stderr sent to /dev/null — so once +# el_runtime.c started calling into the engram siblings, every native test +# reported as FAILED with the real `ld` error invisible. Build the whole set +# once into an archive, then link each test against it. + +# macOS: Homebrew openssl@3 is not on the default include/lib search path, so +# without these the link fails on -lssl/-lcrypto. Empty on Linux/CI. +SSL_INC="" +SSL_LIB="" +if command -v brew >/dev/null 2>&1 && OSSL="$(brew --prefix openssl@3 2>/dev/null)" && [ -n "$OSSL" ]; then + SSL_INC="-I$OSSL/include" + SSL_LIB="-L$OSSL/lib" +fi + +echo "→ Building runtime (compile-once, link-many)..." +HOOK_LIB="/tmp/el_hook_libel.a" +HOOK_OBJ="/tmp/el_hook_obj" +rm -rf "$HOOK_OBJ" && mkdir -p "$HOOK_OBJ" +if ! for src in $("$ROOT/scripts/el-runtime-sources.sh" --check "$RUNTIME"); do + gcc -O2 -c -I "$RUNTIME" $SSL_INC "$src" -o "$HOOK_OBJ/$(basename "${src%.c}").o" || exit 1 + done; then + echo "✗ Pre-commit failed: the runtime does not compile." + echo " Re-run without 2>/dev/null to see the error:" + echo " gcc -O2 -c -I $RUNTIME \$($ROOT/scripts/el-runtime-sources.sh $RUNTIME)" + exit 1 +fi +ar rcs "$HOOK_LIB" "$HOOK_OBJ"/*.o + echo "→ Running El native tests..." PASS=0 FAIL=0 @@ -27,8 +57,8 @@ for test_file in "$LANG_DIR"/tests/native/test_*.el; do tmp_bin="/tmp/el_hook_${name}" if "$ELC" --test "$test_file" > "$tmp_c" 2>/dev/null \ - && gcc -O2 -I "$RUNTIME" "$tmp_c" "$RUNTIME/el_runtime.c" \ - -lcurl -lpthread -lm -o "$tmp_bin" 2>/dev/null \ + && gcc -O2 -I "$RUNTIME" $SSL_INC $SSL_LIB "$tmp_c" "$HOOK_LIB" \ + -lcurl -lssl -lcrypto -lpthread -lm -o "$tmp_bin" 2>/dev/null \ && "$tmp_bin" 2>/dev/null; then PASS=$((PASS + 1)) else diff --git a/AGENTS.md b/AGENTS.md index f2079c2..0fae597 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -199,21 +199,35 @@ wrong, say so with a measurement rather than editing it. All build/test commands run from `lang/` unless noted. Grounded in `.gitea/workflows/sdk-release.yaml`, `lang/install.sh`, and `lang/AGENTS.md`. +> ### The runtime is MULTI-FILE — never link `el_runtime.c` alone +> +> `lang/runtime/el_runtime.c` `#include`s six engram headers and makes hard cross-TU calls into all six sibling `.c` files. **Linking it by itself fails at `ld`** (undefined `engram_ground_json`, `engram_activate_inner`, `eg_find_relation`, `cog_assert_two_axis`, …). The canonical link set lives in exactly one place — **`lang/runtime/SOURCES`** — and is printed by `scripts/el-runtime-sources.sh`: +> +> ```bash +> scripts/el-runtime-sources.sh lang/runtime # ten .c files, in link order +> ``` +> +> Use `$(scripts/el-runtime-sources.sh )` in every link line. Do not spell the list out longhand — it was written out in ~8 places, every copy drifted, and that is why the one-file link line below shipped broken for months. *(Corrected 2026-08-16.)* + **Self-host the compiler** (seed binary → gen2 elc): ```bash cd lang dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c # seed is the committed linux-amd64 binary -gcc -O2 -I el-compiler/runtime dist/elc-gen2.c \ - el-compiler/runtime/el_runtime.c \ +gcc -O2 -I runtime dist/elc-gen2.c \ + $(../scripts/el-runtime-sources.sh runtime) \ -lcurl -lssl -lcrypto -lpthread -lm \ -o dist/platform/elc ``` -On macOS/arm64 the canonical local binary is `dist/platform/elc`; verify self-hosting by recompiling and `diff`ing the emitted `.c` (see `lang/AGENTS.md`). Note: `lang/AGENTS.md` says `el_seed.c` supersedes `el_runtime.c`, but the release workflow still links `el_runtime.c`/`.h` — treat `el_runtime.c` as the published runtime; reconcile which is canonical **(verify)**. +On macOS/arm64 the canonical local binary is `dist/platform/elc`; verify self-hosting by recompiling and `diff`ing the emitted `.c` (see `lang/AGENTS.md`). + +*(Corrected 2026-08-16: this recipe compiled `el-compiler/runtime/el_runtime.c`. That path is a **lagging fork** — the "DO NOT EDIT" list at the top of this file names it as such. Building the canonical compiler from a known-stale fork was a live defect. It now uses `lang/runtime/`, the canonical source.)* + +**Which runtime file is canonical — resolved.** *(This note previously read "`lang/AGENTS.md` says `el_seed.c` supersedes `el_runtime.c`, but the release workflow still links `el_runtime.c`/`.h` — reconcile which is canonical **(verify)**." It is now reconciled.)* **Neither supersedes the other; both ship, together with eight more.** `el_runtime.c` was created on 2026-05-03 as an explicitly temporary build shim — deleted that afternoon, restored 25 minutes later "UNTIL the compiler is updated to emit `#include el_seed.h`" — and the `until` never happened, so it grew to 20.5k lines. The end state remains a seed-only boundary (`elc` emitting `#include "el_seed.h"`, `elb` dropping its hardcoded runtime path); until that lands, **the canonical unit is the set in `lang/runtime/SOURCES`, not any one file.** **Build `elb`** (build coordinator, the `.NET`-style incremental linker — compiles each module independently, no monolithic blobs): ```bash dist/platform/elc elb.el > dist/elb.c -gcc -O2 -I el-compiler/runtime dist/elb.c el-compiler/runtime/el_runtime.c \ +gcc -O2 -I runtime dist/elb.c $(../scripts/el-runtime-sources.sh runtime) \ -lcurl -lssl -lcrypto -lpthread -lm -o dist/bin/elb ``` `epm` and `el-install` are then built via `elb --clean --elc=… --runtime=… --out=…`. @@ -221,10 +235,16 @@ gcc -O2 -I el-compiler/runtime dist/elb.c el-compiler/runtime/el_runtime.c \ **Compile + run an El program:** ```bash elc src/app.el > dist/app.c -cc -std=c11 -O2 -I /el_runtime -o dist/app dist/app.c /el_runtime.c -lcurl -lpthread +cc -std=c11 -O2 -I -o dist/app dist/app.c \ + /el_runtime.c /el_seed.c \ + /engram_store.c /engram_vindex.c /engram_geometry.c \ + /engram_reason.c /engram_verify.c /engram_cognition.c \ + /eg_cosine_batch.c /eg_cosine_batch_strategy_cpu.c \ + -lcurl -lssl -lcrypto -lpthread -lm ``` +(Inside this repo, replace the file list with `$(scripts/el-runtime-sources.sh lang/runtime)`. `install.sh` installs all of these into ``.) -**Tests** — shell suites `bash tests/{text,calendar,time,html_sanitizer}/run.sh` (with `ELC=$(pwd)/dist/platform/elc EL_HOME=$(pwd)`), plus native suites via `elc --test tests/native/test_*.el` (core, text, string, math, state, time, json, env, fs) compiled and run against `el_runtime.c`. +**Tests** — shell suites `bash tests/{text,calendar,time,html_sanitizer}/run.sh` (with `ELC=$(pwd)/dist/platform/elc EL_HOME=$(pwd)`), plus native suites via `elc --test tests/native/test_*.el` (core, text, string, math, state, time, json, env, fs) compiled and run against the full runtime set. **Publishing — how downstream gets the SDK.** On push to `main`, `sdk-release.yaml`: 1. Publishes a Gitea `latest` release with per-file assets `elc`, `el_runtime.c`, `el_runtime.h`, the SDK tarball, and `el-install`. diff --git a/DESIGN.md b/DESIGN.md index fd26324..82b9c42 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -548,9 +548,13 @@ before `main` does anything. That is the dividend of discovery-precedes-executio ``` # once, ever (or when the runtime/framework changes): -cc -c el_runtime.c -o el_runtime.o -elc eltest.el > eltest.c && cc -c eltest.c -o eltest.o -ar rcs libeltest.a el_runtime.o eltest.o +# The runtime is MULTI-FILE — compile every .c named in lang/runtime/SOURCES. +# Linking el_runtime.c alone fails: it calls into the six engram sibling TUs. +for src in $(scripts/el-runtime-sources.sh lang/runtime); do + cc -c "$src" -o "obj/$(basename "${src%.c}").o" +done +elc eltest.el > eltest.c && cc -c eltest.c -o obj/eltest.o +ar rcs libeltest.a obj/*.o # per suite: elc --test foo_test.el > foo_test.c # registry + bodies only diff --git a/lang/AGENTS.md b/lang/AGENTS.md index 099e245..9d3f56d 100644 --- a/lang/AGENTS.md +++ b/lang/AGENTS.md @@ -77,14 +77,28 @@ This is where almost all work belongs. El programs are source files that get com This is the self-contained C OS-boundary layer. It provides the `__`-prefixed primitives that compiled El programs call: libcurl HTTP, pthreads, filesystem I/O, arena allocation, etc. It is **not generated** — it is maintained by hand. -The runtime is native El (`runtime/*.el`) over a C OS-boundary. **Status (verified 2026-08-15):** the migration to a seed-only boundary is *in progress, not done*. Two files exist: -- `runtime/el_runtime.c` (~860 KB) — **LIVE**. Holds the engram store (`EngramStore engram_global`) plus the `http_*`/`json_*`/`state_*`/`engram_*` impls. It is the authoritative single-file link target for the compiler, and `tools/install.sh` compiles it into `libel.a`. This is where a new C builtin's *implementation* must currently live to be linkable. -- `runtime/el_seed.c` — the intended hand-maintained `__`-prefixed seed (thin wrappers over the above). It is compiled alongside `el_runtime.c` by `tools/install.sh`, but does **not** compile standalone yet (see the build-path caveat under "Rebuilding the Compiler"). +The runtime is native El (`runtime/*.el`) over a C OS-boundary. **Status (verified 2026-08-16):** the migration to a seed-only boundary is *in progress, not done*. + +**The runtime is MULTI-FILE. There is no single-file link target and there has not been one for months.** The canonical link set is listed once, in **`runtime/SOURCES`**, and printed by `scripts/el-runtime-sources.sh`. It currently holds ten translation units: `el_runtime.c`, `el_seed.c`, the six `engram_*.c` concern files, and `eg_cosine_batch{,_strategy_cpu}.c`. + +- `runtime/el_runtime.c` (~940 KB, 20.5k lines) — **LIVE, and oversized.** It began life on 2026-05-03 as a temporary build shim: it was deleted that afternoon ("runtime is 100% native El") and restored 25 minutes later, explicitly "UNTIL the compiler is updated to emit `#include el_seed.h`". That `until` never arrived, and in the 3.5 months since, the file doubled. **It is not a volatility unit — it is a dumping ground.** ~47.5% of it is engram code that belongs in the six sibling files that already exist. Do not add to it. See "Where a new C builtin goes" below. +- `runtime/el_seed.c` — the intended hand-maintained `__`-prefixed seed (thin wrappers over the above). +- `runtime/engram_{store,vindex,geometry,reason,verify,cognition}.c` — the engram concerns, each with its own header. `el_runtime.c` `#include`s all six headers and makes hard cross-TU calls into all six. + +> **Linking `el_runtime.c` alone does not work and has not for months.** It fails at `ld` with undefined symbols (`engram_ground_json`, `engram_activate_inner`, `eg_find_relation`, `cog_assert_two_axis`, …). Any recipe, script, or CI step that names `el_runtime.c` by itself is stale — replace it with `$(scripts/el-runtime-sources.sh lang/runtime)`. **Only edit these when you genuinely need OS-level access** (raw sockets, GPU calls, new libcurl features, a new engram store op). For everything else, write El. +#### Where a new C builtin goes + +**Put it in the `.c` that owns the concern — NOT in `el_runtime.c`.** + +*Placement is a link-time concern. The compiler cannot tell which `.c` a symbol came from, and never could.* `builtin_arity` in `el-compiler/src/codegen.el` maps NAME → ARITY INT and nothing else (~413 entries); the El name is emitted as the exact C symbol and resolved by `ld`. Proof, if you want it: `nm lang/dist/platform/elc` on the *shipped* compiler shows `T _engram_geo_reify_index_new` (defined in `engram_geometry.c`), `T _vindex_insert` (`engram_vindex.c`), `T _engram_think` (`engram_cognition.c`), `T _engram_reason_abduce` (`engram_reason.c`). **The shipped compiler is already linked from ten translation units.** A builtin defined in a sibling `.c` is exactly as linkable as one defined in `el_runtime.c`. + +Choose the file by concern: engram store ops → `engram_store.c`; index → `engram_vindex.c`; geometry/priming → `engram_geometry.c`; reasoning → `engram_reason.c`; grounding/consistency → `engram_verify.c`; think/stance → `engram_cognition.c`. **If no existing file owns it, create one** — add the `.c` to `runtime/SOURCES` (one line) and every build path picks it up. For a builtin that belongs to a downstream program rather than the runtime, declare `c_source "path/to/file.c"` in that program's `manifest.el`; `elb` already links it (`parse_manifest_c_sources`, `lang/elb.el:82`). + When you add a C builtin (verbatim-emit recipe — the El name is emitted as the exact C symbol; `builtin_arity` is an arity guard only, not a dispatch table): -1. Implement the C function in `el_runtime.c` (and declare it in `el_runtime.h`). +1. Implement the C function in the **concern-owning `.c`** (and declare it in that file's `.h`). Add the file to `runtime/SOURCES` if it is new. Only put it in `el_runtime.c` if it is genuinely EL core (val/str/map/list/arena) — that is ~8% of what is in there today. 2. Add a `__`-prefixed thin wrapper in `el_seed.c` and declare it in `el_seed.h`. 3. Add the name to `builtin_arity` in `el-compiler/src/codegen.el` — add **both** the plain and `__`-prefixed spellings. 4. Rebuild the elc binary (see below) and confirm the self-host fixpoint is byte-identical. @@ -125,7 +139,9 @@ diff elc-new.c elc-verify.c # must be identical mv dist/platform/elc-new dist/platform/elc ``` -> **Build-path caveat (verified 2026-08-15).** `el_seed.c` is the intended hand-maintained OS-boundary seed, but it does **not** compile standalone under modern clang: it wraps ~16 unprefixed `el_runtime.c` symbols (`http_serve`, `json_*`, `state_*`, `http_response`) without prototypes, and clang treats implicit declarations as errors (C99+). The productionised install (`tools/install.sh`) builds `libel.a` from **both** `el_seed.o` + `el_runtime.o` together, which is why linking succeeds there. To make `el_seed.c` build on its own, add prototypes for those symbols (or `#include "el_runtime.h"`, reconciling the `__http_serve` return-type mismatch first). Until then, `el_runtime.c` is the authoritative single-file link target for the compiler. +> **Build-path caveat (verified 2026-08-15).** `el_seed.c` is the intended hand-maintained OS-boundary seed, but it does **not** compile standalone under modern clang: it wraps ~16 unprefixed `el_runtime.c` symbols (`http_serve`, `json_*`, `state_*`, `http_response`) without prototypes, and clang treats implicit declarations as errors (C99+). The productionised install (`tools/install.sh`) builds `libel.a` from **both** `el_seed.o` + `el_runtime.o` together, which is why linking succeeds there. To make `el_seed.c` build on its own, add prototypes for those symbols (or `#include "el_runtime.h"`, reconciling the `__http_serve` return-type mismatch first). +> +> **There is no single-file link target.** *(Corrected 2026-08-16 — this paragraph previously ended "`el_runtime.c` is the authoritative single-file link target for the compiler". Measured: that is false. Linking `elc-new.c` against `runtime/el_runtime.c` alone fails at `ld` with undefined `engram_ground_json`, `engram_activate_inner`, `eg_find_relation`, `cog_assert_two_axis`, and others, because `el_runtime.c` `#include`s six engram headers and calls into all six sibling `.c` files.)* Link the set in `runtime/SOURCES` via `$(../scripts/el-runtime-sources.sh runtime)`. After changing `el_seed.c` only (no El source changes), rebuild downstream programs but do NOT need to rebuild the compiler binary itself — the seed is linked at the application level, not the compiler level. diff --git a/lang/install.sh b/lang/install.sh index 9604ec3..9317853 100644 --- a/lang/install.sh +++ b/lang/install.sh @@ -49,21 +49,47 @@ download() { TMP_DIR="$(mktemp -d)" trap 'rm -rf "${TMP_DIR}"' EXIT -download "${RELEASE_BASE}/elc" "${TMP_DIR}/elc" -download "${RELEASE_BASE}/el_runtime.c" "${TMP_DIR}/el_runtime.c" -download "${RELEASE_BASE}/el_runtime.h" "${TMP_DIR}/el_runtime.h" +# The runtime is MULTI-FILE. el_runtime.c #includes six engram headers and makes +# hard cross-TU calls into all six sibling .c files, so installing el_runtime.c +# alone produces a lib/ that CANNOT LINK — `ld` fails with undefined +# engram_ground_json / engram_activate_inner / eg_find_relation / cog_assert_two_axis. +# This list mirrors lang/runtime/SOURCES (the in-repo source of truth); keep them +# in step. install.sh is standalone by design — it runs on machines with no repo +# checkout — so it cannot call scripts/el-runtime-sources.sh. +RUNTIME_SOURCES=( + el_runtime.c el_seed.c + engram_store.c engram_vindex.c engram_geometry.c + engram_reason.c engram_verify.c engram_cognition.c + eg_cosine_batch.c eg_cosine_batch_strategy_cpu.c +) +RUNTIME_HEADERS=( + el_runtime.h el_seed.h + engram_store.h engram_vindex.h engram_geometry.h + engram_reason.h engram_verify.h engram_cognition.h + eg_cosine_batch.h eg_cosine_batch_strategy.h +) + +download "${RELEASE_BASE}/elc" "${TMP_DIR}/elc" +for f in "${RUNTIME_SOURCES[@]}" "${RUNTIME_HEADERS[@]}"; do + download "${RELEASE_BASE}/${f}" "${TMP_DIR}/${f}" +done # Install -install -m 755 "${TMP_DIR}/elc" "${BIN_DIR}/elc" -install -m 644 "${TMP_DIR}/el_runtime.c" "${LIB_DIR}/el_runtime.c" -install -m 644 "${TMP_DIR}/el_runtime.h" "${LIB_DIR}/el_runtime.h" +install -m 755 "${TMP_DIR}/elc" "${BIN_DIR}/elc" +for f in "${RUNTIME_SOURCES[@]}" "${RUNTIME_HEADERS[@]}"; do + install -m 644 "${TMP_DIR}/${f}" "${LIB_DIR}/${f}" +done + +# Record the link set so downstream Makefiles can read it instead of hardcoding. +printf '%s\n' "${RUNTIME_SOURCES[@]}" > "${TMP_DIR}/SOURCES" +install -m 644 "${TMP_DIR}/SOURCES" "${LIB_DIR}/SOURCES" echo echo "==> El SDK installed successfully" echo echo " elc binary : ${BIN_DIR}/elc" -echo " runtime : ${LIB_DIR}/el_runtime.c" -echo " header : ${LIB_DIR}/el_runtime.h" +echo " runtime : ${LIB_DIR}/ (${#RUNTIME_SOURCES[@]} .c files, ${#RUNTIME_HEADERS[@]} headers)" +echo " link set : ${LIB_DIR}/SOURCES" echo echo "Add the following to your Makefile to build El programs:" echo @@ -71,10 +97,14 @@ echo " EL_LIB := ${LIB_DIR}" echo " ELC := elc" echo " CC := cc" echo " CFLAGS := -std=c11 -O2 -I\$(EL_LIB)" +echo " LDLIBS := -lcurl -lssl -lcrypto -lpthread -lm" +echo +echo " # The runtime is multi-file — link the whole set, not el_runtime.c alone." +echo " EL_RUNTIME := \$(addprefix \$(EL_LIB)/,\$(shell cat \$(EL_LIB)/SOURCES))" echo echo " dist/myapp.c: src/myapp.el" echo " \t\$(ELC) src/myapp.el > dist/myapp.c" echo echo " dist/myapp: dist/myapp.c" -echo " \t\$(CC) \$(CFLAGS) -o dist/myapp dist/myapp.c \$(EL_LIB)/el_runtime.c -lcurl -lpthread" +echo " \t\$(CC) \$(CFLAGS) -o dist/myapp dist/myapp.c \$(EL_RUNTIME) \$(LDLIBS)" echo diff --git a/lang/runtime/SOURCES b/lang/runtime/SOURCES new file mode 100644 index 0000000..e44c152 --- /dev/null +++ b/lang/runtime/SOURCES @@ -0,0 +1,52 @@ +# SOURCES — the canonical El runtime link set. +# +# THIS FILE IS THE SINGLE SOURCE OF TRUTH for "what do I compile and link to +# get the El runtime". Every build path — CI, install.sh, the SDK release, the +# docs, elb, the engram test harnesses — reads it via scripts/el-runtime-sources.sh +# instead of hardcoding its own list. +# +# WHY THIS FILE EXISTS +# -------------------- +# The runtime has been multi-translation-unit since the engram siblings landed: +# el_runtime.c #includes engram_{store,vindex,geometry,reason,verify,cognition}.h +# and makes hard cross-TU calls into all six. Linking el_runtime.c ALONE has been +# broken since then — `ld` fails with undefined symbols (engram_ground_json, +# engram_activate_inner, eg_find_relation, cog_assert_two_axis, ...). +# +# It stayed broken because the link set was written out longhand in ~8 different +# places, each of which drifted independently. A list copied 8 times is a list +# that is wrong in 8 places. It is now written once, here. +# +# HOW TO USE IT +# ------------- +# scripts/el-runtime-sources.sh # bare names, one per line +# scripts/el-runtime-sources.sh lang/runtime # prefixed with a directory +# cc ... $(scripts/el-runtime-sources.sh lang/runtime) -lcurl -lssl -lcrypto -lpthread -lm +# +# ADDING A FILE +# ------------- +# Add the .c here and it is picked up by every build path at once. That is the +# point: a new concern gets its own translation unit and costs one line, instead +# of being appended to el_runtime.c because appending was the cheaper edit. +# +# Order is link order. Blank lines and `#` comments are ignored. + +# --- EL core language runtime ------------------------------------------------- +el_runtime.c +el_seed.c + +# --- Engram: store, index, geometry, reasoning, verification, cognition ------- +# These are the six concern-owned translation units el_runtime.c calls into. +engram_store.c +engram_vindex.c +engram_geometry.c +engram_reason.c +engram_verify.c +engram_cognition.c + +# --- Vector math: batch cosine + its CPU strategy ---------------------------- +# The ggml strategy (eg_cosine_batch_strategy_ggml.c) is an OPTIONAL swap-in and +# is deliberately NOT in the default set — it needs ggml headers. Link it in +# place of the cpu strategy when you have them. +eg_cosine_batch.c +eg_cosine_batch_strategy_cpu.c diff --git a/lang/spec/language.md b/lang/spec/language.md index 6c15996..b288439 100644 --- a/lang/spec/language.md +++ b/lang/spec/language.md @@ -697,12 +697,22 @@ Every compiled program links against: - `el_runtime.h` — declaration header - `el_runtime.c` — implementation +The runtime is **multi-file**: `el_runtime.c` `#include`s the six `engram_*.h` +headers and calls into all six sibling translation units, so linking it alone +fails at `ld`. The canonical link set is `/SOURCES`. + Compile command: ``` -cc -std=c11 -I -o .c el_runtime.c +cc -std=c11 -I -o .c \ + $(sed 's|^|/|' /SOURCES) \ + -lcurl -lssl -lcrypto -lpthread -lm ``` +Inside this repo, `scripts/el-runtime-sources.sh ` prints that list +(it strips comments; the raw `sed` above works against an installed SDK's +`SOURCES`, which `install.sh` writes comment-free). + ### 13.4 Output Format ```c diff --git a/scripts/el-runtime-sources.sh b/scripts/el-runtime-sources.sh new file mode 100755 index 0000000..fa64a66 --- /dev/null +++ b/scripts/el-runtime-sources.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# el-runtime-sources.sh — print the canonical El runtime link set. +# +# Reads lang/runtime/SOURCES (the single source of truth) and prints one path +# per line, optionally prefixed with a directory. Use it anywhere a link line +# would otherwise spell the runtime .c files out longhand: +# +# cc -std=c11 -O2 -I lang/runtime -o app app.c \ +# $(scripts/el-runtime-sources.sh lang/runtime) \ +# -lcurl -lssl -lcrypto -lpthread -lm +# +# Options: +# --headers print the shipped headers instead of the .c sources +# --check verify every listed file exists; exit non-zero if any is missing +# +# WHY: linking el_runtime.c alone has been broken since el_runtime.c started +# calling into the engram siblings. The list was duplicated across ~8 build +# paths and drifted. It lives in exactly one place now — see lang/runtime/SOURCES. +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +SOURCES="${ROOT}/lang/runtime/SOURCES" + +if [ ! -f "$SOURCES" ]; then + echo "FATAL: canonical runtime source list missing: $SOURCES" >&2 + exit 1 +fi + +MODE="sources" +PREFIX="" +CHECK=0 + +for arg in "$@"; do + case "$arg" in + --headers) MODE="headers" ;; + --check) CHECK=1 ;; + -*) echo "el-runtime-sources.sh: unknown option: $arg" >&2; exit 2 ;; + *) PREFIX="${arg%/}/" ;; + esac +done + +# Strip comments and blank lines. Order is preserved — it is link order. +mapfile -t FILES < <(sed -e 's/#.*//' -e 's/[[:space:]]*$//' "$SOURCES" | grep -v '^$') + +if [ "${#FILES[@]}" -eq 0 ]; then + echo "FATAL: $SOURCES lists no sources" >&2 + exit 1 +fi + +if [ "$MODE" = "headers" ]; then + # Every .c's matching .h, plus the headers that carry no .c of their own. + HDRS=() + for f in "${FILES[@]}"; do + h="${f%.c}.h" + [ -f "${ROOT}/lang/runtime/${h}" ] && HDRS+=("$h") + done + # Interface-only headers: no matching .c, but required to compile against. + for h in eg_cosine_batch_strategy.h el_native_target.h el_platform_win.h; do + [ -f "${ROOT}/lang/runtime/${h}" ] && HDRS+=("$h") + done + FILES=("${HDRS[@]}") +fi + +RC=0 +for f in "${FILES[@]}"; do + if [ "$CHECK" -eq 1 ] && [ ! -f "${ROOT}/lang/runtime/${f}" ]; then + echo "MISSING: lang/runtime/${f} (listed in lang/runtime/SOURCES)" >&2 + RC=1 + fi + printf '%s%s\n' "$PREFIX" "$f" +done + +exit $RC -- 2.52.0