bigmerge
8c2406ff6b
El SDK CI - dev / build-and-test (pull_request) Failing after 5m39s
el_runtime.c was created 2026-05-03 as an explicitly temporary build shim. It
was deleted that afternoon ("runtime is 100% native El") and restored 25 minutes
later "UNTIL the compiler is updated to emit #include el_seed.h". The `until`
never came. 3.5 months on it is 20,527 lines, and nothing was ever set up to
notice — a file scheduled for deletion gets no owner, no budget, no boundary.
What kept it growing is not inertia, it is an instruction. lang/AGENTS.md said
el_runtime.c "is the authoritative single-file link target ... THIS IS WHERE A
NEW C BUILTIN'S IMPLEMENTATION MUST CURRENTLY LIVE TO BE LINKABLE", and made it
step 1 of the add-a-builtin recipe. That is false. Placement is a link-time
concern: builtin_arity maps NAME -> ARITY INT only, the El name is emitted as
the exact C symbol, and `ld` resolves it — the compiler cannot tell which .c a
symbol came from. `nm lang/dist/platform/elc` on the shipped compiler already
shows T _engram_geo_reify_index_new, T _vindex_insert, T _engram_think,
T _engram_reason_abduce: it is linked from ten translation units today. In a
repo where agents write most of the code, a false instruction in the instruction
file is the forcing function. The file grew because the recipe said to grow it.
The multi-file runtime is therefore already real, and the docs and the
distribution never caught up — which left a live, shipped bug:
* Linking el_runtime.c alone FAILS at `ld` (undefined engram_ground_json,
engram_activate_inner, eg_find_relation, cog_assert_two_axis, ...) because
el_runtime.c #includes six engram headers and calls into all six siblings.
* sdk-release.yaml shipped el_runtime.c/.h + engram_store.c/.h and none of the
other five required .c files, so downstream consumers of the el-runtime-c
Artifact Registry package and of install.sh got a lib/ that cannot link.
* .githooks/pre-commit linked el_runtime.c alone with stderr to /dev/null, so
it reported all 13 native suites as FAILED with the real ld error invisible.
* AGENTS.md's self-host recipe compiled el-compiler/runtime/el_runtime.c — a
path the same file's "DO NOT EDIT" list names as a lagging fork.
The root fix is to stop writing the list down eight times:
* lang/runtime/SOURCES — the canonical link set, in one place, in link order.
* scripts/el-runtime-sources.sh — prints it, optionally prefixed; --check
fails loudly on a missing file, --headers for the shipped headers.
* Every link line in AGENTS.md, lang/AGENTS.md, DESIGN.md, lang/spec/language.md,
the three workflows and the pre-commit hook now reads that one list.
* Adding a concern's .c is one line in SOURCES, so a new builtin no longer has
to be appended to el_runtime.c just because appending was the cheaper edit.
Distribution: ship the siblings rather than amalgamate. Amalgamation needs a new
tool and contradicts DESIGN.md's compile-once-link-many; the siblings are already
independently authored and independently tested (engram/test/*.sh link subsets
directly), and engram_store.c was already shipped, so this completes a mechanism
that existed rather than inventing one. Source is also a superset: a consumer
that wants one file can concatenate, one that wants separate TUs cannot undo an
amalgamation. el-runtime-c/-h stay for backward compatibility; el-runtime-src is
added carrying the complete set plus SOURCES.
lang/AGENTS.md now points new C builtins at the concern-owning .c and states
plainly that the compiler cannot tell which .c a symbol came from, with the nm
evidence. AGENTS.md's "reconcile which is canonical (verify)" note is resolved:
neither file supersedes the other, the canonical unit is the set.
Verified locally (the bar; not CI):
* engram/src/server.el compiles and links against the SOURCES set.
* Compile-once-link-many into libel.a links the same program.
* elb builds from the corrected recipe.
* Self-host fixpoint byte-identical (11,110 lines, stage2 == stage3) built
with the SOURCES-driven link line.
* pre-commit hook: 0 of 13 native suites passing -> 8 of 13.
The 5 still-failing suites are PRE-EXISTING and untouched here: test_fs
(fs_list_json undeclared), test_state (state_has, state_get_or undeclared),
test_json (json_build_array/json_build_object/json_escape_string undefined),
test_time (now_ns undefined), test_env (1 assertion). Builtins registered in
builtin_arity with no implementation or no declaration anywhere — the same
recipe defect, now visible because the linker error is no longer suppressed.
Not attempted: making elc emit #include el_seed.h and dropping elb's hardcoded
runtime path. That is the correct long-term fix and finishes the 2026-05-03
migration, but it touches codegen and self-hosting and belongs in its own change.
335 lines
13 KiB
YAML
335 lines
13 KiB
YAML
name: El SDK CI - dev
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- dev
|
|
pull_request:
|
|
branches:
|
|
- dev
|
|
|
|
jobs:
|
|
build-and-test:
|
|
runs-on: ubuntu-latest
|
|
defaults:
|
|
run:
|
|
working-directory: lang
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Install build dependencies
|
|
run: |
|
|
apt-get update -qq
|
|
apt-get install -y gcc libcurl4-openssl-dev apt-transport-https ca-certificates
|
|
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" \
|
|
> /etc/apt/sources.list.d/google-cloud-sdk.list
|
|
apt-get update -qq && apt-get install -y google-cloud-cli
|
|
|
|
# Seed: use the committed linux-amd64 binary as the bootstrap
|
|
- name: Bootstrap from committed linux binary (seed)
|
|
run: |
|
|
chmod +x dist/platform/elc-linux-amd64
|
|
echo "seed elc (committed linux-amd64 binary)"
|
|
dist/platform/elc-linux-amd64 --version || true
|
|
|
|
# Gen2: use seed to self-host compile the El compiler
|
|
- name: Self-host compile El compiler (gen2)
|
|
run: |
|
|
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c
|
|
gcc -O2 \
|
|
-I runtime \
|
|
dist/elc-gen2.c \
|
|
$(../scripts/el-runtime-sources.sh runtime) \
|
|
-lcurl -lssl -lcrypto -lpthread -lm \
|
|
-o dist/platform/elc
|
|
chmod +x dist/platform/elc
|
|
echo "gen2 (self-hosted) elc built"
|
|
dist/platform/elc --version || true
|
|
|
|
# Build elb (needed for Artifact Registry publish and downstream CI)
|
|
- name: Build elb
|
|
run: |
|
|
mkdir -p dist/bin
|
|
dist/platform/elc elb.el > dist/elb.c
|
|
gcc -O2 \
|
|
-I runtime \
|
|
dist/elb.c \
|
|
$(../scripts/el-runtime-sources.sh runtime) \
|
|
-lcurl -lssl -lcrypto -lpthread -lm \
|
|
-o dist/bin/elb
|
|
chmod +x dist/bin/elb
|
|
echo "elb built"
|
|
|
|
- name: Run tests - text
|
|
run: |
|
|
ELC="$(pwd)/dist/platform/elc" \
|
|
EL_HOME="$(pwd)" \
|
|
bash tests/text/run.sh
|
|
|
|
- name: Run tests - calendar
|
|
run: |
|
|
ELC="$(pwd)/dist/platform/elc" \
|
|
EL_HOME="$(pwd)" \
|
|
bash tests/calendar/run.sh
|
|
|
|
- name: Run tests - time
|
|
run: |
|
|
ELC="$(pwd)/dist/platform/elc" \
|
|
EL_HOME="$(pwd)" \
|
|
bash tests/time/run.sh
|
|
|
|
- name: Run tests - html_sanitizer
|
|
run: |
|
|
ELC="$(pwd)/dist/platform/elc" \
|
|
EL_HOME="$(pwd)" \
|
|
bash tests/html_sanitizer/run.sh
|
|
|
|
# Native El test suites (elc --test, compile-link-run)
|
|
# The runtime is MULTI-FILE (see lang/runtime/SOURCES). Every .c is compiled
|
|
# once into /tmp/libel.a and reused by all 8 test modules — compile-once,
|
|
# link-many, as prescribed in DESIGN.md. Linking el_runtime.c alone fails
|
|
# at `ld`: it calls into all six engram sibling TUs.
|
|
- name: Precompile runtime into libel.a
|
|
run: |
|
|
set -euo pipefail
|
|
RUNTIME="$(pwd)/runtime"
|
|
rm -rf /tmp/elrt && mkdir -p /tmp/elrt
|
|
for src in $(../scripts/el-runtime-sources.sh --check "$RUNTIME"); do
|
|
gcc -O2 -c -I "$RUNTIME" "$src" -o "/tmp/elrt/$(basename "${src%.c}").o"
|
|
done
|
|
ar rcs /tmp/libel.a /tmp/elrt/*.o
|
|
echo "libel.a built from $(ls /tmp/elrt/*.o | wc -l) translation units"
|
|
|
|
- name: Run tests - native (core)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core
|
|
/tmp/el_native_core
|
|
|
|
- name: Run tests - native (text)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text
|
|
/tmp/el_native_text
|
|
|
|
- name: Run tests - native (string)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string
|
|
/tmp/el_native_string
|
|
|
|
- name: Run tests - native (math)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math
|
|
/tmp/el_native_math
|
|
|
|
- name: Run tests - native (state)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state
|
|
/tmp/el_native_state
|
|
|
|
- name: Run tests - native (time)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time
|
|
/tmp/el_native_time
|
|
|
|
- name: Run tests - native (json)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json
|
|
/tmp/el_native_json
|
|
|
|
- name: Run tests - native (env)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env
|
|
/tmp/el_native_env
|
|
|
|
- name: Run tests - native (fs)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs
|
|
/tmp/el_native_fs
|
|
|
|
# Build epm binary using elb (epm lives at repo root, not inside lang/)
|
|
- name: Build epm
|
|
run: |
|
|
ABS_ELB="$(pwd)/dist/bin/elb"
|
|
ABS_ELC="$(pwd)/dist/platform/elc"
|
|
ABS_RUNTIME="$(pwd)/runtime"
|
|
ABS_OUT="$(pwd)/dist/bin"
|
|
(cd ../epm && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
|
|
chmod +x dist/bin/epm
|
|
echo "epm built"
|
|
|
|
# Build el-install binary using elb
|
|
- name: Build el-install
|
|
run: |
|
|
ABS_ELB="$(pwd)/dist/bin/elb"
|
|
ABS_ELC="$(pwd)/dist/platform/elc"
|
|
ABS_RUNTIME="$(pwd)/runtime"
|
|
ABS_OUT="$(pwd)/dist/bin"
|
|
(cd tools/install && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
|
|
chmod +x dist/bin/el-install
|
|
echo "el-install built"
|
|
|
|
# Publish only after merge (push event), not on PR validation runs
|
|
- name: Publish El SDK to Artifact Registry (dev)
|
|
if: github.event_name == 'push'
|
|
env:
|
|
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
|
|
run: |
|
|
# Fail loudly: previously this step had no `set -e`, so an auth or
|
|
# upload failure was swallowed (step exited 0 on the trailing echo)
|
|
# and the SDK silently never published. Surface failures now.
|
|
set -euo pipefail
|
|
if [ -z "${GCP_SA_KEY:-}" ]; then
|
|
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
|
|
exit 1
|
|
fi
|
|
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
|
|
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
|
|
gcloud config set project neuron-785695
|
|
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
|
|
|
|
VERSION="${GITHUB_SHA:0:8}"
|
|
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el-elc \
|
|
--version="${VERSION}" \
|
|
--source=dist/platform/elc
|
|
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el-elb \
|
|
--version="${VERSION}" \
|
|
--source=dist/bin/elb
|
|
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el-runtime-c \
|
|
--version="${VERSION}" \
|
|
--source=runtime/el_runtime.c
|
|
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el-runtime-h \
|
|
--version="${VERSION}" \
|
|
--source=runtime/el_runtime.h
|
|
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el-runtime-js \
|
|
--version="${VERSION}" \
|
|
--source=runtime/el_runtime.js
|
|
|
|
echo "Published El SDK version=${VERSION} to foundation-dev"
|
|
# Keep key alive for the ci-base rebuild step below
|
|
# (deleted in that step after docker push)
|
|
|
|
- name: Rebuild ci-base with fresh El SDK (dev)
|
|
# Patches ci-base:dev in-place: pulls the existing image (which has all
|
|
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
|
|
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
|
|
#
|
|
# continue-on-error: this is a CI-cache optimization, NOT the release
|
|
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
|
|
# runner where DinD/Docker availability is fragile. A failure here must
|
|
# never block or redden the job — the SDK publish above is the deliverable.
|
|
continue-on-error: true
|
|
if: github.event_name == 'push'
|
|
env:
|
|
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
|
|
run: |
|
|
set -euo pipefail
|
|
CI_BASE="us-central1-docker.pkg.dev/neuron-785695/neuron-ci/ci-base"
|
|
SHA="${GITHUB_SHA:0:8}"
|
|
|
|
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
|
|
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
|
|
gcloud config set project neuron-785695
|
|
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
|
|
|
|
# Pull existing ci-base:dev (or fall back to :latest on first run)
|
|
BASE_TAG="dev"
|
|
docker pull "${CI_BASE}:dev" || { docker pull "${CI_BASE}:latest" && BASE_TAG="latest"; }
|
|
|
|
# Inline Dockerfile — only replaces the El SDK layer
|
|
cat > /tmp/Dockerfile.ci-base-patch << 'EOF'
|
|
ARG BASE
|
|
FROM ${BASE}
|
|
COPY dist/platform/elc /opt/el/dist/platform/elc
|
|
COPY dist/bin/elb /opt/el/dist/bin/elb
|
|
# Whole runtime link set — el_runtime.c alone does not link (it calls
|
|
# into the six engram sibling TUs). See lang/runtime/SOURCES.
|
|
COPY runtime/ /opt/el/runtime/
|
|
COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js
|
|
RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb
|
|
EOF
|
|
|
|
docker build \
|
|
--build-arg BASE="${CI_BASE}:${BASE_TAG}" \
|
|
--build-arg BUILDKIT_INLINE_CACHE=1 \
|
|
-f /tmp/Dockerfile.ci-base-patch \
|
|
-t "${CI_BASE}:dev" \
|
|
-t "${CI_BASE}:dev-${SHA}" \
|
|
.
|
|
|
|
docker push "${CI_BASE}:dev"
|
|
docker push "${CI_BASE}:dev-${SHA}"
|
|
|
|
echo "ci-base rebuilt: ${CI_BASE}:dev (${SHA})"
|
|
rm -f /tmp/gcp-key.json
|