bigmerge
fe634c4582
scripts/check-single-runtime.sh guards against el_runtime.c being COPIED — it
was written after a lagging fork shipped to prod and dropped learned hebb edges.
Nothing guarded against it GROWING. So it grew: 10,607 -> 20,527 lines, 94% in
3.5 months, the whole time under an explicit commit-message promise that it was
a temporary shim about to be deleted.
Worse, the copy guard was never wired in. Its own footer described the CI
wire-in as a TODO, and the TODO had never been done — the script existed but ran
nowhere, in no workflow and in no hook, so it had caught nothing for as long as
it has been in the tree. A guard that does not run is a comment.
This adds the missing guard and runs both.
* lang/runtime/BUDGET — a RATCHET, not a limit. max_lines is set at the
current 20,527 with NO headroom: the file cannot grow by one line. A second
cap, max_engram_fns (279), counts top-level engram_/eg_/cog_ definitions in
it — ~47.5% of the file is engram code and engram already owns six sibling
.c files, so this is the scoreboard for moving it out. Both may only go DOWN.
* scripts/check-runtime-growth.sh — enforces the ratchet, and three
invariants that keep the multi-file runtime honest: every .c in
lang/runtime/ is either in SOURCES or explicitly platform-optional (an
unaccounted .c is compiled by nothing and is silently dead); install.sh's
hardcoded download list matches SOURCES (it cannot call the helper — it
runs where there is no checkout — so that copy is checked, not trusted);
and an advisory nudge to lower the budget when you have earned it.
* Both guards now run as early steps in ci-dev.yaml, ci-stage.yaml and
sdk-release.yaml, and in .githooks/pre-commit.
The failure message is the point. The guard that existed said what was wrong but
not where the code should go, which makes it easy to "fix" by arguing with the
guard. This one names the destination: the concern-owning .c, or a new .c plus
one line in SOURCES, or c_source in a program's manifest.el — and it prints the
`nm` command that proves placement is link-time and that the shipped compiler
already links from ten translation units. Every runtime file except el_runtime.c
is deliberately uncapped, because that is where code is supposed to go.
Proven with negative controls, per lang/AGENTS.md step 5 — each shown FAILING:
* +1 line to el_runtime.c -> FAIL (20528/20527)
* +1 engram fn, net-zero lines -> FAIL (280/279)
* a new unaccounted lang/runtime/*.c -> FAIL
* engram_store.c removed from install.sh -> FAIL, names the missing file
* el_runtime.c truncated to 20,000 lines -> PASS + "lower max_lines to 20000"
* baseline, tree unmodified -> OK, and both guards green
el_runtime.c is byte-identical after the controls; this commit changes zero
lines of it.
345 lines
13 KiB
YAML
345 lines
13 KiB
YAML
name: El SDK CI - dev
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- dev
|
|
pull_request:
|
|
branches:
|
|
- dev
|
|
|
|
jobs:
|
|
build-and-test:
|
|
runs-on: ubuntu-latest
|
|
defaults:
|
|
run:
|
|
working-directory: lang
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
|
|
# Guards must run from the REPO ROOT — override the job's
|
|
# defaults.run.working-directory: lang
|
|
- name: Guard - single canonical runtime source
|
|
working-directory: ${{ github.workspace }}
|
|
run: bash scripts/check-single-runtime.sh
|
|
|
|
- name: Guard - el_runtime.c growth budget
|
|
working-directory: ${{ github.workspace }}
|
|
run: bash scripts/check-runtime-growth.sh
|
|
|
|
- name: Install build dependencies
|
|
run: |
|
|
apt-get update -qq
|
|
apt-get install -y gcc libcurl4-openssl-dev apt-transport-https ca-certificates
|
|
echo "deb [trusted=yes] https://packages.cloud.google.com/apt cloud-sdk main" \
|
|
> /etc/apt/sources.list.d/google-cloud-sdk.list
|
|
apt-get update -qq && apt-get install -y google-cloud-cli
|
|
|
|
# Seed: use the committed linux-amd64 binary as the bootstrap
|
|
- name: Bootstrap from committed linux binary (seed)
|
|
run: |
|
|
chmod +x dist/platform/elc-linux-amd64
|
|
echo "seed elc (committed linux-amd64 binary)"
|
|
dist/platform/elc-linux-amd64 --version || true
|
|
|
|
# Gen2: use seed to self-host compile the El compiler
|
|
- name: Self-host compile El compiler (gen2)
|
|
run: |
|
|
dist/platform/elc-linux-amd64 elc-cli.el > dist/elc-gen2.c
|
|
gcc -O2 \
|
|
-I runtime \
|
|
dist/elc-gen2.c \
|
|
$(../scripts/el-runtime-sources.sh runtime) \
|
|
-lcurl -lssl -lcrypto -lpthread -lm \
|
|
-o dist/platform/elc
|
|
chmod +x dist/platform/elc
|
|
echo "gen2 (self-hosted) elc built"
|
|
dist/platform/elc --version || true
|
|
|
|
# Build elb (needed for Artifact Registry publish and downstream CI)
|
|
- name: Build elb
|
|
run: |
|
|
mkdir -p dist/bin
|
|
dist/platform/elc elb.el > dist/elb.c
|
|
gcc -O2 \
|
|
-I runtime \
|
|
dist/elb.c \
|
|
$(../scripts/el-runtime-sources.sh runtime) \
|
|
-lcurl -lssl -lcrypto -lpthread -lm \
|
|
-o dist/bin/elb
|
|
chmod +x dist/bin/elb
|
|
echo "elb built"
|
|
|
|
- name: Run tests - text
|
|
run: |
|
|
ELC="$(pwd)/dist/platform/elc" \
|
|
EL_HOME="$(pwd)" \
|
|
bash tests/text/run.sh
|
|
|
|
- name: Run tests - calendar
|
|
run: |
|
|
ELC="$(pwd)/dist/platform/elc" \
|
|
EL_HOME="$(pwd)" \
|
|
bash tests/calendar/run.sh
|
|
|
|
- name: Run tests - time
|
|
run: |
|
|
ELC="$(pwd)/dist/platform/elc" \
|
|
EL_HOME="$(pwd)" \
|
|
bash tests/time/run.sh
|
|
|
|
- name: Run tests - html_sanitizer
|
|
run: |
|
|
ELC="$(pwd)/dist/platform/elc" \
|
|
EL_HOME="$(pwd)" \
|
|
bash tests/html_sanitizer/run.sh
|
|
|
|
# Native El test suites (elc --test, compile-link-run)
|
|
# The runtime is MULTI-FILE (see lang/runtime/SOURCES). Every .c is compiled
|
|
# once into /tmp/libel.a and reused by all 8 test modules — compile-once,
|
|
# link-many, as prescribed in DESIGN.md. Linking el_runtime.c alone fails
|
|
# at `ld`: it calls into all six engram sibling TUs.
|
|
- name: Precompile runtime into libel.a
|
|
run: |
|
|
set -euo pipefail
|
|
RUNTIME="$(pwd)/runtime"
|
|
rm -rf /tmp/elrt && mkdir -p /tmp/elrt
|
|
for src in $(../scripts/el-runtime-sources.sh --check "$RUNTIME"); do
|
|
gcc -O2 -c -I "$RUNTIME" "$src" -o "/tmp/elrt/$(basename "${src%.c}").o"
|
|
done
|
|
ar rcs /tmp/libel.a /tmp/elrt/*.o
|
|
echo "libel.a built from $(ls /tmp/elrt/*.o | wc -l) translation units"
|
|
|
|
- name: Run tests - native (core)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_core.el > /tmp/el_native_core.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_core.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_core
|
|
/tmp/el_native_core
|
|
|
|
- name: Run tests - native (text)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_text.el > /tmp/el_native_text.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_text.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_text
|
|
/tmp/el_native_text
|
|
|
|
- name: Run tests - native (string)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_string.el > /tmp/el_native_string.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_string.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_string
|
|
/tmp/el_native_string
|
|
|
|
- name: Run tests - native (math)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_math.el > /tmp/el_native_math.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_math.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_math
|
|
/tmp/el_native_math
|
|
|
|
- name: Run tests - native (state)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_state.el > /tmp/el_native_state.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_state.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_state
|
|
/tmp/el_native_state
|
|
|
|
- name: Run tests - native (time)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_time.el > /tmp/el_native_time.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_time.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_time
|
|
/tmp/el_native_time
|
|
|
|
- name: Run tests - native (json)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_json.el > /tmp/el_native_json.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_json.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_json
|
|
/tmp/el_native_json
|
|
|
|
- name: Run tests - native (env)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_env.el > /tmp/el_native_env.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_env.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_env
|
|
/tmp/el_native_env
|
|
|
|
- name: Run tests - native (fs)
|
|
run: |
|
|
set -euo pipefail
|
|
ELC="$(pwd)/dist/platform/elc"
|
|
RUNTIME="$(pwd)/runtime"
|
|
"$ELC" --test tests/native/test_fs.el > /tmp/el_native_fs.c
|
|
gcc -O2 -I "$RUNTIME" /tmp/el_native_fs.c /tmp/libel.a \
|
|
-lcurl -lssl -lcrypto -lpthread -lm -o /tmp/el_native_fs
|
|
/tmp/el_native_fs
|
|
|
|
# Build epm binary using elb (epm lives at repo root, not inside lang/)
|
|
- name: Build epm
|
|
run: |
|
|
ABS_ELB="$(pwd)/dist/bin/elb"
|
|
ABS_ELC="$(pwd)/dist/platform/elc"
|
|
ABS_RUNTIME="$(pwd)/runtime"
|
|
ABS_OUT="$(pwd)/dist/bin"
|
|
(cd ../epm && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
|
|
chmod +x dist/bin/epm
|
|
echo "epm built"
|
|
|
|
# Build el-install binary using elb
|
|
- name: Build el-install
|
|
run: |
|
|
ABS_ELB="$(pwd)/dist/bin/elb"
|
|
ABS_ELC="$(pwd)/dist/platform/elc"
|
|
ABS_RUNTIME="$(pwd)/runtime"
|
|
ABS_OUT="$(pwd)/dist/bin"
|
|
(cd tools/install && "$ABS_ELB" --clean --elc="$ABS_ELC" --runtime="$ABS_RUNTIME" --out="$ABS_OUT")
|
|
chmod +x dist/bin/el-install
|
|
echo "el-install built"
|
|
|
|
# Publish only after merge (push event), not on PR validation runs
|
|
- name: Publish El SDK to Artifact Registry (dev)
|
|
if: github.event_name == 'push'
|
|
env:
|
|
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
|
|
run: |
|
|
# Fail loudly: previously this step had no `set -e`, so an auth or
|
|
# upload failure was swallowed (step exited 0 on the trailing echo)
|
|
# and the SDK silently never published. Surface failures now.
|
|
set -euo pipefail
|
|
if [ -z "${GCP_SA_KEY:-}" ]; then
|
|
echo "FATAL: GCP_SA_KEY secret is empty — cannot authenticate to publish" >&2
|
|
exit 1
|
|
fi
|
|
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
|
|
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
|
|
gcloud config set project neuron-785695
|
|
echo "Publishing as active account: $(gcloud config get-value account 2>/dev/null)"
|
|
|
|
VERSION="${GITHUB_SHA:0:8}"
|
|
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el-elc \
|
|
--version="${VERSION}" \
|
|
--source=dist/platform/elc
|
|
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el-elb \
|
|
--version="${VERSION}" \
|
|
--source=dist/bin/elb
|
|
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el-runtime-c \
|
|
--version="${VERSION}" \
|
|
--source=runtime/el_runtime.c
|
|
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el-runtime-h \
|
|
--version="${VERSION}" \
|
|
--source=runtime/el_runtime.h
|
|
|
|
gcloud artifacts generic upload \
|
|
--repository=foundation-dev \
|
|
--location=us-central1 \
|
|
--project=neuron-785695 \
|
|
--package=el-runtime-js \
|
|
--version="${VERSION}" \
|
|
--source=runtime/el_runtime.js
|
|
|
|
echo "Published El SDK version=${VERSION} to foundation-dev"
|
|
# Keep key alive for the ci-base rebuild step below
|
|
# (deleted in that step after docker push)
|
|
|
|
- name: Rebuild ci-base with fresh El SDK (dev)
|
|
# Patches ci-base:dev in-place: pulls the existing image (which has all
|
|
# system deps — Node, Go, gcloud, Docker CLI, etc.) and overlays the freshly
|
|
# built El SDK on top. Keeps the full ci-base rebuild fast and incremental.
|
|
#
|
|
# continue-on-error: this is a CI-cache optimization, NOT the release
|
|
# artifact. It runs Docker (pull/build/push ~600MB) on the host-mode GCE
|
|
# runner where DinD/Docker availability is fragile. A failure here must
|
|
# never block or redden the job — the SDK publish above is the deliverable.
|
|
continue-on-error: true
|
|
if: github.event_name == 'push'
|
|
env:
|
|
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
|
|
run: |
|
|
set -euo pipefail
|
|
CI_BASE="us-central1-docker.pkg.dev/neuron-785695/neuron-ci/ci-base"
|
|
SHA="${GITHUB_SHA:0:8}"
|
|
|
|
echo "${GCP_SA_KEY}" > /tmp/gcp-key.json
|
|
gcloud auth activate-service-account --key-file=/tmp/gcp-key.json
|
|
gcloud config set project neuron-785695
|
|
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
|
|
|
|
# Pull existing ci-base:dev (or fall back to :latest on first run)
|
|
BASE_TAG="dev"
|
|
docker pull "${CI_BASE}:dev" || { docker pull "${CI_BASE}:latest" && BASE_TAG="latest"; }
|
|
|
|
# Inline Dockerfile — only replaces the El SDK layer
|
|
cat > /tmp/Dockerfile.ci-base-patch << 'EOF'
|
|
ARG BASE
|
|
FROM ${BASE}
|
|
COPY dist/platform/elc /opt/el/dist/platform/elc
|
|
COPY dist/bin/elb /opt/el/dist/bin/elb
|
|
# Whole runtime link set — el_runtime.c alone does not link (it calls
|
|
# into the six engram sibling TUs). See lang/runtime/SOURCES.
|
|
COPY runtime/ /opt/el/runtime/
|
|
COPY runtime/el_runtime.js /opt/el/runtime/el_runtime.js
|
|
RUN chmod +x /opt/el/dist/platform/elc /opt/el/dist/bin/elb
|
|
EOF
|
|
|
|
docker build \
|
|
--build-arg BASE="${CI_BASE}:${BASE_TAG}" \
|
|
--build-arg BUILDKIT_INLINE_CACHE=1 \
|
|
-f /tmp/Dockerfile.ci-base-patch \
|
|
-t "${CI_BASE}:dev" \
|
|
-t "${CI_BASE}:dev-${SHA}" \
|
|
.
|
|
|
|
docker push "${CI_BASE}:dev"
|
|
docker push "${CI_BASE}:dev-${SHA}"
|
|
|
|
echo "ci-base rebuilt: ${CI_BASE}:dev (${SHA})"
|
|
rm -f /tmp/gcp-key.json
|