ci: add gitea actions runner on GCP with WIF-backed deploy SA
A single e2-standard-4 GCE VM in us-central1-a runs act_runner, registered to git.neuralplatform.ai. Workload Identity Federation binds the runner to the existing neuron-ci-pusher SA (artifactregistry.writer + run.developer + serviceAccountUser on the runtime SAs). No long-lived JSON keys live on the runner; the GCP_SA_KEY secret remains as a fallback in case the Gitea OIDC issuer isn't reachable from oauth2.googleapis.com. The runner VM has its own minimal-scope SA (gitea-runner-vm) that can only read the registration token from Secret Manager — splitting boot identity from deploy identity so a runner compromise doesn't grant push. SSH is IAP-only (no public ingress on :22). Reference workflow lives at products/web/.gitea/workflows/deploy.yaml and takes manual gcloud out of the loop: push to main triggers build, push to AR, parallel deploy to all 3 marketing prod regions, traffic flip, smoke check.
This commit is contained in:
Will Anderson
@@ -0,0 +1,150 @@
|
||||
# Gitea Actions runner on GCP
|
||||
|
||||
A single GCE VM in `us-central1-a` running [`act_runner`](https://gitea.com/gitea/act_runner)
|
||||
registered against `https://git.neuralplatform.ai`. Workflows that say
|
||||
`runs-on: ubuntu-latest` are scheduled here.
|
||||
|
||||
Source of truth: `servers/gcp/gitea-runner.tf` plus this directory.
|
||||
|
||||
## What this runner does
|
||||
|
||||
- Picks up `.gitea/workflows/*.yaml` jobs from any Gitea repo whose
|
||||
workflow targets one of its labels (`ubuntu-latest`, `ubuntu-22.04`,
|
||||
`ubuntu-24.04`).
|
||||
- Builds Docker images on a 4 vCPU / 16 GiB host with native Docker
|
||||
installed (no Docker-in-Docker — `/var/run/docker.sock` is mounted into
|
||||
the runner user's group).
|
||||
- Authenticates to GCP via Workload Identity Federation. Workflows
|
||||
exchange a Gitea OIDC token for a short-lived access token impersonating
|
||||
the `neuron-ci-pusher` service account. **No long-lived keys live on the
|
||||
runner.**
|
||||
|
||||
## Identity model
|
||||
|
||||
| Identity | Used by | Purpose |
|
||||
|---|---|---|
|
||||
| `gitea-runner-vm@neuron-785695.iam.gserviceaccount.com` | The VM at boot | Read the runner registration token from Secret Manager. Nothing else. |
|
||||
| `neuron-ci-pusher@neuron-785695.iam.gserviceaccount.com` | Workflow steps | Push images to Artifact Registry, deploy Cloud Run revisions, act-as the runtime SAs. Impersonated via WIF. |
|
||||
|
||||
Splitting these means a runner-level compromise does not grant deploy access.
|
||||
|
||||
## Setup checklist (one-time)
|
||||
|
||||
1. **Create the registration token secret** (already done — re-run if rotating):
|
||||
```bash
|
||||
TOKEN=$(curl -s -X GET https://git.neuralplatform.ai/api/v1/admin/runners/registration-token \
|
||||
-H "Authorization: token $(cat ~/Secrets/api-keys/gitea-api-token)" | jq -r .token)
|
||||
echo -n "$TOKEN" | gcloud secrets versions add gitea-runner-token \
|
||||
--project=neuron-785695 --data-file=-
|
||||
```
|
||||
|
||||
2. **`terraform apply`** in `servers/gcp/`. Outputs you will need:
|
||||
```
|
||||
gitea_wif_provider = projects/<num>/locations/global/workloadIdentityPools/gitea-pool/providers/gitea-oidc
|
||||
gitea_wif_deploy_sa = neuron-ci-pusher@neuron-785695.iam.gserviceaccount.com
|
||||
```
|
||||
|
||||
3. **Set Gitea repo secrets** on every repo whose workflows deploy:
|
||||
- `GCP_WIF_PROVIDER` — the `gitea_wif_provider` output above
|
||||
- `GCP_DEPLOY_SA` — the `gitea_wif_deploy_sa` output above
|
||||
|
||||
Via the Gitea UI: repo → Settings → Actions → Secrets, or via API:
|
||||
```bash
|
||||
tea secret set --repo neuron-technologies/neuron-web GCP_WIF_PROVIDER "<provider>"
|
||||
tea secret set --repo neuron-technologies/neuron-web GCP_DEPLOY_SA "<sa-email>"
|
||||
```
|
||||
|
||||
4. **Confirm the runner registered**:
|
||||
```bash
|
||||
curl -s https://git.neuralplatform.ai/api/v1/admin/runners \
|
||||
-H "Authorization: token $(cat ~/Secrets/api-keys/gitea-api-token)"
|
||||
```
|
||||
The runner appears as `gcp-us-central1-runner-1` after the VM finishes its
|
||||
startup script (~3 minutes from `terraform apply`).
|
||||
|
||||
## Adding more runners
|
||||
|
||||
Today the count is hard-coded at one. To add a second runner: copy
|
||||
`google_compute_instance.gitea_runner` in `gitea-runner.tf`, rename to
|
||||
`gitea_runner_2`, give it the name `gitea-runner-2`, and reuse the same
|
||||
SA + startup script. (Eventual TODO: parameterize via `count` or
|
||||
`for_each`.)
|
||||
|
||||
## SSH into the runner (no public exposure)
|
||||
|
||||
```bash
|
||||
gcloud compute ssh gitea-runner-1 \
|
||||
--zone=us-central1-a \
|
||||
--project=neuron-785695 \
|
||||
--tunnel-through-iap
|
||||
```
|
||||
|
||||
The firewall only accepts SSH from GCP's IAP CIDR (`35.235.240.0/20`),
|
||||
so this is the only way in. Your gcloud account needs
|
||||
`roles/iap.tunnelResourceAccessor` (granted automatically to project
|
||||
owners).
|
||||
|
||||
## Debugging a failing build
|
||||
|
||||
On the runner:
|
||||
|
||||
```bash
|
||||
sudo journalctl -u act_runner -f # runner daemon logs
|
||||
sudo cat /var/log/runner-bootstrap.log # boot-time install log
|
||||
sudo systemctl status act_runner # daemon state
|
||||
docker ps # in-flight build containers
|
||||
ls -la /opt/runner # config + .runner state file
|
||||
```
|
||||
|
||||
For a workflow run the live logs are at
|
||||
`https://git.neuralplatform.ai/<owner>/<repo>/actions`.
|
||||
|
||||
## Rotating the registration token
|
||||
|
||||
```bash
|
||||
TOKEN=$(curl -s -X GET https://git.neuralplatform.ai/api/v1/admin/runners/registration-token \
|
||||
-H "Authorization: token $(cat ~/Secrets/api-keys/gitea-api-token)" | jq -r .token)
|
||||
echo -n "$TOKEN" | gcloud secrets versions add gitea-runner-token \
|
||||
--project=neuron-785695 --data-file=-
|
||||
|
||||
# Re-register (token is single-use). Either taint the VM:
|
||||
terraform taint google_compute_instance.gitea_runner
|
||||
terraform apply
|
||||
|
||||
# ...or re-run register on the existing VM via IAP SSH:
|
||||
gcloud compute ssh gitea-runner-1 --zone=us-central1-a --tunnel-through-iap -- "\
|
||||
sudo systemctl stop act_runner && \
|
||||
sudo -u runner /usr/local/bin/act_runner register --no-interactive \
|
||||
--instance https://git.neuralplatform.ai \
|
||||
--token $TOKEN \
|
||||
--name gcp-us-central1-runner-1 \
|
||||
--labels ubuntu-latest:host,ubuntu-22.04:host,ubuntu-24.04:host && \
|
||||
sudo systemctl start act_runner"
|
||||
```
|
||||
|
||||
## Fallback: long-lived JSON key (only if WIF isn't usable)
|
||||
|
||||
If Gitea's OIDC discovery endpoint isn't reachable from
|
||||
`oauth2.googleapis.com` (e.g. CF Access blocks `.well-known/openid-configuration`
|
||||
to anonymous fetchers), workflows can't use WIF. Fall back to the JSON
|
||||
key flow:
|
||||
|
||||
```bash
|
||||
gcloud iam service-accounts keys create /tmp/ci-pusher.json \
|
||||
--iam-account=neuron-ci-pusher@neuron-785695.iam.gserviceaccount.com
|
||||
tea secret set --repo neuron-technologies/neuron-web GCP_SA_KEY < /tmp/ci-pusher.json
|
||||
rm /tmp/ci-pusher.json
|
||||
```
|
||||
|
||||
Then in `deploy.yaml`, swap the `auth` step's `workload_identity_provider`/
|
||||
`service_account` inputs for `credentials_json: ${{ secrets.GCP_SA_KEY }}`.
|
||||
|
||||
The WIF resources stay in Terraform either way — the fallback is purely
|
||||
in the workflow. WIF remains the target.
|
||||
|
||||
## Cost
|
||||
|
||||
`e2-standard-4` in `us-central1` is roughly **$100/mo** sustained-use,
|
||||
plus a few cents for the GCS bucket and the ephemeral IP. Not free. If
|
||||
that's too much, downsize to `e2-standard-2` (~$50/mo) — most builds
|
||||
fit, only the heaviest C++/Rust compiles need 16 GiB.
|
||||
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env bash
|
||||
# servers/gcp/runners/startup.sh
|
||||
#
|
||||
# Boot script for the Gitea Actions runner GCE VM.
|
||||
# - Installs docker + git + jq + the Forgejo/Gitea act_runner binary.
|
||||
# - Pulls the runner registration token from GCP Secret Manager
|
||||
# (the VM's attached service account has secretmanager.secretAccessor on it).
|
||||
# - Registers against https://git.neuralplatform.ai with labels matching
|
||||
# the workflows that target ubuntu-latest hosts.
|
||||
# - Starts act_runner under systemd so it auto-recovers on reboot.
|
||||
#
|
||||
# Logs land at /var/log/runner-bootstrap.log for debugging from IAP SSH.
|
||||
|
||||
set -euxo pipefail
|
||||
exec > >(tee /var/log/runner-bootstrap.log) 2>&1
|
||||
|
||||
apt-get update
|
||||
apt-get install -y curl ca-certificates docker.io git jq
|
||||
|
||||
# Make docker usable by the unprivileged runner user
|
||||
systemctl enable --now docker
|
||||
|
||||
useradd -m -s /bin/bash runner || true
|
||||
usermod -aG docker runner
|
||||
|
||||
# act_runner — pinned to a known-good release. Bump RUNNER_VERSION when
|
||||
# upgrading. The project moved from gitea/act_runner to gitea/runner around
|
||||
# the 0.6.x series; the binary inside the asset is still called act_runner.
|
||||
# Latest releases at: https://gitea.com/gitea/runner/releases
|
||||
RUNNER_VERSION="0.6.1"
|
||||
curl -fsSL \
|
||||
"https://gitea.com/gitea/runner/releases/download/v${RUNNER_VERSION}/act_runner-${RUNNER_VERSION}-linux-amd64" \
|
||||
-o /usr/local/bin/act_runner
|
||||
chmod +x /usr/local/bin/act_runner
|
||||
|
||||
mkdir -p /opt/runner
|
||||
chown -R runner:runner /opt/runner
|
||||
|
||||
# Pull registration token. Fail fast if the secret is missing — the runner
|
||||
# is useless without it, and a silent fallback would leave us debugging an
|
||||
# unregistered VM later.
|
||||
GITEA_RUNNER_TOKEN="$(gcloud secrets versions access latest \
|
||||
--secret=gitea-runner-token \
|
||||
--project=neuron-785695)"
|
||||
|
||||
# Generate a baseline config so the daemon has a sane starting point. The
|
||||
# register step writes the .runner state file into /opt/runner.
|
||||
sudo -u runner /usr/local/bin/act_runner generate-config > /opt/runner/config.yaml
|
||||
chown runner:runner /opt/runner/config.yaml
|
||||
|
||||
cd /opt/runner
|
||||
sudo -u runner /usr/local/bin/act_runner register \
|
||||
--no-interactive \
|
||||
--instance https://git.neuralplatform.ai \
|
||||
--token "${GITEA_RUNNER_TOKEN}" \
|
||||
--name "gcp-us-central1-runner-1" \
|
||||
--labels "ubuntu-latest,ubuntu-22.04,ubuntu-24.04"
|
||||
|
||||
# Belt-and-braces: act_runner v0.6 has a habit of wrapping bare labels with
|
||||
# the default `:docker://docker.gitea.com/runner-images:<label>` shape on
|
||||
# first daemon start. We want host execution (build-stage.sh uses the
|
||||
# host's Docker socket), so re-write the labels in the persisted state
|
||||
# file before we start the daemon.
|
||||
python3 - <<'PY'
|
||||
import json
|
||||
p = "/opt/runner/.runner"
|
||||
with open(p) as f: d = json.load(f)
|
||||
d["labels"] = ["ubuntu-latest", "ubuntu-22.04", "ubuntu-24.04"]
|
||||
with open(p, "w") as f: json.dump(d, f, indent=2)
|
||||
PY
|
||||
chown runner:runner /opt/runner/.runner
|
||||
|
||||
cat > /etc/systemd/system/act_runner.service <<'EOF'
|
||||
[Unit]
|
||||
Description=Gitea Actions runner (act_runner)
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=runner
|
||||
WorkingDirectory=/opt/runner
|
||||
ExecStart=/usr/local/bin/act_runner daemon --config /opt/runner/config.yaml
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
# Pass through GCP metadata-server access so workflows can call gcloud
|
||||
Environment=HOME=/home/runner
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now act_runner
|
||||
|
||||
echo "act_runner registered and running"
|
||||
Reference in New Issue
Block a user