infrastructure/servers/gcp/runners/README.md

# Gitea Actions runner on GCP

A single GCE VM in `us-central1-a` running [`act_runner`](https://gitea.com/gitea/act_runner)
registered against `https://git.neuralplatform.ai`. Workflows that say
`runs-on: ubuntu-latest` are scheduled here.

Source of truth: `servers/gcp/gitea-runner.tf` plus this directory.

## What this runner does

- Picks up `.gitea/workflows/*.yaml` jobs from any Gitea repo whose
  workflow targets one of its labels (`ubuntu-latest`, `ubuntu-22.04`,
  `ubuntu-24.04`).
- Builds Docker images on a 4 vCPU / 16 GiB host with native Docker
  installed (no Docker-in-Docker — `/var/run/docker.sock` is mounted into
  the runner user's group).
- Authenticates to GCP via Workload Identity Federation. Workflows
  exchange a Gitea OIDC token for a short-lived access token impersonating
  the `neuron-ci-pusher` service account. **No long-lived keys live on the
  runner.**

## Identity model

| Identity | Used by | Purpose |
|---|---|---|
| `gitea-runner-vm@neuron-785695.iam.gserviceaccount.com` | The VM at boot | Read the runner registration token from Secret Manager. Nothing else. |
| `neuron-ci-pusher@neuron-785695.iam.gserviceaccount.com` | Workflow steps | Push images to Artifact Registry, deploy Cloud Run revisions, act-as the runtime SAs. Impersonated via WIF. |

Splitting these means a runner-level compromise does not grant deploy access.

## Setup checklist (one-time)

1. **Create the registration token secret** (already done — re-run if rotating):
   ```bash
   TOKEN=$(curl -s -X GET https://git.neuralplatform.ai/api/v1/admin/runners/registration-token \
     -H "Authorization: token $(cat ~/Secrets/api-keys/gitea-api-token)" | jq -r .token)
   echo -n "$TOKEN" | gcloud secrets versions add gitea-runner-token \
     --project=neuron-785695 --data-file=-
   ```

2. **`terraform apply`** in `servers/gcp/`. Outputs you will need:
   ```
   gitea_wif_provider   = projects/<num>/locations/global/workloadIdentityPools/gitea-pool/providers/gitea-oidc
   gitea_wif_deploy_sa  = neuron-ci-pusher@neuron-785695.iam.gserviceaccount.com
   ```

3. **Set Gitea repo secrets** on every repo whose workflows deploy:
   - `GCP_WIF_PROVIDER` — the `gitea_wif_provider` output above
   - `GCP_DEPLOY_SA` — the `gitea_wif_deploy_sa` output above

   Via the Gitea UI: repo → Settings → Actions → Secrets, or via API:
   ```bash
   tea secret set --repo neuron-technologies/neuron-web GCP_WIF_PROVIDER "<provider>"
   tea secret set --repo neuron-technologies/neuron-web GCP_DEPLOY_SA    "<sa-email>"
   ```

4. **Confirm the runner registered**:
   ```bash
   curl -s https://git.neuralplatform.ai/api/v1/admin/runners \
     -H "Authorization: token $(cat ~/Secrets/api-keys/gitea-api-token)"
   ```
   The runner appears as `gcp-us-central1-runner-1` after the VM finishes its
   startup script (~3 minutes from `terraform apply`).

## Adding more runners

Today the count is hard-coded at one. To add a second runner: copy
`google_compute_instance.gitea_runner` in `gitea-runner.tf`, rename to
`gitea_runner_2`, give it the name `gitea-runner-2`, and reuse the same
SA + startup script. (Eventual TODO: parameterize via `count` or
`for_each`.)

## SSH into the runner (no public exposure)

```bash
gcloud compute ssh gitea-runner-1 \
  --zone=us-central1-a \
  --project=neuron-785695 \
  --tunnel-through-iap
```

The firewall only accepts SSH from GCP's IAP CIDR (`35.235.240.0/20`),
so this is the only way in. Your gcloud account needs
`roles/iap.tunnelResourceAccessor` (granted automatically to project
owners).

## Debugging a failing build

On the runner:

```bash
sudo journalctl -u act_runner -f                     # runner daemon logs
sudo cat /var/log/runner-bootstrap.log               # boot-time install log
sudo systemctl status act_runner                     # daemon state
docker ps                                            # in-flight build containers
ls -la /opt/runner                                   # config + .runner state file
```

For a workflow run the live logs are at
`https://git.neuralplatform.ai/<owner>/<repo>/actions`.

## Rotating the registration token

```bash
TOKEN=$(curl -s -X GET https://git.neuralplatform.ai/api/v1/admin/runners/registration-token \
  -H "Authorization: token $(cat ~/Secrets/api-keys/gitea-api-token)" | jq -r .token)
echo -n "$TOKEN" | gcloud secrets versions add gitea-runner-token \
  --project=neuron-785695 --data-file=-

# Re-register (token is single-use). Either taint the VM:
terraform taint google_compute_instance.gitea_runner
terraform apply

# ...or re-run register on the existing VM via IAP SSH:
gcloud compute ssh gitea-runner-1 --zone=us-central1-a --tunnel-through-iap -- "\
  sudo systemctl stop act_runner && \
  sudo -u runner /usr/local/bin/act_runner register --no-interactive \
    --instance https://git.neuralplatform.ai \
    --token $TOKEN \
    --name gcp-us-central1-runner-1 \
    --labels ubuntu-latest:host,ubuntu-22.04:host,ubuntu-24.04:host && \
  sudo systemctl start act_runner"
```

## Fallback: long-lived JSON key (only if WIF isn't usable)

If Gitea's OIDC discovery endpoint isn't reachable from
`oauth2.googleapis.com` (e.g. CF Access blocks `.well-known/openid-configuration`
to anonymous fetchers), workflows can't use WIF. Fall back to the JSON
key flow:

```bash
gcloud iam service-accounts keys create /tmp/ci-pusher.json \
  --iam-account=neuron-ci-pusher@neuron-785695.iam.gserviceaccount.com
tea secret set --repo neuron-technologies/neuron-web GCP_SA_KEY < /tmp/ci-pusher.json
rm /tmp/ci-pusher.json
```

Then in `deploy.yaml`, swap the `auth` step's `workload_identity_provider`/
`service_account` inputs for `credentials_json: ${{ secrets.GCP_SA_KEY }}`.

The WIF resources stay in Terraform either way — the fallback is purely
in the workflow. WIF remains the target.

## Cost

`e2-standard-4` in `us-central1` is roughly **$100/mo** sustained-use,
plus a few cents for the GCS bucket and the ephemeral IP. Not free. If
that's too much, downsize to `e2-standard-2` (~$50/mo) — most builds
fit, only the heaviest C++/Rust compiles need 16 GiB.