Switch gcp/ backend from Cloudflare R2 (S3-compatible) to GCS — uses
Application Default Credentials automatically, no extra env vars needed.
State has been migrated to gs://neuron-785695-terraform-state/gcp/.
Remove GCE Vault node resources (vault-node-1/2/3) from vault-nodes.tf
and Terraform state. GKE Vault (Workload Identity + vault-unseal SA) has
been serving traffic since the NEG cutover; GCE nodes are confirmed dead.
Removed from state and config:
- google_compute_instance.vault_node (x3)
- google_compute_disk.vault_data (x3)
- google_compute_instance_group.vault (x3)
- google_compute_firewall.vault_iap_ssh / vault_raft / vault_api (GCE-targeted)
- google_storage_bucket_object.vault_startup
- google_storage_bucket_iam_member.vault_node_bucket_read
- google_project_iam_member.vault_node_{log_writer,metric_writer,compute_viewer}
Kept: KMS, vault-unseal SA, WI binding, NEG data sources, GKE firewall,
global LB (address, cert, backend service, url maps, proxies, fwd rules).
Set replicas: 0 to prevent crash-loop restarts. The runner requires a Docker
daemon (even in host mode) which Autopilot won't provide. Scale to 1 when a
Standard node pool or external Docker TCP endpoint is available.
Switch to host-mode labels and document the DinD limitation: Autopilot's
Warden blocks privileged containers at the node level. Use internal Gitea
service URL (gitea.gitea.svc.cluster.local) to bypass Cloudflare Access.
Document three resolution paths (Standard node pool, Cloud Build, external
Docker TCP). GCE VM runner remains active in the interim.
Replaces the GCE VM runner with a k8s Deployment in the ci namespace on
neuron-platform GKE. Uses Docker-in-Docker for build isolation since
Autopilot doesn't expose the node socket. Runner token pulled from Secret
Manager via ESO + Workload Identity.
- servers/gcp/k8s/gitea-runner/: namespace, serviceaccount, external-secrets,
deployment manifests (ci namespace, dind sidecar, idempotent registration)
- servers/gcp/k8s/argocd-apps/gitea-runner-gke.yaml: Argo CD Application
- servers/gcp/gitea-runner.tf: gitea-runner-gke GCP SA with secretAccessor
on gitea-runner-token, Workload Identity binding for ci/gitea-runner,
artifactregistry.reader for pulling ci-base image
Build containers run with network: host, so k8s DNS doesn't resolve.
Add extra_hosts entry pointing gitea-proxy.ci.svc.cluster.local to its
ClusterIP (10.43.88.7) in both the will-org and neuron-technologies runner
configs. Without this, actions/checkout fails because Gitea passes the
proxy hostname as GITHUB_SERVER_URL and the build container can't reach it.
Also update stale CF Access comments that no longer reflect the architecture.
Gitea's SSH server is disabled (START_SSH_SERVER=false) so the
SSH URL rewrite breaks git HTTPS clones in build containers. HTTPS
git ops work directly from Legion host network — Cloudflare bypasses
CF Access for smart-HTTP and release asset paths.
CF Access was blocking /api/actions/* even with the correct service-token
headers — the gRPC Actions ping endpoint returns 403 through Cloudflare.
Direct connection to the GKE Network LB IP (34.31.145.131) bypasses CF
entirely. Verified /api/actions/ping returns 200 from the LB IP directly.
Argo CD's bundled schema is missing terminatingReplicas from the
Deployment status spec, causing a ComparisonError on every sync.
Ignoring /status entirely is safe — Argo CD never manages status
fields; they're owned by kube-controller-manager.
gitea-gke and vault-gke were pointing at the dead k3s-internal Gitea
(http://gitea.git.svc.cluster.local:3000). gke-apps already uses
https://git.neuralplatform.ai and syncs successfully — update child apps
to match. Once gke-apps picks this up, gitea-gke will sync the pending
GITEA__actions__ENABLED=true commit.
Without proxy_ssl_server_name on, nginx connects to Cloudflare's IP
without SNI, triggering handshake_failure (alert 40). Cloudflare requires
SNI to route TLS to the correct origin.
Pods 1 and 2 can't unseal because port 8201 (cluster port) only opens
after barrier initialization, but the barrier key transfer requires 8201
to be reachable — deadlock. Running single-node on pod 0 so
vault.neuralplatform.ai reliably routes to the active, unsealed instance.
act_runner cannot inject custom HTTP headers, so CF Access blocks its
unauthenticated calls to git.neuralplatform.ai. Add a gitea-proxy
deployment (nginx:alpine) in the ci namespace that injects the CF
Access service-token headers and proxies to https://git.neuralplatform.ai.
Both runner secrets now point GITEA_INSTANCE_URL at the in-cluster proxy
(http://gitea-proxy.ci.svc.cluster.local:3000). Build containers still
clone via SSH through git-ssh-init.sh — unaffected.
Gitea moved to GKE. The old in-cluster URL
http://gitea.git.svc.cluster.local:3000 now has 0 pods behind it.
The runner daemon gets connection refused on every poll.
Switch both ExternalSecrets to https://git.neuralplatform.ai.
The public URL does not require CF Access tokens — the runner can
connect without them. CF_ACCESS_CLIENT_ID/SECRET remain for git
operations inside CI job containers (via BASH_ENV init script).
Vault:
- Annotate vault-helm-gke Service with cloud.google.com/neg exposed_ports
to create container-native NEGs (k8s1-bfbeff02-vault-...) in 3 zones
- Add vault-api-from-lb-gke firewall rule allowing GCP health check ranges
(130.211.0.0/22, 35.191.0.0/16) to reach GKE pod IPs on port 8200
- Replace GCE instance group backends in google_compute_backend_service.vault
with GKE NEG backends (RATE balancing mode, 100 req/endpoint)
- GCP Global HTTPS LB frontend unchanged — DNS stays at 34.54.164.21
- vault.neuralplatform.ai now terminates at GKE pods (all 3 NEGs HEALTHY)
Gitea:
- Change GKE Gitea Service from ClusterIP to LoadBalancer (external IP: 34.31.145.131)
- Add Cloudflare DNS A record for git.neuralplatform.ai → 34.31.145.131 (proxied)
- Remove git.neuralplatform.ai route from Legion Cloudflare tunnel config
- Add Cloudflare config rule: flexible SSL for git.neuralplatform.ai
(origin serves HTTP/3000, CF proxies HTTPS termination)
- Scale Legion Gitea deployment to 0 replicas (PVC preserved)
- git.neuralplatform.ai now serves from GKE Gitea pod
extra_hosts had stale IP 10.43.1.53 (Gitea is now at 10.43.15.98).
force_pull: true ensures runners always use the latest ci-base image.
Applies to both gitea-runner and neuron-technologies-runner.
SSD quota (SSD_TOTAL_GB) is exhausted at 500/500 GB. pd-ssd PVCs for vault-1
and vault-2 cannot provision. Switching to standard-rwo (pd-balanced) unblocks
the Vault HA cluster immediately.
Quota increase request submitted (Cloud Quotas preference API, preference ID
0aa576b1-5050-4fa7-911d-6f408230df7f, target 1000GB). Once approved, can
optionally migrate back to premium-rwo for all-SSD Vault storage.
vault-1 and vault-2 are stuck Pending because GCE quota (SSD_TOTAL_GB 500/500)
prevents new nodes from provisioning. ScheduleAnyway lets them land in any zone
that has capacity while the quota increase request is pending.
Also adds ESO Workload Identity IAM bindings to Terraform state (previously applied
out-of-band via gcloud; now tracked in cloud-sql.tf).
- Replace GKE_CLUSTER_ENDPOINT placeholder with real endpoint (34.63.89.52)
in all three Argo CD Application manifests
- Fix gitea GCP SA account_id from 'gitea' (5 chars, too short) to 'gitea-gke'
to satisfy GCP's 6-30 char constraint
- Update serviceaccount.yaml annotation to match new SA email (gitea-gke@...)
Gitea lives in the 'git' namespace on Legion k3s, not 'gitea'.
The system gitconfig redirect must use gitea.git.svc.cluster.local:3000.
Update when Gitea migrates to GKE (namespace will be 'gitea' there).
The stale apps/gitea-runner.yaml contained two Deployment manifests
that conflicted with the canonical Deployments owned by the
gitea-runner-config Argo Application (pointing at k8s/gitea-runner/).
Dual ownership caused Argo CD to fight itself — restarting runner
pods mid-job and producing the "context canceled" failures on
neuron-technologies/dharma-el CI.
Canonical Deployments (config-version 2026-05-04-cf-access-public-url,
docker.sock, CF Access env, replicas=2 for nt-runner) live in
k8s/gitea-runner/deployment.yaml and are managed by gitea-runner-config.
- dns-neuralplatform.tf: add vault.neuralplatform.ai A record → 34.54.164.21 (GCP LB)
DNS-only (not proxied) so GCP managed TLS cert can provision correctly
- main.tf: remove vault.neuralplatform.ai from Cloudflare tunnel ingress
(now served directly via GCP Global HTTPS LB)
- main.tf: remove watch.nook.family, jellyfin.nook.family, bazarr.nook.family
from tunnel ingress (nook.family media stack retired; infra is Neuron-focused)
GCE Vault cluster already initialized and running (3-node Raft, active since
2026-05-04T16:05). Secrets migrated 48/48 from k3s vault. ESO ClusterSecretStore
validated against new vault. k3s vault-0 is now superseded.
The containerd socket is mounted from the host at /run/k3s/containerd/containerd.sock
to /var/run/docker.sock inside the container, but act_runner config was pointing to
the host path directly. Fixes CrashLoopBackOff on all runner pods.
Also codifies both deployments (will org + neuron-technologies org) as git-managed
manifests so Argo CD owns them going forward.
The buildah-compiled image (imported directly into k3s) is the working
version. The registry image will catch up after the next CI build.
IfNotPresent ensures the locally-cached correct binary is used.
Soma moved to GCP Cloud Run. The soma-*.yaml files were deleted but
the kustomization still referenced them, blocking all Argo CD syncs
for neuron-prod (including the new allow-daemon-ingress NetworkPolicy).
The ExternalSecret had a stale Gitea URL (192.168.8.148:30322 — old
local IP) and runner tokens were never populated in Vault. Update the
template URL to the cluster-internal Gitea service DNS, and store the
registration tokens in Vault at secret/gitea.runner_token.
Will Anderson
will.anderson