Build containers run with network: host, so k8s DNS doesn't resolve.
Add extra_hosts entry pointing gitea-proxy.ci.svc.cluster.local to its
ClusterIP (10.43.88.7) in both the will-org and neuron-technologies runner
configs. Without this, actions/checkout fails because Gitea passes the
proxy hostname as GITHUB_SERVER_URL and the build container can't reach it.
Also update stale CF Access comments that no longer reflect the architecture.
Gitea's SSH server is disabled (START_SSH_SERVER=false) so the
SSH URL rewrite breaks git HTTPS clones in build containers. HTTPS
git ops work directly from Legion host network — Cloudflare bypasses
CF Access for smart-HTTP and release asset paths.
CF Access was blocking /api/actions/* even with the correct service-token
headers — the gRPC Actions ping endpoint returns 403 through Cloudflare.
Direct connection to the GKE Network LB IP (34.31.145.131) bypasses CF
entirely. Verified /api/actions/ping returns 200 from the LB IP directly.
Without proxy_ssl_server_name on, nginx connects to Cloudflare's IP
without SNI, triggering handshake_failure (alert 40). Cloudflare requires
SNI to route TLS to the correct origin.
act_runner cannot inject custom HTTP headers, so CF Access blocks its
unauthenticated calls to git.neuralplatform.ai. Add a gitea-proxy
deployment (nginx:alpine) in the ci namespace that injects the CF
Access service-token headers and proxies to https://git.neuralplatform.ai.
Both runner secrets now point GITEA_INSTANCE_URL at the in-cluster proxy
(http://gitea-proxy.ci.svc.cluster.local:3000). Build containers still
clone via SSH through git-ssh-init.sh — unaffected.
Gitea moved to GKE. The old in-cluster URL
http://gitea.git.svc.cluster.local:3000 now has 0 pods behind it.
The runner daemon gets connection refused on every poll.
Switch both ExternalSecrets to https://git.neuralplatform.ai.
The public URL does not require CF Access tokens — the runner can
connect without them. CF_ACCESS_CLIENT_ID/SECRET remain for git
operations inside CI job containers (via BASH_ENV init script).
Vault:
- Annotate vault-helm-gke Service with cloud.google.com/neg exposed_ports
to create container-native NEGs (k8s1-bfbeff02-vault-...) in 3 zones
- Add vault-api-from-lb-gke firewall rule allowing GCP health check ranges
(130.211.0.0/22, 35.191.0.0/16) to reach GKE pod IPs on port 8200
- Replace GCE instance group backends in google_compute_backend_service.vault
with GKE NEG backends (RATE balancing mode, 100 req/endpoint)
- GCP Global HTTPS LB frontend unchanged — DNS stays at 34.54.164.21
- vault.neuralplatform.ai now terminates at GKE pods (all 3 NEGs HEALTHY)
Gitea:
- Change GKE Gitea Service from ClusterIP to LoadBalancer (external IP: 34.31.145.131)
- Add Cloudflare DNS A record for git.neuralplatform.ai → 34.31.145.131 (proxied)
- Remove git.neuralplatform.ai route from Legion Cloudflare tunnel config
- Add Cloudflare config rule: flexible SSL for git.neuralplatform.ai
(origin serves HTTP/3000, CF proxies HTTPS termination)
- Scale Legion Gitea deployment to 0 replicas (PVC preserved)
- git.neuralplatform.ai now serves from GKE Gitea pod
extra_hosts had stale IP 10.43.1.53 (Gitea is now at 10.43.15.98).
force_pull: true ensures runners always use the latest ci-base image.
Applies to both gitea-runner and neuron-technologies-runner.
Gitea lives in the 'git' namespace on Legion k3s, not 'gitea'.
The system gitconfig redirect must use gitea.git.svc.cluster.local:3000.
Update when Gitea migrates to GKE (namespace will be 'gitea' there).
The stale apps/gitea-runner.yaml contained two Deployment manifests
that conflicted with the canonical Deployments owned by the
gitea-runner-config Argo Application (pointing at k8s/gitea-runner/).
Dual ownership caused Argo CD to fight itself — restarting runner
pods mid-job and producing the "context canceled" failures on
neuron-technologies/dharma-el CI.
Canonical Deployments (config-version 2026-05-04-cf-access-public-url,
docker.sock, CF Access env, replicas=2 for nt-runner) live in
k8s/gitea-runner/deployment.yaml and are managed by gitea-runner-config.
- dns-neuralplatform.tf: add vault.neuralplatform.ai A record → 34.54.164.21 (GCP LB)
DNS-only (not proxied) so GCP managed TLS cert can provision correctly
- main.tf: remove vault.neuralplatform.ai from Cloudflare tunnel ingress
(now served directly via GCP Global HTTPS LB)
- main.tf: remove watch.nook.family, jellyfin.nook.family, bazarr.nook.family
from tunnel ingress (nook.family media stack retired; infra is Neuron-focused)
GCE Vault cluster already initialized and running (3-node Raft, active since
2026-05-04T16:05). Secrets migrated 48/48 from k3s vault. ESO ClusterSecretStore
validated against new vault. k3s vault-0 is now superseded.
The containerd socket is mounted from the host at /run/k3s/containerd/containerd.sock
to /var/run/docker.sock inside the container, but act_runner config was pointing to
the host path directly. Fixes CrashLoopBackOff on all runner pods.
Also codifies both deployments (will org + neuron-technologies org) as git-managed
manifests so Argo CD owns them going forward.
The buildah-compiled image (imported directly into k3s) is the working
version. The registry image will catch up after the next CI build.
IfNotPresent ensures the locally-cached correct binary is used.
Soma moved to GCP Cloud Run. The soma-*.yaml files were deleted but
the kustomization still referenced them, blocking all Argo CD syncs
for neuron-prod (including the new allow-daemon-ingress NetworkPolicy).
The ExternalSecret had a stale Gitea URL (192.168.8.148:30322 — old
local IP) and runner tokens were never populated in Vault. Update the
template URL to the cluster-internal Gitea service DNS, and store the
registration tokens in Vault at secret/gitea.runner_token.
backup-verify: restic restores /dump/all-databases.sql relative to
restore target, so the check path was wrong (/all-databases.sql →
/dump/all-databases.sql). Also fix the --include filter to match.
redpanda-topics-init: redpanda is intentionally scaled to 0; the init
job will block forever waiting for a broker. Mark it suspended so Argo
CD won't keep creating a stuck job. Re-enable when redpanda is turned on.
The default-deny-all NetworkPolicy was blocking dharma from reaching
neuron-daemon port 7750 (event queue / engram endpoint). Add an explicit
ingress policy for neuron-daemon that allows traffic from the neuron-prod
namespace, so dharma and other internal services can connect at startup.
- Dockerfile for Go DHARMA server (registry.neuralplatform.ai/neuron-technologies/dharma)
- k8s manifests: deployment, service, ExternalSecret (Vault), IngressRoute
- Argo CD app: deploys to neuron-prod namespace
- Cloudflare tunnel rule for dharma.neurontechnologies.ai
- DNS CNAME record for dharma.neurontechnologies.ai → legion tunnel
- Vault secrets already populated (api_key, encryption_key, jwt_secret, engram_key, engram_url)
- Connects to neuron-daemon (engram) at neuron-daemon.neuron-prod.svc.cluster.local:7749
The registry currently allows anonymous cluster-internal pulls so the
imagePullSecrets reference was failing the externalsecret reconcile
(secret/data/mudcraft has no registry creds for this app). Pod was
pulling fine anyway. If the registry later gets locked down, add a
proper externalsecret using the path from secret/data/<app> instead of
borrowing mudcraft's.
The legion-apps Application root was hardcoded to the OLD legion's gitea
cluster IP (10.43.1.53:3000). On the new GCP legion, the gitea service
got a different cluster IP (10.43.15.98), so the umbrella has been
ComparisonError-stuck since the cluster move on 2026-04-28. That meant
new Application manifests in apps/ never got materialized in argocd —
neuron-web was the first new app to need it after the move.
All other apps reference http://gitea.git.svc.cluster.local:3000/...
(the stable DNS name). Bringing the three legion-apps spots in argocd.tf
in line with that convention so the next gitea cluster-IP shuffle
doesn't re-break the umbrella.
Stand up the elb-built native El landing server on Legion k3s alongside
the existing Cloud Run prod marketing site. Image is built on
gitea-runner-1 from elb-emitted .c files + native cc and pushed to
registry.neuralplatform.ai/neuron-web:dev-dfe41234. Deployment exposes
web-stage.neuralplatform.ai via Traefik + Cloudflare tunnel.
The point is to have a working local instance we can iterate against
while we redesign the CI/CD pipeline; nothing here replaces the Cloud
Run deploy yet.
* k8s/neuron-technologies/web/ — namespace, deployment, service, ingress,
registry-pull-secret + externalsecret pulling anthropic/supabase keys
from Vault
* apps/neuron-web.yaml — Argo CD Application
* dns-neuralplatform.tf, main.tf — CNAME + tunnel ingress for
web-stage.neuralplatform.ai
dev.neurontechnologies.ai mirrors stage in shape but looser:
broken builds OK, auto-deploys on every push to dev branch (once
CI lands), dev Supabase project (isolated), test Stripe keys,
RESEND_DRY_RUN=1 so transactional email never actually sends.
Locked behind Cloudflare Access with team policy: anyone with
@neurontechnologies.ai email plus named external collaborators
via the access app's emails list.
Initial image is marketing:fix-gallery-render-1236 (matches stage
at the time of this commit). Future deploys land via CI workflow.
- ExternalSecret: pull neuron_api_key from Vault into NEURON_API_KEY
- ConfigMap: set OTEL_EXPORTER_OTLP_ENDPOINT to alloy-otlp in-cluster service
instead of localhost (fixes 196 restart OTel connection errors)
Will Anderson
will.anderson