Gitea moved to GKE. The old in-cluster URL
http://gitea.git.svc.cluster.local:3000 now has 0 pods behind it.
The runner daemon gets connection refused on every poll.
Switch both ExternalSecrets to https://git.neuralplatform.ai.
The public URL does not require CF Access tokens — the runner can
connect without them. CF_ACCESS_CLIENT_ID/SECRET remain for git
operations inside CI job containers (via BASH_ENV init script).
Vault:
- Annotate vault-helm-gke Service with cloud.google.com/neg exposed_ports
to create container-native NEGs (k8s1-bfbeff02-vault-...) in 3 zones
- Add vault-api-from-lb-gke firewall rule allowing GCP health check ranges
(130.211.0.0/22, 35.191.0.0/16) to reach GKE pod IPs on port 8200
- Replace GCE instance group backends in google_compute_backend_service.vault
with GKE NEG backends (RATE balancing mode, 100 req/endpoint)
- GCP Global HTTPS LB frontend unchanged — DNS stays at 34.54.164.21
- vault.neuralplatform.ai now terminates at GKE pods (all 3 NEGs HEALTHY)
Gitea:
- Change GKE Gitea Service from ClusterIP to LoadBalancer (external IP: 34.31.145.131)
- Add Cloudflare DNS A record for git.neuralplatform.ai → 34.31.145.131 (proxied)
- Remove git.neuralplatform.ai route from Legion Cloudflare tunnel config
- Add Cloudflare config rule: flexible SSL for git.neuralplatform.ai
(origin serves HTTP/3000, CF proxies HTTPS termination)
- Scale Legion Gitea deployment to 0 replicas (PVC preserved)
- git.neuralplatform.ai now serves from GKE Gitea pod
extra_hosts had stale IP 10.43.1.53 (Gitea is now at 10.43.15.98).
force_pull: true ensures runners always use the latest ci-base image.
Applies to both gitea-runner and neuron-technologies-runner.
SSD quota (SSD_TOTAL_GB) is exhausted at 500/500 GB. pd-ssd PVCs for vault-1
and vault-2 cannot provision. Switching to standard-rwo (pd-balanced) unblocks
the Vault HA cluster immediately.
Quota increase request submitted (Cloud Quotas preference API, preference ID
0aa576b1-5050-4fa7-911d-6f408230df7f, target 1000GB). Once approved, can
optionally migrate back to premium-rwo for all-SSD Vault storage.
vault-1 and vault-2 are stuck Pending because GCE quota (SSD_TOTAL_GB 500/500)
prevents new nodes from provisioning. ScheduleAnyway lets them land in any zone
that has capacity while the quota increase request is pending.
Also adds ESO Workload Identity IAM bindings to Terraform state (previously applied
out-of-band via gcloud; now tracked in cloud-sql.tf).
- Replace GKE_CLUSTER_ENDPOINT placeholder with real endpoint (34.63.89.52)
in all three Argo CD Application manifests
- Fix gitea GCP SA account_id from 'gitea' (5 chars, too short) to 'gitea-gke'
to satisfy GCP's 6-30 char constraint
- Update serviceaccount.yaml annotation to match new SA email (gitea-gke@...)
Gitea lives in the 'git' namespace on Legion k3s, not 'gitea'.
The system gitconfig redirect must use gitea.git.svc.cluster.local:3000.
Update when Gitea migrates to GKE (namespace will be 'gitea' there).
The stale apps/gitea-runner.yaml contained two Deployment manifests
that conflicted with the canonical Deployments owned by the
gitea-runner-config Argo Application (pointing at k8s/gitea-runner/).
Dual ownership caused Argo CD to fight itself — restarting runner
pods mid-job and producing the "context canceled" failures on
neuron-technologies/dharma-el CI.
Canonical Deployments (config-version 2026-05-04-cf-access-public-url,
docker.sock, CF Access env, replicas=2 for nt-runner) live in
k8s/gitea-runner/deployment.yaml and are managed by gitea-runner-config.
- dns-neuralplatform.tf: add vault.neuralplatform.ai A record → 34.54.164.21 (GCP LB)
DNS-only (not proxied) so GCP managed TLS cert can provision correctly
- main.tf: remove vault.neuralplatform.ai from Cloudflare tunnel ingress
(now served directly via GCP Global HTTPS LB)
- main.tf: remove watch.nook.family, jellyfin.nook.family, bazarr.nook.family
from tunnel ingress (nook.family media stack retired; infra is Neuron-focused)
GCE Vault cluster already initialized and running (3-node Raft, active since
2026-05-04T16:05). Secrets migrated 48/48 from k3s vault. ESO ClusterSecretStore
validated against new vault. k3s vault-0 is now superseded.
The containerd socket is mounted from the host at /run/k3s/containerd/containerd.sock
to /var/run/docker.sock inside the container, but act_runner config was pointing to
the host path directly. Fixes CrashLoopBackOff on all runner pods.
Also codifies both deployments (will org + neuron-technologies org) as git-managed
manifests so Argo CD owns them going forward.
The buildah-compiled image (imported directly into k3s) is the working
version. The registry image will catch up after the next CI build.
IfNotPresent ensures the locally-cached correct binary is used.
Soma moved to GCP Cloud Run. The soma-*.yaml files were deleted but
the kustomization still referenced them, blocking all Argo CD syncs
for neuron-prod (including the new allow-daemon-ingress NetworkPolicy).
The ExternalSecret had a stale Gitea URL (192.168.8.148:30322 — old
local IP) and runner tokens were never populated in Vault. Update the
template URL to the cluster-internal Gitea service DNS, and store the
registration tokens in Vault at secret/gitea.runner_token.
backup-verify: restic restores /dump/all-databases.sql relative to
restore target, so the check path was wrong (/all-databases.sql →
/dump/all-databases.sql). Also fix the --include filter to match.
redpanda-topics-init: redpanda is intentionally scaled to 0; the init
job will block forever waiting for a broker. Mark it suspended so Argo
CD won't keep creating a stuck job. Re-enable when redpanda is turned on.
The default-deny-all NetworkPolicy was blocking dharma from reaching
neuron-daemon port 7750 (event queue / engram endpoint). Add an explicit
ingress policy for neuron-daemon that allows traffic from the neuron-prod
namespace, so dharma and other internal services can connect at startup.
Legion is offline. Vault needs a GCP-native home. This commit provisions
a 3-node Vault HA cluster on GCE e2-small VMs (us-central1 a/b/c) backed
by Raft consensus with GCP KMS auto-unseal already in place.
- vault-nodes.tf: 3 VMs with separate pd-ssd data disks (prevent_destroy),
IAP-only SSH firewall, Raft peer firewall (8201), global HTTPS LB for
vault.neuralplatform.ai with Google-managed cert, HTTP health checks
against /v1/sys/health
- vault/startup.sh: idempotent boot script — installs Vault from HashiCorp
APT repo, mounts/formats the data disk, writes Raft config with dynamic
retry_join peers resolved via gcloud metadata, starts vault.service
- vault-auth-gcp.tf: GCP Secret Manager secret for init script; outputs
the service account emails needed for GCP auth roles
- vault/configure-vault-auth.sh: post-init script to enable gcp auth
method, write marketing-policy and soma-policy, bind roles to Cloud Run
service account identities
After apply: set vault_lb_ip output as A record for vault.neuralplatform.ai
in Cloudflare, then run vault operator init on vault-node-1 to bootstrap.
vault-kms.tf: KMS keyring/key for Vault auto-unseal + vault-unseal SA
and IAM bindings — all imported from existing GCP resources.
prevent_destroy=true on the crypto key to protect against accidental deletion.
unmanaged-secrets.tf: 8 secrets that were referenced by live Cloud Run
services but had no TF resource — anthropic-api-key, resend-api-key,
supabase-service-key, docuseal-webhook-token, stripe-price-professional,
stripe-price-founding, stripe-price-family-child, gitea-runner-token.
service-account.tf: 4 manually-created SAs — neuron-workspace-dwd,
docuseal-storage, analytics-reader, harmonic-search-console.
DOCUSEAL_WEBHOOK_TOKEN was present on all live marketing services but
missing from cloud-run.tf and cloud-run-stage.tf — next apply would
have stripped it and broken DocuSeal webhook validation.
TOGETHER_API_KEY (soma-together-api-key) was present on soma-prod-us
but missing from cloud-run-soma.tf — next apply would have broken the
Together.ai inference routing path.
- Dockerfile for Go DHARMA server (registry.neuralplatform.ai/neuron-technologies/dharma)
- k8s manifests: deployment, service, ExternalSecret (Vault), IngressRoute
- Argo CD app: deploys to neuron-prod namespace
- Cloudflare tunnel rule for dharma.neurontechnologies.ai
- DNS CNAME record for dharma.neurontechnologies.ai → legion tunnel
- Vault secrets already populated (api_key, encryption_key, jwt_secret, engram_key, engram_url)
- Connects to neuron-daemon (engram) at neuron-daemon.neuron-prod.svc.cluster.local:7749
The registry currently allows anonymous cluster-internal pulls so the
imagePullSecrets reference was failing the externalsecret reconcile
(secret/data/mudcraft has no registry creds for this app). Pod was
pulling fine anyway. If the registry later gets locked down, add a
proper externalsecret using the path from secret/data/<app> instead of
borrowing mudcraft's.
The legion-apps Application root was hardcoded to the OLD legion's gitea
cluster IP (10.43.1.53:3000). On the new GCP legion, the gitea service
got a different cluster IP (10.43.15.98), so the umbrella has been
ComparisonError-stuck since the cluster move on 2026-04-28. That meant
new Application manifests in apps/ never got materialized in argocd —
neuron-web was the first new app to need it after the move.
All other apps reference http://gitea.git.svc.cluster.local:3000/...
(the stable DNS name). Bringing the three legion-apps spots in argocd.tf
in line with that convention so the next gitea cluster-IP shuffle
doesn't re-break the umbrella.
Stand up the elb-built native El landing server on Legion k3s alongside
the existing Cloud Run prod marketing site. Image is built on
gitea-runner-1 from elb-emitted .c files + native cc and pushed to
registry.neuralplatform.ai/neuron-web:dev-dfe41234. Deployment exposes
web-stage.neuralplatform.ai via Traefik + Cloudflare tunnel.
The point is to have a working local instance we can iterate against
while we redesign the CI/CD pipeline; nothing here replaces the Cloud
Run deploy yet.
* k8s/neuron-technologies/web/ — namespace, deployment, service, ingress,
registry-pull-secret + externalsecret pulling anthropic/supabase keys
from Vault
* apps/neuron-web.yaml — Argo CD Application
* dns-neuralplatform.tf, main.tf — CNAME + tunnel ingress for
web-stage.neuralplatform.ai
cloud-run.tf + cloud-run-stage.tf: small alignment edits from the
dev-env agent's work to match the actual deployed Cloud Run shape.
runners/startup.sh: 10-line additions from the gitea-runner agent
during initial provisioning - environment setup adjustments
discovered when the runner came up the first time.
stripe-billing.tf: prior-session work that hadn't been committed,
folding it in now to clean the working tree before further changes.
dev.neurontechnologies.ai mirrors stage in shape but looser:
broken builds OK, auto-deploys on every push to dev branch (once
CI lands), dev Supabase project (isolated), test Stripe keys,
RESEND_DRY_RUN=1 so transactional email never actually sends.
Locked behind Cloudflare Access with team policy: anyone with
@neurontechnologies.ai email plus named external collaborators
via the access app's emails list.
Initial image is marketing:fix-gallery-render-1236 (matches stage
at the time of this commit). Future deploys land via CI workflow.
Locked down to email_domain == neurontechnologies.ai via Cloudflare
Access. Dedicated SA with zero prod IAM. Isolated Supabase project
(not a schema in prod). No outbound email path. /api/share and /said
return 410 Gone in sandbox so the public-artifact surface stays
strictly in prod.
Stub deploy ready. Real soul build pipeline lands separately.
A single e2-standard-4 GCE VM in us-central1-a runs act_runner,
registered to git.neuralplatform.ai. Workload Identity Federation
binds the runner to the existing neuron-ci-pusher SA (artifactregistry.writer
+ run.developer + serviceAccountUser on the runtime SAs). No long-lived
JSON keys live on the runner; the GCP_SA_KEY secret remains as a fallback
in case the Gitea OIDC issuer isn't reachable from oauth2.googleapis.com.
The runner VM has its own minimal-scope SA (gitea-runner-vm) that can
only read the registration token from Secret Manager — splitting boot
identity from deploy identity so a runner compromise doesn't grant push.
SSH is IAP-only (no public ingress on :22). Reference workflow lives at
products/web/.gitea/workflows/deploy.yaml and takes manual gcloud out of
the loop: push to main triggers build, push to AR, parallel deploy to
all 3 marketing prod regions, traffic flip, smoke check.
SPF / DKIM / DMARC for neurontechnologies.ai are managed via the Cloudflare
API rather than Terraform because the cloudflare_record provider rewrites
name="@" on apply, which forces replacement of the apex SPF record and
opens a propagation gap. A duplicate (malformed-quoted) apex SPF that was
causing Gmail to fail SPF was also removed via the API; the surviving
record is the canonical one and is documented here for traceability.
- stripe-billing.tf: Secret Manager resources for all billing IDs:
meters (input/output), plan prices (free/professional/founding),
overage prices per plan per token type
- cloud-run-soma.tf: inject all 13 Stripe env vars into the Soma service
via Secret Manager secret_key_ref; add billing secrets to depends_on
Provisions soma-anthropic-api-key in Secret Manager and injects it as
ANTHROPIC_API_KEY into the Cloud Run service. Also sets ANTHROPIC_MODEL
to claude-sonnet-4-5. Soma's chat handler now uses this to route the
neuron model to Anthropic instead of spinning up GPU VMs.
Will Anderson
will.anderson