Switch gcp/ backend from Cloudflare R2 (S3-compatible) to GCS — uses
Application Default Credentials automatically, no extra env vars needed.
State has been migrated to gs://neuron-785695-terraform-state/gcp/.
Remove GCE Vault node resources (vault-node-1/2/3) from vault-nodes.tf
and Terraform state. GKE Vault (Workload Identity + vault-unseal SA) has
been serving traffic since the NEG cutover; GCE nodes are confirmed dead.
Removed from state and config:
- google_compute_instance.vault_node (x3)
- google_compute_disk.vault_data (x3)
- google_compute_instance_group.vault (x3)
- google_compute_firewall.vault_iap_ssh / vault_raft / vault_api (GCE-targeted)
- google_storage_bucket_object.vault_startup
- google_storage_bucket_iam_member.vault_node_bucket_read
- google_project_iam_member.vault_node_{log_writer,metric_writer,compute_viewer}
Kept: KMS, vault-unseal SA, WI binding, NEG data sources, GKE firewall,
global LB (address, cert, backend service, url maps, proxies, fwd rules).
Set replicas: 0 to prevent crash-loop restarts. The runner requires a Docker
daemon (even in host mode) which Autopilot won't provide. Scale to 1 when a
Standard node pool or external Docker TCP endpoint is available.
Switch to host-mode labels and document the DinD limitation: Autopilot's
Warden blocks privileged containers at the node level. Use internal Gitea
service URL (gitea.gitea.svc.cluster.local) to bypass Cloudflare Access.
Document three resolution paths (Standard node pool, Cloud Build, external
Docker TCP). GCE VM runner remains active in the interim.
Replaces the GCE VM runner with a k8s Deployment in the ci namespace on
neuron-platform GKE. Uses Docker-in-Docker for build isolation since
Autopilot doesn't expose the node socket. Runner token pulled from Secret
Manager via ESO + Workload Identity.
- servers/gcp/k8s/gitea-runner/: namespace, serviceaccount, external-secrets,
deployment manifests (ci namespace, dind sidecar, idempotent registration)
- servers/gcp/k8s/argocd-apps/gitea-runner-gke.yaml: Argo CD Application
- servers/gcp/gitea-runner.tf: gitea-runner-gke GCP SA with secretAccessor
on gitea-runner-token, Workload Identity binding for ci/gitea-runner,
artifactregistry.reader for pulling ci-base image
Argo CD's bundled schema is missing terminatingReplicas from the
Deployment status spec, causing a ComparisonError on every sync.
Ignoring /status entirely is safe — Argo CD never manages status
fields; they're owned by kube-controller-manager.
gitea-gke and vault-gke were pointing at the dead k3s-internal Gitea
(http://gitea.git.svc.cluster.local:3000). gke-apps already uses
https://git.neuralplatform.ai and syncs successfully — update child apps
to match. Once gke-apps picks this up, gitea-gke will sync the pending
GITEA__actions__ENABLED=true commit.
Pods 1 and 2 can't unseal because port 8201 (cluster port) only opens
after barrier initialization, but the barrier key transfer requires 8201
to be reachable — deadlock. Running single-node on pod 0 so
vault.neuralplatform.ai reliably routes to the active, unsealed instance.
Vault:
- Annotate vault-helm-gke Service with cloud.google.com/neg exposed_ports
to create container-native NEGs (k8s1-bfbeff02-vault-...) in 3 zones
- Add vault-api-from-lb-gke firewall rule allowing GCP health check ranges
(130.211.0.0/22, 35.191.0.0/16) to reach GKE pod IPs on port 8200
- Replace GCE instance group backends in google_compute_backend_service.vault
with GKE NEG backends (RATE balancing mode, 100 req/endpoint)
- GCP Global HTTPS LB frontend unchanged — DNS stays at 34.54.164.21
- vault.neuralplatform.ai now terminates at GKE pods (all 3 NEGs HEALTHY)
Gitea:
- Change GKE Gitea Service from ClusterIP to LoadBalancer (external IP: 34.31.145.131)
- Add Cloudflare DNS A record for git.neuralplatform.ai → 34.31.145.131 (proxied)
- Remove git.neuralplatform.ai route from Legion Cloudflare tunnel config
- Add Cloudflare config rule: flexible SSL for git.neuralplatform.ai
(origin serves HTTP/3000, CF proxies HTTPS termination)
- Scale Legion Gitea deployment to 0 replicas (PVC preserved)
- git.neuralplatform.ai now serves from GKE Gitea pod
SSD quota (SSD_TOTAL_GB) is exhausted at 500/500 GB. pd-ssd PVCs for vault-1
and vault-2 cannot provision. Switching to standard-rwo (pd-balanced) unblocks
the Vault HA cluster immediately.
Quota increase request submitted (Cloud Quotas preference API, preference ID
0aa576b1-5050-4fa7-911d-6f408230df7f, target 1000GB). Once approved, can
optionally migrate back to premium-rwo for all-SSD Vault storage.
vault-1 and vault-2 are stuck Pending because GCE quota (SSD_TOTAL_GB 500/500)
prevents new nodes from provisioning. ScheduleAnyway lets them land in any zone
that has capacity while the quota increase request is pending.
Also adds ESO Workload Identity IAM bindings to Terraform state (previously applied
out-of-band via gcloud; now tracked in cloud-sql.tf).
- Replace GKE_CLUSTER_ENDPOINT placeholder with real endpoint (34.63.89.52)
in all three Argo CD Application manifests
- Fix gitea GCP SA account_id from 'gitea' (5 chars, too short) to 'gitea-gke'
to satisfy GCP's 6-30 char constraint
- Update serviceaccount.yaml annotation to match new SA email (gitea-gke@...)
Legion is offline. Vault needs a GCP-native home. This commit provisions
a 3-node Vault HA cluster on GCE e2-small VMs (us-central1 a/b/c) backed
by Raft consensus with GCP KMS auto-unseal already in place.
- vault-nodes.tf: 3 VMs with separate pd-ssd data disks (prevent_destroy),
IAP-only SSH firewall, Raft peer firewall (8201), global HTTPS LB for
vault.neuralplatform.ai with Google-managed cert, HTTP health checks
against /v1/sys/health
- vault/startup.sh: idempotent boot script — installs Vault from HashiCorp
APT repo, mounts/formats the data disk, writes Raft config with dynamic
retry_join peers resolved via gcloud metadata, starts vault.service
- vault-auth-gcp.tf: GCP Secret Manager secret for init script; outputs
the service account emails needed for GCP auth roles
- vault/configure-vault-auth.sh: post-init script to enable gcp auth
method, write marketing-policy and soma-policy, bind roles to Cloud Run
service account identities
After apply: set vault_lb_ip output as A record for vault.neuralplatform.ai
in Cloudflare, then run vault operator init on vault-node-1 to bootstrap.
vault-kms.tf: KMS keyring/key for Vault auto-unseal + vault-unseal SA
and IAM bindings — all imported from existing GCP resources.
prevent_destroy=true on the crypto key to protect against accidental deletion.
unmanaged-secrets.tf: 8 secrets that were referenced by live Cloud Run
services but had no TF resource — anthropic-api-key, resend-api-key,
supabase-service-key, docuseal-webhook-token, stripe-price-professional,
stripe-price-founding, stripe-price-family-child, gitea-runner-token.
service-account.tf: 4 manually-created SAs — neuron-workspace-dwd,
docuseal-storage, analytics-reader, harmonic-search-console.
DOCUSEAL_WEBHOOK_TOKEN was present on all live marketing services but
missing from cloud-run.tf and cloud-run-stage.tf — next apply would
have stripped it and broken DocuSeal webhook validation.
TOGETHER_API_KEY (soma-together-api-key) was present on soma-prod-us
but missing from cloud-run-soma.tf — next apply would have broken the
Together.ai inference routing path.
cloud-run.tf + cloud-run-stage.tf: small alignment edits from the
dev-env agent's work to match the actual deployed Cloud Run shape.
runners/startup.sh: 10-line additions from the gitea-runner agent
during initial provisioning - environment setup adjustments
discovered when the runner came up the first time.
stripe-billing.tf: prior-session work that hadn't been committed,
folding it in now to clean the working tree before further changes.
dev.neurontechnologies.ai mirrors stage in shape but looser:
broken builds OK, auto-deploys on every push to dev branch (once
CI lands), dev Supabase project (isolated), test Stripe keys,
RESEND_DRY_RUN=1 so transactional email never actually sends.
Locked behind Cloudflare Access with team policy: anyone with
@neurontechnologies.ai email plus named external collaborators
via the access app's emails list.
Initial image is marketing:fix-gallery-render-1236 (matches stage
at the time of this commit). Future deploys land via CI workflow.
Locked down to email_domain == neurontechnologies.ai via Cloudflare
Access. Dedicated SA with zero prod IAM. Isolated Supabase project
(not a schema in prod). No outbound email path. /api/share and /said
return 410 Gone in sandbox so the public-artifact surface stays
strictly in prod.
Stub deploy ready. Real soul build pipeline lands separately.
A single e2-standard-4 GCE VM in us-central1-a runs act_runner,
registered to git.neuralplatform.ai. Workload Identity Federation
binds the runner to the existing neuron-ci-pusher SA (artifactregistry.writer
+ run.developer + serviceAccountUser on the runtime SAs). No long-lived
JSON keys live on the runner; the GCP_SA_KEY secret remains as a fallback
in case the Gitea OIDC issuer isn't reachable from oauth2.googleapis.com.
The runner VM has its own minimal-scope SA (gitea-runner-vm) that can
only read the registration token from Secret Manager — splitting boot
identity from deploy identity so a runner compromise doesn't grant push.
SSH is IAP-only (no public ingress on :22). Reference workflow lives at
products/web/.gitea/workflows/deploy.yaml and takes manual gcloud out of
the loop: push to main triggers build, push to AR, parallel deploy to
all 3 marketing prod regions, traffic flip, smoke check.
SPF / DKIM / DMARC for neurontechnologies.ai are managed via the Cloudflare
API rather than Terraform because the cloudflare_record provider rewrites
name="@" on apply, which forces replacement of the apex SPF record and
opens a propagation gap. A duplicate (malformed-quoted) apex SPF that was
causing Gmail to fail SPF was also removed via the API; the surviving
record is the canonical one and is documented here for traceability.
- stripe-billing.tf: Secret Manager resources for all billing IDs:
meters (input/output), plan prices (free/professional/founding),
overage prices per plan per token type
- cloud-run-soma.tf: inject all 13 Stripe env vars into the Soma service
via Secret Manager secret_key_ref; add billing secrets to depends_on
Provisions soma-anthropic-api-key in Secret Manager and injects it as
ANTHROPIC_API_KEY into the Cloud Run service. Also sets ANTHROPIC_MODEL
to claude-sonnet-4-5. Soma's chat handler now uses this to route the
neuron model to Anthropic instead of spinning up GPU VMs.
Creates soma-key-secret in GCP Secret Manager for HMAC key signing, and
adds SOMA_KEY_SECRET + SOMA_BASE_DOMAIN env vars to the soma-prod-us Cloud
Run service so stateless API key validation works in production.
- Add roles/compute.instanceAdmin.v1 for soma SA (provision/terminate GCE VMs)
- Add roles/iam.serviceAccountUser for soma SA (assign SA to inference VMs)
- Add firewall rule: allow TCP 8000 inbound to soma-inference-node tagged VMs
- Add soma-node-secret to Secret Manager (GCE startup script auth)
- Inject SOMA_NODE_SECRET, GCP_PROJECT_ID, GCP_ZONE, SOMA_PUBLIC_URL into
the soma Cloud Run service via env vars
Architecture: intelligence stays on Legion; only compiled artifacts cross
to GCP. Source code and Neuron's knowledge base never leave the system.
Artifact Registry:
- neuron-marketing, neuron-accounts, neuron-api repos in us-central1
- Keep-last-10 cleanup policy; ci-pusher SA with writer access
- Legion CI runners authenticate via GCP_SA_KEY Gitea secret
Cloud SQL (cloud-sql.tf):
- postgres-15 on db-g1-small, us-central1 (scale up to REGIONAL HA at 1k users)
- Point-in-time recovery, 14-day backup retention
- Accounts DB + user; password generated and stored in Secret Manager
- JWT signing key in Secret Manager (shared by accounts + api)
- Cloud Run connects via built-in Auth Proxy (Unix socket volume mount)
Accounts Cloud Run (cloud-run-accounts.tf):
- 3 regions (us-central1, europe-west1, asia-northeast1), min:1 max:50
- Cloud SQL proxy volume mount; secrets via Secret Manager
- Stripe + JWT env vars; health probe on /health
API Cloud Run (cloud-run-api.tf):
- 3 regions, min:1 max:100, cpu_idle=false (always-hot)
- Validates JWTs from accounts service; no direct DB connection
- License admin token from Secret Manager
Load balancer (host-based routing):
- Same global anycast IP for all three services
- URL map routes by Host: neurontechnologies.ai→marketing,
api.neurontechnologies.ai→api, accounts.neurontechnologies.ai→accounts
- New managed SSL certs for api.* and accounts.* added to HTTPS proxy
- Cloud Armor (WAF + rate limit) applied to all backends
Service accounts + IAM:
- neuron-accounts-sa: secretmanager.secretAccessor + cloudsql.client
- neuron-api-sa: secretmanager.secretAccessor
- allUsers invoker on all prod Cloud Run services (LB health checks)
bootstrap.sh:
- One-shot setup: pulls Stripe secrets from Vault → Secret Manager,
creates CI SA JSON key, prints DNS + next-step instructions
Will Anderson