The stale apps/gitea-runner.yaml contained two Deployment manifests
that conflicted with the canonical Deployments owned by the
gitea-runner-config Argo Application (pointing at k8s/gitea-runner/).
Dual ownership caused Argo CD to fight itself — restarting runner
pods mid-job and producing the "context canceled" failures on
neuron-technologies/dharma-el CI.
Canonical Deployments (config-version 2026-05-04-cf-access-public-url,
docker.sock, CF Access env, replicas=2 for nt-runner) live in
k8s/gitea-runner/deployment.yaml and are managed by gitea-runner-config.
- dns-neuralplatform.tf: add vault.neuralplatform.ai A record → 34.54.164.21 (GCP LB)
DNS-only (not proxied) so GCP managed TLS cert can provision correctly
- main.tf: remove vault.neuralplatform.ai from Cloudflare tunnel ingress
(now served directly via GCP Global HTTPS LB)
- main.tf: remove watch.nook.family, jellyfin.nook.family, bazarr.nook.family
from tunnel ingress (nook.family media stack retired; infra is Neuron-focused)
GCE Vault cluster already initialized and running (3-node Raft, active since
2026-05-04T16:05). Secrets migrated 48/48 from k3s vault. ESO ClusterSecretStore
validated against new vault. k3s vault-0 is now superseded.
The containerd socket is mounted from the host at /run/k3s/containerd/containerd.sock
to /var/run/docker.sock inside the container, but act_runner config was pointing to
the host path directly. Fixes CrashLoopBackOff on all runner pods.
Also codifies both deployments (will org + neuron-technologies org) as git-managed
manifests so Argo CD owns them going forward.
The buildah-compiled image (imported directly into k3s) is the working
version. The registry image will catch up after the next CI build.
IfNotPresent ensures the locally-cached correct binary is used.
Soma moved to GCP Cloud Run. The soma-*.yaml files were deleted but
the kustomization still referenced them, blocking all Argo CD syncs
for neuron-prod (including the new allow-daemon-ingress NetworkPolicy).
The ExternalSecret had a stale Gitea URL (192.168.8.148:30322 — old
local IP) and runner tokens were never populated in Vault. Update the
template URL to the cluster-internal Gitea service DNS, and store the
registration tokens in Vault at secret/gitea.runner_token.
backup-verify: restic restores /dump/all-databases.sql relative to
restore target, so the check path was wrong (/all-databases.sql →
/dump/all-databases.sql). Also fix the --include filter to match.
redpanda-topics-init: redpanda is intentionally scaled to 0; the init
job will block forever waiting for a broker. Mark it suspended so Argo
CD won't keep creating a stuck job. Re-enable when redpanda is turned on.
The default-deny-all NetworkPolicy was blocking dharma from reaching
neuron-daemon port 7750 (event queue / engram endpoint). Add an explicit
ingress policy for neuron-daemon that allows traffic from the neuron-prod
namespace, so dharma and other internal services can connect at startup.
Legion is offline. Vault needs a GCP-native home. This commit provisions
a 3-node Vault HA cluster on GCE e2-small VMs (us-central1 a/b/c) backed
by Raft consensus with GCP KMS auto-unseal already in place.
- vault-nodes.tf: 3 VMs with separate pd-ssd data disks (prevent_destroy),
IAP-only SSH firewall, Raft peer firewall (8201), global HTTPS LB for
vault.neuralplatform.ai with Google-managed cert, HTTP health checks
against /v1/sys/health
- vault/startup.sh: idempotent boot script — installs Vault from HashiCorp
APT repo, mounts/formats the data disk, writes Raft config with dynamic
retry_join peers resolved via gcloud metadata, starts vault.service
- vault-auth-gcp.tf: GCP Secret Manager secret for init script; outputs
the service account emails needed for GCP auth roles
- vault/configure-vault-auth.sh: post-init script to enable gcp auth
method, write marketing-policy and soma-policy, bind roles to Cloud Run
service account identities
After apply: set vault_lb_ip output as A record for vault.neuralplatform.ai
in Cloudflare, then run vault operator init on vault-node-1 to bootstrap.
vault-kms.tf: KMS keyring/key for Vault auto-unseal + vault-unseal SA
and IAM bindings — all imported from existing GCP resources.
prevent_destroy=true on the crypto key to protect against accidental deletion.
unmanaged-secrets.tf: 8 secrets that were referenced by live Cloud Run
services but had no TF resource — anthropic-api-key, resend-api-key,
supabase-service-key, docuseal-webhook-token, stripe-price-professional,
stripe-price-founding, stripe-price-family-child, gitea-runner-token.
service-account.tf: 4 manually-created SAs — neuron-workspace-dwd,
docuseal-storage, analytics-reader, harmonic-search-console.
DOCUSEAL_WEBHOOK_TOKEN was present on all live marketing services but
missing from cloud-run.tf and cloud-run-stage.tf — next apply would
have stripped it and broken DocuSeal webhook validation.
TOGETHER_API_KEY (soma-together-api-key) was present on soma-prod-us
but missing from cloud-run-soma.tf — next apply would have broken the
Together.ai inference routing path.
- Dockerfile for Go DHARMA server (registry.neuralplatform.ai/neuron-technologies/dharma)
- k8s manifests: deployment, service, ExternalSecret (Vault), IngressRoute
- Argo CD app: deploys to neuron-prod namespace
- Cloudflare tunnel rule for dharma.neurontechnologies.ai
- DNS CNAME record for dharma.neurontechnologies.ai → legion tunnel
- Vault secrets already populated (api_key, encryption_key, jwt_secret, engram_key, engram_url)
- Connects to neuron-daemon (engram) at neuron-daemon.neuron-prod.svc.cluster.local:7749
The registry currently allows anonymous cluster-internal pulls so the
imagePullSecrets reference was failing the externalsecret reconcile
(secret/data/mudcraft has no registry creds for this app). Pod was
pulling fine anyway. If the registry later gets locked down, add a
proper externalsecret using the path from secret/data/<app> instead of
borrowing mudcraft's.
The legion-apps Application root was hardcoded to the OLD legion's gitea
cluster IP (10.43.1.53:3000). On the new GCP legion, the gitea service
got a different cluster IP (10.43.15.98), so the umbrella has been
ComparisonError-stuck since the cluster move on 2026-04-28. That meant
new Application manifests in apps/ never got materialized in argocd —
neuron-web was the first new app to need it after the move.
All other apps reference http://gitea.git.svc.cluster.local:3000/...
(the stable DNS name). Bringing the three legion-apps spots in argocd.tf
in line with that convention so the next gitea cluster-IP shuffle
doesn't re-break the umbrella.
Stand up the elb-built native El landing server on Legion k3s alongside
the existing Cloud Run prod marketing site. Image is built on
gitea-runner-1 from elb-emitted .c files + native cc and pushed to
registry.neuralplatform.ai/neuron-web:dev-dfe41234. Deployment exposes
web-stage.neuralplatform.ai via Traefik + Cloudflare tunnel.
The point is to have a working local instance we can iterate against
while we redesign the CI/CD pipeline; nothing here replaces the Cloud
Run deploy yet.
* k8s/neuron-technologies/web/ — namespace, deployment, service, ingress,
registry-pull-secret + externalsecret pulling anthropic/supabase keys
from Vault
* apps/neuron-web.yaml — Argo CD Application
* dns-neuralplatform.tf, main.tf — CNAME + tunnel ingress for
web-stage.neuralplatform.ai
cloud-run.tf + cloud-run-stage.tf: small alignment edits from the
dev-env agent's work to match the actual deployed Cloud Run shape.
runners/startup.sh: 10-line additions from the gitea-runner agent
during initial provisioning - environment setup adjustments
discovered when the runner came up the first time.
stripe-billing.tf: prior-session work that hadn't been committed,
folding it in now to clean the working tree before further changes.
dev.neurontechnologies.ai mirrors stage in shape but looser:
broken builds OK, auto-deploys on every push to dev branch (once
CI lands), dev Supabase project (isolated), test Stripe keys,
RESEND_DRY_RUN=1 so transactional email never actually sends.
Locked behind Cloudflare Access with team policy: anyone with
@neurontechnologies.ai email plus named external collaborators
via the access app's emails list.
Initial image is marketing:fix-gallery-render-1236 (matches stage
at the time of this commit). Future deploys land via CI workflow.
Locked down to email_domain == neurontechnologies.ai via Cloudflare
Access. Dedicated SA with zero prod IAM. Isolated Supabase project
(not a schema in prod). No outbound email path. /api/share and /said
return 410 Gone in sandbox so the public-artifact surface stays
strictly in prod.
Stub deploy ready. Real soul build pipeline lands separately.
A single e2-standard-4 GCE VM in us-central1-a runs act_runner,
registered to git.neuralplatform.ai. Workload Identity Federation
binds the runner to the existing neuron-ci-pusher SA (artifactregistry.writer
+ run.developer + serviceAccountUser on the runtime SAs). No long-lived
JSON keys live on the runner; the GCP_SA_KEY secret remains as a fallback
in case the Gitea OIDC issuer isn't reachable from oauth2.googleapis.com.
The runner VM has its own minimal-scope SA (gitea-runner-vm) that can
only read the registration token from Secret Manager — splitting boot
identity from deploy identity so a runner compromise doesn't grant push.
SSH is IAP-only (no public ingress on :22). Reference workflow lives at
products/web/.gitea/workflows/deploy.yaml and takes manual gcloud out of
the loop: push to main triggers build, push to AR, parallel deploy to
all 3 marketing prod regions, traffic flip, smoke check.
SPF / DKIM / DMARC for neurontechnologies.ai are managed via the Cloudflare
API rather than Terraform because the cloudflare_record provider rewrites
name="@" on apply, which forces replacement of the apex SPF record and
opens a propagation gap. A duplicate (malformed-quoted) apex SPF that was
causing Gmail to fail SPF was also removed via the API; the surviving
record is the canonical one and is documented here for traceability.
- stripe-billing.tf: Secret Manager resources for all billing IDs:
meters (input/output), plan prices (free/professional/founding),
overage prices per plan per token type
- cloud-run-soma.tf: inject all 13 Stripe env vars into the Soma service
via Secret Manager secret_key_ref; add billing secrets to depends_on
Provisions soma-anthropic-api-key in Secret Manager and injects it as
ANTHROPIC_API_KEY into the Cloud Run service. Also sets ANTHROPIC_MODEL
to claude-sonnet-4-5. Soma's chat handler now uses this to route the
neuron model to Anthropic instead of spinning up GPU VMs.
- ExternalSecret: pull neuron_api_key from Vault into NEURON_API_KEY
- ConfigMap: set OTEL_EXPORTER_OTLP_ENDPOINT to alloy-otlp in-cluster service
instead of localhost (fixes 196 restart OTel connection errors)
Creates soma-key-secret in GCP Secret Manager for HMAC key signing, and
adds SOMA_KEY_SECRET + SOMA_BASE_DOMAIN env vars to the soma-prod-us Cloud
Run service so stateless API key validation works in production.
- Add roles/compute.instanceAdmin.v1 for soma SA (provision/terminate GCE VMs)
- Add roles/iam.serviceAccountUser for soma SA (assign SA to inference VMs)
- Add firewall rule: allow TCP 8000 inbound to soma-inference-node tagged VMs
- Add soma-node-secret to Secret Manager (GCE startup script auth)
- Inject SOMA_NODE_SECRET, GCP_PROJECT_ID, GCP_ZONE, SOMA_PUBLIC_URL into
the soma Cloud Run service via env vars
- neuron-dev: update neuron-rest and neuron-mcp from dev-d7a587e2 to v0.18.14
- neuron-stage: update neuron-rest/mcp from stage-0f5b2328 to v0.18.14; marketing from :latest to 1e94e8ae
- swarm-alpha/beta/gamma: update neuron-rest and neuron-mcp from v0.15.3 to v0.18.14
- ci runners: fix docker socket path from /var/run/docker.sock to /run/k3s/containerd/containerd.sock (k3s node has no Docker daemon)
Will Anderson
will.anderson