The stale apps/gitea-runner.yaml contained two Deployment manifests
that conflicted with the canonical Deployments owned by the
gitea-runner-config Argo Application (pointing at k8s/gitea-runner/).
Dual ownership caused Argo CD to fight itself — restarting runner
pods mid-job and producing the "context canceled" failures on
neuron-technologies/dharma-el CI.
Canonical Deployments (config-version 2026-05-04-cf-access-public-url,
docker.sock, CF Access env, replicas=2 for nt-runner) live in
k8s/gitea-runner/deployment.yaml and are managed by gitea-runner-config.
- dns-neuralplatform.tf: add vault.neuralplatform.ai A record → 34.54.164.21 (GCP LB)
DNS-only (not proxied) so GCP managed TLS cert can provision correctly
- main.tf: remove vault.neuralplatform.ai from Cloudflare tunnel ingress
(now served directly via GCP Global HTTPS LB)
- main.tf: remove watch.nook.family, jellyfin.nook.family, bazarr.nook.family
from tunnel ingress (nook.family media stack retired; infra is Neuron-focused)
GCE Vault cluster already initialized and running (3-node Raft, active since
2026-05-04T16:05). Secrets migrated 48/48 from k3s vault. ESO ClusterSecretStore
validated against new vault. k3s vault-0 is now superseded.
The containerd socket is mounted from the host at /run/k3s/containerd/containerd.sock
to /var/run/docker.sock inside the container, but act_runner config was pointing to
the host path directly. Fixes CrashLoopBackOff on all runner pods.
Also codifies both deployments (will org + neuron-technologies org) as git-managed
manifests so Argo CD owns them going forward.
The buildah-compiled image (imported directly into k3s) is the working
version. The registry image will catch up after the next CI build.
IfNotPresent ensures the locally-cached correct binary is used.
Soma moved to GCP Cloud Run. The soma-*.yaml files were deleted but
the kustomization still referenced them, blocking all Argo CD syncs
for neuron-prod (including the new allow-daemon-ingress NetworkPolicy).
The ExternalSecret had a stale Gitea URL (192.168.8.148:30322 — old
local IP) and runner tokens were never populated in Vault. Update the
template URL to the cluster-internal Gitea service DNS, and store the
registration tokens in Vault at secret/gitea.runner_token.
backup-verify: restic restores /dump/all-databases.sql relative to
restore target, so the check path was wrong (/all-databases.sql →
/dump/all-databases.sql). Also fix the --include filter to match.
redpanda-topics-init: redpanda is intentionally scaled to 0; the init
job will block forever waiting for a broker. Mark it suspended so Argo
CD won't keep creating a stuck job. Re-enable when redpanda is turned on.
The default-deny-all NetworkPolicy was blocking dharma from reaching
neuron-daemon port 7750 (event queue / engram endpoint). Add an explicit
ingress policy for neuron-daemon that allows traffic from the neuron-prod
namespace, so dharma and other internal services can connect at startup.
Legion is offline. Vault needs a GCP-native home. This commit provisions
a 3-node Vault HA cluster on GCE e2-small VMs (us-central1 a/b/c) backed
by Raft consensus with GCP KMS auto-unseal already in place.
- vault-nodes.tf: 3 VMs with separate pd-ssd data disks (prevent_destroy),
IAP-only SSH firewall, Raft peer firewall (8201), global HTTPS LB for
vault.neuralplatform.ai with Google-managed cert, HTTP health checks
against /v1/sys/health
- vault/startup.sh: idempotent boot script — installs Vault from HashiCorp
APT repo, mounts/formats the data disk, writes Raft config with dynamic
retry_join peers resolved via gcloud metadata, starts vault.service
- vault-auth-gcp.tf: GCP Secret Manager secret for init script; outputs
the service account emails needed for GCP auth roles
- vault/configure-vault-auth.sh: post-init script to enable gcp auth
method, write marketing-policy and soma-policy, bind roles to Cloud Run
service account identities
After apply: set vault_lb_ip output as A record for vault.neuralplatform.ai
in Cloudflare, then run vault operator init on vault-node-1 to bootstrap.
vault-kms.tf: KMS keyring/key for Vault auto-unseal + vault-unseal SA
and IAM bindings — all imported from existing GCP resources.
prevent_destroy=true on the crypto key to protect against accidental deletion.
unmanaged-secrets.tf: 8 secrets that were referenced by live Cloud Run
services but had no TF resource — anthropic-api-key, resend-api-key,
supabase-service-key, docuseal-webhook-token, stripe-price-professional,
stripe-price-founding, stripe-price-family-child, gitea-runner-token.
service-account.tf: 4 manually-created SAs — neuron-workspace-dwd,
docuseal-storage, analytics-reader, harmonic-search-console.
DOCUSEAL_WEBHOOK_TOKEN was present on all live marketing services but
missing from cloud-run.tf and cloud-run-stage.tf — next apply would
have stripped it and broken DocuSeal webhook validation.
TOGETHER_API_KEY (soma-together-api-key) was present on soma-prod-us
but missing from cloud-run-soma.tf — next apply would have broken the
Together.ai inference routing path.
- Dockerfile for Go DHARMA server (registry.neuralplatform.ai/neuron-technologies/dharma)
- k8s manifests: deployment, service, ExternalSecret (Vault), IngressRoute
- Argo CD app: deploys to neuron-prod namespace
- Cloudflare tunnel rule for dharma.neurontechnologies.ai
- DNS CNAME record for dharma.neurontechnologies.ai → legion tunnel
- Vault secrets already populated (api_key, encryption_key, jwt_secret, engram_key, engram_url)
- Connects to neuron-daemon (engram) at neuron-daemon.neuron-prod.svc.cluster.local:7749
The registry currently allows anonymous cluster-internal pulls so the
imagePullSecrets reference was failing the externalsecret reconcile
(secret/data/mudcraft has no registry creds for this app). Pod was
pulling fine anyway. If the registry later gets locked down, add a
proper externalsecret using the path from secret/data/<app> instead of
borrowing mudcraft's.
The legion-apps Application root was hardcoded to the OLD legion's gitea
cluster IP (10.43.1.53:3000). On the new GCP legion, the gitea service
got a different cluster IP (10.43.15.98), so the umbrella has been
ComparisonError-stuck since the cluster move on 2026-04-28. That meant
new Application manifests in apps/ never got materialized in argocd —
neuron-web was the first new app to need it after the move.
All other apps reference http://gitea.git.svc.cluster.local:3000/...
(the stable DNS name). Bringing the three legion-apps spots in argocd.tf
in line with that convention so the next gitea cluster-IP shuffle
doesn't re-break the umbrella.
Stand up the elb-built native El landing server on Legion k3s alongside
the existing Cloud Run prod marketing site. Image is built on
gitea-runner-1 from elb-emitted .c files + native cc and pushed to
registry.neuralplatform.ai/neuron-web:dev-dfe41234. Deployment exposes
web-stage.neuralplatform.ai via Traefik + Cloudflare tunnel.
The point is to have a working local instance we can iterate against
while we redesign the CI/CD pipeline; nothing here replaces the Cloud
Run deploy yet.
* k8s/neuron-technologies/web/ — namespace, deployment, service, ingress,
registry-pull-secret + externalsecret pulling anthropic/supabase keys
from Vault
* apps/neuron-web.yaml — Argo CD Application
* dns-neuralplatform.tf, main.tf — CNAME + tunnel ingress for
web-stage.neuralplatform.ai
cloud-run.tf + cloud-run-stage.tf: small alignment edits from the
dev-env agent's work to match the actual deployed Cloud Run shape.
runners/startup.sh: 10-line additions from the gitea-runner agent
during initial provisioning - environment setup adjustments
discovered when the runner came up the first time.
stripe-billing.tf: prior-session work that hadn't been committed,
folding it in now to clean the working tree before further changes.
dev.neurontechnologies.ai mirrors stage in shape but looser:
broken builds OK, auto-deploys on every push to dev branch (once
CI lands), dev Supabase project (isolated), test Stripe keys,
RESEND_DRY_RUN=1 so transactional email never actually sends.
Locked behind Cloudflare Access with team policy: anyone with
@neurontechnologies.ai email plus named external collaborators
via the access app's emails list.
Initial image is marketing:fix-gallery-render-1236 (matches stage
at the time of this commit). Future deploys land via CI workflow.
Locked down to email_domain == neurontechnologies.ai via Cloudflare
Access. Dedicated SA with zero prod IAM. Isolated Supabase project
(not a schema in prod). No outbound email path. /api/share and /said
return 410 Gone in sandbox so the public-artifact surface stays
strictly in prod.
Stub deploy ready. Real soul build pipeline lands separately.
A single e2-standard-4 GCE VM in us-central1-a runs act_runner,
registered to git.neuralplatform.ai. Workload Identity Federation
binds the runner to the existing neuron-ci-pusher SA (artifactregistry.writer
+ run.developer + serviceAccountUser on the runtime SAs). No long-lived
JSON keys live on the runner; the GCP_SA_KEY secret remains as a fallback
in case the Gitea OIDC issuer isn't reachable from oauth2.googleapis.com.
The runner VM has its own minimal-scope SA (gitea-runner-vm) that can
only read the registration token from Secret Manager — splitting boot
identity from deploy identity so a runner compromise doesn't grant push.
SSH is IAP-only (no public ingress on :22). Reference workflow lives at
products/web/.gitea/workflows/deploy.yaml and takes manual gcloud out of
the loop: push to main triggers build, push to AR, parallel deploy to
all 3 marketing prod regions, traffic flip, smoke check.
SPF / DKIM / DMARC for neurontechnologies.ai are managed via the Cloudflare
API rather than Terraform because the cloudflare_record provider rewrites
name="@" on apply, which forces replacement of the apex SPF record and
opens a propagation gap. A duplicate (malformed-quoted) apex SPF that was
causing Gmail to fail SPF was also removed via the API; the surviving
record is the canonical one and is documented here for traceability.
- stripe-billing.tf: Secret Manager resources for all billing IDs:
meters (input/output), plan prices (free/professional/founding),
overage prices per plan per token type
- cloud-run-soma.tf: inject all 13 Stripe env vars into the Soma service
via Secret Manager secret_key_ref; add billing secrets to depends_on
Provisions soma-anthropic-api-key in Secret Manager and injects it as
ANTHROPIC_API_KEY into the Cloud Run service. Also sets ANTHROPIC_MODEL
to claude-sonnet-4-5. Soma's chat handler now uses this to route the
neuron model to Anthropic instead of spinning up GPU VMs.
- ExternalSecret: pull neuron_api_key from Vault into NEURON_API_KEY
- ConfigMap: set OTEL_EXPORTER_OTLP_ENDPOINT to alloy-otlp in-cluster service
instead of localhost (fixes 196 restart OTel connection errors)
Creates soma-key-secret in GCP Secret Manager for HMAC key signing, and
adds SOMA_KEY_SECRET + SOMA_BASE_DOMAIN env vars to the soma-prod-us Cloud
Run service so stateless API key validation works in production.
- Add roles/compute.instanceAdmin.v1 for soma SA (provision/terminate GCE VMs)
- Add roles/iam.serviceAccountUser for soma SA (assign SA to inference VMs)
- Add firewall rule: allow TCP 8000 inbound to soma-inference-node tagged VMs
- Add soma-node-secret to Secret Manager (GCE startup script auth)
- Inject SOMA_NODE_SECRET, GCP_PROJECT_ID, GCP_ZONE, SOMA_PUBLIC_URL into
the soma Cloud Run service via env vars
- neuron-dev: update neuron-rest and neuron-mcp from dev-d7a587e2 to v0.18.14
- neuron-stage: update neuron-rest/mcp from stage-0f5b2328 to v0.18.14; marketing from :latest to 1e94e8ae
- swarm-alpha/beta/gamma: update neuron-rest and neuron-mcp from v0.15.3 to v0.18.14
- ci runners: fix docker socket path from /var/run/docker.sock to /run/k3s/containerd/containerd.sock (k3s node has no Docker daemon)
Expose the MCP endpoint directly at the root domain so Bearer token
clients can reach it without going through the OAuth subdomain.
The Spring Boot MCP layer already accepts ntn_* API keys via
ApiKeyAuthenticationFilter; this ingress rule enables access via
https://neurontechnologies.ai/mcp.
Will Anderson
will.anderson