116 Commits

Author SHA1 Message Date
Will Anderson 479190ef47 migrate all Argo CD repo sources from Legion Gitea to GKE Gitea 2026-05-05 02:25:14 -05:00
Will Anderson 872a834989 Cut over Vault and Gitea traffic from GCE/Legion to GKE
Vault:
- Annotate vault-helm-gke Service with cloud.google.com/neg exposed_ports
  to create container-native NEGs (k8s1-bfbeff02-vault-...) in 3 zones
- Add vault-api-from-lb-gke firewall rule allowing GCP health check ranges
  (130.211.0.0/22, 35.191.0.0/16) to reach GKE pod IPs on port 8200
- Replace GCE instance group backends in google_compute_backend_service.vault
  with GKE NEG backends (RATE balancing mode, 100 req/endpoint)
- GCP Global HTTPS LB frontend unchanged — DNS stays at 34.54.164.21
- vault.neuralplatform.ai now terminates at GKE pods (all 3 NEGs HEALTHY)

Gitea:
- Change GKE Gitea Service from ClusterIP to LoadBalancer (external IP: 34.31.145.131)
- Add Cloudflare DNS A record for git.neuralplatform.ai → 34.31.145.131 (proxied)
- Remove git.neuralplatform.ai route from Legion Cloudflare tunnel config
- Add Cloudflare config rule: flexible SSL for git.neuralplatform.ai
  (origin serves HTTP/3000, CF proxies HTTPS termination)
- Scale Legion Gitea deployment to 0 replicas (PVC preserved)
- git.neuralplatform.ai now serves from GKE Gitea pod
2026-05-04 23:46:20 -05:00
Will Anderson 9330107fcc migrate Vault and Gitea to GKE Autopilot cluster
Implements Option A: move Vault (3x GCE e2-small) and Gitea (Legion k8s)
onto a new GKE Autopilot cluster (neuron-platform, us-central1) managed
through Legion Argo CD.

Terraform (servers/gcp/):
- gke.tf: GKE Autopilot cluster, Workload Identity bindings for Vault (KMS)
  and Gitea (Cloud SQL)
- cloud-sql.tf: gitea database + user on neuron-prod-pg15, gitea GCP SA,
  gitea-database-url and gitea-db-password Secret Manager secrets
- vault-nodes.tf: PENDING DECOMMISSION comment with migration checklist

k8s manifests (servers/gcp/k8s/):
- vault/: namespace.yaml
- gitea/: namespace, serviceaccount (Workload Identity annotation), pvc (50Gi
  standard-rwo), deployment (Gitea + Cloud SQL Auth Proxy sidecar), service,
  configmap (custom CSS carried from Legion), external-secrets (GCP SM provider)
- argocd-apps/: vault-gke.yaml, vault-helm-gke.yaml (Helm chart, HA Raft 3
  replicas, GCP KMS auto-unseal, topologySpread across zones, 10Gi premium-rwo),
  gitea-gke.yaml — all target GKE_CLUSTER_ENDPOINT placeholder

Legion (servers/legion/):
- apps/gke-apps.yaml: App-of-Apps entry point on Legion Argo CD that syncs the
  GKE Application manifests
- k8s/gitea-runner/Dockerfile: add system-level git insteadOf so GKE CI runners
  resolve Gitea in-cluster without Cloudflare Access headers
2026-05-04 20:40:48 -05:00
Will Anderson b4b05bfe40 fix(ci): remove duplicate runner Deployment from apps/
The stale apps/gitea-runner.yaml contained two Deployment manifests
that conflicted with the canonical Deployments owned by the
gitea-runner-config Argo Application (pointing at k8s/gitea-runner/).

Dual ownership caused Argo CD to fight itself — restarting runner
pods mid-job and producing the "context canceled" failures on
neuron-technologies/dharma-el CI.

Canonical Deployments (config-version 2026-05-04-cf-access-public-url,
docker.sock, CF Access env, replicas=2 for nt-runner) live in
k8s/gitea-runner/deployment.yaml and are managed by gitea-runner-config.
2026-05-04 17:55:28 -05:00
Will Anderson b70fef611a deploy DHARMA registry to neuron-prod
- Dockerfile for Go DHARMA server (registry.neuralplatform.ai/neuron-technologies/dharma)
- k8s manifests: deployment, service, ExternalSecret (Vault), IngressRoute
- Argo CD app: deploys to neuron-prod namespace
- Cloudflare tunnel rule for dharma.neurontechnologies.ai
- DNS CNAME record for dharma.neurontechnologies.ai → legion tunnel
- Vault secrets already populated (api_key, encryption_key, jwt_secret, engram_key, engram_url)
- Connects to neuron-daemon (engram) at neuron-daemon.neuron-prod.svc.cluster.local:7749
2026-05-03 13:26:54 -05:00
Will Anderson 3daf615dd2 add neuron-web phase-1 deploy on Legion
Stand up the elb-built native El landing server on Legion k3s alongside
the existing Cloud Run prod marketing site. Image is built on
gitea-runner-1 from elb-emitted .c files + native cc and pushed to
registry.neuralplatform.ai/neuron-web:dev-dfe41234. Deployment exposes
web-stage.neuralplatform.ai via Traefik + Cloudflare tunnel.

The point is to have a working local instance we can iterate against
while we redesign the CI/CD pipeline; nothing here replaces the Cloud
Run deploy yet.

* k8s/neuron-technologies/web/ — namespace, deployment, service, ingress,
  registry-pull-secret + externalsecret pulling anthropic/supabase keys
  from Vault
* apps/neuron-web.yaml — Argo CD Application
* dns-neuralplatform.tf, main.tf — CNAME + tunnel ingress for
  web-stage.neuralplatform.ai
2026-05-03 10:45:18 -05:00
Will Anderson 60f4e0693e Remove adguard — home DNS server, no purpose on GCP 2026-04-27 18:27:21 -05:00
Will Anderson fb5b93f9d7 fix broken pods in GCP k3s cluster
- neuron-dev: update neuron-rest and neuron-mcp from dev-d7a587e2 to v0.18.14
- neuron-stage: update neuron-rest/mcp from stage-0f5b2328 to v0.18.14; marketing from :latest to 1e94e8ae
- swarm-alpha/beta/gamma: update neuron-rest and neuron-mcp from v0.15.3 to v0.18.14
- ci runners: fix docker socket path from /var/run/docker.sock to /run/k3s/containerd/containerd.sock (k3s node has no Docker daemon)
2026-04-27 18:17:32 -05:00
Will Anderson f5a691be3c Remove media (torrents) and headscale VPN — no longer needed on GCP 2026-04-27 17:44:08 -05:00
Will Anderson 498979ec90 fix(redis): remove image tag override, use chart default 2026-04-27 17:18:46 -05:00
Will Anderson bba12f3156 fix: remove static volumeNames for GCP migration, fix redis image tag
Dynamic PVC provisioning needed on new GCP instance.
Redis image 7.4.3-debian-12-r0 was removed from Docker Hub;
use floating 7.4-debian-12 tag instead.
2026-04-27 16:53:01 -05:00
Will Anderson 55a38bb693 fix: update Gitea URL to new GCP cluster instance
Legion hardware died April 27, 2026. New GCP k3s instance at 104.197.5.199.
Gitea now at gitea.git.svc.cluster.local:3000 within cluster.
2026-04-27 16:20:46 -05:00
Will Anderson 714a7c7a8d launch: deploy daemon StatefulSet, fix neuron-rest security context, add daemon Argo CD app
- Add neuron-daemon Argo CD Application (apps/neuron-daemon.yaml) pointing to
  k8s/neuron-technologies/daemon/ on main — daemon will deploy once image is built
- Add daemon k8s manifests: StatefulSet (2 replicas, imagePullPolicy: Always),
  Service (ClusterIP :7749, session affinity), ExternalSecret (Anthropic API key from Vault)
- Fix neuron-rest runAsNonRoot: remove constraint since v0.15.3 image runs as root,
  clears the CreateContainerConfigError on the 11h-old stale pod
- Add NEURON_DAEMON_WS_URL env var to marketing deployment for chat UI connectivity
- Add neuron.neurontechnologies.ai Cloudflare DNS record and tunnel ingress rule (Terraform)
2026-04-26 10:13:48 -05:00
Will Anderson 2550786965 Revert CI to host Docker socket, remove buildkitd
BuildKit rootless failed on k3s (mount propagation), privileged mode
fixed that but buildkitd gRPC is incompatible with the Docker REST API
that forgejo-runner needs to manage job containers. Net security change
vs. the original socket approach was zero (privileged ≈ socket access).

Remove buildkitd.yaml entirely. Restore docker-sock hostPath mount on
both runners. Builds work again.
2026-04-26 02:50:27 -05:00
Will Anderson 74dd054a36 feat(swarm): add swarm self-improvement loop infrastructure
Adds complete k8s manifests, ArgoCD app, and Terraform namespace
resources for the Neuron swarm self-improvement loop system.

Each variant (alpha, beta, gamma) gets its own isolated namespace,
PVC, MCP/REST deployments, ExternalSecrets from Vault, RBAC for CI,
and a SQLite clone Job template for session startup.
2026-04-26 00:58:43 -05:00
Will Anderson 7adab317d4 fix: restore postgres chart to 18.x (was wrongly pinned to 16.x)
Chart 16.x deploys PG17 images that do not exist on Docker Hub.
Existing data directory is PG18. Restoring >=18.0.0,<19.0.0 range.
2026-04-26 04:51:07 +00:00
Will Anderson 93358505fc Harden prod: security, autoscaling, observability, BuildKit CI
Security:
- Drop ALL capabilities, enforce non-root, RuntimeDefault seccomp on
  neuron-mcp, neuron-rest, neuron-marketing pods
- Add startup probes (150s window for JVM) so liveness doesn't fire early
- Replace docker-sock hostPath with BuildKit rootless TCP endpoint
  (moby/buildkit:v0.19.0-rootless) — removes node root access from CI
- Document full ESO AppRole migration path in cluster-secret-store.yaml

Autoscaling & availability:
- HPAs on mcp (1–6), rest (1–4), marketing (2–8) at 65–70% CPU
- PodDisruptionBudgets (minAvailable: 1) on all three services
- NetworkPolicy: default-deny-all in neuron-prod, explicit allow rules
  for Traefik ingress, intra-namespace, and egress to DNS/platform/vault

Observability:
- ServiceMonitors for mcp, rest, marketing (cross-namespace enabled in
  kube-prometheus-stack with serviceMonitorSelectorNilUsesHelmValues:false)
- PrometheusRules: high error rate, high latency, crash loops, replica
  shortage, Postgres down/connections, backup failure, backup staleness

Chart version pinning:
- kube-prometheus-stack, loki, tempo, redis, alloy, postgres — all pinned
  to major-version ranges to block silent breaking upgrades

Backup hardening:
- restic:latest → restic:0.17.3 (deterministic image)
- Weekly backup-verify CronJob: restores latest snapshot and validates
  SQL dump structure (≥5 CREATE TABLE, pg_dump header check)

ArgoCD:
- neuron-prod AppProject: scopes deploys to neuron-prod + platform ns,
  blacklists ClusterRole/ClusterRoleBinding/Namespace creation,
  automated sync window 2–6am UTC, manual always allowed
2026-04-25 22:54:18 -05:00
Will Anderson 5d4cc17829 chore(neuron): retire Python backend — keep cloudflared only
Remove neuron/neuron-alpha/neuron-beta/neuron-gamma Deployments and
Services, the slot LB service, Python ingress, slot PVCs, snapshot job,
neuron-config ConfigMap, and neuron-secrets ExternalSecret. The Python
MCP backend has been superseded by the Spring Boot backend in neuron-prod.

Cloudflared Deployment and cloudflared-secret ExternalSecret are
retained in the neuron namespace. The neuron-data PVC is preserved
for data safety.
2026-04-24 20:39:17 -05:00
Will Anderson 91bfc42989 ops: scale down all non-critical services — dedicate resources to Neuron
Scaled to replicas: 0: fornax (coordinator/ui/workers), jellyfin via media,
overseerr, plex, radarr, sonarr, bazarr, prowlarr, flaresolverr, ollama,
harmonic-wordpress, docuseal, listmonk, memos, plane, redpanda, wp-coordinator
2026-04-24 20:07:41 -05:00
Will Anderson 2d98df75ad Deploy Plane project management to k8s with Cloudflare Access gate
Adds all k8s manifests (namespace, postgres, redis, api, worker, beat,
web, ingress, externalsecret, configmap, kustomization), Argo CD Application,
and Cloudflare Zero Trust access policy for plane.neuralplatform.ai.
2026-04-24 14:21:02 -05:00
Will Anderson 0f9448e8ea deploy neuron marketing site to neurontechnologies.ai
Add marketing/ manifests (Deployment, Service, ExternalSecret) for the
Next.js marketing site in neuron-prod namespace, an Argo CD app pointing
to the new path, and update the prod ingress to serve / from neuron-marketing
instead of neuron-rest.
2026-04-24 13:09:29 -05:00
Will Anderson 336a9333a2 prepare neuron-tim namespace for Tim's future Neuron instance
Adds the neuron-tim namespace (via Terraform) and a full Argo CD
app + k8s manifests mirroring neuron-prod. All deployments set to
replicas=0 — nothing runs until Tim's imprint is ready and Vault
secrets are populated at secret/neuron-technologies/tim.
2026-04-24 12:52:36 -05:00
Will Anderson 92743b1efc add redpanda to legion cluster
Deploy single-broker Redpanda in dedicated namespace with 20Gi storage,
1 CPU / 2Gi memory limits, and a one-shot topic init job for neuron.swarm,
neuron.ci, neuron.vcs, and neuron.webhooks with appropriate retention.
2026-04-24 12:39:02 -05:00
Will Anderson cd0cb30933 ci: add neuron-technologies runner for Gitea Actions 2026-04-23 21:19:26 -05:00
Will Anderson b2c25e4e0f chore(neuron): promote 617dcd1d to prod (fix begin_work NOT NULL data column) 2026-04-24 00:26:24 +00:00
Gitea CI c30da5d1a8 chore(neuron): deploy 617dcd1d5c1d148838b8e73df1bba8296257c0b6 to alpha/beta/gamma 2026-04-24 00:21:40 +00:00
Gitea CI 48964eccac chore(neuron): deploy 3e230d550235599d21edaa1df0a22949ccbf695a to alpha/beta/gamma 2026-04-23 19:35:30 +00:00
Gitea CI 3d739b6c5b chore(neuron): deploy 539fc9e7f333dfa0cd31772fc09a976b3079090c to alpha 2026-04-23 19:30:27 +00:00
Will Anderson a8ef7e1782 feat(neuron): set NEURON_SERVER_NAME per slot so each advertises a unique MCP identity 2026-04-23 14:27:03 -05:00
Will Anderson cc4db4dc11 feat(neuron): gamma zone, all-slot ingress, snapshot job, self-improvement loop script 2026-04-23 13:57:13 -05:00
Gitea CI a9758b6eda chore(neuron): deploy 7d29e699c061c3cdb907fc2326b67ad6601298a9 to alpha 2026-04-23 18:51:52 +00:00
Will Anderson 27b4b1e066 feat(neuron): add alpha slot + lb Service; standardize all slots to port 8001 2026-04-23 13:47:21 -05:00
Gitea CI 6dd67730b7 chore(neuron): deploy bddce6b2bd11428efe3eac0dcb269da5490014c4 2026-04-23 18:36:29 +00:00
Will Anderson 882c4a79d4 feat(neuron): add neuron-beta deployment on port 8002 with separate PVC 2026-04-23 13:34:55 -05:00
Will Anderson 60d0d81144 chore(ci-runner): roll pod to pick up NodePort URL from updated secret 2026-04-23 13:16:18 -05:00
Will Anderson 39808ab964 fix(ci-runner): add extra_hosts for cluster DNS in CI Docker containers 2026-04-23 12:50:53 -05:00
Will Anderson 5a62077b99 fix(gitea): allow unauthenticated API ping for runner registration 2026-04-23 12:43:47 -05:00
Will Anderson 095d874552 Remove docuseal-old temp instance 2026-04-21 18:15:49 -05:00
Will Anderson 0cbb06660a Add temporary docuseal-old at sign.neuralplatform.ai for license transfer 2026-04-21 17:33:40 -05:00
Will Anderson d40fb8c3f1 Add Docuseal e-signature service at sign.neuralplatform.ai
- k8s manifests: Deployment, Service, PVC, Ingress, ExternalSecret
- Argo CD app watching servers/legion/k8s/docuseal
- CF tunnel ingress rule for sign.neuralplatform.ai
- Vault secret at secret/docuseal (secret_key_base seeded)
2026-04-21 14:26:01 -05:00
Will Anderson 3b3811942c Update Legion LAN IP from 192.168.68.77 to 192.168.8.148
Static IP assigned on new network segment. Updates:
- variables.tf default legion_ip
- headscale nameserver config
- bootstrap.sh default target
- README/RUNBOOK documentation
- media ingress comment
2026-04-21 10:32:39 -05:00
Gitea CI 26c7d2259e chore(neuron): deploy efc982c605cacc6929f29382f293e2a2d4a4cf95 2026-04-18 20:28:36 +00:00
Will Anderson 193bea7f3f fix(neuron): add ephemeral-storage limits to prevent eviction cascade 2026-04-17 09:59:21 -05:00
Will Anderson 1b8ea63541 feat(neuron): add stage and prod k8s manifests + Argo CD apps
Completes the three-env setup (dev was added previously). Each env has
deployments, services, ingress, PVC, ConfigMap, and ExternalSecret pulling
from Vault. Prod has larger resource limits and 10Gi PVC. Vault secret at
secret/neuron-technologies/prod seeded separately.
2026-04-17 09:43:23 -05:00
Will Anderson 912e129e0d feat(neuron-dev): add k8s manifests and Argo CD app for neuron-technologies dev environment 2026-04-17 08:46:47 -05:00
Will Anderson ad2a1db3e2 Remove conflicting fornax-worker Argo CD apps — media app is sole owner
The fornax-worker-tx253 and fornax-worker-tx34 Argo CD apps were fighting
the media app (which also manages the same Deployments, Services, PVCs,
and ExternalSecrets via fornax-workers.yaml). This caused 73+ rolling
restarts per 2 hours. Since neither app had a cascade-delete finalizer,
removing these Application CRDs leaves existing resources intact and
transfers ownership to the media app.
2026-04-14 22:25:25 -05:00
Will Anderson 4f406e13c9 add fornax-ui Argo CD application
Deploys fornax/ui:latest (nginx serving React SPA) to the media
namespace. Image is now pushed by the fornax CI pipeline.
2026-04-14 12:48:42 -05:00
Will Anderson bdba692db2 Fornax: COORDINATOR_URL env, port-lease webhook, Grafana dashboard
- Worker deployment: add COORDINATOR_URL and WORKER_ID env vars to
  portforward-helper; webhook calls coordinator /api/v2/workers/:id/port-lease
  after each NAT-PMP renewal so coordinator tracks current forwarded port
- Worker overlays: patch WORKER_ID env var to actual worker ID (tx253/tx34)
- Grafana: fornax-grafana Argo CD app pointing to k8s/fornax/grafana/
  ConfigMap with grafana_dashboard=1 label for auto-discovery by sidecar;
  7 panels covering bandwidth, active torrents, worker health, request rate,
  and error rate
2026-04-14 05:30:01 +00:00
Will Anderson 180fb91c95 add fornax coordinator and worker k8s manifests 2026-04-13 23:57:22 +00:00
Gitea CI 0aa1f4333c chore(neuron): deploy e102af6f86599361b911a2e7e5aa0ec737b54527 2026-04-12 19:58:46 +00:00