649 Commits

Author SHA1 Message Date
Will Anderson 5f984e1fc2 gitea-runner: revert to pantheon-runner (container: directive unsupported in host mode) 2026-03-26 07:53:48 -05:00
Will Anderson 4e9d3287d4 gitea-runner: use lean legion-runner base image 2026-03-26 07:51:21 -05:00
Will Anderson f409c06e65 gitea-runner: use pantheon-runner image (act_runner + all CI tools) 2026-03-25 20:39:32 -05:00
Will Anderson 08d4ab1d72 gitea-runner: set default_image in container config to use pantheon-ci for all jobs 2026-03-25 20:37:17 -05:00
Will Anderson d5c5d753c7 ci: use custom pantheon-ci image; fix registry 413 with Traefik buffering middleware 2026-03-25 20:36:20 -05:00
Will Anderson 9904bce343 gitea-runner: use internal cluster URL to avoid CF tunnel timeouts on runner long-polling 2026-03-25 20:29:23 -05:00
Will Anderson ebb42a6dd2 gitea-runner: switch to native host execution with Legion tool mounts 2026-03-25 20:27:58 -05:00
Will Anderson 672a890659 upgrade gitea 1.23 → 1.25.5 2026-03-25 20:22:54 -05:00
Will Anderson 35055ec783 upgrade gitea to 1.23 2026-03-25 20:04:05 -05:00
Will Anderson e19cadf971 docs: sync infrastructure docs with actual running state
- Fix direnv path typo in AGENTS.md (servers/servers → servers)
- Correct Prometheus/Alertmanager as internal-only (no public ingresses)
- Add Alloy public URL (alloy.neuralplatform.ai)
- Add external-secrets namespace to namespaces table
- Add contexthub org to Gitea orgs list
- Document Gitea SSH config (Host gitea → 192.168.68.77:30022)
- Specify GPU model: GTX 1660 Ti 6GB
- Add devpi to Legion README services table
2026-03-25 15:59:12 -05:00
Will Anderson b7f8d70fd3 docs: update AGENTS.md — headscale, monitoring, tea CLI, namespace details
- Add Headscale VPN entry to services table with note about direct TLS
- Add Grafana/Prometheus/Alertmanager/Alloy to platform services
- Add VPN section with Headscale DNS configuration notes
- Add tea CLI section (replaces raw Gitea API examples)
- Expand namespace table with service detail and cert-manager note
2026-03-25 13:53:25 -05:00
Will Anderson 680b176e45 docs: update AGENTS.md to reflect Terraform->ArgoCD migration and ESO secrets flow
- Terraform now owns infra only (namespaces, Cloudflare, R2), not k8s config
- Argo CD manages all k8s resources via git
- Secrets: Vault -> ExternalSecret -> ESO -> k8s Secret (no Terraform secrets)
- Updated Common Operations for new GitOps flow
- Added tea CLI note, corrected repo paths
2026-03-25 11:05:52 -05:00
Will Anderson 24f2dcbf9b Fix headscale DNS nameservers: use LAN IP and Cloudflare fallback 2026-03-25 10:59:45 -05:00
Will Anderson 371e60d3bc fix: add admin-user key to grafana-admin-secret ExternalSecret 2026-03-25 10:52:10 -05:00
Will Anderson 3737ff0bcb fix: pin PVCs to existing PVs; fix kube-prometheus-stack grafana auth
PVC manifests: add volumeName to bind new PVCs to existing PVs whose
old PVCs were accidentally deleted during Terraform->ArgoCD migration.
All PVs patched to Retain to prevent data loss on pod restart.

kube-prometheus-stack: replace invalid adminPassword.valueFrom.secretKeyRef
(Helm values don't support k8s secretKeyRef syntax) with correct
admin.existingSecret + admin.passwordKey pattern.
2026-03-25 10:48:10 -05:00
Will Anderson f1fc1fafea Fix ddclient image: use linuxserver/ddclient (correct image) 2026-03-25 10:36:23 -05:00
Will Anderson 6c073029a3 Fix gitea-runner ExternalSecret: use template for static GITEA_INSTANCE_URL 2026-03-25 10:34:11 -05:00
Will Anderson 4d0cfb1bbf Migrate k8s config from Terraform to Argo CD + ESO
Phase 1: Install External Secrets Operator via Argo CD app
- apps/external-secrets.yaml — ESO Helm chart install
- apps/external-secrets-config.yaml — ClusterSecretStore deployment
- k8s/external-secrets/cluster-secret-store.yaml — Vault backend using vault-token Secret

Phase 2: Create k8s manifests for all services
- k8s/neuron/ — PVC, ConfigMap, ExternalSecrets (neuron-secrets, cloudflared-secret), Ingress
- k8s/gitea/ — PVC, ConfigMap (custom CSS), ExternalSecret (gitea-db), Ingress
- k8s/github-runner/ — ExternalSecret (github-runner-secret)
- k8s/gitea-runner/ — ExternalSecret (gitea-runner-secret)
- k8s/monitoring/ — ExternalSecrets (grafana, slack), Alloy OTLP service+middleware, datasources ConfigMap, Ingress
- k8s/postgres/ — ExternalSecret (postgres-passwords)
- k8s/vault/ — ExternalSecret (vault-gcp-sa from Vault)
- k8s/adguard/ — PVCs, ConfigMap, Certificate, Ingress, ddclient Deployment+ExternalSecret
- k8s/ollama/ — PVC, Ingress
- k8s/headscale/ — PVC
- k8s/packages/ — PVCs, ConfigMap, Ingresses
- k8s/registry/ — PVC, Ingresses
- k8s/backup/ — CronJob, ExternalSecret (backup-credentials)

New Argo CD apps for Helm releases:
- apps/kube-prometheus-stack.yaml, loki.yaml, tempo.yaml, alloy.yaml
- apps/postgres.yaml, redis.yaml, vault.yaml

New Argo CD apps for k8s config paths:
- apps/neuron-config, gitea-config, ci-config, gitea-runner-config
- apps/monitoring-config, adguard-config, ollama-config, headscale-config
- apps/packages-config, registry-config, postgres-config, vault-config, backup

Phase 3: Strip Terraform to infrastructure-only
- All kubernetes_* and helm_release resources removed from service .tf files
- Each service .tf now contains only kubernetes_namespace (bootstrap dependency)
- variables.tf stripped to only cloudflare_api_key, cloudflare_email, gitea_api_token
- namespaces.tf gains external-secrets namespace
- ingress.tf, backup.tf, ddclient.tf emptied (resources in k8s/)
- cert-manager.tf, argocd.tf, traefik.tf unchanged (bootstrap)
2026-03-25 10:29:14 -05:00
Will Anderson dd0f8a49a3 monitoring: expose Alloy OTLP for external log ingestion from RunPod pods
- Alloy config: route OTLP logs → Loki (was traces-only)
- Alloy config: fix River syntax (semicolons in rule{} blocks are invalid)
- Add alloy-otlp Service (ClusterIP:4318)
- Add alloy-otlp-auth Secret (bcrypt htpasswd for Traefik BasicAuth)
- Add alloy-otlp-basicauth-middleware Traefik Middleware
- Add alloy.neuralplatform.ai Ingress (TLS via cert-manager)
  Auth: Authorization: Basic base64(pantheon:<push_token>)
  Token stored in Vault at secret/alloy push_token

RunPod pods push OTLP logs to https://alloy.neuralplatform.ai/v1/logs
Alloy routes → Loki → queryable in Grafana
2026-03-25 10:15:41 -05:00
Will Anderson 6ebf8a9b3e Fix gitea_domain variable and add VPN/Headscale infrastructure
- Fix gitea_domain default from git.neuralplatform.dev → git.neuralplatform.ai
  so Traefik serves the correct Let's Encrypt cert when accessed via VPN/LAN
- Add docs/vpn-comparison.md: full evaluation of WireGuard/Tailscale/Headscale/Netbird/Netmaker
2026-03-25 08:44:20 -05:00
Will Anderson abe2dad093 headscale: fix base_domain conflict, set ts.neuralplatform.ai for MagicDNS 2026-03-25 08:35:59 -05:00
Will Anderson a461172473 headscale: rename coordination endpoint to vpn.neuralplatform.ai 2026-03-25 08:16:47 -05:00
Will Anderson 5334881d3f infra: add Headscale — self-hosted Tailscale control plane
- headscale.tf: namespace + 1Gi PVC for SQLite state
- apps/headscale.yaml: Deployment (headscale/headscale:0.23) + Service + ConfigMap
  - server_url: https://headscale.neuralplatform.ai
  - sqlite3 backend, DERP via Tailscale's map
  - MagicDNS base: vpn.neuralplatform.ai
- Cloudflare tunnel: headscale.neuralplatform.ai → headscale.headscale.svc:8080
- DNS CNAME added via API

Next: install tailscaled on Legion host, enroll as subnet router
2026-03-25 07:59:46 -05:00
Will Anderson 1312be1a03 runner: bind-mount host kubectl into job containers; registry domain → neuralplatform.ai
gitea-runner: add container.options to mount /usr/local/bin/kubectl:ro
from the Legion host into every act_runner job container. Eliminates
per-run curl download in CI workflows.

ingress: change registry ingress hostname from registry.nook.family to
registry.neuralplatform.ai. Matches the domain used in Neuron CI and
neuron.yaml image reference. DNS CNAME + tunnel route already added.
2026-03-24 14:39:57 -05:00
Will Anderson 7c08543ca4 fix: gitea runner uses public URL via CF Access bypass
- Added Cloudflare Access bypass for git.neuralplatform.ai/api/actions/*
  restricted to Legion public IP (72.204.75.49)
- Runner now registers and communicates with git.neuralplatform.ai directly
- Job containers clone from https://git.neuralplatform.ai (proper name)
2026-03-24 13:40:31 -05:00
Will Anderson 455797745c fix: runner container network=host for k8s DNS resolution
Job containers use bridge network by default, so they can't resolve
gitea.git.svc.cluster.local. host network inherits k3s host routing
which can reach cluster service IPs directly.
2026-03-24 13:20:44 -05:00
Will Anderson 05973b50bd fix: remove duplicate docker.sock mount in runner config
The socket is already mounted via the pod volume mount; specifying it
again in container.options causes 'Duplicate mount point' on job containers.
2026-03-24 13:14:43 -05:00
Will Anderson e91d6c0b09 fix: gitea runner Docker execution mode with catthehacker/ubuntu
Switch from :host to :docker execution so jobs run inside containers.
- Labels use docker://ghcr.io/catthehacker/ubuntu:act-22.04 (Python 3.10,
  Node 18, Docker CLI — everything CI jobs need pre-installed)
- config.yaml passes docker.sock through to job containers for build/push
- Capacity set to 2 parallel jobs

Fixes: 'Cannot find: node in PATH' — actions/checkout@v4 requires Node
2026-03-24 13:07:56 -05:00
Will Anderson 0341b570ab fix: gitea runner workingDir, README updates, monitoring domain
- Fix act-runner workingDir: /data on both init and main containers so
  .runner registration file is found at daemon startup
- Add servers/legion/README.md with full service catalog and deploy docs
- Update root README.md to be an index pointing to key docs
- Fix monitoring.tf comment: grafana.neuralplatform.ai (not nook.family)
- Fix AGENTS.md router model to TP-Link Deco BE5000
2026-03-24 12:55:29 -05:00
Will Anderson 5bd7702874 Add Gitea act-runner and fix ddclient deployment
- gitea-runner.tf: Secret for Gitea runner registration token
- apps/gitea-runner.yaml: Gitea act-runner deployment (init container
  registers with Gitea, main container runs daemon) — serves all Gitea
  Actions CI jobs on self-hosted,linux,x64,legion labels
- variables.tf: Add gitea_runner_token variable
- ddclient.tf: Switch to official ghcr.io/ddclient/ddclient image,
  remove linuxserver wrapper that was exhausting inotify instances
2026-03-24 12:51:22 -05:00
Will Anderson 087c5a79f4 Add native DNS-over-TLS (DoT) on dot.nook.family:853
- ddclient.tf: dynamic DNS daemon that keeps dot.nook.family A record
  pointing at the home public IP via Cloudflare API (updates every 5m)
- adguard.tf: cert-manager Certificate for dot.nook.family via DNS-01,
  updated TLS config block with DoT paths and port 853
- apps/adguard.yaml: mount adguard-dot-tls secret, expose port 853,
  init container enforces TLS/DoT settings on every restart
- adguard.tf: fix stale neuralplatform.dev → neuralplatform.ai rewrite

Android Private DNS: set to dot.nook.family
Router: forward TCP 853 → 192.168.68.77
2026-03-24 09:27:21 -05:00
Will Anderson 3518620360 fix: PGPASSWORD env var, neuron registry, gitea domain
- backup.tf: rename POSTGRES_PASSWORD → PGPASSWORD so pg_dumpall
  can authenticate (was silently failing for 2 days)
- neuron.yaml: fix image registry nook.family → neuralplatform.ai
- gitea.yaml: fix domain neuralplatform.dev → neuralplatform.ai
2026-03-24 08:08:40 -05:00
Will Anderson 1c8d7cecd8 monitoring: wire Alertmanager to #infrastructure-alerts in Slack
Routes all warning/critical alerts to the Neural Platform Slack workspace.
Suppresses Watchdog and info-level noise. Groups by namespace+alertname,
repeats every 4h. Uses bot token auth via chat.postMessage API.
2026-03-24 01:45:06 -05:00
Will Anderson f8bc853136 DR: full bootstrap script + recovery runbook
bootstrap.sh now covers a complete fresh Ubuntu 24.04 setup:
- NVIDIA driver 580 + container toolkit + k3s GPU config
- k3s install with nvidia as default runtime + device plugin
- Host cloudflared (Gitea SSH tunnel)
- Terraform + Helm
- systemd-resolved disabled, resolv.conf pinned

RUNBOOK.md documents the full 4-step recovery:
bootstrap → terraform apply → Gitea/Postgres restore → verify
Target: under 1 hour on a new machine.
2026-03-24 00:46:36 -05:00
Will Anderson 82b902f7c4 Move AdGuard to dns.nook.family, fix Agents.md service domains 2026-03-23 10:43:38 -05:00
Will Anderson bf2ec73983 Expose AdGuard DoH at dns.neuralplatform.ai 2026-03-23 10:29:42 -05:00
Will Anderson 8e208045de Fix init container: install pyyaml before import 2026-03-23 10:11:51 -05:00
Will Anderson 0e6b924e70 AdGuard: enforce filter lists and DNS upstreams via init container
ConfigMap adguard-defaults holds desired settings (upstreams, bootstrap DNS,
filter lists, rewrites). Init container merges them into the PVC config on
every pod start — no more manual kubectl edits needed.

- Switch upstreams to Cloudflare + Google DoH (Quad9 was failing)
- Bootstrap DNS IPv4-only (removes IPv6 timeout errors)
- Add OISD Big, EasyList, EasyPrivacy filter lists
- Fix DNS rewrites pointing to old IP 192.168.0.105
2026-03-23 10:05:41 -05:00
Will Anderson 6e856993a0 Add Agents.md to infrastructure repo root 2026-03-23 08:51:45 -05:00
Will Anderson f50b1b34c1 Move Agents.md to top-level infrastructure repo 2026-03-23 08:38:24 -05:00
Will Anderson bca06c7216 Add Agents.md — system reference for AI agents 2026-03-23 08:37:41 -05:00
Will Anderson afe01d2ad9 Migrate all app deployments from Terraform to Argo CD
Move gitea, github-runner, neuron, cloudflared, ollama, verdaccio,
devpi, registry, and registry-ui deployments+services to apps/*.yaml
so Argo CD manages the app layer. Terraform retains namespaces, PVCs,
ConfigMaps, Secrets, and Ingresses.

New Secrets in Terraform:
- kubernetes_secret.github_runner_secret (ci/github-runner-secret)
- kubernetes_secret.cloudflared_secret (neuron/cloudflared-secret)

Hardcoded service names in ingress.tf and neuron.tf to remove
dependency on removed kubernetes_service resources.
2026-03-23 08:15:17 -05:00
Will Anderson b2b04c231c Migrate AdGuard deployment+service to Argo CD apps/ 2026-03-23 08:05:18 -05:00
Will Anderson 5231adc3a0 Configure Argo CD to watch servers/legion/apps/ in will/infrastructure 2026-03-23 08:00:37 -05:00
Will Anderson ef2ae715ee Argo CD on neuralplatform.ai, fix domain_suffix vs infra_domain usage 2026-03-23 07:57:10 -05:00
Will Anderson 13e71fd491 Add Argo CD — GitOps controller for legion app deployments 2026-03-23 07:52:07 -05:00
Will Anderson 7f022d3a67 Remove null_resources, add bootstrap.sh, fix k3s cert for new LAN IP 2026-03-23 07:49:57 -05:00
Will Anderson 0b4a236a88 Restructure: servers/legion/ layout, rename repo to infrastructure 2026-03-23 07:38:42 -05:00
Will Anderson 7751eddd73 Initial commit — migrate legion Terraform from infrastructure repo 2026-03-23 07:35:03 -05:00