Commit Graph

613 Commits

Author SHA1 Message Date
Will Anderson e4db8bbda5 ci: switch to Forgejo runner for proper DEFAULT_ACTIONS_URL support 2026-03-26 12:58:22 -05:00
Will Anderson a40897fd5d ci: add ubuntu-latest, ubuntu-24.04 labels to runner 2026-03-26 12:48:38 -05:00
Will Anderson c9bee67e67 feat: make ci-base equivalent to ubuntu-latest
Comprehensive runner image with Python, Node.js 20, Ruby, Go 1.22,
Docker CLI, kubectl, Helm, yq, gh, make, jq, rsync, zstd, and all
standard tools. Mirrors GitHub ubuntu-24.04 hosted runner so workflows
are swappable between Gitea and GitHub with no friction.
2026-03-26 12:11:38 -05:00
Will Anderson 798cf8611d feat: make ci-base a comprehensive runner image like ubuntu-latest
Add Node.js 20 LTS (required for all Forgejo JS actions), Python 3.12,
Ruby, Go 1.22, Docker CLI, kubectl, Helm, yq, gh, zstd, and common
build/system tools. Mirrors GitHub ubuntu-24.04 hosted runner capability
so jobs work without needing container: overrides for standard tasks.
2026-03-26 12:08:16 -05:00
Will Anderson db08c5a208 ci: set runner default_image to ubuntu:24.04
Replace pantheon-ci:latest (custom, potentially stale) with ubuntu:24.04
as the fallback container for jobs that don't specify container:.
2026-03-26 11:41:55 -05:00
Will Anderson 978639601c fix: correct act_runner binary path in ci-base Dockerfile 2026-03-26 11:35:37 -05:00
Will Anderson 9595a18f3a refactor: rebase CI runner on Ubuntu 24.04, rename to ci-base
Replace Alpine-based legion-runner with a minimal Ubuntu 24.04 image.
The act_runner binary is copied from the official image via multi-stage
build. Job-specific tooling (Ruby, Node, Python, etc.) belongs in
per-job container images specified via container: in workflows, not
in the base runner image.

Rename: legion-runner → ci-base
2026-03-26 11:32:52 -05:00
Will Anderson d163a92ec6 fix: replace registry Ingress+Buffering with IngressRoute for streaming
The Buffering middleware was causing 413 on large Docker layer pushes
by spooling entire request bodies to disk before forwarding. Docker's
chunked blob upload protocol (PATCH/PUT with Content-Range) requires
streaming, not buffering.

Replace with IngressRoute CRDs + Certificate resources for direct
streaming passthrough. responseForwarding.flushInterval=100ms ensures
prompt chunk delivery to the registry backend.
2026-03-26 11:13:25 -05:00
Will Anderson 206873e83b fix: set DEFAULT_ACTIONS_URL to Forgejo action mirrors
code.forgejo.org maintains up-to-date mirrors of common GitHub Actions
(checkout, upload-artifact, etc.) and is the recommended source for
Gitea Actions runners.
2026-03-26 11:06:43 -05:00
Will Anderson ce45384f81 feat: add Ruby and bundler to legion-runner image
Required for Jekyll-based doc builds in CI.
2026-03-26 10:59:36 -05:00
Will Anderson 8425f0988c fix: set Gitea DEFAULT_ACTIONS_URL to self
https://gitea.com is no longer a valid value in Gitea 1.25 — causes
startup crash. Use 'self' to point action references at this instance.
2026-03-26 10:41:54 -05:00
Will Anderson d8f2c822f8 fix: DEFAULT_ACTIONS_URL must be a full URL, not 'gitea' 2026-03-26 10:36:31 -05:00
Will Anderson f3477ac102 ci: fix gitea runner URL and set DEFAULT_ACTIONS_URL
- external-secrets.yaml: GITEA_INSTANCE_URL changed from cluster-internal
  URL to https://git.neuralplatform.ai so runner can register externally
- gitea.yaml: add GITEA__actions__DEFAULT_ACTIONS_URL=gitea so all
  actions/* references resolve from gitea.com, not github.com
- Dockerfile: track runner image (Python 3.12 + build deps) in repo
2026-03-26 10:35:13 -05:00
Will Anderson 737fb27a5c gitea-runner: use lean legion-runner base image 2026-03-26 08:01:41 -05:00
Will Anderson 5f984e1fc2 gitea-runner: revert to pantheon-runner (container: directive unsupported in host mode) 2026-03-26 07:53:48 -05:00
Will Anderson 4e9d3287d4 gitea-runner: use lean legion-runner base image 2026-03-26 07:51:21 -05:00
Will Anderson f409c06e65 gitea-runner: use pantheon-runner image (act_runner + all CI tools) 2026-03-25 20:39:32 -05:00
Will Anderson 08d4ab1d72 gitea-runner: set default_image in container config to use pantheon-ci for all jobs 2026-03-25 20:37:17 -05:00
Will Anderson d5c5d753c7 ci: use custom pantheon-ci image; fix registry 413 with Traefik buffering middleware 2026-03-25 20:36:20 -05:00
Will Anderson 9904bce343 gitea-runner: use internal cluster URL to avoid CF tunnel timeouts on runner long-polling 2026-03-25 20:29:23 -05:00
Will Anderson ebb42a6dd2 gitea-runner: switch to native host execution with Legion tool mounts 2026-03-25 20:27:58 -05:00
Will Anderson 672a890659 upgrade gitea 1.23 → 1.25.5 2026-03-25 20:22:54 -05:00
Will Anderson 35055ec783 upgrade gitea to 1.23 2026-03-25 20:04:05 -05:00
Will Anderson e19cadf971 docs: sync infrastructure docs with actual running state
- Fix direnv path typo in AGENTS.md (servers/servers → servers)
- Correct Prometheus/Alertmanager as internal-only (no public ingresses)
- Add Alloy public URL (alloy.neuralplatform.ai)
- Add external-secrets namespace to namespaces table
- Add contexthub org to Gitea orgs list
- Document Gitea SSH config (Host gitea → 192.168.68.77:30022)
- Specify GPU model: GTX 1660 Ti 6GB
- Add devpi to Legion README services table
2026-03-25 15:59:12 -05:00
Will Anderson b7f8d70fd3 docs: update AGENTS.md — headscale, monitoring, tea CLI, namespace details
- Add Headscale VPN entry to services table with note about direct TLS
- Add Grafana/Prometheus/Alertmanager/Alloy to platform services
- Add VPN section with Headscale DNS configuration notes
- Add tea CLI section (replaces raw Gitea API examples)
- Expand namespace table with service detail and cert-manager note
2026-03-25 13:53:25 -05:00
Will Anderson 680b176e45 docs: update AGENTS.md to reflect Terraform->ArgoCD migration and ESO secrets flow
- Terraform now owns infra only (namespaces, Cloudflare, R2), not k8s config
- Argo CD manages all k8s resources via git
- Secrets: Vault -> ExternalSecret -> ESO -> k8s Secret (no Terraform secrets)
- Updated Common Operations for new GitOps flow
- Added tea CLI note, corrected repo paths
2026-03-25 11:05:52 -05:00
Will Anderson 24f2dcbf9b Fix headscale DNS nameservers: use LAN IP and Cloudflare fallback 2026-03-25 10:59:45 -05:00
Will Anderson 371e60d3bc fix: add admin-user key to grafana-admin-secret ExternalSecret 2026-03-25 10:52:10 -05:00
Will Anderson 3737ff0bcb fix: pin PVCs to existing PVs; fix kube-prometheus-stack grafana auth
PVC manifests: add volumeName to bind new PVCs to existing PVs whose
old PVCs were accidentally deleted during Terraform->ArgoCD migration.
All PVs patched to Retain to prevent data loss on pod restart.

kube-prometheus-stack: replace invalid adminPassword.valueFrom.secretKeyRef
(Helm values don't support k8s secretKeyRef syntax) with correct
admin.existingSecret + admin.passwordKey pattern.
2026-03-25 10:48:10 -05:00
Will Anderson f1fc1fafea Fix ddclient image: use linuxserver/ddclient (correct image) 2026-03-25 10:36:23 -05:00
Will Anderson 6c073029a3 Fix gitea-runner ExternalSecret: use template for static GITEA_INSTANCE_URL 2026-03-25 10:34:11 -05:00
Will Anderson 4d0cfb1bbf Migrate k8s config from Terraform to Argo CD + ESO
Phase 1: Install External Secrets Operator via Argo CD app
- apps/external-secrets.yaml — ESO Helm chart install
- apps/external-secrets-config.yaml — ClusterSecretStore deployment
- k8s/external-secrets/cluster-secret-store.yaml — Vault backend using vault-token Secret

Phase 2: Create k8s manifests for all services
- k8s/neuron/ — PVC, ConfigMap, ExternalSecrets (neuron-secrets, cloudflared-secret), Ingress
- k8s/gitea/ — PVC, ConfigMap (custom CSS), ExternalSecret (gitea-db), Ingress
- k8s/github-runner/ — ExternalSecret (github-runner-secret)
- k8s/gitea-runner/ — ExternalSecret (gitea-runner-secret)
- k8s/monitoring/ — ExternalSecrets (grafana, slack), Alloy OTLP service+middleware, datasources ConfigMap, Ingress
- k8s/postgres/ — ExternalSecret (postgres-passwords)
- k8s/vault/ — ExternalSecret (vault-gcp-sa from Vault)
- k8s/adguard/ — PVCs, ConfigMap, Certificate, Ingress, ddclient Deployment+ExternalSecret
- k8s/ollama/ — PVC, Ingress
- k8s/headscale/ — PVC
- k8s/packages/ — PVCs, ConfigMap, Ingresses
- k8s/registry/ — PVC, Ingresses
- k8s/backup/ — CronJob, ExternalSecret (backup-credentials)

New Argo CD apps for Helm releases:
- apps/kube-prometheus-stack.yaml, loki.yaml, tempo.yaml, alloy.yaml
- apps/postgres.yaml, redis.yaml, vault.yaml

New Argo CD apps for k8s config paths:
- apps/neuron-config, gitea-config, ci-config, gitea-runner-config
- apps/monitoring-config, adguard-config, ollama-config, headscale-config
- apps/packages-config, registry-config, postgres-config, vault-config, backup

Phase 3: Strip Terraform to infrastructure-only
- All kubernetes_* and helm_release resources removed from service .tf files
- Each service .tf now contains only kubernetes_namespace (bootstrap dependency)
- variables.tf stripped to only cloudflare_api_key, cloudflare_email, gitea_api_token
- namespaces.tf gains external-secrets namespace
- ingress.tf, backup.tf, ddclient.tf emptied (resources in k8s/)
- cert-manager.tf, argocd.tf, traefik.tf unchanged (bootstrap)
2026-03-25 10:29:14 -05:00
Will Anderson dd0f8a49a3 monitoring: expose Alloy OTLP for external log ingestion from RunPod pods
- Alloy config: route OTLP logs → Loki (was traces-only)
- Alloy config: fix River syntax (semicolons in rule{} blocks are invalid)
- Add alloy-otlp Service (ClusterIP:4318)
- Add alloy-otlp-auth Secret (bcrypt htpasswd for Traefik BasicAuth)
- Add alloy-otlp-basicauth-middleware Traefik Middleware
- Add alloy.neuralplatform.ai Ingress (TLS via cert-manager)
  Auth: Authorization: Basic base64(pantheon:<push_token>)
  Token stored in Vault at secret/alloy push_token

RunPod pods push OTLP logs to https://alloy.neuralplatform.ai/v1/logs
Alloy routes → Loki → queryable in Grafana
2026-03-25 10:15:41 -05:00
Will Anderson 6ebf8a9b3e Fix gitea_domain variable and add VPN/Headscale infrastructure
- Fix gitea_domain default from git.neuralplatform.dev → git.neuralplatform.ai
  so Traefik serves the correct Let's Encrypt cert when accessed via VPN/LAN
- Add docs/vpn-comparison.md: full evaluation of WireGuard/Tailscale/Headscale/Netbird/Netmaker
2026-03-25 08:44:20 -05:00
Will Anderson abe2dad093 headscale: fix base_domain conflict, set ts.neuralplatform.ai for MagicDNS 2026-03-25 08:35:59 -05:00
Will Anderson a461172473 headscale: rename coordination endpoint to vpn.neuralplatform.ai 2026-03-25 08:16:47 -05:00
Will Anderson 5334881d3f infra: add Headscale — self-hosted Tailscale control plane
- headscale.tf: namespace + 1Gi PVC for SQLite state
- apps/headscale.yaml: Deployment (headscale/headscale:0.23) + Service + ConfigMap
  - server_url: https://headscale.neuralplatform.ai
  - sqlite3 backend, DERP via Tailscale's map
  - MagicDNS base: vpn.neuralplatform.ai
- Cloudflare tunnel: headscale.neuralplatform.ai → headscale.headscale.svc:8080
- DNS CNAME added via API

Next: install tailscaled on Legion host, enroll as subnet router
2026-03-25 07:59:46 -05:00
Will Anderson 1312be1a03 runner: bind-mount host kubectl into job containers; registry domain → neuralplatform.ai
gitea-runner: add container.options to mount /usr/local/bin/kubectl:ro
from the Legion host into every act_runner job container. Eliminates
per-run curl download in CI workflows.

ingress: change registry ingress hostname from registry.nook.family to
registry.neuralplatform.ai. Matches the domain used in Neuron CI and
neuron.yaml image reference. DNS CNAME + tunnel route already added.
2026-03-24 14:39:57 -05:00
Will Anderson 7c08543ca4 fix: gitea runner uses public URL via CF Access bypass
- Added Cloudflare Access bypass for git.neuralplatform.ai/api/actions/*
  restricted to Legion public IP (72.204.75.49)
- Runner now registers and communicates with git.neuralplatform.ai directly
- Job containers clone from https://git.neuralplatform.ai (proper name)
2026-03-24 13:40:31 -05:00
Will Anderson 455797745c fix: runner container network=host for k8s DNS resolution
Job containers use bridge network by default, so they can't resolve
gitea.git.svc.cluster.local. host network inherits k3s host routing
which can reach cluster service IPs directly.
2026-03-24 13:20:44 -05:00
Will Anderson 05973b50bd fix: remove duplicate docker.sock mount in runner config
The socket is already mounted via the pod volume mount; specifying it
again in container.options causes 'Duplicate mount point' on job containers.
2026-03-24 13:14:43 -05:00
Will Anderson e91d6c0b09 fix: gitea runner Docker execution mode with catthehacker/ubuntu
Switch from :host to :docker execution so jobs run inside containers.
- Labels use docker://ghcr.io/catthehacker/ubuntu:act-22.04 (Python 3.10,
  Node 18, Docker CLI — everything CI jobs need pre-installed)
- config.yaml passes docker.sock through to job containers for build/push
- Capacity set to 2 parallel jobs

Fixes: 'Cannot find: node in PATH' — actions/checkout@v4 requires Node
2026-03-24 13:07:56 -05:00
Will Anderson 0341b570ab fix: gitea runner workingDir, README updates, monitoring domain
- Fix act-runner workingDir: /data on both init and main containers so
  .runner registration file is found at daemon startup
- Add servers/legion/README.md with full service catalog and deploy docs
- Update root README.md to be an index pointing to key docs
- Fix monitoring.tf comment: grafana.neuralplatform.ai (not nook.family)
- Fix AGENTS.md router model to TP-Link Deco BE5000
2026-03-24 12:55:29 -05:00
Will Anderson 5bd7702874 Add Gitea act-runner and fix ddclient deployment
- gitea-runner.tf: Secret for Gitea runner registration token
- apps/gitea-runner.yaml: Gitea act-runner deployment (init container
  registers with Gitea, main container runs daemon) — serves all Gitea
  Actions CI jobs on self-hosted,linux,x64,legion labels
- variables.tf: Add gitea_runner_token variable
- ddclient.tf: Switch to official ghcr.io/ddclient/ddclient image,
  remove linuxserver wrapper that was exhausting inotify instances
2026-03-24 12:51:22 -05:00
Will Anderson 087c5a79f4 Add native DNS-over-TLS (DoT) on dot.nook.family:853
- ddclient.tf: dynamic DNS daemon that keeps dot.nook.family A record
  pointing at the home public IP via Cloudflare API (updates every 5m)
- adguard.tf: cert-manager Certificate for dot.nook.family via DNS-01,
  updated TLS config block with DoT paths and port 853
- apps/adguard.yaml: mount adguard-dot-tls secret, expose port 853,
  init container enforces TLS/DoT settings on every restart
- adguard.tf: fix stale neuralplatform.dev → neuralplatform.ai rewrite

Android Private DNS: set to dot.nook.family
Router: forward TCP 853 → 192.168.68.77
2026-03-24 09:27:21 -05:00
Will Anderson 3518620360 fix: PGPASSWORD env var, neuron registry, gitea domain
- backup.tf: rename POSTGRES_PASSWORD → PGPASSWORD so pg_dumpall
  can authenticate (was silently failing for 2 days)
- neuron.yaml: fix image registry nook.family → neuralplatform.ai
- gitea.yaml: fix domain neuralplatform.dev → neuralplatform.ai
2026-03-24 08:08:40 -05:00
Will Anderson 1c8d7cecd8 monitoring: wire Alertmanager to #infrastructure-alerts in Slack
Routes all warning/critical alerts to the Neural Platform Slack workspace.
Suppresses Watchdog and info-level noise. Groups by namespace+alertname,
repeats every 4h. Uses bot token auth via chat.postMessage API.
2026-03-24 01:45:06 -05:00
Will Anderson f8bc853136 DR: full bootstrap script + recovery runbook
bootstrap.sh now covers a complete fresh Ubuntu 24.04 setup:
- NVIDIA driver 580 + container toolkit + k3s GPU config
- k3s install with nvidia as default runtime + device plugin
- Host cloudflared (Gitea SSH tunnel)
- Terraform + Helm
- systemd-resolved disabled, resolv.conf pinned

RUNBOOK.md documents the full 4-step recovery:
bootstrap → terraform apply → Gitea/Postgres restore → verify
Target: under 1 hour on a new machine.
2026-03-24 00:46:36 -05:00
Will Anderson 82b902f7c4 Move AdGuard to dns.nook.family, fix Agents.md service domains 2026-03-23 10:43:38 -05:00
Will Anderson bf2ec73983 Expose AdGuard DoH at dns.neuralplatform.ai 2026-03-23 10:29:42 -05:00