Commit Graph

567 Commits

Author SHA1 Message Date
Gitea CI e510207857 chore(neuron): deploy 56e092b61e9eb1b92ba753c0ba17914212ad58af 2026-03-27 16:05:11 +00:00
Gitea CI 58448a445f chore(neuron): deploy 87b0bad5c66abf7218dd51319d23eea3d1bfbd29 2026-03-27 16:00:05 +00:00
Gitea CI ff2b9238fd chore(neuron): deploy 8e87d7057235151e430bbc78015732f463d6d323 2026-03-27 15:51:49 +00:00
Gitea CI f78b9b0d75 chore(neuron): deploy c8eeba5783de3f388ed0266d850fc0b26db219fd 2026-03-27 15:39:40 +00:00
Will Anderson 485e09532e scale down github-runner — no longer needed 2026-03-27 10:38:33 -05:00
Gitea CI 7a53d14cf0 chore(neuron): deploy d69b7bf14a8b641c5685adc3171e7acf9b3b7112 2026-03-27 14:36:29 +00:00
Will Anderson 076b3562f0 ci: map runner labels to ci-base image via docker:// scheme 2026-03-26 13:39:25 -05:00
Will Anderson aff34f061f ci: set default_image to ci-base so all jobs get full tooling 2026-03-26 13:17:19 -05:00
Will Anderson 2cb89b715f ci: copy forgejo-runner binary from correct path 2026-03-26 13:00:42 -05:00
Will Anderson 23131ef04f ci: use data.forgejo.org for runner image 2026-03-26 13:00:01 -05:00
Will Anderson e4db8bbda5 ci: switch to Forgejo runner for proper DEFAULT_ACTIONS_URL support 2026-03-26 12:58:22 -05:00
Will Anderson a40897fd5d ci: add ubuntu-latest, ubuntu-24.04 labels to runner 2026-03-26 12:48:38 -05:00
Will Anderson c9bee67e67 feat: make ci-base equivalent to ubuntu-latest
Comprehensive runner image with Python, Node.js 20, Ruby, Go 1.22,
Docker CLI, kubectl, Helm, yq, gh, make, jq, rsync, zstd, and all
standard tools. Mirrors GitHub ubuntu-24.04 hosted runner so workflows
are swappable between Gitea and GitHub with no friction.
2026-03-26 12:11:38 -05:00
Will Anderson 798cf8611d feat: make ci-base a comprehensive runner image like ubuntu-latest
Add Node.js 20 LTS (required for all Forgejo JS actions), Python 3.12,
Ruby, Go 1.22, Docker CLI, kubectl, Helm, yq, gh, zstd, and common
build/system tools. Mirrors GitHub ubuntu-24.04 hosted runner capability
so jobs work without needing container: overrides for standard tasks.
2026-03-26 12:08:16 -05:00
Will Anderson db08c5a208 ci: set runner default_image to ubuntu:24.04
Replace pantheon-ci:latest (custom, potentially stale) with ubuntu:24.04
as the fallback container for jobs that don't specify container:.
2026-03-26 11:41:55 -05:00
Will Anderson 978639601c fix: correct act_runner binary path in ci-base Dockerfile 2026-03-26 11:35:37 -05:00
Will Anderson 9595a18f3a refactor: rebase CI runner on Ubuntu 24.04, rename to ci-base
Replace Alpine-based legion-runner with a minimal Ubuntu 24.04 image.
The act_runner binary is copied from the official image via multi-stage
build. Job-specific tooling (Ruby, Node, Python, etc.) belongs in
per-job container images specified via container: in workflows, not
in the base runner image.

Rename: legion-runner → ci-base
2026-03-26 11:32:52 -05:00
Will Anderson d163a92ec6 fix: replace registry Ingress+Buffering with IngressRoute for streaming
The Buffering middleware was causing 413 on large Docker layer pushes
by spooling entire request bodies to disk before forwarding. Docker's
chunked blob upload protocol (PATCH/PUT with Content-Range) requires
streaming, not buffering.

Replace with IngressRoute CRDs + Certificate resources for direct
streaming passthrough. responseForwarding.flushInterval=100ms ensures
prompt chunk delivery to the registry backend.
2026-03-26 11:13:25 -05:00
Will Anderson 206873e83b fix: set DEFAULT_ACTIONS_URL to Forgejo action mirrors
code.forgejo.org maintains up-to-date mirrors of common GitHub Actions
(checkout, upload-artifact, etc.) and is the recommended source for
Gitea Actions runners.
2026-03-26 11:06:43 -05:00
Will Anderson ce45384f81 feat: add Ruby and bundler to legion-runner image
Required for Jekyll-based doc builds in CI.
2026-03-26 10:59:36 -05:00
Will Anderson 8425f0988c fix: set Gitea DEFAULT_ACTIONS_URL to self
https://gitea.com is no longer a valid value in Gitea 1.25 — causes
startup crash. Use 'self' to point action references at this instance.
2026-03-26 10:41:54 -05:00
Will Anderson d8f2c822f8 fix: DEFAULT_ACTIONS_URL must be a full URL, not 'gitea' 2026-03-26 10:36:31 -05:00
Will Anderson f3477ac102 ci: fix gitea runner URL and set DEFAULT_ACTIONS_URL
- external-secrets.yaml: GITEA_INSTANCE_URL changed from cluster-internal
  URL to https://git.neuralplatform.ai so runner can register externally
- gitea.yaml: add GITEA__actions__DEFAULT_ACTIONS_URL=gitea so all
  actions/* references resolve from gitea.com, not github.com
- Dockerfile: track runner image (Python 3.12 + build deps) in repo
2026-03-26 10:35:13 -05:00
Will Anderson 737fb27a5c gitea-runner: use lean legion-runner base image 2026-03-26 08:01:41 -05:00
Will Anderson 5f984e1fc2 gitea-runner: revert to pantheon-runner (container: directive unsupported in host mode) 2026-03-26 07:53:48 -05:00
Will Anderson 4e9d3287d4 gitea-runner: use lean legion-runner base image 2026-03-26 07:51:21 -05:00
Will Anderson f409c06e65 gitea-runner: use pantheon-runner image (act_runner + all CI tools) 2026-03-25 20:39:32 -05:00
Will Anderson 08d4ab1d72 gitea-runner: set default_image in container config to use pantheon-ci for all jobs 2026-03-25 20:37:17 -05:00
Will Anderson d5c5d753c7 ci: use custom pantheon-ci image; fix registry 413 with Traefik buffering middleware 2026-03-25 20:36:20 -05:00
Will Anderson 9904bce343 gitea-runner: use internal cluster URL to avoid CF tunnel timeouts on runner long-polling 2026-03-25 20:29:23 -05:00
Will Anderson ebb42a6dd2 gitea-runner: switch to native host execution with Legion tool mounts 2026-03-25 20:27:58 -05:00
Will Anderson 672a890659 upgrade gitea 1.23 → 1.25.5 2026-03-25 20:22:54 -05:00
Will Anderson 35055ec783 upgrade gitea to 1.23 2026-03-25 20:04:05 -05:00
Will Anderson e19cadf971 docs: sync infrastructure docs with actual running state
- Fix direnv path typo in AGENTS.md (servers/servers → servers)
- Correct Prometheus/Alertmanager as internal-only (no public ingresses)
- Add Alloy public URL (alloy.neuralplatform.ai)
- Add external-secrets namespace to namespaces table
- Add contexthub org to Gitea orgs list
- Document Gitea SSH config (Host gitea → 192.168.68.77:30022)
- Specify GPU model: GTX 1660 Ti 6GB
- Add devpi to Legion README services table
2026-03-25 15:59:12 -05:00
Will Anderson 24f2dcbf9b Fix headscale DNS nameservers: use LAN IP and Cloudflare fallback 2026-03-25 10:59:45 -05:00
Will Anderson 371e60d3bc fix: add admin-user key to grafana-admin-secret ExternalSecret 2026-03-25 10:52:10 -05:00
Will Anderson 3737ff0bcb fix: pin PVCs to existing PVs; fix kube-prometheus-stack grafana auth
PVC manifests: add volumeName to bind new PVCs to existing PVs whose
old PVCs were accidentally deleted during Terraform->ArgoCD migration.
All PVs patched to Retain to prevent data loss on pod restart.

kube-prometheus-stack: replace invalid adminPassword.valueFrom.secretKeyRef
(Helm values don't support k8s secretKeyRef syntax) with correct
admin.existingSecret + admin.passwordKey pattern.
2026-03-25 10:48:10 -05:00
Will Anderson f1fc1fafea Fix ddclient image: use linuxserver/ddclient (correct image) 2026-03-25 10:36:23 -05:00
Will Anderson 6c073029a3 Fix gitea-runner ExternalSecret: use template for static GITEA_INSTANCE_URL 2026-03-25 10:34:11 -05:00
Will Anderson 4d0cfb1bbf Migrate k8s config from Terraform to Argo CD + ESO
Phase 1: Install External Secrets Operator via Argo CD app
- apps/external-secrets.yaml — ESO Helm chart install
- apps/external-secrets-config.yaml — ClusterSecretStore deployment
- k8s/external-secrets/cluster-secret-store.yaml — Vault backend using vault-token Secret

Phase 2: Create k8s manifests for all services
- k8s/neuron/ — PVC, ConfigMap, ExternalSecrets (neuron-secrets, cloudflared-secret), Ingress
- k8s/gitea/ — PVC, ConfigMap (custom CSS), ExternalSecret (gitea-db), Ingress
- k8s/github-runner/ — ExternalSecret (github-runner-secret)
- k8s/gitea-runner/ — ExternalSecret (gitea-runner-secret)
- k8s/monitoring/ — ExternalSecrets (grafana, slack), Alloy OTLP service+middleware, datasources ConfigMap, Ingress
- k8s/postgres/ — ExternalSecret (postgres-passwords)
- k8s/vault/ — ExternalSecret (vault-gcp-sa from Vault)
- k8s/adguard/ — PVCs, ConfigMap, Certificate, Ingress, ddclient Deployment+ExternalSecret
- k8s/ollama/ — PVC, Ingress
- k8s/headscale/ — PVC
- k8s/packages/ — PVCs, ConfigMap, Ingresses
- k8s/registry/ — PVC, Ingresses
- k8s/backup/ — CronJob, ExternalSecret (backup-credentials)

New Argo CD apps for Helm releases:
- apps/kube-prometheus-stack.yaml, loki.yaml, tempo.yaml, alloy.yaml
- apps/postgres.yaml, redis.yaml, vault.yaml

New Argo CD apps for k8s config paths:
- apps/neuron-config, gitea-config, ci-config, gitea-runner-config
- apps/monitoring-config, adguard-config, ollama-config, headscale-config
- apps/packages-config, registry-config, postgres-config, vault-config, backup

Phase 3: Strip Terraform to infrastructure-only
- All kubernetes_* and helm_release resources removed from service .tf files
- Each service .tf now contains only kubernetes_namespace (bootstrap dependency)
- variables.tf stripped to only cloudflare_api_key, cloudflare_email, gitea_api_token
- namespaces.tf gains external-secrets namespace
- ingress.tf, backup.tf, ddclient.tf emptied (resources in k8s/)
- cert-manager.tf, argocd.tf, traefik.tf unchanged (bootstrap)
2026-03-25 10:29:14 -05:00
Will Anderson dd0f8a49a3 monitoring: expose Alloy OTLP for external log ingestion from RunPod pods
- Alloy config: route OTLP logs → Loki (was traces-only)
- Alloy config: fix River syntax (semicolons in rule{} blocks are invalid)
- Add alloy-otlp Service (ClusterIP:4318)
- Add alloy-otlp-auth Secret (bcrypt htpasswd for Traefik BasicAuth)
- Add alloy-otlp-basicauth-middleware Traefik Middleware
- Add alloy.neuralplatform.ai Ingress (TLS via cert-manager)
  Auth: Authorization: Basic base64(pantheon:<push_token>)
  Token stored in Vault at secret/alloy push_token

RunPod pods push OTLP logs to https://alloy.neuralplatform.ai/v1/logs
Alloy routes → Loki → queryable in Grafana
2026-03-25 10:15:41 -05:00
Will Anderson 6ebf8a9b3e Fix gitea_domain variable and add VPN/Headscale infrastructure
- Fix gitea_domain default from git.neuralplatform.dev → git.neuralplatform.ai
  so Traefik serves the correct Let's Encrypt cert when accessed via VPN/LAN
- Add docs/vpn-comparison.md: full evaluation of WireGuard/Tailscale/Headscale/Netbird/Netmaker
2026-03-25 08:44:20 -05:00
Will Anderson abe2dad093 headscale: fix base_domain conflict, set ts.neuralplatform.ai for MagicDNS 2026-03-25 08:35:59 -05:00
Will Anderson a461172473 headscale: rename coordination endpoint to vpn.neuralplatform.ai 2026-03-25 08:16:47 -05:00
Will Anderson 5334881d3f infra: add Headscale — self-hosted Tailscale control plane
- headscale.tf: namespace + 1Gi PVC for SQLite state
- apps/headscale.yaml: Deployment (headscale/headscale:0.23) + Service + ConfigMap
  - server_url: https://headscale.neuralplatform.ai
  - sqlite3 backend, DERP via Tailscale's map
  - MagicDNS base: vpn.neuralplatform.ai
- Cloudflare tunnel: headscale.neuralplatform.ai → headscale.headscale.svc:8080
- DNS CNAME added via API

Next: install tailscaled on Legion host, enroll as subnet router
2026-03-25 07:59:46 -05:00
Will Anderson 1312be1a03 runner: bind-mount host kubectl into job containers; registry domain → neuralplatform.ai
gitea-runner: add container.options to mount /usr/local/bin/kubectl:ro
from the Legion host into every act_runner job container. Eliminates
per-run curl download in CI workflows.

ingress: change registry ingress hostname from registry.nook.family to
registry.neuralplatform.ai. Matches the domain used in Neuron CI and
neuron.yaml image reference. DNS CNAME + tunnel route already added.
2026-03-24 14:39:57 -05:00
Will Anderson 7c08543ca4 fix: gitea runner uses public URL via CF Access bypass
- Added Cloudflare Access bypass for git.neuralplatform.ai/api/actions/*
  restricted to Legion public IP (72.204.75.49)
- Runner now registers and communicates with git.neuralplatform.ai directly
- Job containers clone from https://git.neuralplatform.ai (proper name)
2026-03-24 13:40:31 -05:00
Will Anderson 455797745c fix: runner container network=host for k8s DNS resolution
Job containers use bridge network by default, so they can't resolve
gitea.git.svc.cluster.local. host network inherits k3s host routing
which can reach cluster service IPs directly.
2026-03-24 13:20:44 -05:00
Will Anderson 05973b50bd fix: remove duplicate docker.sock mount in runner config
The socket is already mounted via the pod volume mount; specifying it
again in container.options causes 'Duplicate mount point' on job containers.
2026-03-24 13:14:43 -05:00
Will Anderson e91d6c0b09 fix: gitea runner Docker execution mode with catthehacker/ubuntu
Switch from :host to :docker execution so jobs run inside containers.
- Labels use docker://ghcr.io/catthehacker/ubuntu:act-22.04 (Python 3.10,
  Node 18, Docker CLI — everything CI jobs need pre-installed)
- config.yaml passes docker.sock through to job containers for build/push
- Capacity set to 2 parallel jobs

Fixes: 'Cannot find: node in PATH' — actions/checkout@v4 requires Node
2026-03-24 13:07:56 -05:00