will.anderson
0006380c27
route runner build container clones via public URL with CF Access ( #7 )
2026-05-04 21:37:53 +00:00
will.anderson
cbb564ccf5
revert(ci): runner public URL — CF Access blocks registration ( #4 )
2026-05-04 21:05:29 +00:00
will.anderson
be0508037a
fix(dharma): drop letsencrypt certResolver from IngressRoute
2026-05-04 20:56:29 +00:00
will.anderson
6f5d041440
fix(ci): point Gitea Actions runners at public instance URL
2026-05-04 20:56:26 +00:00
4754c69e01
fix: mount /var/run/docker.sock (Docker Engine) not containerd socket
2026-05-04 14:10:25 -05:00
0444d265b9
fix: gitea runner docker_host path mismatch — use /var/run/docker.sock
...
The containerd socket is mounted from the host at /run/k3s/containerd/containerd.sock
to /var/run/docker.sock inside the container, but act_runner config was pointing to
the host path directly. Fixes CrashLoopBackOff on all runner pods.
Also codifies both deployments (will org + neuron-technologies org) as git-managed
manifests so Argo CD owns them going forward.
2026-05-04 13:44:46 -05:00
44faf74b61
fix CI runner secrets: correct Gitea URL and use Vault-backed tokens
...
The ExternalSecret had a stale Gitea URL (192.168.8.148:30322 — old
local IP) and runner tokens were never populated in Vault. Update the
template URL to the cluster-internal Gitea service DNS, and store the
registration tokens in Vault at secret/gitea.runner_token.
2026-05-04 11:07:43 -05:00
2550786965
Revert CI to host Docker socket, remove buildkitd
...
BuildKit rootless failed on k3s (mount propagation), privileged mode
fixed that but buildkitd gRPC is incompatible with the Docker REST API
that forgejo-runner needs to manage job containers. Net security change
vs. the original socket approach was zero (privileged ≈ socket access).
Remove buildkitd.yaml entirely. Restore docker-sock hostPath mount on
both runners. Builds work again.
2026-04-26 02:50:27 -05:00
ffac2161bc
Use TCP readiness probe for BuildKit — exec fails on k3s privileged containers
2026-04-26 01:23:52 -05:00
6e67f62ffe
Switch BuildKit to privileged mode — rootless fails on k3s mount propagation
2026-04-26 01:20:45 -05:00
e138c45d51
Fix BuildKit rootless: allow privilege escalation for newuidmap/newgidmap
2026-04-26 01:17:41 -05:00
93358505fc
Harden prod: security, autoscaling, observability, BuildKit CI
...
Security:
- Drop ALL capabilities, enforce non-root, RuntimeDefault seccomp on
neuron-mcp, neuron-rest, neuron-marketing pods
- Add startup probes (150s window for JVM) so liveness doesn't fire early
- Replace docker-sock hostPath with BuildKit rootless TCP endpoint
(moby/buildkit:v0.19.0-rootless) — removes node root access from CI
- Document full ESO AppRole migration path in cluster-secret-store.yaml
Autoscaling & availability:
- HPAs on mcp (1–6), rest (1–4), marketing (2–8) at 65–70% CPU
- PodDisruptionBudgets (minAvailable: 1) on all three services
- NetworkPolicy: default-deny-all in neuron-prod, explicit allow rules
for Traefik ingress, intra-namespace, and egress to DNS/platform/vault
Observability:
- ServiceMonitors for mcp, rest, marketing (cross-namespace enabled in
kube-prometheus-stack with serviceMonitorSelectorNilUsesHelmValues:false)
- PrometheusRules: high error rate, high latency, crash loops, replica
shortage, Postgres down/connections, backup failure, backup staleness
Chart version pinning:
- kube-prometheus-stack, loki, tempo, redis, alloy, postgres — all pinned
to major-version ranges to block silent breaking upgrades
Backup hardening:
- restic:latest → restic:0.17.3 (deterministic image)
- Weekly backup-verify CronJob: restores latest snapshot and validates
SQL dump structure (≥5 CREATE TABLE, pg_dump header check)
ArgoCD:
- neuron-prod AppProject: scopes deploys to neuron-prod + platform ns,
blacklists ClusterRole/ClusterRoleBinding/Namespace creation,
automated sync window 2–6am UTC, manual always allowed
2026-04-25 22:54:18 -05:00
cd0cb30933
ci: add neuron-technologies runner for Gitea Actions
2026-04-23 21:19:26 -05:00
43737ce5cb
chore(ci-runner): restore ESO refresh interval to 1h
2026-04-23 13:32:18 -05:00
c8ee5e66ee
fix(ci-runner): use NodePort HTTP URL to bypass Cloudflare gRPC issue
2026-04-23 13:15:16 -05:00
e69052d89e
fix(ci-runner): use external Gitea URL now that REQUIRE_SIGNIN_VIEW is disabled
2026-04-23 12:58:51 -05:00
28fb4e29f4
fix(ci-runner): use internal cluster URL for runner registration (gRPC)
2026-04-23 12:57:11 -05:00
b1898a7a0f
fix(ci-runner): revert to external Gitea URL for git clone compatibility
2026-04-23 12:53:41 -05:00
521079590b
chore(ci-runner): restore ESO refresh interval to 1h
2026-04-23 12:45:34 -05:00
0d9f74f68f
fix(ci-runner): use internal cluster URL for Gitea to bypass Cloudflare Access
2026-04-23 12:45:00 -05:00
415ee9eb68
chore(ci): restore ESO refresh interval to 1h after token sync
2026-04-23 12:33:01 -05:00
b31aad85a7
chore(ci): shorten ESO refresh to 10s to force runner token update
2026-04-23 12:31:11 -05:00
e84c586282
chore(ci): force ESO resync for gitea-runner-secret with fresh registration token
2026-04-23 12:29:58 -05:00
2cb89b715f
ci: copy forgejo-runner binary from correct path
2026-03-26 13:00:42 -05:00
23131ef04f
ci: use data.forgejo.org for runner image
2026-03-26 13:00:01 -05:00
e4db8bbda5
ci: switch to Forgejo runner for proper DEFAULT_ACTIONS_URL support
2026-03-26 12:58:22 -05:00
c9bee67e67
feat: make ci-base equivalent to ubuntu-latest
...
Comprehensive runner image with Python, Node.js 20, Ruby, Go 1.22,
Docker CLI, kubectl, Helm, yq, gh, make, jq, rsync, zstd, and all
standard tools. Mirrors GitHub ubuntu-24.04 hosted runner so workflows
are swappable between Gitea and GitHub with no friction.
2026-03-26 12:11:38 -05:00
798cf8611d
feat: make ci-base a comprehensive runner image like ubuntu-latest
...
Add Node.js 20 LTS (required for all Forgejo JS actions), Python 3.12,
Ruby, Go 1.22, Docker CLI, kubectl, Helm, yq, gh, zstd, and common
build/system tools. Mirrors GitHub ubuntu-24.04 hosted runner capability
so jobs work without needing container: overrides for standard tasks.
2026-03-26 12:08:16 -05:00
978639601c
fix: correct act_runner binary path in ci-base Dockerfile
2026-03-26 11:35:37 -05:00
9595a18f3a
refactor: rebase CI runner on Ubuntu 24.04, rename to ci-base
...
Replace Alpine-based legion-runner with a minimal Ubuntu 24.04 image.
The act_runner binary is copied from the official image via multi-stage
build. Job-specific tooling (Ruby, Node, Python, etc.) belongs in
per-job container images specified via container: in workflows, not
in the base runner image.
Rename: legion-runner → ci-base
2026-03-26 11:32:52 -05:00
ce45384f81
feat: add Ruby and bundler to legion-runner image
...
Required for Jekyll-based doc builds in CI.
2026-03-26 10:59:36 -05:00
f3477ac102
ci: fix gitea runner URL and set DEFAULT_ACTIONS_URL
...
- external-secrets.yaml: GITEA_INSTANCE_URL changed from cluster-internal
URL to https://git.neuralplatform.ai so runner can register externally
- gitea.yaml: add GITEA__actions__DEFAULT_ACTIONS_URL=gitea so all
actions/* references resolve from gitea.com, not github.com
- Dockerfile: track runner image (Python 3.12 + build deps) in repo
2026-03-26 10:35:13 -05:00
9904bce343
gitea-runner: use internal cluster URL to avoid CF tunnel timeouts on runner long-polling
2026-03-25 20:29:23 -05:00
6c073029a3
Fix gitea-runner ExternalSecret: use template for static GITEA_INSTANCE_URL
2026-03-25 10:34:11 -05:00
4d0cfb1bbf
Migrate k8s config from Terraform to Argo CD + ESO
...
Phase 1: Install External Secrets Operator via Argo CD app
- apps/external-secrets.yaml — ESO Helm chart install
- apps/external-secrets-config.yaml — ClusterSecretStore deployment
- k8s/external-secrets/cluster-secret-store.yaml — Vault backend using vault-token Secret
Phase 2: Create k8s manifests for all services
- k8s/neuron/ — PVC, ConfigMap, ExternalSecrets (neuron-secrets, cloudflared-secret), Ingress
- k8s/gitea/ — PVC, ConfigMap (custom CSS), ExternalSecret (gitea-db), Ingress
- k8s/github-runner/ — ExternalSecret (github-runner-secret)
- k8s/gitea-runner/ — ExternalSecret (gitea-runner-secret)
- k8s/monitoring/ — ExternalSecrets (grafana, slack), Alloy OTLP service+middleware, datasources ConfigMap, Ingress
- k8s/postgres/ — ExternalSecret (postgres-passwords)
- k8s/vault/ — ExternalSecret (vault-gcp-sa from Vault)
- k8s/adguard/ — PVCs, ConfigMap, Certificate, Ingress, ddclient Deployment+ExternalSecret
- k8s/ollama/ — PVC, Ingress
- k8s/headscale/ — PVC
- k8s/packages/ — PVCs, ConfigMap, Ingresses
- k8s/registry/ — PVC, Ingresses
- k8s/backup/ — CronJob, ExternalSecret (backup-credentials)
New Argo CD apps for Helm releases:
- apps/kube-prometheus-stack.yaml, loki.yaml, tempo.yaml, alloy.yaml
- apps/postgres.yaml, redis.yaml, vault.yaml
New Argo CD apps for k8s config paths:
- apps/neuron-config, gitea-config, ci-config, gitea-runner-config
- apps/monitoring-config, adguard-config, ollama-config, headscale-config
- apps/packages-config, registry-config, postgres-config, vault-config, backup
Phase 3: Strip Terraform to infrastructure-only
- All kubernetes_* and helm_release resources removed from service .tf files
- Each service .tf now contains only kubernetes_namespace (bootstrap dependency)
- variables.tf stripped to only cloudflare_api_key, cloudflare_email, gitea_api_token
- namespaces.tf gains external-secrets namespace
- ingress.tf, backup.tf, ddclient.tf emptied (resources in k8s/)
- cert-manager.tf, argocd.tf, traefik.tf unchanged (bootstrap)
2026-03-25 10:29:14 -05:00
will.anderson
Will Anderson