 Will Anderson
|
93358505fc
|
Harden prod: security, autoscaling, observability, BuildKit CI
Security:
- Drop ALL capabilities, enforce non-root, RuntimeDefault seccomp on
neuron-mcp, neuron-rest, neuron-marketing pods
- Add startup probes (150s window for JVM) so liveness doesn't fire early
- Replace docker-sock hostPath with BuildKit rootless TCP endpoint
(moby/buildkit:v0.19.0-rootless) — removes node root access from CI
- Document full ESO AppRole migration path in cluster-secret-store.yaml
Autoscaling & availability:
- HPAs on mcp (1–6), rest (1–4), marketing (2–8) at 65–70% CPU
- PodDisruptionBudgets (minAvailable: 1) on all three services
- NetworkPolicy: default-deny-all in neuron-prod, explicit allow rules
for Traefik ingress, intra-namespace, and egress to DNS/platform/vault
Observability:
- ServiceMonitors for mcp, rest, marketing (cross-namespace enabled in
kube-prometheus-stack with serviceMonitorSelectorNilUsesHelmValues:false)
- PrometheusRules: high error rate, high latency, crash loops, replica
shortage, Postgres down/connections, backup failure, backup staleness
Chart version pinning:
- kube-prometheus-stack, loki, tempo, redis, alloy, postgres — all pinned
to major-version ranges to block silent breaking upgrades
Backup hardening:
- restic:latest → restic:0.17.3 (deterministic image)
- Weekly backup-verify CronJob: restores latest snapshot and validates
SQL dump structure (≥5 CREATE TABLE, pg_dump header check)
ArgoCD:
- neuron-prod AppProject: scopes deploys to neuron-prod + platform ns,
blacklists ClusterRole/ClusterRoleBinding/Namespace creation,
automated sync window 2–6am UTC, manual always allowed
|
2026-04-25 22:54:18 -05:00 |
|
|
Will Anderson
|
4d0cfb1bbf
|
Migrate k8s config from Terraform to Argo CD + ESO
Phase 1: Install External Secrets Operator via Argo CD app
- apps/external-secrets.yaml — ESO Helm chart install
- apps/external-secrets-config.yaml — ClusterSecretStore deployment
- k8s/external-secrets/cluster-secret-store.yaml — Vault backend using vault-token Secret
Phase 2: Create k8s manifests for all services
- k8s/neuron/ — PVC, ConfigMap, ExternalSecrets (neuron-secrets, cloudflared-secret), Ingress
- k8s/gitea/ — PVC, ConfigMap (custom CSS), ExternalSecret (gitea-db), Ingress
- k8s/github-runner/ — ExternalSecret (github-runner-secret)
- k8s/gitea-runner/ — ExternalSecret (gitea-runner-secret)
- k8s/monitoring/ — ExternalSecrets (grafana, slack), Alloy OTLP service+middleware, datasources ConfigMap, Ingress
- k8s/postgres/ — ExternalSecret (postgres-passwords)
- k8s/vault/ — ExternalSecret (vault-gcp-sa from Vault)
- k8s/adguard/ — PVCs, ConfigMap, Certificate, Ingress, ddclient Deployment+ExternalSecret
- k8s/ollama/ — PVC, Ingress
- k8s/headscale/ — PVC
- k8s/packages/ — PVCs, ConfigMap, Ingresses
- k8s/registry/ — PVC, Ingresses
- k8s/backup/ — CronJob, ExternalSecret (backup-credentials)
New Argo CD apps for Helm releases:
- apps/kube-prometheus-stack.yaml, loki.yaml, tempo.yaml, alloy.yaml
- apps/postgres.yaml, redis.yaml, vault.yaml
New Argo CD apps for k8s config paths:
- apps/neuron-config, gitea-config, ci-config, gitea-runner-config
- apps/monitoring-config, adguard-config, ollama-config, headscale-config
- apps/packages-config, registry-config, postgres-config, vault-config, backup
Phase 3: Strip Terraform to infrastructure-only
- All kubernetes_* and helm_release resources removed from service .tf files
- Each service .tf now contains only kubernetes_namespace (bootstrap dependency)
- variables.tf stripped to only cloudflare_api_key, cloudflare_email, gitea_api_token
- namespaces.tf gains external-secrets namespace
- ingress.tf, backup.tf, ddclient.tf emptied (resources in k8s/)
- cert-manager.tf, argocd.tf, traefik.tf unchanged (bootstrap)
|
2026-03-25 10:29:14 -05:00 |
|