Harden prod: security, autoscaling, observability, BuildKit CI

Security:
- Drop ALL capabilities, enforce non-root, RuntimeDefault seccomp on
  neuron-mcp, neuron-rest, neuron-marketing pods
- Add startup probes (150s window for JVM) so liveness doesn't fire early
- Replace docker-sock hostPath with BuildKit rootless TCP endpoint
  (moby/buildkit:v0.19.0-rootless) — removes node root access from CI
- Document full ESO AppRole migration path in cluster-secret-store.yaml

Autoscaling & availability:
- HPAs on mcp (1–6), rest (1–4), marketing (2–8) at 65–70% CPU
- PodDisruptionBudgets (minAvailable: 1) on all three services
- NetworkPolicy: default-deny-all in neuron-prod, explicit allow rules
  for Traefik ingress, intra-namespace, and egress to DNS/platform/vault

Observability:
- ServiceMonitors for mcp, rest, marketing (cross-namespace enabled in
  kube-prometheus-stack with serviceMonitorSelectorNilUsesHelmValues:false)
- PrometheusRules: high error rate, high latency, crash loops, replica
  shortage, Postgres down/connections, backup failure, backup staleness

Chart version pinning:
- kube-prometheus-stack, loki, tempo, redis, alloy, postgres — all pinned
  to major-version ranges to block silent breaking upgrades

Backup hardening:
- restic:latest → restic:0.17.3 (deterministic image)
- Weekly backup-verify CronJob: restores latest snapshot and validates
  SQL dump structure (≥5 CREATE TABLE, pg_dump header check)

ArgoCD:
- neuron-prod AppProject: scopes deploys to neuron-prod + platform ns,
  blacklists ClusterRole/ClusterRoleBinding/Namespace creation,
  automated sync window 2–6am UTC, manual always allowed
This commit is contained in:
Will Anderson
2026-04-25 22:34:59 -05:00
parent 8fd3d12907
commit 93358505fc
20 changed files with 852 additions and 37 deletions
@@ -0,0 +1,71 @@
---
# BuildKit daemon — rootless image builder for CI runners.
# Replaces the /var/run/docker.sock hostPath mount that gave containers
# full root access to the Legion host node.
#
# Runners connect via TCP: DOCKER_HOST=tcp://buildkitd.ci.svc.cluster.local:1234
apiVersion: apps/v1
kind: Deployment
metadata:
name: buildkitd
namespace: ci
labels:
app: buildkitd
spec:
replicas: 1
selector:
matchLabels:
app: buildkitd
template:
metadata:
labels:
app: buildkitd
spec:
containers:
- name: buildkitd
image: moby/buildkit:v0.19.0-rootless
args:
- --oci-worker-no-process-sandbox
- --addr
- tcp://0.0.0.0:1234
- --addr
- unix:///run/buildkit/buildkitd.sock
ports:
- name: tcp
containerPort: 1234
securityContext:
seccompProfile:
type: Unconfined # BuildKit rootless requires this
runAsUser: 1000
runAsGroup: 1000
readinessProbe:
exec:
command: ["buildctl", "debug", "workers"]
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: "4"
memory: 4Gi
volumeMounts:
- name: buildkit-storage
mountPath: /home/user/.local/share/buildkit
volumes:
- name: buildkit-storage
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: buildkitd
namespace: ci
spec:
selector:
app: buildkitd
ports:
- name: tcp
port: 1234
targetPort: 1234