Harden prod: security, autoscaling, observability, BuildKit CI

Security:
- Drop ALL capabilities, enforce non-root, RuntimeDefault seccomp on
  neuron-mcp, neuron-rest, neuron-marketing pods
- Add startup probes (150s window for JVM) so liveness doesn't fire early
- Replace docker-sock hostPath with BuildKit rootless TCP endpoint
  (moby/buildkit:v0.19.0-rootless) — removes node root access from CI
- Document full ESO AppRole migration path in cluster-secret-store.yaml

Autoscaling & availability:
- HPAs on mcp (1–6), rest (1–4), marketing (2–8) at 65–70% CPU
- PodDisruptionBudgets (minAvailable: 1) on all three services
- NetworkPolicy: default-deny-all in neuron-prod, explicit allow rules
  for Traefik ingress, intra-namespace, and egress to DNS/platform/vault

Observability:
- ServiceMonitors for mcp, rest, marketing (cross-namespace enabled in
  kube-prometheus-stack with serviceMonitorSelectorNilUsesHelmValues:false)
- PrometheusRules: high error rate, high latency, crash loops, replica
  shortage, Postgres down/connections, backup failure, backup staleness

Chart version pinning:
- kube-prometheus-stack, loki, tempo, redis, alloy, postgres — all pinned
  to major-version ranges to block silent breaking upgrades

Backup hardening:
- restic:latest → restic:0.17.3 (deterministic image)
- Weekly backup-verify CronJob: restores latest snapshot and validates
  SQL dump structure (≥5 CREATE TABLE, pg_dump header check)

ArgoCD:
- neuron-prod AppProject: scopes deploys to neuron-prod + platform ns,
  blacklists ClusterRole/ClusterRoleBinding/Namespace creation,
  automated sync window 2–6am UTC, manual always allowed
This commit is contained in:
Will Anderson
2026-04-25 22:34:59 -05:00
parent 8fd3d12907
commit 93358505fc
20 changed files with 852 additions and 37 deletions
+77 -1
View File
@@ -43,8 +43,10 @@ spec:
limits:
memory: 256Mi
containers:
# Pin to explicit version — never use latest for backup tooling.
# Bump this intentionally after testing; do not let it float.
- name: backup
image: restic/restic:latest
image: restic/restic:0.17.3
command:
- /bin/sh
- -c
@@ -93,3 +95,77 @@ spec:
readOnly: true
- name: dump
emptyDir: {}
---
# Weekly restore-verify job — proves the backup is actually restorable.
# Runs Sunday 4am. Restores latest snapshot to /tmp, validates the SQL dump.
# Alerts via Alertmanager if it fails (BackupJobFailed PrometheusRule watches git namespace).
apiVersion: batch/v1
kind: CronJob
metadata:
name: backup-verify
namespace: git
spec:
schedule: "0 4 * * 0"
concurrencyPolicy: Forbid
failedJobsHistoryLimit: 2
successfulJobsHistoryLimit: 1
jobTemplate:
spec:
backoffLimit: 0
activeDeadlineSeconds: 1800 # 30 min max
template:
spec:
restartPolicy: Never
containers:
- name: verify
image: restic/restic:0.17.3
command:
- /bin/sh
- -c
- |
set -e
echo "=== Backup Restore Verification ==="
echo "Listing latest snapshots..."
restic snapshots --latest 3
echo "Restoring latest snapshot to /tmp/restore-verify..."
mkdir -p /tmp/restore-verify
restic restore latest \
--target /tmp/restore-verify \
--include /all-databases.sql
SQL_FILE="/tmp/restore-verify/all-databases.sql"
if [ ! -f "$SQL_FILE" ]; then
echo "ERROR: SQL dump not found in restored snapshot"
exit 1
fi
SIZE=$(du -sh "$SQL_FILE" | cut -f1)
echo "SQL dump restored: $SIZE"
# Basic structural validation
if ! grep -q "PostgreSQL database cluster dump" "$SQL_FILE"; then
echo "ERROR: SQL dump does not look like a valid pg_dumpall output"
exit 1
fi
TABLE_COUNT=$(grep -c "^CREATE TABLE" "$SQL_FILE" || true)
echo "Tables found in dump: $TABLE_COUNT"
if [ "$TABLE_COUNT" -lt 5 ]; then
echo "ERROR: Too few tables in dump (expected >= 5, got $TABLE_COUNT)"
exit 1
fi
echo "=== Verification PASSED: backup is restorable ==="
rm -rf /tmp/restore-verify
envFrom:
- secretRef:
name: backup-credentials
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 512Mi