Harden prod: security, autoscaling, observability, BuildKit CI
Security: - Drop ALL capabilities, enforce non-root, RuntimeDefault seccomp on neuron-mcp, neuron-rest, neuron-marketing pods - Add startup probes (150s window for JVM) so liveness doesn't fire early - Replace docker-sock hostPath with BuildKit rootless TCP endpoint (moby/buildkit:v0.19.0-rootless) — removes node root access from CI - Document full ESO AppRole migration path in cluster-secret-store.yaml Autoscaling & availability: - HPAs on mcp (1–6), rest (1–4), marketing (2–8) at 65–70% CPU - PodDisruptionBudgets (minAvailable: 1) on all three services - NetworkPolicy: default-deny-all in neuron-prod, explicit allow rules for Traefik ingress, intra-namespace, and egress to DNS/platform/vault Observability: - ServiceMonitors for mcp, rest, marketing (cross-namespace enabled in kube-prometheus-stack with serviceMonitorSelectorNilUsesHelmValues:false) - PrometheusRules: high error rate, high latency, crash loops, replica shortage, Postgres down/connections, backup failure, backup staleness Chart version pinning: - kube-prometheus-stack, loki, tempo, redis, alloy, postgres — all pinned to major-version ranges to block silent breaking upgrades Backup hardening: - restic:latest → restic:0.17.3 (deterministic image) - Weekly backup-verify CronJob: restores latest snapshot and validates SQL dump structure (≥5 CREATE TABLE, pg_dump header check) ArgoCD: - neuron-prod AppProject: scopes deploys to neuron-prod + platform ns, blacklists ClusterRole/ClusterRoleBinding/Namespace creation, automated sync window 2–6am UTC, manual always allowed
This commit is contained in:
@@ -43,8 +43,10 @@ spec:
|
||||
limits:
|
||||
memory: 256Mi
|
||||
containers:
|
||||
# Pin to explicit version — never use latest for backup tooling.
|
||||
# Bump this intentionally after testing; do not let it float.
|
||||
- name: backup
|
||||
image: restic/restic:latest
|
||||
image: restic/restic:0.17.3
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
@@ -93,3 +95,77 @@ spec:
|
||||
readOnly: true
|
||||
- name: dump
|
||||
emptyDir: {}
|
||||
---
|
||||
# Weekly restore-verify job — proves the backup is actually restorable.
|
||||
# Runs Sunday 4am. Restores latest snapshot to /tmp, validates the SQL dump.
|
||||
# Alerts via Alertmanager if it fails (BackupJobFailed PrometheusRule watches git namespace).
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: backup-verify
|
||||
namespace: git
|
||||
spec:
|
||||
schedule: "0 4 * * 0"
|
||||
concurrencyPolicy: Forbid
|
||||
failedJobsHistoryLimit: 2
|
||||
successfulJobsHistoryLimit: 1
|
||||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
activeDeadlineSeconds: 1800 # 30 min max
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: verify
|
||||
image: restic/restic:0.17.3
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
set -e
|
||||
|
||||
echo "=== Backup Restore Verification ==="
|
||||
echo "Listing latest snapshots..."
|
||||
restic snapshots --latest 3
|
||||
|
||||
echo "Restoring latest snapshot to /tmp/restore-verify..."
|
||||
mkdir -p /tmp/restore-verify
|
||||
restic restore latest \
|
||||
--target /tmp/restore-verify \
|
||||
--include /all-databases.sql
|
||||
|
||||
SQL_FILE="/tmp/restore-verify/all-databases.sql"
|
||||
if [ ! -f "$SQL_FILE" ]; then
|
||||
echo "ERROR: SQL dump not found in restored snapshot"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SIZE=$(du -sh "$SQL_FILE" | cut -f1)
|
||||
echo "SQL dump restored: $SIZE"
|
||||
|
||||
# Basic structural validation
|
||||
if ! grep -q "PostgreSQL database cluster dump" "$SQL_FILE"; then
|
||||
echo "ERROR: SQL dump does not look like a valid pg_dumpall output"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TABLE_COUNT=$(grep -c "^CREATE TABLE" "$SQL_FILE" || true)
|
||||
echo "Tables found in dump: $TABLE_COUNT"
|
||||
|
||||
if [ "$TABLE_COUNT" -lt 5 ]; then
|
||||
echo "ERROR: Too few tables in dump (expected >= 5, got $TABLE_COUNT)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Verification PASSED: backup is restorable ==="
|
||||
rm -rf /tmp/restore-verify
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: backup-credentials
|
||||
resources:
|
||||
requests:
|
||||
memory: 256Mi
|
||||
cpu: 100m
|
||||
limits:
|
||||
memory: 512Mi
|
||||
|
||||
Reference in New Issue
Block a user