will.anderson/infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden prod: security, autoscaling, observability, BuildKit CI

Browse Source 
Security:
- Drop ALL capabilities, enforce non-root, RuntimeDefault seccomp on
  neuron-mcp, neuron-rest, neuron-marketing pods
- Add startup probes (150s window for JVM) so liveness doesn't fire early
- Replace docker-sock hostPath with BuildKit rootless TCP endpoint
  (moby/buildkit:v0.19.0-rootless) — removes node root access from CI
- Document full ESO AppRole migration path in cluster-secret-store.yaml

Autoscaling & availability:
- HPAs on mcp (1–6), rest (1–4), marketing (2–8) at 65–70% CPU
- PodDisruptionBudgets (minAvailable: 1) on all three services
- NetworkPolicy: default-deny-all in neuron-prod, explicit allow rules
  for Traefik ingress, intra-namespace, and egress to DNS/platform/vault

Observability:
- ServiceMonitors for mcp, rest, marketing (cross-namespace enabled in
  kube-prometheus-stack with serviceMonitorSelectorNilUsesHelmValues:false)
- PrometheusRules: high error rate, high latency, crash loops, replica
  shortage, Postgres down/connections, backup failure, backup staleness

Chart version pinning:
- kube-prometheus-stack, loki, tempo, redis, alloy, postgres — all pinned
  to major-version ranges to block silent breaking upgrades

Backup hardening:
- restic:latest → restic:0.17.3 (deterministic image)
- Weekly backup-verify CronJob: restores latest snapshot and validates
  SQL dump structure (≥5 CREATE TABLE, pg_dump header check)

ArgoCD:
- neuron-prod AppProject: scopes deploys to neuron-prod + platform ns,
  blacklists ClusterRole/ClusterRoleBinding/Namespace creation,
  automated sync window 2–6am UTC, manual always allowed
This commit is contained in:
Will Anderson
2026-04-25 22:34:59 -05:00
parent 8fd3d12907
commit 93358505fc
20 changed files with 852 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/legion/k8s/neuron-technologies/prod/hpa.yaml
+98
View File
@@ -0,0 +1,98 @@
---
# HorizontalPodAutoscalers for neuron-prod
# Single Legion node today — minReplicas=1 prevents unnecessary idle waste.
# When GCP GKE is live these minimums should go to 2 for HA.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: neuron-mcp-hpa
namespace: neuron-prod
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: neuron-mcp-blue
minReplicas: 1
maxReplicas: 6
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 65
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
behavior:
scaleDown:
stabilizationWindowSeconds: 300 # Don't thrash — wait 5m before scaling down
policies:
- type: Pods
value: 1
periodSeconds: 120
scaleUp:
stabilizationWindowSeconds: 30
policies:
- type: Pods
value: 2
periodSeconds: 60
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: neuron-rest-hpa
namespace: neuron-prod
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: neuron-rest
minReplicas: 1
maxReplicas: 4
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 65
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
behavior:
scaleDown:
stabilizationWindowSeconds: 300
scaleUp:
stabilizationWindowSeconds: 30
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: neuron-marketing-hpa
namespace: neuron-prod
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: neuron-marketing
minReplicas: 2
maxReplicas: 8
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
behavior:
scaleDown:
stabilizationWindowSeconds: 120
scaleUp:
stabilizationWindowSeconds: 15
servers/legion/k8s/neuron-technologies/prod/kustomization.yaml
+4
View File
@@ -15,3 +15,7 @@ resources:
- backup-cronjob.yaml
- runpod-inference.yaml
- runpod-lb-configmap.yaml
# Hardening additions
- hpa.yaml
- pdb.yaml
- network-policy.yaml
servers/legion/k8s/neuron-technologies/prod/mcp-deployment.yaml
+19 -2
View File
@@ -25,6 +25,10 @@ spec:
kubectl.kubernetes.io/restartedAt: "2026-04-25T08:40:00Z"
spec:
terminationGracePeriodSeconds: 30
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: neuron-mcp
image: registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.15.3
@@ -32,6 +36,12 @@ spec:
ports:
- name: http
containerPort: 8080
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: false # JVM writes temp files; restrict paths via allowedPaths once profiled
runAsNonRoot: true
capabilities:
drop: ["ALL"]
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://alloy-otlp.monitoring.svc.cluster.local:4318"
@@ -57,18 +67,25 @@ spec:
exec:
# Give Traefik time to drain in-flight connections before pod terminates
command: ["sh", "-c", "sleep 8"]
# startupProbe gives the JVM up to 150s to start before liveness takes over.
# Prevents false crash loops during slow cold starts.
startupProbe:
httpGet:
path: /actuator/health/liveness
port: http
failureThreshold: 30
periodSeconds: 5
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: http
initialDelaySeconds: 30
periodSeconds: 30
failureThreshold: 3
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: http
initialDelaySeconds: 10
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
volumes:
servers/legion/k8s/neuron-technologies/prod/network-policy.yaml
+124
View File
@@ -0,0 +1,124 @@
---
# NetworkPolicies for neuron-prod
# Default-deny all ingress and egress, then explicitly allow only what's needed.
# Enforced by k3s's built-in NetworkPolicy controller.
# ── Baseline: deny all traffic in namespace ───────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: neuron-prod
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# ── neuron-mcp: accept traffic from Traefik (kube-system) only ───────────────
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-mcp-ingress
namespace: neuron-prod
spec:
podSelector:
matchLabels:
app: neuron-mcp
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: neuron-prod
---
# ── neuron-rest: accept from mcp, kube-system (Traefik), monitoring (Alloy) ──
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-rest-ingress
namespace: neuron-prod
spec:
podSelector:
matchLabels:
app: neuron-rest
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: neuron-prod
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
---
# ── neuron-marketing: accept from Traefik only ───────────────────────────────
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-marketing-ingress
namespace: neuron-prod
spec:
podSelector:
matchLabels:
app: neuron-marketing
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
---
# ── Egress: all prod pods may reach platform (postgres/redis), vault,
# monitoring (alloy OTLP), kube-dns, and the internet (external APIs) ─
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-prod-egress
namespace: neuron-prod
spec:
podSelector: {}
policyTypes:
- Egress
egress:
# Kubernetes DNS
- ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
# platform namespace (Postgres, Redis, accounts service)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: platform
# vault namespace
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: vault
# monitoring namespace (Alloy OTLP)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
# intra-namespace (mcp ↔ rest ↔ license)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: neuron-prod
# external internet (Stripe, OAuth providers, Cloudflare, etc.)
- ports:
- port: 443
protocol: TCP
- port: 80
protocol: TCP
servers/legion/k8s/neuron-technologies/prod/pdb.yaml
+39
View File
@@ -0,0 +1,39 @@
---
# PodDisruptionBudgets for neuron-prod
# Ensures at least 1 replica of each service survives node drains and upgrades.
# With replicas=1 today, minAvailable=1 means drain will block until HPA scales up
# to 2+ — this is intentional and provides a natural safety gate.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: neuron-mcp-pdb
namespace: neuron-prod
spec:
minAvailable: 1
selector:
matchLabels:
app: neuron-mcp
slot: blue
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: neuron-rest-pdb
namespace: neuron-prod
spec:
minAvailable: 1
selector:
matchLabels:
app: neuron-rest
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: neuron-marketing-pdb
namespace: neuron-prod
spec:
# Marketing runs 2 replicas — keep at least 1 alive during drains.
minAvailable: 1
selector:
matchLabels:
app: neuron-marketing
servers/legion/k8s/neuron-technologies/prod/rest-deployment.yaml
+17 -2
View File
@@ -19,6 +19,10 @@ spec:
annotations:
kubectl.kubernetes.io/restartedAt: "2026-04-24T10:00:00Z"
spec:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: neuron-rest
image: registry.neuralplatform.ai/neuron-technologies/neuron-rest:v0.15.3
@@ -26,6 +30,12 @@ spec:
ports:
- name: http
containerPort: 8081
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: false
runAsNonRoot: true
capabilities:
drop: ["ALL"]
envFrom:
- configMapRef:
name: neuron-prod-config
@@ -41,18 +51,23 @@ spec:
limits:
cpu: 1000m
memory: 1Gi
startupProbe:
httpGet:
path: /actuator/health/liveness
port: http
failureThreshold: 30
periodSeconds: 5
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: http
initialDelaySeconds: 30
periodSeconds: 30
failureThreshold: 3
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: http
initialDelaySeconds: 15
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
volumes: