infrastructure/servers/legion/RUNBOOK.md

Legion Disaster Recovery Runbook

Complete recovery from total loss of the Legion server. Target: running cluster in under 2 hours on a fresh machine.

---

Prerequisites (stored offsite)

What	Where
~/Secrets (encrypted)	R2 bucket legion-secrets-backup
Terraform state	R2 bucket legion-terraform-state
Gitea repos + DB dump	R2 bucket legion-gitea-backup (restic)
age decryption key	Printed copy / separate secure storage

Note: The age private key (~/Secrets/age-backup.key) is inside the encrypted archive — which means you need the key to decrypt the archive that contains the key. Store the age private key somewhere independent (password manager, printed sheet, etc.).

---

Step 1 — Recover Secrets (on Mac or any machine with age + aws)

# Install age if needed
brew install age

# Configure R2 access (you'll need the R2 access key from your password manager)
export AWS_ACCESS_KEY_ID=<r2-access-key-id>
export AWS_SECRET_ACCESS_KEY=<r2-secret-access-key>
R2_ENDPOINT="https://651161e0a3d321561b4c90b5bcd5f15b.r2.cloudflarestorage.com"

# Download the latest encrypted backup
aws s3 cp s3://legion-secrets-backup/secrets-latest.tar.gz.age /tmp/secrets.tar.gz.age \
  --endpoint-url "$R2_ENDPOINT"

# Decrypt and extract (you need the age private key for this)
age --decrypt -i /path/to/age-backup.key /tmp/secrets.tar.gz.age | tar -xz -C ~/

# Verify
ls ~/Secrets/credentials/
source ~/Secrets/credentials/infrastructure.env

---

Step 2 — Provision Fresh Machine

Fresh Ubuntu 24.04 LTS, same LAN IP (192.168.8.148).

# From Mac — copy SSH key and cloudflared credentials
ssh-copy-id will@192.168.8.148
scp ~/Secrets/certs/cloudflared-gitea.json legion:/tmp/cloudflared-gitea.json

# Run bootstrap
ssh legion "bash -s" < ~/Development/infrastructure/servers/legion/bootstrap.sh

# Copy kubeconfig back to Mac
scp legion:/home/will/.kube/config ~/.kube/legion-config
export KUBECONFIG=~/.kube/legion-config
kubectl get nodes  # should show legion as Ready

Bootstrap installs:

• NVIDIA driver 580 + container toolkit
• k3s (latest stable) with NVIDIA as default runtime
• NVIDIA device plugin
• Cloudflared (host-level, Gitea SSH tunnel)
• Terraform + Helm

---

Step 3 — Apply Terraform

Terraform state lives in R2 — it will re-create all k8s resources from scratch.

cd ~/Development/infrastructure/servers/legion/
direnv exec . terraform init   # pulls state from R2
direnv exec . terraform plan   # review what will be created
direnv exec . terraform apply  # recreates everything: namespaces, PVCs, secrets, Helm releases, Argo CD

Wait for Argo CD to sync (~2 minutes after apply). Check:

kubectl get pods -A  # all pods should reach Running

---

Step 4 — Restore Gitea Data

Gitea repos and the Postgres DB are backed up nightly via restic to legion-gitea-backup.

# Set up restic env
set -a; source ~/Secrets/credentials/infrastructure.env; set +a
export RESTIC_REPOSITORY="s3:$(secret r2-endpoint)/legion-gitea-backup"
export RESTIC_PASSWORD="gitea-restic-2026"
export AWS_ACCESS_KEY_ID=$(secret r2-access-key)
export AWS_SECRET_ACCESS_KEY=$(secret r2-secret-key)

# List available snapshots
restic snapshots

# Restore Gitea data volume to a local dir
restic restore latest --target /tmp/gitea-restore --include /gitea-data

# Copy into the PVC via kubectl
GITEA_POD=$(kubectl get pod -n git -l app=gitea -o jsonpath='{.items[0].metadata.name}')
kubectl cp /tmp/gitea-restore/gitea-data/. git/$GITEA_POD:/data/gitea/

# Restore Postgres DB dump
restic restore latest --target /tmp/gitea-restore --include /dump/all-databases.sql
kubectl exec -n platform deploy/postgres-postgresql -- \
  psql -U postgres < /tmp/gitea-restore/dump/all-databases.sql

# Restart Gitea to pick up restored data
kubectl rollout restart deployment/gitea -n git

---

Step 5 — Verify

# k8s cluster healthy
kubectl get nodes
kubectl get pods -A | grep -v Running | grep -v Completed

# Argo CD synced
kubectl get applications -n argocd

# DNS working (AdGuard)
dig @192.168.8.148 git.neuralplatform.ai

# Services reachable
curl -sk https://git.neuralplatform.ai/api/v1/version
curl -sk https://argocd.neuralplatform.ai/healthz

# GPU available
kubectl describe node legion | grep nvidia.com/gpu

---

Key Reference

Secret	Vault path	Field
R2 access key	secret/r2	access_key_id
R2 secret	secret/r2	secret_access_key
Cloudflare API key	secret/cloudflare	api_key
Gitea admin token	secret/gitea	api_token
Postgres password	secret/postgres	password
Vault root token	~/Secrets/tokens/vault-root-token	—

Tunnel IDs:

• Platform tunnel (k8s pod): 54bc9b05-3953-47a2-9c3e-adecdcc53d51
• Gitea SSH tunnel (host cloudflared): b1bf80bf-5448-4c33-8667-d6fce8a82c7a

---

Estimated Recovery Time

Step	Time
Recover secrets	5 min
Provision OS + bootstrap	20-30 min (driver install)
Terraform apply	10-15 min
Gitea data restore	10 min
Verify	5 min
Total	~1 hour