Will Anderson
3b3811942c
Static IP assigned on new network segment. Updates: - variables.tf default legion_ip - headscale nameserver config - bootstrap.sh default target - README/RUNBOOK documentation - media ingress comment
177 lines
5.1 KiB
Markdown
177 lines
5.1 KiB
Markdown
# Legion Disaster Recovery Runbook
|
|
|
|
Complete recovery from total loss of the Legion server.
|
|
Target: running cluster in under 2 hours on a fresh machine.
|
|
|
|
---
|
|
|
|
## Prerequisites (stored offsite)
|
|
|
|
| What | Where |
|
|
|------|-------|
|
|
| `~/Secrets` (encrypted) | R2 bucket `legion-secrets-backup` |
|
|
| Terraform state | R2 bucket `legion-terraform-state` |
|
|
| Gitea repos + DB dump | R2 bucket `legion-gitea-backup` (restic) |
|
|
| age decryption key | Printed copy / separate secure storage |
|
|
|
|
> **Note:** The age private key (`~/Secrets/age-backup.key`) is inside the encrypted
|
|
> archive — which means you need the key to decrypt the archive that contains the key.
|
|
> Store the age private key somewhere independent (password manager, printed sheet, etc.).
|
|
|
|
---
|
|
|
|
## Step 1 — Recover Secrets (on Mac or any machine with `age` + `aws`)
|
|
|
|
```bash
|
|
# Install age if needed
|
|
brew install age
|
|
|
|
# Configure R2 access (you'll need the R2 access key from your password manager)
|
|
export AWS_ACCESS_KEY_ID=<r2-access-key-id>
|
|
export AWS_SECRET_ACCESS_KEY=<r2-secret-access-key>
|
|
R2_ENDPOINT="https://651161e0a3d321561b4c90b5bcd5f15b.r2.cloudflarestorage.com"
|
|
|
|
# Download the latest encrypted backup
|
|
aws s3 cp s3://legion-secrets-backup/secrets-latest.tar.gz.age /tmp/secrets.tar.gz.age \
|
|
--endpoint-url "$R2_ENDPOINT"
|
|
|
|
# Decrypt and extract (you need the age private key for this)
|
|
age --decrypt -i /path/to/age-backup.key /tmp/secrets.tar.gz.age | tar -xz -C ~/
|
|
|
|
# Verify
|
|
ls ~/Secrets/credentials/
|
|
source ~/Secrets/credentials/infrastructure.env
|
|
```
|
|
|
|
---
|
|
|
|
## Step 2 — Provision Fresh Machine
|
|
|
|
Fresh Ubuntu 24.04 LTS, same LAN IP (`192.168.8.148`).
|
|
|
|
```bash
|
|
# From Mac — copy SSH key and cloudflared credentials
|
|
ssh-copy-id will@192.168.8.148
|
|
scp ~/Secrets/certs/cloudflared-gitea.json legion:/tmp/cloudflared-gitea.json
|
|
|
|
# Run bootstrap
|
|
ssh legion "bash -s" < ~/Development/infrastructure/servers/legion/bootstrap.sh
|
|
|
|
# Copy kubeconfig back to Mac
|
|
scp legion:/home/will/.kube/config ~/.kube/legion-config
|
|
export KUBECONFIG=~/.kube/legion-config
|
|
kubectl get nodes # should show legion as Ready
|
|
```
|
|
|
|
**Bootstrap installs:**
|
|
- NVIDIA driver 580 + container toolkit
|
|
- k3s (latest stable) with NVIDIA as default runtime
|
|
- NVIDIA device plugin
|
|
- Cloudflared (host-level, Gitea SSH tunnel)
|
|
- Terraform + Helm
|
|
|
|
---
|
|
|
|
## Step 3 — Apply Terraform
|
|
|
|
Terraform state lives in R2 — it will re-create all k8s resources from scratch.
|
|
|
|
```bash
|
|
cd ~/Development/infrastructure/servers/legion/
|
|
direnv exec . terraform init # pulls state from R2
|
|
direnv exec . terraform plan # review what will be created
|
|
direnv exec . terraform apply # recreates everything: namespaces, PVCs, secrets, Helm releases, Argo CD
|
|
```
|
|
|
|
Wait for Argo CD to sync (~2 minutes after apply). Check:
|
|
|
|
```bash
|
|
kubectl get pods -A # all pods should reach Running
|
|
```
|
|
|
|
---
|
|
|
|
## Step 4 — Restore Gitea Data
|
|
|
|
Gitea repos and the Postgres DB are backed up nightly via restic to `legion-gitea-backup`.
|
|
|
|
```bash
|
|
# Set up restic env
|
|
set -a; source ~/Secrets/credentials/infrastructure.env; set +a
|
|
export RESTIC_REPOSITORY="s3:$(secret r2-endpoint)/legion-gitea-backup"
|
|
export RESTIC_PASSWORD="gitea-restic-2026"
|
|
export AWS_ACCESS_KEY_ID=$(secret r2-access-key)
|
|
export AWS_SECRET_ACCESS_KEY=$(secret r2-secret-key)
|
|
|
|
# List available snapshots
|
|
restic snapshots
|
|
|
|
# Restore Gitea data volume to a local dir
|
|
restic restore latest --target /tmp/gitea-restore --include /gitea-data
|
|
|
|
# Copy into the PVC via kubectl
|
|
GITEA_POD=$(kubectl get pod -n git -l app=gitea -o jsonpath='{.items[0].metadata.name}')
|
|
kubectl cp /tmp/gitea-restore/gitea-data/. git/$GITEA_POD:/data/gitea/
|
|
|
|
# Restore Postgres DB dump
|
|
restic restore latest --target /tmp/gitea-restore --include /dump/all-databases.sql
|
|
kubectl exec -n platform deploy/postgres-postgresql -- \
|
|
psql -U postgres < /tmp/gitea-restore/dump/all-databases.sql
|
|
|
|
# Restart Gitea to pick up restored data
|
|
kubectl rollout restart deployment/gitea -n git
|
|
```
|
|
|
|
---
|
|
|
|
## Step 5 — Verify
|
|
|
|
```bash
|
|
# k8s cluster healthy
|
|
kubectl get nodes
|
|
kubectl get pods -A | grep -v Running | grep -v Completed
|
|
|
|
# Argo CD synced
|
|
kubectl get applications -n argocd
|
|
|
|
# DNS working (AdGuard)
|
|
dig @192.168.8.148 git.neuralplatform.ai
|
|
|
|
# Services reachable
|
|
curl -sk https://git.neuralplatform.ai/api/v1/version
|
|
curl -sk https://argocd.neuralplatform.ai/healthz
|
|
|
|
# GPU available
|
|
kubectl describe node legion | grep nvidia.com/gpu
|
|
```
|
|
|
|
---
|
|
|
|
## Key Reference
|
|
|
|
| Secret | Vault path | Field |
|
|
|--------|------------|-------|
|
|
| R2 access key | `secret/r2` | `access_key_id` |
|
|
| R2 secret | `secret/r2` | `secret_access_key` |
|
|
| Cloudflare API key | `secret/cloudflare` | `api_key` |
|
|
| Gitea admin token | `secret/gitea` | `api_token` |
|
|
| Postgres password | `secret/postgres` | `password` |
|
|
| Vault root token | `~/Secrets/tokens/vault-root-token` | — |
|
|
|
|
Tunnel IDs:
|
|
- Platform tunnel (k8s pod): `54bc9b05-3953-47a2-9c3e-adecdcc53d51`
|
|
- Gitea SSH tunnel (host cloudflared): `b1bf80bf-5448-4c33-8667-d6fce8a82c7a`
|
|
|
|
---
|
|
|
|
## Estimated Recovery Time
|
|
|
|
| Step | Time |
|
|
|------|------|
|
|
| Recover secrets | 5 min |
|
|
| Provision OS + bootstrap | 20-30 min (driver install) |
|
|
| Terraform apply | 10-15 min |
|
|
| Gitea data restore | 10 min |
|
|
| Verify | 5 min |
|
|
| **Total** | **~1 hour** |
|