# Legion Disaster Recovery Runbook

Complete recovery from total loss of the Legion server.
Target: running cluster in under 2 hours on a fresh machine.

---

## Prerequisites (stored offsite)

| What | Where |
|------|-------|
| `~/Secrets` (encrypted) | R2 bucket `legion-secrets-backup` |
| Terraform state | R2 bucket `legion-terraform-state` |
| Gitea repos + DB dump | R2 bucket `legion-gitea-backup` (restic) |
| age decryption key | Printed copy / separate secure storage |

> **Note:** The age private key (`~/Secrets/age-backup.key`) is inside the encrypted
> archive — which means you need the key to decrypt the archive that contains the key.
> Store the age private key somewhere independent (password manager, printed sheet, etc.).

---

## Step 1 — Recover Secrets (on Mac or any machine with `age` + `aws`)

```bash
# Install age if needed
brew install age

# Configure R2 access (you'll need the R2 access key from your password manager)
export AWS_ACCESS_KEY_ID=<r2-access-key-id>
export AWS_SECRET_ACCESS_KEY=<r2-secret-access-key>
R2_ENDPOINT="https://651161e0a3d321561b4c90b5bcd5f15b.r2.cloudflarestorage.com"

# Download the latest encrypted backup
aws s3 cp s3://legion-secrets-backup/secrets-latest.tar.gz.age /tmp/secrets.tar.gz.age \
  --endpoint-url "$R2_ENDPOINT"

# Decrypt and extract (you need the age private key for this)
age --decrypt -i /path/to/age-backup.key /tmp/secrets.tar.gz.age | tar -xz -C ~/

# Verify
ls ~/Secrets/credentials/
source ~/Secrets/credentials/infrastructure.env
```

---

## Step 2 — Provision Fresh Machine

Fresh Ubuntu 24.04 LTS, same LAN IP (`192.168.8.148`).

```bash
# From Mac — copy SSH key and cloudflared credentials
ssh-copy-id will@192.168.8.148
scp ~/Secrets/certs/cloudflared-gitea.json legion:/tmp/cloudflared-gitea.json

# Run bootstrap
ssh legion "bash -s" < ~/Development/infrastructure/servers/legion/bootstrap.sh

# Copy kubeconfig back to Mac
scp legion:/home/will/.kube/config ~/.kube/legion-config
export KUBECONFIG=~/.kube/legion-config
kubectl get nodes  # should show legion as Ready
```

**Bootstrap installs:**
- NVIDIA driver 580 + container toolkit
- k3s (latest stable) with NVIDIA as default runtime
- NVIDIA device plugin
- Cloudflared (host-level, Gitea SSH tunnel)
- Terraform + Helm

---

## Step 3 — Apply Terraform

Terraform state lives in R2 — it will re-create all k8s resources from scratch.

```bash
cd ~/Development/infrastructure/servers/legion/
direnv exec . terraform init   # pulls state from R2
direnv exec . terraform plan   # review what will be created
direnv exec . terraform apply  # recreates everything: namespaces, PVCs, secrets, Helm releases, Argo CD
```

Wait for Argo CD to sync (~2 minutes after apply). Check:

```bash
kubectl get pods -A  # all pods should reach Running
```

---

## Step 4 — Restore Gitea Data

Gitea repos and the Postgres DB are backed up nightly via restic to `legion-gitea-backup`.

```bash
# Set up restic env
set -a; source ~/Secrets/credentials/infrastructure.env; set +a
export RESTIC_REPOSITORY="s3:$(secret r2-endpoint)/legion-gitea-backup"
export RESTIC_PASSWORD="gitea-restic-2026"
export AWS_ACCESS_KEY_ID=$(secret r2-access-key)
export AWS_SECRET_ACCESS_KEY=$(secret r2-secret-key)

# List available snapshots
restic snapshots

# Restore Gitea data volume to a local dir
restic restore latest --target /tmp/gitea-restore --include /gitea-data

# Copy into the PVC via kubectl
GITEA_POD=$(kubectl get pod -n git -l app=gitea -o jsonpath='{.items[0].metadata.name}')
kubectl cp /tmp/gitea-restore/gitea-data/. git/$GITEA_POD:/data/gitea/

# Restore Postgres DB dump
restic restore latest --target /tmp/gitea-restore --include /dump/all-databases.sql
kubectl exec -n platform deploy/postgres-postgresql -- \
  psql -U postgres < /tmp/gitea-restore/dump/all-databases.sql

# Restart Gitea to pick up restored data
kubectl rollout restart deployment/gitea -n git
```

---

## Step 5 — Verify

```bash
# k8s cluster healthy
kubectl get nodes
kubectl get pods -A | grep -v Running | grep -v Completed

# Argo CD synced
kubectl get applications -n argocd

# DNS working (AdGuard)
dig @192.168.8.148 git.neuralplatform.ai

# Services reachable
curl -sk https://git.neuralplatform.ai/api/v1/version
curl -sk https://argocd.neuralplatform.ai/healthz

# GPU available
kubectl describe node legion | grep nvidia.com/gpu
```

---

## Key Reference

| Secret | Vault path | Field |
|--------|------------|-------|
| R2 access key | `secret/r2` | `access_key_id` |
| R2 secret | `secret/r2` | `secret_access_key` |
| Cloudflare API key | `secret/cloudflare` | `api_key` |
| Gitea admin token | `secret/gitea` | `api_token` |
| Postgres password | `secret/postgres` | `password` |
| Vault root token | `~/Secrets/tokens/vault-root-token` | — |

Tunnel IDs:
- Platform tunnel (k8s pod): `54bc9b05-3953-47a2-9c3e-adecdcc53d51`
- Gitea SSH tunnel (host cloudflared): `b1bf80bf-5448-4c33-8667-d6fce8a82c7a`

---

## Estimated Recovery Time

| Step | Time |
|------|------|
| Recover secrets | 5 min |
| Provision OS + bootstrap | 20-30 min (driver install) |
| Terraform apply | 10-15 min |
| Gitea data restore | 10 min |
| Verify | 5 min |
| **Total** | **~1 hour** |