Will Anderson
3b3811942c
Static IP assigned on new network segment. Updates: - variables.tf default legion_ip - headscale nameserver config - bootstrap.sh default target - README/RUNBOOK documentation - media ingress comment
123 lines
4.2 KiB
Markdown
123 lines
4.2 KiB
Markdown
# Legion
|
|
|
|
Ubuntu 24.04 home server running k3s. Hosts all personal and platform infrastructure.
|
|
|
|
**LAN IP:** `192.168.8.148`
|
|
**SSH:** `ssh legion`
|
|
**Kubeconfig:** `~/.kube/legion-config`
|
|
|
|
---
|
|
|
|
## Services
|
|
|
|
### Git & Code
|
|
| Service | URL | What it does |
|
|
|---------|-----|--------------|
|
|
| Gitea | https://git.neuralplatform.ai | Self-hosted Git — all repos live here |
|
|
| GitHub Actions Runner | — | Runs CI jobs from harmonic-framework org |
|
|
| Verdaccio | https://npm.neuralplatform.ai | Private npm registry |
|
|
| devpi | https://pypi.neuralplatform.ai | Private PyPI registry |
|
|
|
|
### Platform
|
|
| Service | URL | What it does |
|
|
|---------|-----|--------------|
|
|
| Argo CD | https://argocd.neuralplatform.ai | Deploys everything in `apps/` from Gitea |
|
|
| Vault | https://vault.neuralplatform.ai | Secret storage |
|
|
| Docker Registry | https://registry.neuralplatform.ai | Private container image registry |
|
|
|
|
### AI & Apps
|
|
| Service | URL | What it does |
|
|
|---------|-----|--------------|
|
|
| Neuron | internal | MCP server + webhook handler for Axon events |
|
|
| Ollama | http://ollama.legion.local | Local LLM inference (GPU) |
|
|
|
|
### Networking & DNS
|
|
| Service | URL | What it does |
|
|
|---------|-----|--------------|
|
|
| AdGuard Home | https://dns.nook.family | LAN DNS with ad blocking |
|
|
| DoT | `dot.nook.family:853` | DNS-over-TLS for mobile devices (Android Private DNS) |
|
|
| Traefik | — | Ingress controller, terminates TLS for all services |
|
|
| Cloudflare Tunnel | — | Exposes public services without opening ports |
|
|
|
|
### Observability
|
|
| Service | URL | What it does |
|
|
|---------|-----|--------------|
|
|
| Grafana | https://grafana.neuralplatform.ai | Dashboards, log search, trace viewer |
|
|
| Prometheus | internal | Metrics collection and alerting |
|
|
| Alertmanager | internal | Routes alerts to #infrastructure-alerts in Slack |
|
|
| Loki | internal | Log aggregation (fed by Alloy) |
|
|
| Tempo | internal | Distributed tracing backend |
|
|
| Alloy | internal | Collects pod logs → Loki, forwards OTLP traces → Tempo |
|
|
|
|
### Data
|
|
| Service | What it does |
|
|
|---------|--------------|
|
|
| Postgres | Shared database (Gitea, Kong, others) |
|
|
| Redis | Shared cache |
|
|
|
|
---
|
|
|
|
## How Deployment Works
|
|
|
|
- **Infrastructure** (namespaces, secrets, Helm charts, PVCs): managed by Terraform in this directory
|
|
- **Apps** (Deployments, Services, ConfigMaps): live in `apps/`, deployed by Argo CD watching this repo
|
|
- Never apply changes with `kubectl` directly — use Terraform or push to `apps/`
|
|
|
|
```bash
|
|
# Apply infra changes
|
|
cd servers/legion/
|
|
direnv exec . terraform apply
|
|
|
|
# Deploy app changes — just push to Gitea
|
|
git push # Argo CD auto-syncs within ~30s
|
|
```
|
|
|
|
---
|
|
|
|
## Networking
|
|
|
|
**Router:** TP-Link Deco BE5000 mesh network, router mode. Subnet `192.168.68.x`, DHCP DNS points to AdGuard (`192.168.8.148`).
|
|
|
|
External traffic reaches Legion two ways:
|
|
|
|
1. **Cloudflare Tunnel** — public services (`*.neuralplatform.ai`) route through a tunnel pod in k8s. No ports open on the router.
|
|
2. **Direct port forward** — DoT on port 853 (TCP) is forwarded from the router to `192.168.8.148:853`. Required because Cloudflare can't proxy raw TCP.
|
|
|
|
LAN devices use AdGuard as their DNS server (`192.168.8.148`). AdGuard rewrites `*.nook.family` and `*.neuralplatform.ai` to the LAN IP so local traffic stays local.
|
|
|
|
ddclient keeps the `dot.nook.family` A record updated if the home IP changes.
|
|
|
|
---
|
|
|
|
## GPU
|
|
|
|
Legion has an NVIDIA GeForce GTX 1660 Ti (6GB VRAM). k3s is configured to use it as the default container runtime. Ollama uses it for local inference.
|
|
|
|
```bash
|
|
kubectl describe node legion | grep nvidia.com/gpu
|
|
```
|
|
|
|
---
|
|
|
|
## Backups
|
|
|
|
Nightly at 2am (cron job in `git` namespace):
|
|
- Dumps all Postgres databases
|
|
- Backs up Gitea data volume + DB dump via restic → R2 bucket `legion-gitea-backup`
|
|
- Retains 30 daily / 12 weekly / 6 monthly snapshots
|
|
|
|
`~/Secrets` on the Mac is encrypted with age and uploaded to R2 bucket `legion-secrets-backup` nightly.
|
|
|
|
See `RUNBOOK.md` for full disaster recovery procedure.
|
|
|
|
---
|
|
|
|
## Key Files
|
|
|
|
| File | What it does |
|
|
|------|--------------|
|
|
| `bootstrap.sh` | Provisions a fresh Ubuntu 24.04 machine from scratch |
|
|
| `RUNBOOK.md` | Step-by-step disaster recovery |
|
|
| `variables.tf` | All configurable values and secrets |
|
|
| `apps/` | Kubernetes manifests deployed by Argo CD |
|