Commit Graph

19 Commits

Author SHA1 Message Date
Will Anderson 91bfc42989 ops: scale down all non-critical services — dedicate resources to Neuron
Scaled to replicas: 0: fornax (coordinator/ui/workers), jellyfin via media,
overseerr, plex, radarr, sonarr, bazarr, prowlarr, flaresolverr, ollama,
harmonic-wordpress, docuseal, listmonk, memos, plane, redpanda, wp-coordinator
2026-04-24 20:07:41 -05:00
Will Anderson e12fd9781a Fix VPN routing: give portforward-helper NET_ADMIN to delete gluetun rule 100
gluetun adds 'ip rule priority 100 from <pod-IP> lookup 200' which routes
all pod traffic via eth0. The iptables OUTPUT DROP policy then blocks it.
Rule 101 (not fwmark 0xca6c -> tun0) never fires because rule 100 matches
first. Result: qBittorrent has no internet access through the VPN.

Fix: portforward-helper waits for gluetun to be ready (polls :9999), then
deletes rule 100. Application traffic then falls through to rule 101 and
routes correctly via tun0.
2026-04-15 01:40:02 -05:00
Will Anderson ef8dc293e5 Set HEALTH_ICMP_TARGET_IPS to WireGuard gateway for periodic health check
ProtonVPN does not NAT ICMP to external IPs (1.1.1.1, 8.8.8.8), causing
the periodic small health check to always fail and restart the VPN loop.
10.2.0.1 (WireGuard gateway) responds to ICMP at ~28ms and remains
reachable as long as the WireGuard tunnel itself is up.
2026-04-14 22:08:40 -05:00
Will Anderson 214afed0a2 Set DNS_UPSTREAM_RESOLVER_TYPE=plain to actually disable DoT in gluetun
DOT=off alone does not override DNS_UPSTREAM_RESOLVER_TYPE in this version
of gluetun — it defaults to DoT regardless, causing DNS failures through
ProtonVPN (port 853 connection reset). Setting the resolver type directly
fixes plain DNS routing to k8s CoreDNS.
2026-04-14 22:04:36 -05:00
Will Anderson 41ab2eaf0e Fix DNS_UPSTREAM_PLAIN_ADDRESSES format: requires ip:port not bare ip 2026-04-14 22:01:50 -05:00
Will Anderson 10dd561ec0 Fix gluetun healthcheck targets and DNS env var on fornax workers
Change HEALTH_TARGET_ADDRESS to ProtonVPN API endpoints so TLS startup
check doesn't fail against cloudflare.com/github.com (rejected from
ProtonVPN exit IPs). Rename deprecated DNS_ADDRESS to
DNS_UPSTREAM_PLAIN_ADDRESSES in fornax-workers.yaml.
2026-04-14 21:58:52 -05:00
Will Anderson a1623eda39 Fix gluetun DNS: disable DoT, use k8s coredns as plaintext resolver
DoT (port 853) to 1.1.1.1 through ProtonVPN VPN tunnel gets TCP RST,
causing gluetun healthcheck to fail (can't resolve github.com /
cloudflare.com). k8s coredns at 10.43.0.10 is reachable via eth0
within FIREWALL_OUTBOUND_SUBNETS, bypassing the VPN for DNS while
letting all other traffic tunnel correctly.

Also rename VPN_ENDPOINT_IP/PORT to WIREGUARD_ENDPOINT_IP/PORT to
suppress gluetun deprecation warnings.
2026-04-14 21:32:57 -05:00
Will Anderson 741191a7e6 Switch workers to custom WireGuard provider with correct endpoint IPs
gluetun's bundled protonvpn server list has stale IPs for US-TX#179
(37.19.200.26) and US-TX#220 (95.173.217.2). The WIREGUARD_ENDPOINT_IP
env var is a filter not an override, so there's no way to redirect
protonvpn provider to a new IP.

Switch to custom WireGuard provider with hardcoded endpoint IPs from
fresh ProtonVPN configs (95.173.217.29 / 146.70.58.130). Add
DNS_KEEP_NAMESERVER=on so gluetun leaves k8s DNS intact instead of
routing DNS through its own proxy (which breaks in-cluster).

Port forwarding is not available with custom provider; will restore
once gluetun releases an updated server list image.
2026-04-14 21:29:29 -05:00
Will Anderson a2239ab89e Switch back to protonvpn provider with US-TX#253 and US-TX#34 2026-04-14 21:17:20 -05:00
Will Anderson 88c4d795e7 Switch DNS to 8.8.8.8 — ProtonVPN internal DNS returning errors 2026-04-14 21:12:33 -05:00
Will Anderson 027667e7f7 Remove custom health target — use ProtonVPN internal DNS for resolution 2026-04-14 21:10:42 -05:00
Will Anderson e6d1350ca8 Use ProtonVPN internal DNS and direct IP healthcheck to diagnose VPN 2026-04-14 21:08:55 -05:00
Will Anderson 9778f79845 Configure per-worker VPN endpoints: tx253→US-TX#253, tx34→US-TX#34 2026-04-14 21:02:01 -05:00
Will Anderson fcff5f62e5 Switch worker base to custom WireGuard provider for US-IL#267 2026-04-14 20:31:03 -05:00
Will Anderson bdba692db2 Fornax: COORDINATOR_URL env, port-lease webhook, Grafana dashboard
- Worker deployment: add COORDINATOR_URL and WORKER_ID env vars to
  portforward-helper; webhook calls coordinator /api/v2/workers/:id/port-lease
  after each NAT-PMP renewal so coordinator tracks current forwarded port
- Worker overlays: patch WORKER_ID env var to actual worker ID (tx253/tx34)
- Grafana: fornax-grafana Argo CD app pointing to k8s/fornax/grafana/
  ConfigMap with grafana_dashboard=1 label for auto-discovery by sidecar;
  7 panels covering bandwidth, active torrents, worker health, request rate,
  and error rate
2026-04-14 05:30:01 +00:00
Will Anderson 4a0ebededf fix: revert to protonvpn provider, use server name from secret 2026-04-14 00:07:11 +00:00
Will Anderson 1cc783e704 fix: use custom gluetun provider with endpoint IP for ProtonVPN workers 2026-04-14 00:04:50 +00:00
Will Anderson eebd0226de fix: worker gluetun endpoint IP approach, add VPN key secrets 2026-04-14 00:01:43 +00:00
Will Anderson 180fb91c95 add fornax coordinator and worker k8s manifests 2026-04-13 23:57:22 +00:00