Commit Graph

146 Commits

Author SHA1 Message Date
Will Anderson 59f551663b Switch gluetun to custom provider — use explicit WireGuard config for US-IL#267
Avoids gluetun's ProtonVPN server name lookup entirely. All WireGuard
parameters (endpoint, peer key, client address, keepalive) come directly
from the downloaded ProtonVPN config.
2026-04-14 20:22:31 -05:00
Will Anderson 9a523da831 Use SERVER_COUNTRIES=US with explicit endpoint — gluetun has no US-IL naming 2026-04-14 20:20:47 -05:00
Will Anderson c283110e70 Switch workers to US-IL#267 with explicit WireGuard endpoint and peer key
US-TX#179 and US-TX#220 were failing — WireGuard handshake completing
but no traffic passing (silent drop). Switching both workers to US-IL#267
and pinning the endpoint IP and peer public key directly to avoid relying
on gluetun's built-in server list lookup.
2026-04-14 20:18:57 -05:00
Will Anderson 7f7e2e26da Force coordinator rollout to pick up new image with downloadSpeedBps 2026-04-14 20:10:08 -05:00
Will Anderson b5f481d9a4 Fix qbt-config-patch for Python 3.14: use lambda in re.sub replacements
Python 3.14 raises PatternError for backslash sequences like \P in
re.sub replacement strings. Switch to lambda replacements which bypass
that interpretation. Also pin to python:3.13-alpine to avoid future
surprises from pulling :3-alpine.
2026-04-14 20:00:24 -05:00
Will Anderson 07f75c7dc8 Force UI pod rollout to pick up new nginx image 2026-04-14 19:57:32 -05:00
Will Anderson 87c4b63cca Route /api/ui through nginx to bypass Cloudflare Access; bump restartedAt
The /api/ui path is now routed to the fornax-ui nginx pod (which proxies
internally to the coordinator) rather than directly to the coordinator.
This fixes the browser fetch being blocked by Cloudflare Access.

Both coordinator and UI deployments get a restartedAt bump to pull the
new images after CI builds.
2026-04-14 19:44:23 -05:00
Will Anderson 3110903c81 Force rollout to pick up new images with real qBT wiring 2026-04-14 19:30:23 -05:00
Will Anderson a1615df3e9 Fix overseerr.yaml YAML indentation error blocking media app sync 2026-04-14 19:18:13 -05:00
Will Anderson 2a21c9d57e Fix fornax service port and add qBittorrent worker env vars
- fornax service: targetPort 8080→3000 (Express listens on 3000, not 8080)
  This was silently dropping all Radarr/Sonarr requests
- coordinator deployment: add QBT_WORKER_ADDRS, QBT_USER, QBT_PASS
  so /api/ui/state can query real worker qBittorrent instances
- remove Python coordinator ConfigMap and Deployment (superseded)
2026-04-14 19:17:00 -05:00
Will Anderson 788f103c0e Force UI rollout to pick up new image with CSS stylesheet 2026-04-14 19:02:48 -05:00
Will Anderson 7e2a7c7126 Force coordinator rollout to pick up new image with crash fix 2026-04-14 19:02:18 -05:00
Will Anderson 27d9d9c436 fornax: fix crash-looping coordinator, add UI deployment
- Fix liveness/readiness probe paths from /api/v2/app/version (qBittorrent)
  to /health and /health/ready (Fornax Express server). Pod was restarting
  every ~30s due to probe failures against wrong endpoint.
- Add fornax-ui deployment and service for React dashboard
  (image: fornax/ui:latest, nginx on port 80)
- Split ingress: /api and /health → coordinator:3000, / → ui:80
- Update kustomization to include UI resources
2026-04-14 08:45:21 -05:00
Will Anderson bdba692db2 Fornax: COORDINATOR_URL env, port-lease webhook, Grafana dashboard
- Worker deployment: add COORDINATOR_URL and WORKER_ID env vars to
  portforward-helper; webhook calls coordinator /api/v2/workers/:id/port-lease
  after each NAT-PMP renewal so coordinator tracks current forwarded port
- Worker overlays: patch WORKER_ID env var to actual worker ID (tx253/tx34)
- Grafana: fornax-grafana Argo CD app pointing to k8s/fornax/grafana/
  ConfigMap with grafana_dashboard=1 label for auto-discovery by sidecar;
  7 panels covering bandwidth, active torrents, worker health, request rate,
  and error rate
2026-04-14 05:30:01 +00:00
Will Anderson 4a0ebededf fix: revert to protonvpn provider, use server name from secret 2026-04-14 00:07:11 +00:00
Will Anderson 1cc783e704 fix: use custom gluetun provider with endpoint IP for ProtonVPN workers 2026-04-14 00:04:50 +00:00
Will Anderson eebd0226de fix: worker gluetun endpoint IP approach, add VPN key secrets 2026-04-14 00:01:43 +00:00
Will Anderson 180fb91c95 add fornax coordinator and worker k8s manifests 2026-04-13 23:57:22 +00:00
Will Anderson db980af27a monitoring: add WireGuard key expiry reminders for voidstash GL.iNet
Posts to #infrastructure-alerts via Slack bot on escalating schedule:
- 2027-03-11: 1 month out
- 2027-03-25, 04-01, 04-04: weekly
- 2027-04-05 through 04-10: daily

Key: neural_net-US-IL-267, expires 2027-04-11
2026-04-11 13:29:38 -05:00
Will Anderson 4ef625ef3d qbittorrent: bake performance settings into initContainer
Switch qbt-config-patch from busybox/sed to python:3-alpine so we can
cleanly handle INI keys with backslashes. Now seeds both password hash
and high-throughput defaults (3000 max connections, 50 active downloads,
unlimited rate) on fresh PVC deployments. Existing configs are updated
in-place; API-applied values are preserved on restart.
2026-04-11 12:24:16 -05:00
Will Anderson 259b1bdade overseerr: remove Kids & Family library from Jellyfin config 2026-04-11 12:17:49 -05:00
Will Anderson 494b72229c overseerr: seed Sonarr and Radarr config in restore.js
Sonarr and Radarr service settings live in settings.json (not the SQLite
database). Extend the restore.js initContainer to seed both services on
first boot or after a PVC wipe, matching current live config.
2026-04-11 12:17:18 -05:00
Will Anderson c4cd91920a Fix qBittorrent auth: stamp known password hash via initContainer
LocalhostAuthEnabled=false isn't honored in this qBT version. Instead, initContainer
stamps the PBKDF2 hash for admin:adminadmin before startup so all three pods have
consistent credentials. portforward-helper and coordinator restored to cookie-based
SID auth.
2026-04-11 11:14:23 -05:00
Will Anderson 531df76a90 Patch qBittorrent config via initContainer before startup
Adds qbt-config-patch initContainer that sets WebUI\LocalhostAuthEnabled=false
in qBittorrent.conf before the main containers start. Prevents qBittorrent from
overwriting the setting (previously edited at runtime, lost on pod restart).
2026-04-11 11:11:23 -05:00
Will Anderson 854ccd1344 Disable qBittorrent localhost auth; remove credentials from helpers
LocalhostAuthEnabled=false set in all three qbt configs. portforward-helper and
coordinator now call the API directly without auth — no more ban risk from failed
login attempts.
2026-04-11 11:08:54 -05:00
Will Anderson ffd3068f78 Fix port forwarding: read file instead of HTTP API, fix server names
- portforward-helper now reads /tmp/gluetun/forwarded_port via shared emptyDir
  volume instead of polling gluetun HTTP API (which returned Unauthorized)
- Main qbt: SERVER_NAMES=US-IL#1 (valid entry in gluetun server list)
- Both manifests cleaned up
2026-04-11 10:12:19 -05:00
Will Anderson 69976ca172 Fix main qbt server selection; bump portforward-helper memory limit
- Main qbt: switch from pinned US-IL#149 (no PF) to SERVER_REGIONS=Illinois so
  gluetun picks any IL server with port forwarding enabled
- Fornax workers: portforward-helper 32Mi→96Mi to stop OOMKill
2026-04-11 10:03:52 -05:00
Will Anderson 0aff4523af Upgrade coordinator to least-loaded routing (checks active torrent count per worker) 2026-04-11 10:02:45 -05:00
Will Anderson ff6c1f1ee4 Add Fornax coordinator — simple round-robin qBittorrent proxy
Python proxy running on python:3-alpine via ConfigMap. Routes /api/v2/torrents/add
across all three workers in round-robin; proxies status/monitoring to primary.
Manages independent SID sessions to each backend with auto-refresh on 403.
Service: fornax.media.svc:8080 — point Sonarr/Radarr here instead of qbittorrent.
2026-04-11 10:01:03 -05:00
Will Anderson cce8c0a0d4 Fix server names to match gluetun's ProtonVPN server list
US-TX#253 and US-TX#34 aren't in gluetun's built-in list. Switching to:
- Main qbt: US-IL#149 (was US-IL#267)
- Fornax TX253: US-TX#179
- Fornax TX34: US-TX#220
Private keys are per-account and work with any ProtonVPN WireGuard server.
2026-04-11 09:59:23 -05:00
Will Anderson e1996d4396 Switch to gluetun native ProtonVPN port forwarding (VPN_PORT_FORWARDING=on)
Replaces DIY natpmpc sidecar with gluetun's built-in NAT-PMP handling for the protonvpn
provider. The natpmpc UDP response was being dropped by gluetun's firewall since conntrack
doesn't track stateless UDP from the gateway. With VPN_PORT_FORWARDING=on, gluetun handles
the NAT-PMP exchange internally and exposes the port at :8000/v1/openvpn/portforwarded.
Helper sidecar now just polls that endpoint.
2026-04-11 09:57:07 -05:00
Will Anderson 24e308202b Allow NAT-PMP response port through gluetun firewall (FIREWALL_INPUT_PORTS=5351) 2026-04-11 09:50:55 -05:00
Will Anderson 03011a73e8 Add Fornax worker pods — US-TX#253 and US-TX#34
Two initial Fornax distributed torrent workers, each a gluetun+qBittorrent+natpmpc-helper
pod on a different ProtonVPN TX server with NAT-PMP enabled. VPN private keys stored in
Vault at secret/fornax/worker-tx253 and secret/fornax/worker-tx34, surfaced via
ExternalSecrets. Workers share the media-data PVC; each has its own config PVC.

Services: fornax-worker-tx253:8080 and fornax-worker-tx34:8080 (ClusterIP, media ns)
2026-04-11 09:46:18 -05:00
Will Anderson c6aa006ec7 Switch VPN to US-IL#267 (NAT-PMP port forwarding supported) 2026-04-11 08:41:06 -05:00
Will Anderson 3f33a76ed2 media: natpmpc-helper sleep before apk install (VPN must be up first) 2026-04-11 08:29:54 -05:00
Will Anderson ca5f9e90da media: fix natpmpc-helper package name (libnatpmp not natpmpc) 2026-04-11 08:27:53 -05:00
Will Anderson f99965da0a media: fix natpmpc-helper OOM and API call format
Increase memory limit 32→64Mi (apk install was hitting the 32Mi ceiling).
Fix qBittorrent setPreferences call: use form-encoded json=<string> not
json body nesting.
2026-04-11 08:24:19 -05:00
Will Anderson fd08987e12 media: add natpmpc-helper sidecar for ProtonVPN port forwarding
Adds a lightweight Alpine sidecar that runs natpmpc against the ProtonVPN
NAT-PMP gateway (10.2.0.1) every 45s to maintain a forwarded port, then
updates qBittorrent's listen port via API. Without inbound port forwarding
peers can only be dialed out to, which severely limits download speeds on
a gigabit connection.
2026-04-11 08:21:37 -05:00
Will Anderson 615f455eff fix(bazarr): merge duplicate metadata blocks in pod template 2026-04-11 04:40:21 -05:00
Will Anderson 0f15063260 fix(bazarr): patch ffprobe refiner to handle BoxList audio language
Audio tracks in some video files return language as a BoxList (list of
Language objects) from the video metadata parser. The existing code
tries to add this directly to a Python set, which fails because BoxList
is not hashable — crashing subtitle search for every movie.

Fix: check if lang has __iter__ and iterate over it, or call .alpha3
directly if it's a single Language object.

Delivered via ConfigMap volumeMount so it survives pod restarts.
2026-04-11 04:39:33 -05:00
Will Anderson 7c17b8a93e fix(gluetun): remove shell wrapper, use default entrypoint
The /gluetun binary has restrictive permissions and cannot be exec'd
from a shell wrapper. The OOM fix (512Mi limit) prevents the crashes
that caused stale ip rules, so the wrapper is not needed.
2026-04-11 04:31:01 -05:00
Will Anderson 203233beca fix(gluetun): prevent OOM crash and stale ip rule on container restart
- Increase memory limit 128Mi → 512Mi (DNS block list download needs ~200MB)
- Add command wrapper that cleans stale ip rules before exec'ing gluetun
  so container restarts within the same pod don't fail with "file exists"
- Fix deprecated VPN_ENDPOINT_IP/PORT → WIREGUARD_ENDPOINT_IP/PORT
2026-04-11 04:28:32 -05:00
Will Anderson 28a108a1d8 fix(gluetun): pin to US-TX#457 server to restore downloads
The protonvpn named provider rotates through P2P servers and DNS
health checks were failing on every server it tried. Switching back
to custom provider pinned to the specific US-TX#457 server/key combo
from the ProtonVPN conf file, which is confirmed to work.

This sacrifices automatic port forwarding but restores stable downloads.
2026-04-11 04:24:35 -05:00
Will Anderson e5785f3359 gluetun: fix ip rule cleanup in init container for VPN restart stability
Delete ip rule by priority 101 before table 51820 to prevent 'file exists'
errors when gluetun internally restarts the WireGuard connection.
2026-04-11 04:18:56 -05:00
Will Anderson 9cd3dd0134 gluetun: switch to protonvpn provider with port forwarding enabled
Custom WireGuard mode doesn't support NAT-PMP port forwarding — peers
can't connect inbound so most torrents stall. Switching to protonvpn
provider with SERVER_FEATURES=p2p picks a port-forwarding-capable server
and VPN_PORT_FORWARDING=on handles the NAT-PMP handshake. Up command
updates qBittorrent listen port automatically when gluetun gets the port.
2026-04-11 04:09:29 -05:00
Will Anderson d59af5b6aa seerr: fix mediaServerType to numeric 2 (JELLYFIN enum value)
Seerr's scheduler compares mediaServerType against numeric enum values
(PLEX=1, JELLYFIN=2, EMBY=3). restore.js was setting the string 'JELLYFIN'
which never matched, so Jellyfin scan jobs were never registered in the
scheduler — causing 'Watch on Jellyfin' buttons to never appear.
2026-04-11 02:52:15 -05:00
Will Anderson ba339ee5c9 bazarr: increase liveness probe delay to 120s — startup takes 90+s
Bazarr initializes scheduler, DB, and Radarr/Sonarr connections before
serving HTTP. 30s initialDelay was too short, causing restart loops.
Also add readinessProbe and failureThreshold for resilience.
2026-04-11 02:33:41 -05:00
Will Anderson c8d240e131 fix(seerr): persist applicationUrl so Jellyfin OAuth callback works 2026-04-11 02:16:57 -05:00
Will Anderson 7225f56fbc fix(seerr): set public.initialized=true to skip setup wizard on every start 2026-04-11 02:01:54 -05:00
Will Anderson c1d4390393 fix(seerr): persist library config in ConfigMap, restore libraries on pod restart 2026-04-11 01:58:06 -05:00