infrastructure/Agents.md

Agent System Reference

Quick reference for AI agents working in Will's infrastructure. Read this before doing any infrastructure work.

---

Machines

Machine	Role	IP	SSH
Local Mac	Dev workstation	192.168.68.55	—
Legion	Ubuntu 24.04, k3s server	192.168.68.77	ssh legion

Legion runs all production services: k3s, Postgres, Redis, Ollama (GPU), Docker Registry, GitHub Actions runner, Vault, Gitea, AdGuard, Neuron.

---

Secrets — Always Use Vault

Vault is the source of truth for all credentials. Never hardcode secrets. Never ask the user for a secret value.

CLI access (on local Mac)

# Load all secrets into env
set -a; source ~/Secrets/credentials/infrastructure.env; set +a

# Or with direnv (auto-loads in legion/ directory):
cd ~/Development/infrastructure/servers/legion/
# direnv auto-sources infrastructure.env

# Friendly alias (after sourcing infrastructure.env):
secret civitai          # → civitai API key
secret slack-bot-token  # → Slack bot token
secret cf-key           # → Cloudflare API key

# Direct vault lookup:
vault kv get -field=api_key secret/ai          # CivitAI
vault kv get -field=bot_token secret/slack      # Slack
vault kv get -field=api_key secret/cloudflare   # Cloudflare

Full path+field table is in ~/Secrets/credentials/infrastructure.env — look at the _v() calls.

Vault connection

VAULT_ADDR=https://vault.neuralplatform.ai
VAULT_TOKEN=$(cat ~/Secrets/tokens/vault-root-token)

If Vault is unreachable (CF tunnel down), port-forward:

ssh legion -L 8200:localhost:8200  # in background
export VAULT_ADDR=http://localhost:8200

How secrets flow to pods

Vault (source of truth) → ExternalSecret (in git) → ESO operator → k8s Secret → pod env

All k8s Secrets are managed by External Secrets Operator (ESO) pulling from Vault. No secrets are stored in Terraform. Do not add kubernetes_secret resources to Terraform.

---

Infrastructure Management

The split: Terraform vs Argo CD

Layer	Tool	What it owns
Infrastructure	Terraform	Namespaces, Cloudflare DNS/tunnels, R2 storage, Vault bootstrap
Configuration	Argo CD	All k8s resources: Deployments, Services, ConfigMaps, Ingresses, PVCs, ExternalSecrets, Helm releases
Secrets	ESO → Vault	All k8s Secrets — ExternalSecret manifests in git, values pulled from Vault at runtime

Never modify k8s resources directly with kubectl. Always go through Terraform or Argo CD (Git push).

Secrets flow:

Vault (source of truth) → ExternalSecret (in git) → ESO operator → k8s Secret → pod env

Adding a new secret:

1. vault kv put secret/<path> field=value
2. Add ExternalSecret manifest to k8s/<service>/ referencing that path
3. Push → Argo CD applies ExternalSecret → ESO syncs Secret within ~60s

Running Terraform

cd ~/Development/infrastructure/servers/legion/
terraform plan    # uses direnv to load credentials
terraform apply

State is stored in Cloudflare R2 bucket legion-terraform-state.

Argo CD

Root app legion-apps watches servers/legion/apps/ in the will/infrastructure Gitea repo. Push to main → Argo CD syncs within ~30 seconds.

• App manifests: ~/Development/infrastructure/servers/legion/apps/*.yaml
• k8s config: ~/Development/infrastructure/servers/legion/k8s/<service>/*.yaml
• Argo CD UI: https://argocd.neuralplatform.ai
• Gitea CLI: tea pr ls, tea issue ls (configured, default login: neuralplatform)

Repo layout

~/Development/
  infrastructure/                      ← this repo (local only, no GitHub remote)
    AGENTS.md                          ← you are here
    servers/                           ← nested repo (will/infrastructure on Gitea)
      legion/
        *.tf                           ← Terraform (infra layer: namespaces, Cloudflare, R2)
        apps/*.yaml                    ← Argo CD Application manifests (app definitions)
        k8s/<service>/*.yaml           ← k8s config synced by Argo CD (PVCs, ExternalSecrets, etc.)
  neural-platform/
    neuron/                            ← github: harmonic-framework/neuron, gitea: neural-platform/neuron
  harmonic-framework/
    harmonic-framework.com/            ← github: harmonic-framework/harmonic-framework.com
  projects/
    personal/
      prism/                           ← github: harmonic-framework/prism, gitea: neural-platform/prism
    clients/
      ilih.life/                       ← github: harmonic-framework/ilih.life, gitea: will/ilih.life

---

Services & Domains

Family (nook.family — via Cloudflare tunnel)

Service	URL	Notes
AdGuard	https://dns.nook.family	DNS + ad blocking (DoH: https://dns.nook.family/dns-query)

Platform (neuralplatform.ai — via Cloudflare tunnel)

Service	URL	Notes
Argo CD	https://argocd.neuralplatform.ai	GitOps UI
Vault	https://vault.neuralplatform.ai	Secrets
Gitea	https://git.neuralplatform.ai	Git server
Ollama	https://ollama.neuralplatform.ai	LLM API
Neuron MCP	https://neuron.neuralplatform.ai	MCP server (port 8001)
Axon webhooks	https://axon.neuralplatform.ai	Webhook hub (port 3847)
npm registry	https://npm.neuralplatform.ai	Verdaccio
PyPI registry	https://pypi.neuralplatform.ai	devpi
Docker Registry	https://registry.neuralplatform.ai	Push images here
Registry UI	https://docker.neuralplatform.ai	Docker registry browser
Headscale VPN	https://vpn.neuralplatform.ai	Tailscale control plane (direct TLS, not CF-proxied)
Grafana	https://grafana.neuralplatform.ai	Metrics + logs dashboards
Prometheus	—	Metrics (kube-prometheus-stack, internal only)
Alertmanager	—	Alert routing → Slack (internal only)
Alloy	https://alloy.neuralplatform.ai	OTLP ingest for Loki/Tempo

VPN (Headscale / Tailscale)

Headscale runs at vpn.neuralplatform.ai (DNS-only, no CF proxy — required for Tailscale TS2021 WebSocket upgrades). Magic DNS base domain: ts.neuralplatform.ai. DNS resolvers: 192.168.68.77 (AdGuard) + 1.1.1.1.

NodePort services (direct to Legion IP)

Service	Port	Notes
Gitea SSH	30022	git@gitea:org/repo.git (via Host gitea SSH config)
Ollama API	31434	http://192.168.68.77:31434

Gitea SSH config (Mac ~/.ssh/config)

Host gitea
  HostName 192.168.68.77
  Port 30022
  User git
  IdentityFile ~/.ssh/id_ed25519

Use for git remotes: git@gitea:will/infrastructure.git. Works on LAN and over Headscale VPN.

---

Access Patterns

SSH

ssh legion                        # Ubuntu 24.04, user: will
ssh legion "kubectl get pods -A"  # run kubectl remotely

kubectl (local)

export KUBECONFIG=~/.kube/legion-config
kubectl get pods -A
kubectl logs -n neuron deployment/neuron

Gitea CLI (tea)

Use tea (installed on both Mac and Legion, default login: neuralplatform):

tea repo ls                      # list repos
tea pr ls --repo will/infrastructure
tea issue ls --repo neural-platform/neuron

Gitea API (direct)

CF Access blocks direct calls from Mac. Use tea or SSH to Legion:

TOKEN=$(vault kv get -field=api_token secret/gitea)
ssh legion "curl -s -H 'Authorization: token $TOKEN' http://10.43.1.53:3000/api/v1/repos/search?limit=50"

---

Key Paths

What	Where
This repo	~/Development/infrastructure/
Infrastructure (Terraform + Argo CD)	~/Development/infrastructure/servers/
Neural Platform projects	~/Development/neural-platform/
Harmonic Framework projects	~/Development/harmonic-framework/
Personal projects	~/Development/projects/personal/
Client projects	~/Development/projects/clients/
Knowledge base	~/Knowledge/
Secrets & tokens	~/Secrets/
Infrastructure env	~/Secrets/credentials/infrastructure.env
kubeconfig	~/.kube/legion-config
Agent directives	~/.claude/projects/-Users-will/memory/directives.md
Agent memory index	~/.claude/projects/-Users-will/memory/MEMORY.md

---

GitHub / Gitea

GitHub (github.com/harmonic-framework) — open-source, CI runners, public-facing

Gitea (git.neuralplatform.ai) — private/infra, four orgs:

• will — personal infra (will/infrastructure)
• neural-platform — platform projects (neural-platform/neuron, neural-platform/prism)
• harmonic-framework — org site/assets (harmonic-framework/harmonic-framework.com)
• contexthub — archived reference repos (ContextHub, shut down 2021)

Runner labels: self-hosted,linux,x64,legion

---

Namespaces (k8s)

Namespace	What
dns	AdGuard (DNS + ad-blocking, port 53)
git	Gitea
neuron	Neuron + cloudflared
ollama	Ollama (GPU inference)
ci	GitHub Actions runner
packages	Verdaccio (npm) + devpi (PyPI)
registry	Docker registry + UI
platform	Postgres, Redis
monitoring	kube-prometheus-stack (Prometheus, Grafana, Alertmanager) + Loki + Tempo + Alloy
headscale	Headscale VPN control plane
vault	HashiCorp Vault
argocd	Argo CD
cert-manager	cert-manager (Let's Encrypt via HTTP-01)
external-secrets	External Secrets Operator — syncs Vault secrets → k8s Secrets

---

Common Operations

Deploy a config change to an app

1. Edit servers/legion/apps/<app>.yaml or servers/legion/k8s/<service>/<file>.yaml
2. git commit && git push (from servers/)
3. Argo CD syncs in ~30s

Add a new secret to a pod

1. Store the secret: vault kv put secret/path field=value
2. Add/update an ExternalSecret in servers/legion/k8s/<service>/external-secrets.yaml
3. Reference secretKeyRef: name: <secret-name> in the app manifest
4. Push → ESO syncs the k8s Secret within ~60s

Add a new Helm release

Add an Argo CD Application manifest to servers/legion/apps/<name>.yaml with spec.source.chart and spec.source.helm.values. Push to deploy.

Rotate a secret

1. vault kv patch secret/path field=newvalue
2. Force ESO refresh: kubectl annotate externalsecret -n <ns> <name> force-sync=$(date +%s) --overwrite
3. Restart pod if needed: kubectl rollout restart deployment/<name> -n <ns>

---

Network

• Router: TP-Link Deco BE5000 mesh, router mode
• Subnet: 192.168.68.x
• DHCP DNS: 192.168.68.77 (AdGuard) + 1.1.1.1 fallback
• AdGuard provides *.nook.family resolution → 192.168.68.77
• Cloudflare tunnel ID: 54bc9b05-3953-47a2-9c3e-adecdcc53d51
• systemd-resolved is disabled on Legion (conflicts with AdGuard port 53)

---

What NOT To Do

• kubectl apply directly — push to Git and let Argo CD sync
• kubectl edit — edit the file in servers/legion/, push to Git
• Hardcode any secret value — store in Vault, reference via ExternalSecret
• Add kubernetes_secret to Terraform — secrets belong in ESO/Vault, not Terraform
• Commit terraform.tfstate — state is in R2, never local
• Call Gitea API from Mac directly — CF Access blocks it; use tea CLI or SSH to Legion
• Use var.X pattern for secrets in Terraform — Terraform no longer owns k8s Secrets