# Agent System Reference

Quick reference for AI agents working in Will's infrastructure. Read this before doing any infrastructure work.

---

## Machines

| Machine | Role | IP | SSH |
|---------|------|----|-----|
| Local Mac | Dev workstation | 192.168.68.55 | — |
| Legion | Ubuntu 24.04, k3s server | 192.168.68.77 | `ssh legion` |

Legion runs all production services: k3s, Postgres, Redis, Ollama (GPU), Docker Registry, GitHub Actions runner, Vault, Gitea, AdGuard, Neuron.

---

## Secrets — Always Use Vault

**Vault is the source of truth for all credentials.** Never hardcode secrets. Never ask the user for a secret value.

### CLI access (on local Mac)

```bash
# Load all secrets into env
set -a; source ~/Secrets/credentials/infrastructure.env; set +a

# Or with direnv (auto-loads in legion/ directory):
cd ~/Development/infrastructure/servers/legion/
# direnv auto-sources infrastructure.env

# Friendly alias (after sourcing infrastructure.env):
secret civitai          # → civitai API key
secret slack-bot-token  # → Slack bot token
secret cf-key           # → Cloudflare API key

# Direct vault lookup:
vault kv get -field=api_key secret/ai          # CivitAI
vault kv get -field=bot_token secret/slack      # Slack
vault kv get -field=api_key secret/cloudflare   # Cloudflare
```

Full path+field table is in `~/Secrets/credentials/infrastructure.env` — look at the `_v()` calls.

### Vault connection

```bash
VAULT_ADDR=https://vault.neuralplatform.ai
VAULT_TOKEN=$(cat ~/Secrets/tokens/vault-root-token)
```

If Vault is unreachable (CF tunnel down), port-forward:
```bash
ssh legion -L 8200:localhost:8200  # in background
export VAULT_ADDR=http://localhost:8200
```

### How secrets flow to pods

```
Vault (source of truth) → ExternalSecret (in git) → ESO operator → k8s Secret → pod env
```

All k8s Secrets are managed by External Secrets Operator (ESO) pulling from Vault.
No secrets are stored in Terraform. Do not add `kubernetes_secret` resources to Terraform.

---

## Infrastructure Management

### The split: Terraform vs Argo CD

| Layer | Tool | What it owns |
|-------|------|-------------|
| Infrastructure | **Terraform** | Namespaces, Cloudflare DNS/tunnels, R2 storage, Vault bootstrap |
| Configuration | **Argo CD** | All k8s resources: Deployments, Services, ConfigMaps, Ingresses, PVCs, ExternalSecrets, Helm releases |
| Secrets | **ESO → Vault** | All k8s Secrets — ExternalSecret manifests in git, values pulled from Vault at runtime |

**Never modify k8s resources directly with kubectl.** Always go through Terraform or Argo CD (Git push).

**Secrets flow:**
```
Vault (source of truth) → ExternalSecret (in git) → ESO operator → k8s Secret → pod env
```

Adding a new secret:
1. `vault kv put secret/<path> field=value`
2. Add ExternalSecret manifest to `k8s/<service>/` referencing that path
3. Push → Argo CD applies ExternalSecret → ESO syncs Secret within ~60s

### Running Terraform

```bash
cd ~/Development/infrastructure/servers/legion/
terraform plan    # uses direnv to load credentials
terraform apply
```

State is stored in Cloudflare R2 bucket `legion-terraform-state`.

### Argo CD

Root app `legion-apps` watches `servers/legion/apps/` in the `will/infrastructure` Gitea repo.
Push to main → Argo CD syncs within ~30 seconds.

- App manifests: `~/Development/infrastructure/servers/legion/apps/*.yaml`
- k8s config: `~/Development/infrastructure/servers/legion/k8s/<service>/*.yaml`
- Argo CD UI: https://argocd.neuralplatform.ai
- Gitea CLI: `tea pr ls`, `tea issue ls` (configured, default login: neuralplatform)

### Repo layout

```
~/Development/
  infrastructure/                      ← this repo (local only, no GitHub remote)
    AGENTS.md                          ← you are here
    servers/                           ← nested repo (will/infrastructure on Gitea)
      legion/
        *.tf                           ← Terraform (infra layer: namespaces, Cloudflare, R2)
        apps/*.yaml                    ← Argo CD Application manifests (app definitions)
        k8s/<service>/*.yaml           ← k8s config synced by Argo CD (PVCs, ExternalSecrets, etc.)
  neural-platform/
    neuron/                            ← github: harmonic-framework/neuron, gitea: neural-platform/neuron
  harmonic-framework/
    harmonic-framework.com/            ← github: harmonic-framework/harmonic-framework.com
  projects/
    personal/
      prism/                           ← github: harmonic-framework/prism, gitea: neural-platform/prism
    clients/
      ilih.life/                       ← github: harmonic-framework/ilih.life, gitea: will/ilih.life
```

---

## Services & Domains

### Family (nook.family — via Cloudflare tunnel)

| Service | URL | Notes |
|---------|-----|-------|
| AdGuard | https://dns.nook.family | DNS + ad blocking (DoH: https://dns.nook.family/dns-query) |

### Platform (neuralplatform.ai — via Cloudflare tunnel)

| Service | URL | Notes |
|---------|-----|-------|
| Argo CD | https://argocd.neuralplatform.ai | GitOps UI |
| Vault | https://vault.neuralplatform.ai | Secrets |
| Gitea | https://git.neuralplatform.ai | Git server |
| Ollama | https://ollama.neuralplatform.ai | LLM API |
| Neuron MCP | https://neuron.neuralplatform.ai | MCP server (port 8001) |
| Axon webhooks | https://axon.neuralplatform.ai | Webhook hub (port 3847) |
| npm registry | https://npm.neuralplatform.ai | Verdaccio |
| PyPI registry | https://pypi.neuralplatform.ai | devpi |
| Docker Registry | https://registry.neuralplatform.ai | Push images here |
| Registry UI | https://docker.neuralplatform.ai | Docker registry browser |
| Headscale VPN | https://vpn.neuralplatform.ai | Tailscale control plane (direct TLS, not CF-proxied) |
| Grafana | https://grafana.neuralplatform.ai | Metrics + logs dashboards |
| Prometheus | — | Metrics (kube-prometheus-stack, internal only) |
| Alertmanager | — | Alert routing → Slack (internal only) |
| Alloy | https://alloy.neuralplatform.ai | OTLP ingest for Loki/Tempo |

### VPN (Headscale / Tailscale)

Headscale runs at `vpn.neuralplatform.ai` (DNS-only, no CF proxy — required for Tailscale TS2021 WebSocket upgrades). Magic DNS base domain: `ts.neuralplatform.ai`. DNS resolvers: `192.168.68.77` (AdGuard) + `1.1.1.1`.

### NodePort services (direct to Legion IP)

| Service | Port | Notes |
|---------|------|-------|
| Gitea SSH | 30022 | `git@gitea:org/repo.git` (via `Host gitea` SSH config) |
| Ollama API | 31434 | `http://192.168.68.77:31434` |

### Gitea SSH config (Mac `~/.ssh/config`)

```
Host gitea
  HostName 192.168.68.77
  Port 30022
  User git
  IdentityFile ~/.ssh/id_ed25519
```

Use for git remotes: `git@gitea:will/infrastructure.git`. Works on LAN and over Headscale VPN.

---

## Access Patterns

### SSH

```bash
ssh legion                        # Ubuntu 24.04, user: will
ssh legion "kubectl get pods -A"  # run kubectl remotely
```

### kubectl (local)

```bash
export KUBECONFIG=~/.kube/legion-config
kubectl get pods -A
kubectl logs -n neuron deployment/neuron
```

### Gitea CLI (tea)

Use `tea` (installed on both Mac and Legion, default login: `neuralplatform`):

```bash
tea repo ls                      # list repos
tea pr ls --repo will/infrastructure
tea issue ls --repo neural-platform/neuron
```

### Gitea API (direct)

CF Access blocks direct calls from Mac. Use `tea` or SSH to Legion:

```bash
TOKEN=$(vault kv get -field=api_token secret/gitea)
ssh legion "curl -s -H 'Authorization: token $TOKEN' http://10.43.1.53:3000/api/v1/repos/search?limit=50"
```

---

## Key Paths

| What | Where |
|------|-------|
| This repo | `~/Development/infrastructure/` |
| Infrastructure (Terraform + Argo CD) | `~/Development/infrastructure/servers/` |
| Neural Platform projects | `~/Development/neural-platform/` |
| Harmonic Framework projects | `~/Development/harmonic-framework/` |
| Personal projects | `~/Development/projects/personal/` |
| Client projects | `~/Development/projects/clients/` |
| Knowledge base | `~/Knowledge/` |
| Secrets & tokens | `~/Secrets/` |
| Infrastructure env | `~/Secrets/credentials/infrastructure.env` |
| kubeconfig | `~/.kube/legion-config` |
| Agent directives | `~/.claude/projects/-Users-will/memory/directives.md` |
| Agent memory index | `~/.claude/projects/-Users-will/memory/MEMORY.md` |

---

## GitHub / Gitea

**GitHub** (`github.com/harmonic-framework`) — open-source, CI runners, public-facing

**Gitea** (`git.neuralplatform.ai`) — private/infra, four orgs:
- `will` — personal infra (`will/infrastructure`)
- `neural-platform` — platform projects (`neural-platform/neuron`, `neural-platform/prism`)
- `harmonic-framework` — org site/assets (`harmonic-framework/harmonic-framework.com`)
- `contexthub` — archived reference repos (ContextHub, shut down 2021)

Runner labels: `self-hosted,linux,x64,legion`

---

## Namespaces (k8s)

| Namespace | What |
|-----------|------|
| `dns` | AdGuard (DNS + ad-blocking, port 53) |
| `git` | Gitea |
| `neuron` | Neuron + cloudflared |
| `ollama` | Ollama (GPU inference) |
| `ci` | GitHub Actions runner |
| `packages` | Verdaccio (npm) + devpi (PyPI) |
| `registry` | Docker registry + UI |
| `platform` | Postgres, Redis |
| `monitoring` | kube-prometheus-stack (Prometheus, Grafana, Alertmanager) + Loki + Tempo + Alloy |
| `headscale` | Headscale VPN control plane |
| `vault` | HashiCorp Vault |
| `argocd` | Argo CD |
| `cert-manager` | cert-manager (Let's Encrypt via HTTP-01) |
| `external-secrets` | External Secrets Operator — syncs Vault secrets → k8s Secrets |

---

## Common Operations

### Deploy a config change to an app

1. Edit `servers/legion/apps/<app>.yaml` or `servers/legion/k8s/<service>/<file>.yaml`
2. `git commit && git push` (from `servers/`)
3. Argo CD syncs in ~30s

### Add a new secret to a pod

1. Store the secret: `vault kv put secret/path field=value`
2. Add/update an ExternalSecret in `servers/legion/k8s/<service>/external-secrets.yaml`
3. Reference `secretKeyRef: name: <secret-name>` in the app manifest
4. Push → ESO syncs the k8s Secret within ~60s

### Add a new Helm release

Add an Argo CD Application manifest to `servers/legion/apps/<name>.yaml` with `spec.source.chart` and `spec.source.helm.values`. Push to deploy.

### Rotate a secret

1. `vault kv patch secret/path field=newvalue`
2. Force ESO refresh: `kubectl annotate externalsecret -n <ns> <name> force-sync=$(date +%s) --overwrite`
3. Restart pod if needed: `kubectl rollout restart deployment/<name> -n <ns>`

---

## Network

- Router: TP-Link Deco BE5000 mesh, router mode
- Subnet: `192.168.68.x`
- DHCP DNS: `192.168.68.77` (AdGuard) + `1.1.1.1` fallback
- AdGuard provides `*.nook.family` resolution → `192.168.68.77`
- Cloudflare tunnel ID: `54bc9b05-3953-47a2-9c3e-adecdcc53d51`
- systemd-resolved is **disabled** on Legion (conflicts with AdGuard port 53)

---

## What NOT To Do

- `kubectl apply` directly — push to Git and let Argo CD sync
- `kubectl edit` — edit the file in `servers/legion/`, push to Git
- Hardcode any secret value — store in Vault, reference via ExternalSecret
- Add `kubernetes_secret` to Terraform — secrets belong in ESO/Vault, not Terraform
- Commit `terraform.tfstate` — state is in R2, never local
- Call Gitea API from Mac directly — CF Access blocks it; use `tea` CLI or SSH to Legion
- Use `var.X` pattern for secrets in Terraform — Terraform no longer owns k8s Secrets