456 Commits

Author SHA1 Message Date
Will Anderson 086a7b0bf0 add DocuSeal to GKE — manifests, Terraform, Argo CD app; update ci-base Dockerfile 2026-05-05 10:38:45 -05:00
Will Anderson 4dc687f2ae Fix runner extra_hosts: inject gitea-proxy ClusterIP for build containers
Build containers run with network: host, so k8s DNS doesn't resolve.
Add extra_hosts entry pointing gitea-proxy.ci.svc.cluster.local to its
ClusterIP (10.43.88.7) in both the will-org and neuron-technologies runner
configs. Without this, actions/checkout fails because Gitea passes the
proxy hostname as GITHUB_SERVER_URL and the build container can't reach it.

Also update stale CF Access comments that no longer reflect the architecture.
2026-05-05 05:41:59 -05:00
Will Anderson c94721b70b Remove SSH insteadOf rewrite from git-ssh-init.sh
Gitea's SSH server is disabled (START_SSH_SERVER=false) so the
SSH URL rewrite breaks git HTTPS clones in build containers. HTTPS
git ops work directly from Legion host network — Cloudflare bypasses
CF Access for smart-HTTP and release asset paths.
2026-05-05 05:27:44 -05:00
Will Anderson a301302138 Route gitea-proxy directly to GKE LB IP, drop CF Access
CF Access was blocking /api/actions/* even with the correct service-token
headers — the gRPC Actions ping endpoint returns 403 through Cloudflare.
Direct connection to the GKE Network LB IP (34.31.145.131) bypasses CF
entirely. Verified /api/actions/ping returns 200 from the LB IP directly.
2026-05-05 05:12:48 -05:00
Will Anderson 158689dd0e Roll runner pods to pick up updated GITEA_INSTANCE_URL 2026-05-05 04:45:38 -05:00
Will Anderson b5c7125c13 Roll gitea-proxy pod for SNI fix 2026-05-05 04:42:08 -05:00
Will Anderson 464b33a176 Fix nginx SNI for Cloudflare upstream TLS
Without proxy_ssl_server_name on, nginx connects to Cloudflare's IP
without SNI, triggering handshake_failure (alert 40). Cloudflare requires
SNI to route TLS to the correct origin.
2026-05-05 04:41:46 -05:00
Will Anderson 939e66bfbb Route runner daemon through nginx CF Access proxy
act_runner cannot inject custom HTTP headers, so CF Access blocks its
unauthenticated calls to git.neuralplatform.ai. Add a gitea-proxy
deployment (nginx:alpine) in the ci namespace that injects the CF
Access service-token headers and proxies to https://git.neuralplatform.ai.

Both runner secrets now point GITEA_INSTANCE_URL at the in-cluster proxy
(http://gitea-proxy.ci.svc.cluster.local:3000). Build containers still
clone via SSH through git-ssh-init.sh — unaffected.
2026-05-05 04:17:52 -05:00
Will Anderson ae3257525e wire soul to web_demo_key for Anthropic API isolation 2026-05-05 04:09:41 -05:00
Will Anderson c2900400a4 ci: switch runner git clones from HTTPS+CF Access to SSH deploy key 2026-05-05 04:07:35 -05:00
Will Anderson 3f08c6dc01 fix(runner): bump config-version to trigger rollout with new Gitea URL 2026-05-05 03:48:15 -05:00
Will Anderson 01507ad8ae fix(runner): point GITEA_INSTANCE_URL at GKE Gitea public URL
Gitea moved to GKE. The old in-cluster URL
http://gitea.git.svc.cluster.local:3000 now has 0 pods behind it.
The runner daemon gets connection refused on every poll.

Switch both ExternalSecrets to https://git.neuralplatform.ai.
The public URL does not require CF Access tokens — the runner can
connect without them. CF_ACCESS_CLIENT_ID/SECRET remain for git
operations inside CI job containers (via BASH_ENV init script).
2026-05-05 03:39:17 -05:00
Will Anderson 479190ef47 migrate all Argo CD repo sources from Legion Gitea to GKE Gitea 2026-05-05 02:25:14 -05:00
Will Anderson 010d81d6d9 fix(runner): correct Gitea ClusterIP and enable force_pull
extra_hosts had stale IP 10.43.1.53 (Gitea is now at 10.43.15.98).
force_pull: true ensures runners always use the latest ci-base image.
Applies to both gitea-runner and neuron-technologies-runner.
2026-05-04 23:32:09 -05:00
Will Anderson 4ef5e99f31 fix(ci-base): correct Gitea namespace in git insteadOf redirect
Gitea lives in the 'git' namespace on Legion k3s, not 'gitea'.
The system gitconfig redirect must use gitea.git.svc.cluster.local:3000.
Update when Gitea migrates to GKE (namespace will be 'gitea' there).
2026-05-04 20:42:27 -05:00
Will Anderson 9330107fcc migrate Vault and Gitea to GKE Autopilot cluster
Implements Option A: move Vault (3x GCE e2-small) and Gitea (Legion k8s)
onto a new GKE Autopilot cluster (neuron-platform, us-central1) managed
through Legion Argo CD.

Terraform (servers/gcp/):
- gke.tf: GKE Autopilot cluster, Workload Identity bindings for Vault (KMS)
  and Gitea (Cloud SQL)
- cloud-sql.tf: gitea database + user on neuron-prod-pg15, gitea GCP SA,
  gitea-database-url and gitea-db-password Secret Manager secrets
- vault-nodes.tf: PENDING DECOMMISSION comment with migration checklist

k8s manifests (servers/gcp/k8s/):
- vault/: namespace.yaml
- gitea/: namespace, serviceaccount (Workload Identity annotation), pvc (50Gi
  standard-rwo), deployment (Gitea + Cloud SQL Auth Proxy sidecar), service,
  configmap (custom CSS carried from Legion), external-secrets (GCP SM provider)
- argocd-apps/: vault-gke.yaml, vault-helm-gke.yaml (Helm chart, HA Raft 3
  replicas, GCP KMS auto-unseal, topologySpread across zones, 10Gi premium-rwo),
  gitea-gke.yaml — all target GKE_CLUSTER_ENDPOINT placeholder

Legion (servers/legion/):
- apps/gke-apps.yaml: App-of-Apps entry point on Legion Argo CD that syncs the
  GKE Application manifests
- k8s/gitea-runner/Dockerfile: add system-level git insteadOf so GKE CI runners
  resolve Gitea in-cluster without Cloudflare Access headers
2026-05-04 20:40:48 -05:00
will.anderson 0006380c27 route runner build container clones via public URL with CF Access (#7) 2026-05-04 21:37:53 +00:00
will.anderson 23fc64e7b7 fix(neuron-prod): set neuron-marketing-hpa minReplicas to 1 (#8) 2026-05-04 21:21:16 +00:00
will.anderson a64860064b fix(neuron-prod): add allow-dharma-ingress NetworkPolicy (#6) 2026-05-04 21:16:01 +00:00
will.anderson cbb564ccf5 revert(ci): runner public URL — CF Access blocks registration (#4) 2026-05-04 21:05:29 +00:00
will.anderson be0508037a fix(dharma): drop letsencrypt certResolver from IngressRoute 2026-05-04 20:56:29 +00:00
will.anderson 6f5d041440 fix(ci): point Gitea Actions runners at public instance URL 2026-05-04 20:56:26 +00:00
Will Anderson 4754c69e01 fix: mount /var/run/docker.sock (Docker Engine) not containerd socket 2026-05-04 14:10:25 -05:00
Will Anderson 0444d265b9 fix: gitea runner docker_host path mismatch — use /var/run/docker.sock
The containerd socket is mounted from the host at /run/k3s/containerd/containerd.sock
to /var/run/docker.sock inside the container, but act_runner config was pointing to
the host path directly. Fixes CrashLoopBackOff on all runner pods.

Also codifies both deployments (will org + neuron-technologies org) as git-managed
manifests so Argo CD owns them going forward.
2026-05-04 13:44:46 -05:00
Will Anderson 84688aec9d restore Always pull policy for engram-server: registry image now works
The registry image now compiles with -rdynamic/-ldl so dlsym can find
handle_request at runtime. IfNotPresent workaround no longer needed.
2026-05-04 11:43:52 -05:00
Will Anderson 10c3af4afe fix engram-server: use IfNotPresent pull policy to use local k3s cache
The buildah-compiled image (imported directly into k3s) is the working
version. The registry image will catch up after the next CI build.
IfNotPresent ensures the locally-cached correct binary is used.
2026-05-04 11:42:10 -05:00
Will Anderson 283e335e0a deploy engram-server to neuron-prod: dharma storage backend
- Add engram-server Deployment, Service, PVC to neuron-prod
- engram-server is the graph memory substrate for DHARMA registry
- Add allow-engram-ingress NetworkPolicy (ingress from neuron-prod)
- ENGRAM_URL in Vault updated to http://engram-server.neuron-prod.svc.cluster.local:8742
- Image: registry.neuralplatform.ai/neuron-technologies/engram-server:latest
  compiled from foundation/engram dist/engram.c + el_runtime
2026-05-04 11:28:45 -05:00
Will Anderson f2f20c1f2c remove missing soma manifests from prod kustomization
Soma moved to GCP Cloud Run. The soma-*.yaml files were deleted but
the kustomization still referenced them, blocking all Argo CD syncs
for neuron-prod (including the new allow-daemon-ingress NetworkPolicy).
2026-05-04 11:11:18 -05:00
Will Anderson 44faf74b61 fix CI runner secrets: correct Gitea URL and use Vault-backed tokens
The ExternalSecret had a stale Gitea URL (192.168.8.148:30322 — old
local IP) and runner tokens were never populated in Vault. Update the
template URL to the cluster-internal Gitea service DNS, and store the
registration tokens in Vault at secret/gitea.runner_token.
2026-05-04 11:07:43 -05:00
Will Anderson 2d655e0966 fix backup-verify path and suspend redpanda-topics-init
backup-verify: restic restores /dump/all-databases.sql relative to
restore target, so the check path was wrong (/all-databases.sql →
/dump/all-databases.sql). Also fix the --include filter to match.

redpanda-topics-init: redpanda is intentionally scaled to 0; the init
job will block forever waiting for a broker. Mark it suspended so Argo
CD won't keep creating a stuck job. Re-enable when redpanda is turned on.
2026-05-04 10:54:31 -05:00
Will Anderson c7da8ca727 fix dharma startup: allow neuron-daemon ingress from neuron-prod namespace
The default-deny-all NetworkPolicy was blocking dharma from reaching
neuron-daemon port 7750 (event queue / engram endpoint). Add an explicit
ingress policy for neuron-daemon that allows traffic from the neuron-prod
namespace, so dharma and other internal services can connect at startup.
2026-05-04 10:49:55 -05:00
Will Anderson d0a6ca753c expose engram port 7750 on neuron-daemon service 2026-05-03 13:32:35 -05:00
Will Anderson b70fef611a deploy DHARMA registry to neuron-prod
- Dockerfile for Go DHARMA server (registry.neuralplatform.ai/neuron-technologies/dharma)
- k8s manifests: deployment, service, ExternalSecret (Vault), IngressRoute
- Argo CD app: deploys to neuron-prod namespace
- Cloudflare tunnel rule for dharma.neurontechnologies.ai
- DNS CNAME record for dharma.neurontechnologies.ai → legion tunnel
- Vault secrets already populated (api_key, encryption_key, jwt_secret, engram_key, engram_url)
- Connects to neuron-daemon (engram) at neuron-daemon.neuron-prod.svc.cluster.local:7749
2026-05-03 13:26:54 -05:00
Will Anderson 73004fa287 neuron-web: drop unused registry-pull-secret
The registry currently allows anonymous cluster-internal pulls so the
imagePullSecrets reference was failing the externalsecret reconcile
(secret/data/mudcraft has no registry creds for this app). Pod was
pulling fine anyway. If the registry later gets locked down, add a
proper externalsecret using the path from secret/data/<app> instead of
borrowing mudcraft's.
2026-05-03 10:49:24 -05:00
Will Anderson 3daf615dd2 add neuron-web phase-1 deploy on Legion
Stand up the elb-built native El landing server on Legion k3s alongside
the existing Cloud Run prod marketing site. Image is built on
gitea-runner-1 from elb-emitted .c files + native cc and pushed to
registry.neuralplatform.ai/neuron-web:dev-dfe41234. Deployment exposes
web-stage.neuralplatform.ai via Traefik + Cloudflare tunnel.

The point is to have a working local instance we can iterate against
while we redesign the CI/CD pipeline; nothing here replaces the Cloud
Run deploy yet.

* k8s/neuron-technologies/web/ — namespace, deployment, service, ingress,
  registry-pull-secret + externalsecret pulling anthropic/supabase keys
  from Vault
* apps/neuron-web.yaml — Argo CD Application
* dns-neuralplatform.tf, main.tf — CNAME + tunnel ingress for
  web-stage.neuralplatform.ai
2026-05-03 10:45:18 -05:00
Will Anderson 961dad90c0 Set neuron-marketing HPA minReplicas to 0 — site down 2026-04-28 18:57:09 -05:00
Will Anderson a823b0c1d3 Scale neuron-marketing to 0 — taking site down 2026-04-28 18:55:39 -05:00
Will Anderson e69ecd2240 Scale neuron-marketing to 1 replica for demo launch 2026-04-28 18:52:21 -05:00
Will Anderson 4943ccfc74 Add NEURON_API_KEY to neuron-dev secrets and fix OTLP endpoint
- ExternalSecret: pull neuron_api_key from Vault into NEURON_API_KEY
- ConfigMap: set OTEL_EXPORTER_OTLP_ENDPOINT to alloy-otlp in-cluster service
  instead of localhost (fixes 196 restart OTel connection errors)
2026-04-28 15:17:16 -05:00
Will Anderson 1a8e039ecf Rename inference endpoint to neuron.neurontechnologies.ai, add wildcard DNS for customer orgs 2026-04-28 11:15:11 -05:00
Will Anderson f2b025a433 Deploy soma to GCP Cloud Run at ai.neurontechnologies.ai
- Add cloud-run-soma.tf: soma-prod-us Cloud Run service in us-central1,
  neuron-soma-sa service account, soma Artifact Registry repo, Secret
  Manager secrets for HF token and operator key, serverless NEG, backend
  service, SSL cert
- Add dns-gcp.tf: Cloudflare A record for ai.neurontechnologies.ai pointing
  to GCP LB IP; Cloudflare provider added to main.tf/variables.tf
- Update load-balancer.tf: soma host rule + path matcher, soma SSL cert
  added to HTTPS proxy
- Update outputs.tf: soma service URL and artifact registry URL outputs
- Remove legion soma k8s manifests (Legion is gone)
- Update AGENTS.md to reflect GCP as primary production environment
2026-04-28 10:15:15 -05:00
Will Anderson 65dd9884ae Fix soma model ID: NeuronTechnologiesAI/Neuron (correct HF org casing) 2026-04-28 09:35:37 -05:00
Will Anderson 88d7799911 deploy soma inference gateway with neuron model on ai.neurontechnologies.ai 2026-04-28 09:31:46 -05:00
Will Anderson 60f4e0693e Remove adguard — home DNS server, no purpose on GCP 2026-04-27 18:27:21 -05:00
Will Anderson fb5b93f9d7 fix broken pods in GCP k3s cluster
- neuron-dev: update neuron-rest and neuron-mcp from dev-d7a587e2 to v0.18.14
- neuron-stage: update neuron-rest/mcp from stage-0f5b2328 to v0.18.14; marketing from :latest to 1e94e8ae
- swarm-alpha/beta/gamma: update neuron-rest and neuron-mcp from v0.15.3 to v0.18.14
- ci runners: fix docker socket path from /var/run/docker.sock to /run/k3s/containerd/containerd.sock (k3s node has no Docker daemon)
2026-04-27 18:17:32 -05:00
Will Anderson aee77bbfc0 Bump secret-hash annotation to force MCP pod restart
The NEURON_API_KEY secret was updated in Vault to the real ntn_ key;
bumping this annotation triggers a rolling restart so pods pick up
the new value.
2026-04-27 18:05:47 -05:00
Will Anderson 409a04ce80 Route neurontechnologies.ai/mcp to neuron-mcp:8080
Expose the MCP endpoint directly at the root domain so Bearer token
clients can reach it without going through the OAuth subdomain.
The Spring Boot MCP layer already accepts ntn_* API keys via
ApiKeyAuthenticationFilter; this ingress rule enables access via
https://neurontechnologies.ai/mcp.
2026-04-27 18:05:04 -05:00
Will Anderson 20a4a2c37f Take marketing site offline until launch 2026-04-27 17:58:33 -05:00
Will Anderson dda07ae269 Route /oauth2, /connect, /.well-known at neurontechnologies.ai to MCP 2026-04-27 17:51:59 -05:00
Will Anderson 03495962e0 fix(daemon): use neuron-rest cluster service instead of local JVM (--jar was circular proxy) 2026-04-27 17:26:58 -05:00