601 Commits

Author SHA1 Message Date
Neuron CI bd7d5efa66 ci(neuron-prod): blue-green flip to green@v0.18.8 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.18.8) 2026-04-26 16:49:02 +00:00
Neuron CI 726510d7d9 ci(daemon): deploy main neuron-daemon@5dc2d833 2026-04-26 16:34:07 +00:00
Will Anderson fdbbfdb0ad iam: grant ci-pusher serviceAccountUser on all Cloud Run runtime SAs 2026-04-26 11:18:50 -05:00
will.anderson 66bffc5a4b Merge pull request 'fix(rest): run as UID 1000 to match PVC file ownership' (#35) from feat/daemon-image-c190e6e5 into main 2026-04-26 16:07:02 +00:00
Will Anderson c6245d1170 fix(rest): run as UID 1000 to match PVC file ownership — SQLite needs write access 2026-04-26 11:06:23 -05:00
will.anderson f056806cbd Merge pull request 'fix(rest): Recreate strategy to avoid SQLite write lock on rollout' (#34) from feat/daemon-image-c190e6e5 into main 2026-04-26 16:05:24 +00:00
Will Anderson 58513835e2 fix(rest): use Recreate strategy to avoid SQLite write lock on rollout 2026-04-26 11:04:16 -05:00
Neuron CI f525437c77 ci(daemon): deploy main neuron-daemon@b2bed846 2026-04-26 16:03:29 +00:00
Will Anderson e22dce6b5a iam: grant ci-pusher run.developer for Cloud Run deploys 2026-04-26 10:53:21 -05:00
Will Anderson baf147543c ci(daemon): deploy main neuron-daemon@c190e6e5 2026-04-26 10:37:29 -05:00
Will Anderson 714a7c7a8d launch: deploy daemon StatefulSet, fix neuron-rest security context, add daemon Argo CD app
- Add neuron-daemon Argo CD Application (apps/neuron-daemon.yaml) pointing to
  k8s/neuron-technologies/daemon/ on main — daemon will deploy once image is built
- Add daemon k8s manifests: StatefulSet (2 replicas, imagePullPolicy: Always),
  Service (ClusterIP :7749, session affinity), ExternalSecret (Anthropic API key from Vault)
- Fix neuron-rest runAsNonRoot: remove constraint since v0.15.3 image runs as root,
  clears the CreateContainerConfigError on the 11h-old stale pod
- Add NEURON_DAEMON_WS_URL env var to marketing deployment for chat UI connectivity
- Add neuron.neurontechnologies.ai Cloudflare DNS record and tunnel ingress rule (Terraform)
2026-04-26 10:13:48 -05:00
Will Anderson 655ab38558 Deploy neuron-mcp:a837369 — VBD marketplace architecture 2026-04-26 08:51:07 -05:00
Will Anderson 6041ac3b25 Deploy neuron-mcp:514db58 — STATELESS, built from current main 2026-04-26 08:33:03 -05:00
Will Anderson 623a6828e8 Deploy neuron-mcp:05d978f — STATELESS from main branch 2026-04-26 08:24:55 -05:00
Will Anderson d804d71040 Remove sticky session from MCP IngressRoute — STATELESS is sessionless 2026-04-26 08:21:16 -05:00
Will Anderson 5f1c9adaa6 Deploy neuron-mcp:1b26e3e — STATELESS HTTP transport
Removes STREAMABLE session state. No sticky sessions needed.
Works across all pods, survives restarts, scales cleanly.
2026-04-26 08:18:07 -05:00
Will Anderson 08ef13c40d Add sticky session affinity to MCP IngressRoute
STREAMABLE transport holds session state in-memory per-pod. With HPA
scaling to 2+ replicas and round-robin routing, requests hit different
pods and fail with "Session not found". Pin each client to one pod via
a secure HttpOnly cookie.
2026-04-26 08:07:23 -05:00
Will Anderson 20397e9831 Restore neuron-mcp:97bd19ba (rollback to v0.15.3 was incompatible) 2026-04-26 04:28:19 -05:00
Will Anderson 5fa8cf45d5 Rollback neuron-mcp to v0.15.3 to restore MCP connectivity 2026-04-26 04:20:57 -05:00
Will Anderson 83b8d0768e Add Traefik IngressRoute for neuron.neurontechnologies.ai MCP endpoint 2026-04-26 03:52:24 -05:00
Will Anderson 6ac4faabc5 Deploy neuron-mcp:97bd19ba — Postgres marketplace datasource 2026-04-26 03:27:46 -05:00
Will Anderson 2550786965 Revert CI to host Docker socket, remove buildkitd
BuildKit rootless failed on k3s (mount propagation), privileged mode
fixed that but buildkitd gRPC is incompatible with the Docker REST API
that forgejo-runner needs to manage job containers. Net security change
vs. the original socket approach was zero (privileged ≈ socket access).

Remove buildkitd.yaml entirely. Restore docker-sock hostPath mount on
both runners. Builds work again.
2026-04-26 02:50:27 -05:00
Will Anderson e18d953ff9 Wire marketplace Postgres datasource for neuron-mcp
Add MARKETPLACE_DB_URL/USER to ConfigMap and MARKETPLACE_DB_PASSWORD
ExternalSecret (sourced from secret/legion-db in Vault). Remove the
SQLite subPath volume mount and fix-data-ownership initContainer from
the blue deployment — marketplace storage is now in Postgres.
2026-04-26 02:33:26 -05:00
Will Anderson 952e781550 Pre-create neuron-marketplace.db in initContainer for subPath mount
subPath mounts require the source file to exist on the PVC before the
main container starts. Add 'touch /data/neuron-marketplace.db' to the
initContainer so the subPath mount can bind the file at /app/neuron-marketplace.db.
2026-04-26 02:06:02 -05:00
Will Anderson 702faca691 Mount neuron-marketplace.db from PVC via subPath at /app/neuron-marketplace.db
App hard-codes relative path 'neuron-marketplace.db' from working dir /app.
Mount the PVC file at /app/neuron-marketplace.db via subPath so the app can
write to it as UID 1000. PVC persists the file across pod restarts.
Also keep NEURON_MARKETPLACE_DB_PATH in ConfigMap for future app versions.
2026-04-26 02:03:09 -05:00
Will Anderson a6b0dc9211 Force pod restart to pick up NEURON_MARKETPLACE_DB_PATH configmap change
Add config-hash annotation to template to cycle the ReplicaSet and
ensure pods restart with the updated ConfigMap env var.
2026-04-26 01:52:19 -05:00
Will Anderson ddaab28638 Fix marketplace DB: revert workingDir, add NEURON_MARKETPLACE_DB_PATH
workingDir: /data broke the JVM (app.jar is a relative path in /app).
Add NEURON_MARKETPLACE_DB_PATH=/data/neuron-marketplace.db env var
following the same pattern as NEURON_DB_PATH for the core DB.
2026-04-26 01:49:48 -05:00
Will Anderson 26ed92ca38 Set workingDir: /data so relative DB paths resolve to PVC
neuron-marketplace.db was opening relative to /app (image root,
not writable by UID 1000). Set workingDir to /data so all relative
file opens land on the writable PVC volume.
2026-04-26 01:47:02 -05:00
Will Anderson 754cd72df5 Fix SQLite READONLY: chown /data to UID 1000 via initContainer
PVC was written as root before runAsUser: 1000 was added; SQLite
refused writes with SQLITE_READONLY. initContainer runs as root to
chown the volume, then the app runs as UID 1000 as required.
2026-04-26 01:43:37 -05:00
Will Anderson 01245a6278 Fix Neuron MCP: runAsUser + move to neuron.neurontechnologies.ai
Pod was crash-looping: image has no USER directive so kubelet rejected it
with runAsNonRoot. Add runAsUser: 1000 at both pod and container level.

MCP paths moved from neurontechnologies.ai (now a GCP Cloud Run A record,
unreachable on Legion) to private subdomain neuron.neurontechnologies.ai.
Cloudflare tunnel and DNS CNAME updated out-of-band.
2026-04-26 01:35:20 -05:00
Will Anderson ffac2161bc Use TCP readiness probe for BuildKit — exec fails on k3s privileged containers 2026-04-26 01:23:52 -05:00
Will Anderson 6e67f62ffe Switch BuildKit to privileged mode — rootless fails on k3s mount propagation 2026-04-26 01:20:45 -05:00
Will Anderson e138c45d51 Fix BuildKit rootless: allow privilege escalation for newuidmap/newgidmap 2026-04-26 01:17:41 -05:00
Will Anderson 5b16aebdcb Remove stage environment from GCP — staging is local only 2026-04-26 01:09:44 -05:00
Will Anderson 74dd054a36 feat(swarm): add swarm self-improvement loop infrastructure
Adds complete k8s manifests, ArgoCD app, and Terraform namespace
resources for the Neuron swarm self-improvement loop system.

Each variant (alpha, beta, gamma) gets its own isolated namespace,
PVC, MCP/REST deployments, ExternalSecrets from Vault, RBAC for CI,
and a SQLite clone Job template for session startup.
2026-04-26 00:58:43 -05:00
Will Anderson 53f2ad3c16 Fix Cloud Run probe ports and LB timeout_sec
- accounts: add ACCOUNTS_PORT=8080 env var (service defaults to 7753)
- api: add SERVER_PORT=8080 env var, change probe path to /actuator/health
- LB backends: remove timeout_sec (not supported for serverless NEG backends)
2026-04-26 00:28:33 -05:00
Will Anderson 36cfd3738d Fix Cloud Run: remove reserved PORT env, increase startup probe tolerance 2026-04-25 23:55:33 -05:00
Will Anderson 7adab317d4 fix: restore postgres chart to 18.x (was wrongly pinned to 16.x)
Chart 16.x deploys PG17 images that do not exist on Docker Hub.
Existing data directory is PG18. Restoring >=18.0.0,<19.0.0 range.
2026-04-26 04:51:07 +00:00
Will Anderson bb583e3ccb Fix HCL syntax errors in accounts and api Cloud Run definitions 2026-04-25 22:54:18 -05:00
Will Anderson d4c65d5857 Expand GCP infra: accounts + API services, Cloud SQL, Artifact Registry
Architecture: intelligence stays on Legion; only compiled artifacts cross
to GCP. Source code and Neuron's knowledge base never leave the system.

Artifact Registry:
- neuron-marketing, neuron-accounts, neuron-api repos in us-central1
- Keep-last-10 cleanup policy; ci-pusher SA with writer access
- Legion CI runners authenticate via GCP_SA_KEY Gitea secret

Cloud SQL (cloud-sql.tf):
- postgres-15 on db-g1-small, us-central1 (scale up to REGIONAL HA at 1k users)
- Point-in-time recovery, 14-day backup retention
- Accounts DB + user; password generated and stored in Secret Manager
- JWT signing key in Secret Manager (shared by accounts + api)
- Cloud Run connects via built-in Auth Proxy (Unix socket volume mount)

Accounts Cloud Run (cloud-run-accounts.tf):
- 3 regions (us-central1, europe-west1, asia-northeast1), min:1 max:50
- Cloud SQL proxy volume mount; secrets via Secret Manager
- Stripe + JWT env vars; health probe on /health

API Cloud Run (cloud-run-api.tf):
- 3 regions, min:1 max:100, cpu_idle=false (always-hot)
- Validates JWTs from accounts service; no direct DB connection
- License admin token from Secret Manager

Load balancer (host-based routing):
- Same global anycast IP for all three services
- URL map routes by Host: neurontechnologies.ai→marketing,
  api.neurontechnologies.ai→api, accounts.neurontechnologies.ai→accounts
- New managed SSL certs for api.* and accounts.* added to HTTPS proxy
- Cloud Armor (WAF + rate limit) applied to all backends

Service accounts + IAM:
- neuron-accounts-sa: secretmanager.secretAccessor + cloudsql.client
- neuron-api-sa: secretmanager.secretAccessor
- allUsers invoker on all prod Cloud Run services (LB health checks)

bootstrap.sh:
- One-shot setup: pulls Stripe secrets from Vault → Secret Manager,
  creates CI SA JSON key, prints DNS + next-step instructions
2026-04-25 22:54:18 -05:00
Will Anderson 93358505fc Harden prod: security, autoscaling, observability, BuildKit CI
Security:
- Drop ALL capabilities, enforce non-root, RuntimeDefault seccomp on
  neuron-mcp, neuron-rest, neuron-marketing pods
- Add startup probes (150s window for JVM) so liveness doesn't fire early
- Replace docker-sock hostPath with BuildKit rootless TCP endpoint
  (moby/buildkit:v0.19.0-rootless) — removes node root access from CI
- Document full ESO AppRole migration path in cluster-secret-store.yaml

Autoscaling & availability:
- HPAs on mcp (1–6), rest (1–4), marketing (2–8) at 65–70% CPU
- PodDisruptionBudgets (minAvailable: 1) on all three services
- NetworkPolicy: default-deny-all in neuron-prod, explicit allow rules
  for Traefik ingress, intra-namespace, and egress to DNS/platform/vault

Observability:
- ServiceMonitors for mcp, rest, marketing (cross-namespace enabled in
  kube-prometheus-stack with serviceMonitorSelectorNilUsesHelmValues:false)
- PrometheusRules: high error rate, high latency, crash loops, replica
  shortage, Postgres down/connections, backup failure, backup staleness

Chart version pinning:
- kube-prometheus-stack, loki, tempo, redis, alloy, postgres — all pinned
  to major-version ranges to block silent breaking upgrades

Backup hardening:
- restic:latest → restic:0.17.3 (deterministic image)
- Weekly backup-verify CronJob: restores latest snapshot and validates
  SQL dump structure (≥5 CREATE TABLE, pg_dump header check)

ArgoCD:
- neuron-prod AppProject: scopes deploys to neuron-prod + platform ns,
  blacklists ClusterRole/ClusterRoleBinding/Namespace creation,
  automated sync window 2–6am UTC, manual always allowed
2026-04-25 22:54:18 -05:00
Will Anderson 8fd3d12907 simplify neuron self-improve loop to blue/green + stage
Replace the aspirational alpha/beta/gamma model with the actual
deployment topology: prod runs blue/green in neuron-prod namespace,
stage is the single experiment slot in neuron-stage namespace.

The old script referenced neuron-alpha/beta/gamma deployments that
never existed. The new script uses blue-green-deploy.sh for prod
promotion and kubectl set image for stage experiments.

Loop: snapshot → deploy stage → evaluate → promote via blue/green.
2026-04-25 22:54:18 -05:00
Will Anderson 7eeff54a11 Wire GCS backup to neuron-db-backup-prod bucket (neuron-494301)
Bucket created, SA key stored in Vault at secret/gcs.
CronJob ExternalSecret updated to pull from secret/gcs.
Hourly restic backup now runs to both R2 and GCS.
2026-04-25 17:51:57 -05:00
Will Anderson 8d97bbd802 Merge branch 'main' of https://git.neuralplatform.ai/will/infrastructure 2026-04-25 17:50:32 -05:00
Will Anderson 2d0ce77518 Scale Docuseal up to 1 replica 2026-04-25 15:59:03 -05:00
Will Anderson a37deca724 Add GCS backup bucket + dual-destination hourly backup (R2 + GCS)
Provision Google Cloud Storage bucket for neuron prod DB backups via Terraform.
Create dedicated backup service account with objectAdmin on the bucket.
Update neuron-prod backup CronJob to run restic against both R2 and GCS hourly —
R2 as primary, GCS as secondary, independent credentials and repositories.
2026-04-25 15:23:51 -05:00
Neuron CI 8de866a8b9 ci(neuron-prod): update rest+license to v0.15.3 2026-04-25 20:06:27 +00:00
Neuron CI df85598df2 ci(neuron-prod): blue-green flip to blue@v0.15.3 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.15.3) 2026-04-25 20:06:23 +00:00
Neuron CI 950f00586f ci(neuron-prod): update rest+license to v0.15.2 2026-04-25 19:53:53 +00:00
Neuron CI 35e346d94c ci(neuron-prod): blue-green flip to green@v0.15.2 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.15.2) 2026-04-25 19:53:50 +00:00