12 Commits

Author SHA1 Message Date
Will Anderson 283e335e0a deploy engram-server to neuron-prod: dharma storage backend
- Add engram-server Deployment, Service, PVC to neuron-prod
- engram-server is the graph memory substrate for DHARMA registry
- Add allow-engram-ingress NetworkPolicy (ingress from neuron-prod)
- ENGRAM_URL in Vault updated to http://engram-server.neuron-prod.svc.cluster.local:8742
- Image: registry.neuralplatform.ai/neuron-technologies/engram-server:latest
  compiled from foundation/engram dist/engram.c + el_runtime
2026-05-04 11:28:45 -05:00
Will Anderson f2f20c1f2c remove missing soma manifests from prod kustomization
Soma moved to GCP Cloud Run. The soma-*.yaml files were deleted but
the kustomization still referenced them, blocking all Argo CD syncs
for neuron-prod (including the new allow-daemon-ingress NetworkPolicy).
2026-05-04 11:11:18 -05:00
Will Anderson 88d7799911 deploy soma inference gateway with neuron model on ai.neurontechnologies.ai 2026-04-28 09:31:46 -05:00
Will Anderson 83b8d0768e Add Traefik IngressRoute for neuron.neurontechnologies.ai MCP endpoint 2026-04-26 03:52:24 -05:00
Will Anderson 93358505fc Harden prod: security, autoscaling, observability, BuildKit CI
Security:
- Drop ALL capabilities, enforce non-root, RuntimeDefault seccomp on
  neuron-mcp, neuron-rest, neuron-marketing pods
- Add startup probes (150s window for JVM) so liveness doesn't fire early
- Replace docker-sock hostPath with BuildKit rootless TCP endpoint
  (moby/buildkit:v0.19.0-rootless) — removes node root access from CI
- Document full ESO AppRole migration path in cluster-secret-store.yaml

Autoscaling & availability:
- HPAs on mcp (1–6), rest (1–4), marketing (2–8) at 65–70% CPU
- PodDisruptionBudgets (minAvailable: 1) on all three services
- NetworkPolicy: default-deny-all in neuron-prod, explicit allow rules
  for Traefik ingress, intra-namespace, and egress to DNS/platform/vault

Observability:
- ServiceMonitors for mcp, rest, marketing (cross-namespace enabled in
  kube-prometheus-stack with serviceMonitorSelectorNilUsesHelmValues:false)
- PrometheusRules: high error rate, high latency, crash loops, replica
  shortage, Postgres down/connections, backup failure, backup staleness

Chart version pinning:
- kube-prometheus-stack, loki, tempo, redis, alloy, postgres — all pinned
  to major-version ranges to block silent breaking upgrades

Backup hardening:
- restic:latest → restic:0.17.3 (deterministic image)
- Weekly backup-verify CronJob: restores latest snapshot and validates
  SQL dump structure (≥5 CREATE TABLE, pg_dump header check)

ArgoCD:
- neuron-prod AppProject: scopes deploys to neuron-prod + platform ns,
  blacklists ClusterRole/ClusterRoleBinding/Namespace creation,
  automated sync window 2–6am UTC, manual always allowed
2026-04-25 22:54:18 -05:00
Will Anderson a6421a5f98 feat(api): expose api.neurontechnologies.ai + add Stripe and R2 secrets
- Add api.neurontechnologies.ai DNS CNAME → CF tunnel
- Add tunnel ingress rule for api.neurontechnologies.ai → Traefik
- Create rest-ingressroute.yaml: api.neurontechnologies.ai → neuron-rest:8081
- Add STRIPE_SECRET_KEY to neuron-prod ExternalSecret (from marketing-test Vault)
2026-04-25 01:25:39 -05:00
Will Anderson 8eb88a3116 feat(runpod): add inference pod templates, nginx LB, and provisioner script
Infrastructure readiness for RunPod inference workloads:
- runpod-inference.yaml: ConfigMap with pod creation payloads for RTX 4090,
  A40 (single+dual), and custom templates
- runpod-lb-configmap.yaml: nginx least-conn load balancer for inference
  endpoint distribution (Deployment + ClusterIP Service)
- runpod-provision.sh: bash provisioner script — reads RUNPOD_API_KEY/HF_TOKEN,
  creates pods via GraphQL, polls until RUNNING, outputs endpoint URLs.
  Does NOT spin up any pods (dry-run flag available).
2026-04-25 01:20:55 -05:00
Will Anderson 1d527a9365 chore(neuron-prod): add green deployment to kustomization
Green manifest was committed but missing from kustomization.yaml so Argo CD
never created the resource. Wires it in so the blue/green slot swap can work.
2026-04-24 22:22:20 -05:00
Will Anderson 95b4669839 feat(neuron-prod): add IngressRoute for license API at /api/v1/* paths 2026-04-24 15:12:37 -05:00
Will Anderson 59aa0f0dfb deploy neuron-license service to prod
Adds license deployment, service, ingress route (/license), and
Unkey + admin token secrets via ExternalSecret. Image built and
pushed by CI on merge of neuron PR #30.
2026-04-24 01:42:07 -05:00
Will Anderson 5085840a79 Add neuron prod DB backup and tighten actuator surface
- Add hourly restic backup CronJob to neuron-prod namespace
  - Backs up /data/neuron-prod.db to Cloudflare R2 (legion-neuron-backup bucket)
  - Retention: 24 hourly, 7 daily, 4 weekly — <$0.01/month at current DB size
  - ExternalSecret pulls restic password + R2 creds from Vault secret/r2
- Actuator exposure per environment:
  - prod: health,info only / show-details=never (no internals exposed)
  - stage: health,info,metrics,loggers / show-details=always (pre-prod visibility)
  - dev: full (health,info,metrics,env,beans,loggers,mappings) / show-details=always
2026-04-23 21:43:20 -05:00
Will Anderson 1b8ea63541 feat(neuron): add stage and prod k8s manifests + Argo CD apps
Completes the three-env setup (dev was added previously). Each env has
deployments, services, ingress, PVC, ConfigMap, and ExternalSecret pulling
from Vault. Prod has larger resource limits and 10Gi PVC. Vault secret at
secret/neuron-technologies/prod seeded separately.
2026-04-17 09:43:23 -05:00