702faca691
Mount neuron-marketplace.db from PVC via subPath at /app/neuron-marketplace.db
...
App hard-codes relative path 'neuron-marketplace.db' from working dir /app.
Mount the PVC file at /app/neuron-marketplace.db via subPath so the app can
write to it as UID 1000. PVC persists the file across pod restarts.
Also keep NEURON_MARKETPLACE_DB_PATH in ConfigMap for future app versions.
2026-04-26 02:03:09 -05:00
a6b0dc9211
Force pod restart to pick up NEURON_MARKETPLACE_DB_PATH configmap change
...
Add config-hash annotation to template to cycle the ReplicaSet and
ensure pods restart with the updated ConfigMap env var.
2026-04-26 01:52:19 -05:00
ddaab28638
Fix marketplace DB: revert workingDir, add NEURON_MARKETPLACE_DB_PATH
...
workingDir: /data broke the JVM (app.jar is a relative path in /app).
Add NEURON_MARKETPLACE_DB_PATH=/data/neuron-marketplace.db env var
following the same pattern as NEURON_DB_PATH for the core DB.
2026-04-26 01:49:48 -05:00
26ed92ca38
Set workingDir: /data so relative DB paths resolve to PVC
...
neuron-marketplace.db was opening relative to /app (image root,
not writable by UID 1000). Set workingDir to /data so all relative
file opens land on the writable PVC volume.
2026-04-26 01:47:02 -05:00
754cd72df5
Fix SQLite READONLY: chown /data to UID 1000 via initContainer
...
PVC was written as root before runAsUser: 1000 was added; SQLite
refused writes with SQLITE_READONLY. initContainer runs as root to
chown the volume, then the app runs as UID 1000 as required.
2026-04-26 01:43:37 -05:00
01245a6278
Fix Neuron MCP: runAsUser + move to neuron.neurontechnologies.ai
...
Pod was crash-looping: image has no USER directive so kubelet rejected it
with runAsNonRoot. Add runAsUser: 1000 at both pod and container level.
MCP paths moved from neurontechnologies.ai (now a GCP Cloud Run A record,
unreachable on Legion) to private subdomain neuron.neurontechnologies.ai.
Cloudflare tunnel and DNS CNAME updated out-of-band.
2026-04-26 01:35:20 -05:00
93358505fc
Harden prod: security, autoscaling, observability, BuildKit CI
...
Security:
- Drop ALL capabilities, enforce non-root, RuntimeDefault seccomp on
neuron-mcp, neuron-rest, neuron-marketing pods
- Add startup probes (150s window for JVM) so liveness doesn't fire early
- Replace docker-sock hostPath with BuildKit rootless TCP endpoint
(moby/buildkit:v0.19.0-rootless) — removes node root access from CI
- Document full ESO AppRole migration path in cluster-secret-store.yaml
Autoscaling & availability:
- HPAs on mcp (1–6), rest (1–4), marketing (2–8) at 65–70% CPU
- PodDisruptionBudgets (minAvailable: 1) on all three services
- NetworkPolicy: default-deny-all in neuron-prod, explicit allow rules
for Traefik ingress, intra-namespace, and egress to DNS/platform/vault
Observability:
- ServiceMonitors for mcp, rest, marketing (cross-namespace enabled in
kube-prometheus-stack with serviceMonitorSelectorNilUsesHelmValues:false)
- PrometheusRules: high error rate, high latency, crash loops, replica
shortage, Postgres down/connections, backup failure, backup staleness
Chart version pinning:
- kube-prometheus-stack, loki, tempo, redis, alloy, postgres — all pinned
to major-version ranges to block silent breaking upgrades
Backup hardening:
- restic:latest → restic:0.17.3 (deterministic image)
- Weekly backup-verify CronJob: restores latest snapshot and validates
SQL dump structure (≥5 CREATE TABLE, pg_dump header check)
ArgoCD:
- neuron-prod AppProject: scopes deploys to neuron-prod + platform ns,
blacklists ClusterRole/ClusterRoleBinding/Namespace creation,
automated sync window 2–6am UTC, manual always allowed
2026-04-25 22:54:18 -05:00
df85598df2
ci(neuron-prod): blue-green flip to blue@v0.15.3 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.15.3)
2026-04-25 20:06:23 +00:00
35e346d94c
ci(neuron-prod): blue-green flip to green@v0.15.2 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.15.2)
2026-04-25 19:53:50 +00:00
9b057411db
ci(neuron-prod): blue-green flip to blue@v0.15.1 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.15.1)
2026-04-25 19:39:29 +00:00
8b56eb5290
ci(neuron-prod): blue-green flip to green@v0.15.0 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.15.0)
2026-04-25 19:20:35 +00:00
6b782f04e9
ci(neuron-prod): blue-green flip to blue@v0.14.12 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.12)
2026-04-25 10:18:28 +00:00
acf9f07dbd
ci(neuron-prod): blue-green flip to green@v0.14.11 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.11)
2026-04-25 10:08:12 +00:00
663abd5188
ci(neuron-prod): blue-green flip to blue@v0.14.10 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.10)
2026-04-25 10:02:20 +00:00
fed551185e
ci(neuron-prod): blue-green flip to green@v0.14.9 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.9)
2026-04-25 09:47:46 +00:00
8d02a1164b
ci(neuron-prod): blue-green flip to blue@v0.14.8 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.8)
2026-04-25 09:14:22 +00:00
302f37ad29
ci(neuron-prod): blue-green flip to green@v0.14.7 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.7)
2026-04-25 09:08:17 +00:00
12da58e716
ci(neuron-prod): blue-green flip to blue@v0.14.6 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.6)
2026-04-25 09:03:32 +00:00
244ba40c20
ci(neuron-prod): blue-green flip to green@v0.14.5 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.5)
2026-04-25 08:50:35 +00:00
46fd91b1fe
ops(neuron-prod): rollback blue to v0.14.3 — v0.14.4 crashes on empty updated_at (v0.14.5 fix in build)
2026-04-25 03:47:17 -05:00
47bcef9bc2
ops(neuron-prod): restore blue service while v0.14.5 builds (green crashed on empty updated_at)
2026-04-25 03:45:09 -05:00
3185cb5062
ci(neuron-prod): deploy v0.14.4 to blue (DB schema migrations + session persistence fixes)
2026-04-25 03:38:57 -05:00
d0b8a812f0
ci(neuron-prod): blue-green flip to green@v0.14.4 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.4)
2026-04-25 08:36:18 +00:00
e201fa7dbb
ops(neuron-prod): restart blue pod to recover new MCP session after compact
2026-04-25 03:23:09 -05:00
d8919f869b
ci(neuron-prod): blue-green flip to blue@v0.14.3 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.3)
2026-04-25 08:20:19 +00:00
7931d54e40
ci(neuron-prod): blue-green flip to green@v0.14.2 (registry.neuralplatform.ai/neuron-technologies/neuron-mcp:v0.14.2)
2026-04-25 07:59:17 +00:00
f831da7f17
ci(neuron-prod): deploy neuron@v0.13.1
2026-04-25 06:44:48 +00:00
11e9962431
ci(neuron-prod): deploy neuron@v0.12.2
2026-04-25 06:25:49 +00:00
c76eb09aa5
ci(neuron-prod): deploy neuron@v0.12.1
2026-04-25 06:11:44 +00:00
5ff525e1c8
ci(neuron-prod): deploy neuron@v0.12.0
2026-04-25 05:50:26 +00:00
ec66233644
ci(neuron-prod): deploy neuron@v0.11.1
2026-04-25 05:33:25 +00:00
abf3453d8d
ci(neuron-prod): deploy neuron@v0.10.0
2026-04-25 04:53:36 +00:00
1a9dd13b0a
Fix neuron-mcp OTLP endpoint: alloy → alloy-otlp service
...
The `alloy` ClusterIP service only exposes port 12345 (UI/metrics).
OTLP HTTP ingestion on port 4318 is on the separate `alloy-otlp` service.
Pointing OTEL_EXPORTER_OTLP_ENDPOINT at `alloy:4318` caused continuous
SocketTimeoutException in neuron-mcp. Updated both blue and green deployments.
2026-04-24 23:36:48 -05:00
c9ffa55863
ci(neuron-prod): deploy neuron@v0.9.1
2026-04-25 03:35:33 +00:00
67de9ac463
ci(neuron-prod): deploy neuron@v0.9.0
2026-04-25 03:17:56 +00:00
f86ee04467
fix(neuron-mcp): blue/green deployment, memory limits, preStop drain hook
...
- Rename Deployment to neuron-mcp-blue with slot=blue label
- Add neuron-mcp-green Deployment scaffold (replicas=0, ready to receive new image)
- Service selector now routes by slot label; annotation tracks active slot
- Memory: 768Mi → 1536Mi limit, 256Mi → 512Mi request (fixes OOM pod restarts / 502s)
- Add JAVA_TOOL_OPTIONS: container-aware GC + ExitOnOutOfMemoryError
- Add preStop sleep 8s so Traefik drains in-flight connections before pod terminates
- terminationGracePeriodSeconds: 30
- Add scripts/blue-green-deploy.sh: automated slot-swap with health check and rollback
2026-04-24 22:08:12 -05:00
b411f2a7b0
ci(neuron-prod): deploy neuron@v0.8.3
2026-04-25 01:35:26 +00:00
719c66ea82
ci(neuron-prod): deploy neuron@v0.8.2
2026-04-25 01:13:52 +00:00
6160e8e6b8
ci(neuron-prod): deploy neuron@v0.8.1
2026-04-25 00:57:54 +00:00
af8a178b8c
ci(neuron-prod): deploy neuron@v0.8.0
2026-04-25 00:36:02 +00:00
cc489e4047
ci(neuron-prod): deploy neuron@v0.7.7
2026-04-25 00:22:52 +00:00
0e590df537
fix(neuron-prod): reduce mcp memory request to unblock scheduling
2026-04-24 18:27:18 -05:00
851e64a561
ci(neuron-prod): deploy neuron@v0.7.6
2026-04-24 23:22:38 +00:00
cf6f390756
ci(neuron-prod): deploy neuron@v0.7.5
2026-04-24 22:26:03 +00:00
2befcc088a
ci(neuron-prod): deploy neuron@v0.7.4
2026-04-24 22:07:54 +00:00
b36de8190e
ci(neuron-prod): deploy neuron@v0.7.3
2026-04-24 21:59:07 +00:00
060995e762
ci(neuron-prod): deploy neuron@v0.7.2
2026-04-24 21:26:01 +00:00
2a0011c479
ci(neuron-prod): deploy neuron@v0.7.1
2026-04-24 20:57:42 +00:00
fe17d020ef
fix(k8s): use Recreate strategy for neuron-mcp deployments
...
Single-node k3s cluster cannot schedule a second pod during rolling update
(512Mi request × 2 exceeds available memory). Recreate terminates the old
pod before starting the new one, trading brief downtime for schedulability.
2026-04-24 15:19:10 -05:00
02db9c3d43
feat(observability): wire OpenTelemetry into MCP backend and marketing site
...
- Add OTEL_EXPORTER_OTLP_ENDPOINT env var to prod and stage MCP deployments pointing at Alloy in-cluster
- Add Neuron Platform Overview Grafana dashboard (request rate, latency percentiles, error rate, memory nodes, HTTP status distribution)
- Allow grafana/dashboards/*.json through .gitignore (was blocked by secrets rule)
2026-04-24 14:21:34 -05:00
Will Anderson