neuron/docs/architecture/04-runtime-and-deployment.md

Neuron — Runtime & Deployment

Process/port topology, the end-to-end MCP request path, local vs GKE blue/green production, and a high-level view of secrets/config. Grounded in entrypoint.sh, scripts/blue-green-deploy.sh, the k8s manifests under infrastructure/platform/k8s/neuron-mcp/, and .gitea/workflows/.

Process & port topology

A running neuron is two processes in one container: the soul and the engram, started by entrypoint.sh.

 container (one pod)
 ┌──────────────────────────────────────────────────────────┐
 │  entrypoint.sh                                             │
 │    1. start engram  (background)   ── listens :8742       │
 │    2. wait /health up to 60s                              │
 │    3. exec soul (PID 1 foreground)  ── listens :7770      │
 │                                                           │
 │   soul  :7770  ──HTTP──►  engram  :8742                    │
 │   (ENGRAM_URL=http://localhost:8742, HTTP mode)           │
 │                                                           │
 │   /data  (PVC mount) ◄── engram snapshot.json             │
 └──────────────────────────────────────────────────────────┘

• entrypoint.sh starts engram with ENGRAM_BIND=:8742 and ENGRAM_DATA_DIR=/data, polls http://localhost:8742/health (up to 60s; Autopilot cold starts are slow), then execs the soul. SOUL_ENGRAM_PATH is deliberately unset so ENGRAM_URL triggers HTTP mode (soul talks to engram over localhost HTTP, not an in-process embed).
• EL HTTP runtime is tuned down for co-located calls: EL_HTTP_TIMEOUT_MS=10000, EL_HTTP_CONNECT_TIMEOUT_MS=3000 (entrypoint.sh).

Full port map

Port	Process	Role	Source
7779	mcp-proxy	MCP client entry; byte-forward + retry	mcp-proxy/src/main.el
17779	mcp-wrapper	MCP JSON-RPC ⇄ soul REST; tool catalog	mcp-wrapper/src/main.el
7770	soul	HTTP cognitive API + handle_request	NEURON_PORT, deployment-blue.yaml
8742	engram	graph store HTTP	entrypoint.sh, server.el:711
7771	neuron-connectd	MCP connector bridges	routes.el connectd_*

Local vs prod, an important distinction. The proxy → wrapper chain is the local developer adapter: a stdio MCP client (Claude Code) needs to reach an HTTP soul, so the proxy/wrapper translate and add resilience. In production, the neuron-mcp Kubernetes Service is a ClusterIP that targets the soul's :7770 directly (service.yaml) — external access is "to be wired via Cloudflare Tunnel later" (annotation, same file). So in prod the MCP/HTTP boundary is the soul's own HTTP surface; the proxy/wrapper are not (yet) in the cluster path. (inference from the ClusterIP-only Service + the local-only proxy/wrapper binaries.)

The MCP request path (end to end)

A single tools/call from an MCP client, local topology:

 client                proxy :7779         wrapper :17779        soul :7770            engram :8742
   │  JSON-RPC             │                    │                    │                     │
   │ tools/call ─────────► │  forward+retry     │                    │                     │
   │                       │ ─────────────────► │  map tool→REST     │                     │
   │                       │                    │ ─── HTTP POST ────► │  handle_request     │
   │                       │                    │   /api/neuron/...   │   → handle_api_*    │
   │                       │                    │                    │  engram_* builtin   │
   │                       │                    │                    │ ── (HTTP mode) ───► │  activate/search/
   │                       │                    │                    │                     │  save
   │                       │                    │                    │ ◄─── nodes/edges ── │
   │                       │                    │ ◄── JSON result ── │                     │
   │                       │                    │  fire_activation    │                     │
   │                       │                    │  /recall warm-up ─► soul (side effect)    │
   │ ◄──── result ──────── │ ◄───────────────── │                    │                     │

Responsibilities per hop, and the volatility each isolates (VBD reading):

1. proxy — transport resilience. Isolates client connection volatility (drops, retries, health) from everything above. No MCP semantics.
2. wrapper — protocol translation. Isolates the MCP protocol from the soul: owns initialize/tools/list/tools/call, the ~90-tool catalog, and dispatch_tool_call. Also fires the fire_activation /recall side effect so tool use warms working memory.
3. soul — cognition. handle_request dispatch → handle_api_* → engram builtins. In HTTP mode it reaches engram over localhost; otherwise embedded.
4. engram — the graph. Spreading activation, Hebbian edges, snapshot persistence.

For the user-facing chat pipeline (not tool calls), /api/chat enters layered_cycle (soul.el) — L1 safety → L2 stewardship → L3 imprint — described in 02-components.md §3c.

Production: GKE blue/green

Neuron prod runs on GKE cluster neuron-platform (Autopilot, us-central1), namespace neuron-prod. Two Deployments, neuron-mcp-blue and neuron-mcp-green, share one Service selector that names the active slot.

• Deployments (deployment-blue.yaml / deployment-green.yaml): one container soul, image pinned by digest (not :latest) so Argo CD can't drift the active slot to an untested build (see the pin comment in deployment-blue.yaml). strategy: Recreate — the PVC is RWO so only one pod can hold it at a time. Probes hit /health on :7770.
• Service (service.yaml): ClusterIP neuron-mcp, port 7770 → 7770, selector: {app: neuron-mcp, slot: blue}. The blue/green script patches slot.
• Storage (pvc.yaml): neuron-engram-data, standard-rwo (pd-balanced), 10Gi, RWO. Engram data is the single snapshot.json (~8MB active).
• The swap (scripts/blue-green-deploy.sh): (1) set image on the target slot; (2) scale target to 1, wait for rollout; (3) patch the Service selector to the new slot (traffic flip); (4) scale the old slot to 0. Imperative kubectl for the live swap, then git-update the Argo manifests so a sync doesn't revert replica counts.
• Backup (backup-cronjob.yaml): every 15 min, tar /data → GCS, keep 96.

Resource sizing (learned the hard way)

deployment-blue.yaml documents the memory history in comments: idle soul RSS ~860Mi; the beginSession call (loads memories + backlog + preferences) spikes past 1Gi and OOM-killed the pod mid-request (client socket closed). Current setting: requests = limits = 2Gi, cpu 250m/1000m. This is why the cognitive API projects/compacts payloads so aggressively (doc 02 §2, doc 01 axis 1) — the memory ceiling is real and close.

CI/CD

Two Gitea Actions workflows (.gitea/workflows/), serialized on a single GCE runner (concurrency: neuron-runner).

• ci.yaml (push/PR to main):
  • build: free disk → checkout → install gcc/libcurl/gcloud → download el-runtime-c/el-runtime-h from Artifact Registry foundation-prod (elc/elb intentionally not downloaded) → compile the committed dist/soul.c directly: cc -O2 -DHAVE_CURL dist/soul.c el_runtime.c -lssl -lcrypto -lcurl -lpthread -lm -o dist/neuron → strip -s → smoke test dist/neuron --help → publish neuron-soul@<sha8> to AR (push only).
  • deploy (push-to-main only): auth GCP → get-credentials neuron-platform → determine idle slot (the deployment at 0 replicas) → prepare artifacts (soul binary + elc + runtime for the Docker build) → clone the engram repo into ./engram/ (Dockerfile builds engram from source) → docker build+push neuron-soul:<sha> → scripts/blue-green-deploy.sh --image --slot → git-push updated infra manifests → kubectl rollout status → verify neuron-mcp endpoints.
• deploy-gke.yaml (workflow_dispatch only, slot default green) — manual rollback / forced-slot deploy without a rebuild; same auth → slot → docker → blue-green → manifest-sync → verify steps.

The Docker image (Dockerfile) is a two-stage build: stage 1 compiles engram/src/server.el → engram.c → engram binary via elc + cc; stage 2 is an Ubuntu 24.04 runtime (GLIBC 2.39 satisfies both binaries) with soul + engram + entrypoint.sh.

Config & secrets (high level)

Runtime configuration is injected as environment, sourced from a Kubernetes Secret neuron-soul-secrets via ExternalSecret (ESO → GCP Secret Manager, Workload Identity — no key files). From deployment-blue.yaml:

Env	Meaning
NEURON_PORT	soul HTTP port (7770)
NEURON_LLM_0_URL / _KEY / _FORMAT	primary LLM endpoint (Anthropic format)
SOUL_CGI_ID / SOUL_IDENTITY	CGI id + identity seed (→ seed_persona_from_env, soul.el:250)
NEURON_TOKEN	auth token (present in env; note the HTTP dispatch does not currently check it — doc 01 Divergence 3)
NEURON_API_URL	self-callback URL (http://neuron-mcp.neuron-prod.svc.cluster.local:7770)
ENGRAM_URL / ENGRAM_DATA_DIR	http://localhost:8742 / /data

There is also an in-graph config surface: ConfigEntry nodes read/written by inspect_config / tune_config (neuron-api.el:616-653) — runtime-tunable persona/behavior keys stored in the engram rather than the environment.

Operational note to flag. The deployment-blue.yaml image pin comment (dated Jul 2026) records that :latest resolved to an untested build lacking a mem_save/genesis-SIGSEGV fix, which is why the active slot is pinned to a digest. Any promotion must (a) rebuild a good soul and (b) update the digest in git so Argo CD and blue-green-deploy.sh agree. (state as-of the manifests read; verify current slot before deploying.)

Performance & retrieval cost (MEASURED, 2026-08-14; ANN index PLANNED)

Measured envelope of a live mind, and where the time goes:

• Working footprint: a live mind is ~1 GB resident.
• Retrieval is the bottleneck. Retrieval today does brute-force cosine over all nodes — ~330 ms at ~13k nodes — and that scan dominates request latency (the geometric-retrieval path of 03 §Retrieval / 06 §2.5 improved quality, not the scan cost).
• Planned fix — an HNSW approximate-nearest-neighbour index (backlog d3d0d644): turns the linear scan into ≈O(D·log N), so a 100× larger graph costs ≈1.5× rather than ≈100×. PLANNED, not built — brute-force is the live behavior; do not present the ANN speedup as shipped.