will.anderson
65dd2cf097
The architecture docs describe four things the design spec has since ruled out, and each one is a supervisor invented for something that should be a property of the substrate: grounding modelled as a subsystem rather than as the edge weight it already is; faculties modelled as parameters of a read when abduce is a write; wonder materialized as a maintained manifest when it is the boundary of the structure; and consolidation implemented eleven times behind tickers when a brain has no cron job. Left standing rather than deleted, per the repo's own supersession discipline — the trail of how the understanding matured is the point. Each stale passage is marked inline and points at a new 06 §12 that transcribes the corrections and records the measured consolidation inventory. Authority: foundation/el, branch design/correspondence-and-censorship, lang/spec/correspondence-and-censorship.md.
202 lines
12 KiB
Markdown
202 lines
12 KiB
Markdown
# Neuron — Runtime & Deployment
|
||
|
||
> Process/port topology, the end-to-end MCP request path, local vs GKE
|
||
> blue/green production, and a high-level view of secrets/config. Grounded in
|
||
> `entrypoint.sh`, `scripts/blue-green-deploy.sh`, the k8s manifests under
|
||
> `infrastructure/platform/k8s/neuron-mcp/`, and `.gitea/workflows/`.
|
||
|
||
## Process & port topology
|
||
|
||
A running neuron is **two processes in one container**: the soul and the engram,
|
||
started by `entrypoint.sh`.
|
||
|
||
```
|
||
container (one pod)
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ entrypoint.sh │
|
||
│ 1. start engram (background) ── listens :8742 │
|
||
│ 2. wait /health up to 60s │
|
||
│ 3. exec soul (PID 1 foreground) ── listens :7770 │
|
||
│ │
|
||
│ soul :7770 ──HTTP──► engram :8742 │
|
||
│ (ENGRAM_URL=http://localhost:8742, HTTP mode) │
|
||
│ │
|
||
│ /data (PVC mount) ◄── engram snapshot.json │
|
||
└──────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
- `entrypoint.sh` starts engram with `ENGRAM_BIND=:8742` and
|
||
`ENGRAM_DATA_DIR=/data`, polls `http://localhost:8742/health` (up to 60s;
|
||
Autopilot cold starts are slow), then `exec`s the soul. `SOUL_ENGRAM_PATH` is
|
||
deliberately unset so `ENGRAM_URL` triggers **HTTP mode** (soul talks to engram
|
||
over localhost HTTP, not an in-process embed).
|
||
- EL HTTP runtime is tuned down for co-located calls: `EL_HTTP_TIMEOUT_MS=10000`,
|
||
`EL_HTTP_CONNECT_TIMEOUT_MS=3000` (`entrypoint.sh`).
|
||
|
||
### Full port map
|
||
|
||
| Port | Process | Role | Source |
|
||
|---|---|---|---|
|
||
| 7779 | mcp-proxy | MCP client entry; byte-forward + retry | `mcp-proxy/src/main.el` |
|
||
| 17779 | mcp-wrapper | MCP JSON-RPC ⇄ soul REST; tool catalog | `mcp-wrapper/src/main.el` |
|
||
| 7770 | soul | HTTP cognitive API + `handle_request` | `NEURON_PORT`, `deployment-blue.yaml` |
|
||
| 8742 | engram | graph store HTTP | `entrypoint.sh`, `server.el:711` |
|
||
| 7771 | neuron-connectd | MCP connector bridges | `routes.el` `connectd_*` |
|
||
|
||
**Local vs prod, an important distinction.** The proxy → wrapper chain is the
|
||
**local developer adapter**: a stdio MCP client (Claude Code) needs to reach an
|
||
HTTP soul, so the proxy/wrapper translate and add resilience. In **production**,
|
||
the `neuron-mcp` Kubernetes Service is a ClusterIP that targets the soul's
|
||
`:7770` directly (`service.yaml`) — external access is "to be wired via
|
||
Cloudflare Tunnel later" (annotation, same file). So in prod the MCP/HTTP
|
||
boundary is the soul's own HTTP surface; the proxy/wrapper are not (yet) in the
|
||
cluster path. *(inference from the ClusterIP-only Service + the local-only
|
||
proxy/wrapper binaries.)*
|
||
|
||
## The MCP request path (end to end)
|
||
|
||
A single `tools/call` from an MCP client, local topology:
|
||
|
||
```
|
||
client proxy :7779 wrapper :17779 soul :7770 engram :8742
|
||
│ JSON-RPC │ │ │ │
|
||
│ tools/call ─────────► │ forward+retry │ │ │
|
||
│ │ ─────────────────► │ map tool→REST │ │
|
||
│ │ │ ─── HTTP POST ────► │ handle_request │
|
||
│ │ │ /api/neuron/... │ → handle_api_* │
|
||
│ │ │ │ engram_* builtin │
|
||
│ │ │ │ ── (HTTP mode) ───► │ activate/search/
|
||
│ │ │ │ │ save
|
||
│ │ │ │ ◄─── nodes/edges ── │
|
||
│ │ │ ◄── JSON result ── │ │
|
||
│ │ │ fire_activation │ │
|
||
│ │ │ /recall warm-up ─► soul (side effect) │
|
||
│ ◄──── result ──────── │ ◄───────────────── │ │ │
|
||
```
|
||
|
||
Responsibilities per hop, and the volatility each isolates (VBD reading):
|
||
|
||
1. **proxy** — transport resilience. Isolates *client connection volatility*
|
||
(drops, retries, health) from everything above. No MCP semantics.
|
||
2. **wrapper** — protocol translation. Isolates the *MCP protocol* from the soul:
|
||
owns `initialize`/`tools/list`/`tools/call`, the ~90-tool catalog, and
|
||
`dispatch_tool_call`. Also fires the `fire_activation` `/recall` side effect so
|
||
tool use warms working memory.
|
||
3. **soul** — cognition. `handle_request` dispatch → `handle_api_*` → engram
|
||
builtins. In HTTP mode it reaches engram over localhost; otherwise embedded.
|
||
4. **engram** — the graph. Spreading activation, Hebbian edges, snapshot
|
||
persistence.
|
||
|
||
For the user-facing chat pipeline (not tool calls), `/api/chat` enters
|
||
`layered_cycle` (soul.el) — L1 safety → L2 stewardship → L3 imprint — described
|
||
in `02-components.md §3c`.
|
||
|
||
## Production: GKE blue/green
|
||
|
||
Neuron prod runs on GKE cluster **`neuron-platform`** (Autopilot, us-central1),
|
||
namespace **`neuron-prod`**. Two Deployments, `neuron-mcp-blue` and
|
||
`neuron-mcp-green`, share one Service selector that names the *active slot*.
|
||
|
||
- **Deployments** (`deployment-blue.yaml` / `deployment-green.yaml`): one
|
||
container `soul`, image pinned by **digest** (not `:latest`) so Argo CD can't
|
||
drift the active slot to an untested build (see the pin comment in
|
||
`deployment-blue.yaml`). `strategy: Recreate` — the PVC is RWO so only one pod
|
||
can hold it at a time. Probes hit `/health` on `:7770`.
|
||
- **Service** (`service.yaml`): ClusterIP `neuron-mcp`, port 7770 → 7770,
|
||
`selector: {app: neuron-mcp, slot: blue}`. The blue/green script patches
|
||
`slot`.
|
||
- **Storage** (`pvc.yaml`): `neuron-engram-data`, `standard-rwo` (pd-balanced),
|
||
10Gi, RWO. Engram data is the single `snapshot.json` (~8MB active).
|
||
- **The swap** (`scripts/blue-green-deploy.sh`): (1) set image on the target
|
||
slot; (2) scale target to 1, wait for rollout; (3) **patch the Service selector
|
||
to the new slot** (traffic flip); (4) scale the old slot to 0. Imperative
|
||
`kubectl` for the live swap, then git-update the Argo manifests so a sync
|
||
doesn't revert replica counts.
|
||
- **Backup** (`backup-cronjob.yaml`): every 15 min, tar `/data` → GCS, keep 96.
|
||
> **Ticker, but not consolidation (2026-08-16).** Flagged only because `06`
|
||
> §12.4 sequences *"no tickers, no cron"* and an auditor sweeping for them will
|
||
> land here. This is **ops/backup, not cognition** — it does not consolidate and
|
||
> must not be folded into the dreamer. Local counterparts measured 2026-08-16:
|
||
> `ai.neuron.engram-backup` (`StartInterval = 3600`),
|
||
> `ai.neuron.snapshot-backup` (`StartInterval = 900`),
|
||
> `ai.neuron.act-runner-watchdog` (`StartInterval = 120`). Also measured:
|
||
> `crontab -l` contains **zero** neuron entries — every neuron schedule on this
|
||
> machine is launchd `StartInterval` / `StartCalendarInterval`, not cron.
|
||
|
||
### Resource sizing (learned the hard way)
|
||
|
||
`deployment-blue.yaml` documents the memory history in comments: idle soul RSS
|
||
~860Mi; the `beginSession` call (loads memories + backlog + preferences) spikes
|
||
past 1Gi and OOM-killed the pod mid-request (client socket closed). Current
|
||
setting: `requests = limits = 2Gi`, cpu 250m/1000m. This is *why* the cognitive
|
||
API projects/compacts payloads so aggressively (doc 02 §2, doc 01 axis 1) — the
|
||
memory ceiling is real and close.
|
||
|
||
## CI/CD
|
||
|
||
Two Gitea Actions workflows (`.gitea/workflows/`), serialized on a single GCE
|
||
runner (`concurrency: neuron-runner`).
|
||
|
||
- **`ci.yaml`** (push/PR to `main`):
|
||
- **build:** free disk → checkout → install gcc/libcurl/gcloud → download
|
||
`el-runtime-c`/`el-runtime-h` from Artifact Registry `foundation-prod`
|
||
(`elc`/`elb` intentionally **not** downloaded) → compile the committed
|
||
`dist/soul.c` directly: `cc -O2 -DHAVE_CURL dist/soul.c el_runtime.c -lssl
|
||
-lcrypto -lcurl -lpthread -lm -o dist/neuron` → `strip -s` → smoke test
|
||
`dist/neuron --help` → publish `neuron-soul@<sha8>` to AR (push only).
|
||
- **deploy** (push-to-main only): auth GCP → `get-credentials neuron-platform`
|
||
→ **determine idle slot** (the deployment at 0 replicas) → prepare artifacts
|
||
(soul binary + `elc` + runtime for the Docker build) → **clone the engram
|
||
repo** into `./engram/` (Dockerfile builds engram from source) → `docker
|
||
build`+push `neuron-soul:<sha>` → `scripts/blue-green-deploy.sh --image
|
||
--slot` → git-push updated infra manifests → `kubectl rollout status` →
|
||
verify `neuron-mcp` endpoints.
|
||
- **`deploy-gke.yaml`** (`workflow_dispatch` only, slot default `green`) — manual
|
||
rollback / forced-slot deploy without a rebuild; same auth → slot → docker →
|
||
blue-green → manifest-sync → verify steps.
|
||
|
||
The Docker image (`Dockerfile`) is a two-stage build: stage 1 compiles
|
||
`engram/src/server.el` → `engram.c` → `engram` binary via `elc` + `cc`; stage 2
|
||
is an Ubuntu 24.04 runtime (GLIBC 2.39 satisfies both binaries) with `soul` +
|
||
`engram` + `entrypoint.sh`.
|
||
|
||
## Config & secrets (high level)
|
||
|
||
Runtime configuration is injected as environment, sourced from a Kubernetes
|
||
Secret `neuron-soul-secrets` via ExternalSecret (ESO → GCP Secret Manager,
|
||
Workload Identity — no key files). From `deployment-blue.yaml`:
|
||
|
||
| Env | Meaning |
|
||
|---|---|
|
||
| `NEURON_PORT` | soul HTTP port (7770) |
|
||
| `NEURON_LLM_0_URL` / `_KEY` / `_FORMAT` | primary LLM endpoint (Anthropic format) |
|
||
| `SOUL_CGI_ID` / `SOUL_IDENTITY` | CGI id + identity seed (→ `seed_persona_from_env`, `soul.el:250`) |
|
||
| `NEURON_TOKEN` | auth token *(present in env; note the HTTP dispatch does not currently check it — doc 01 Divergence 3)* |
|
||
| `NEURON_API_URL` | self-callback URL (`http://neuron-mcp.neuron-prod.svc.cluster.local:7770`) |
|
||
| `ENGRAM_URL` / `ENGRAM_DATA_DIR` | `http://localhost:8742` / `/data` |
|
||
|
||
There is also an in-graph config surface: `ConfigEntry` nodes read/written by
|
||
`inspect_config` / `tune_config` (`neuron-api.el:616-653`) — runtime-tunable
|
||
persona/behavior keys stored *in* the engram rather than the environment.
|
||
|
||
> **Operational note to flag.** The `deployment-blue.yaml` image pin comment
|
||
> (dated Jul 2026) records that `:latest` resolved to an untested build lacking a
|
||
> `mem_save`/genesis-SIGSEGV fix, which is why the active slot is pinned to a
|
||
> digest. Any promotion must (a) rebuild a good soul and (b) update the digest in
|
||
> git so Argo CD and `blue-green-deploy.sh` agree. *(state as-of the manifests
|
||
> read; verify current slot before deploying.)*
|
||
|
||
## Performance & retrieval cost (MEASURED, 2026-08-14; ANN index PLANNED)
|
||
|
||
Measured envelope of a live mind, and where the time goes:
|
||
|
||
- **Working footprint:** a live mind is **~1 GB** resident.
|
||
- **Retrieval is the bottleneck.** Retrieval today does **brute-force cosine over
|
||
all nodes** — **~330 ms at ~13k nodes** — and that scan dominates request
|
||
latency (the geometric-retrieval path of `03` §Retrieval / `06` §2.5 improved
|
||
*quality*, not the scan cost).
|
||
- **Planned fix — an HNSW approximate-nearest-neighbour index** (backlog
|
||
`d3d0d644`): turns the linear scan into ≈`O(D·log N)`, so a **100× larger graph
|
||
costs ≈1.5×** rather than ≈100×. **PLANNED, not built** — brute-force is the
|
||
live behavior; do not present the ANN speedup as shipped.
|