will.anderson
4bff40fa4a
High-fanout identity anchors (voice, writing-imprint, self-root) have ~670KB
neighborhoods. inspect_graph returned the full traversal, which overflowed the
MCP client's context and socket-closed the wrapper mid self-load -- the soul
could not traverse its own identity graph.
handle_api_inspect_graph gains an opt-in `compact` projection (compact=1|true):
the neighborhood is relevance-ranked, the top K (default 12) keep a UTF-8-safe
content snippet (default snip=600), and the remainder collapse to lightweight
{id,label,node_type,tier,edge,pointer:true} stubs. This bounds the voice node
from 669,799B -> 25,353B (HTTP 200, valid JSON) and the wrapper's soul-load no
longer socket-closes. New helpers: api_compact_neighbors, api_neigh_full,
api_neigh_pointer, api_neigh_rank, api_neigh_better, api_float_or.
The flag is gated: ABSENT it, the response is byte-identical to the old plain
traversal, so the studio app (which never sends it) is unaffected. The MCP
wrapper (mcp-wrapper/src/main.el) appends &compact=1 on its inspectGraph and
fetch-by-id paths.
dist/soul.c is REGENERATED so CI ships the fix: CI compiles the committed
single-TU dist/soul.c directly (running elb/elc on the Linux runner OOM-kills
it), so an .el-only change would build the OLD behavior. Regenerated and verified
on macOS -- compiles with the CI cc line (0 errors) and, on a throwaway soul over
a copy of the live snapshot, serves compact ~25KB / non-compact ~670KB. The regen
also syncs the amalgamation to this branch's .el sources, which had drifted
several self-review commits ahead of the previously-committed soul.c.
Docs: docs/architecture/00-05 added; 01/02/05 corrected so the relevance-ranked
inspect_graph projection reads as committed source, not an in-flight concern.
279 lines
16 KiB
Markdown
279 lines
16 KiB
Markdown
# Neuron — Component Detail
|
||
|
||
> Per-subsystem detail: routing/dispatch, the cognitive API, the memory &
|
||
> activation engine, and the MCP transport chain. For the *why* behind these
|
||
> boundaries read `01-vbd-decomposition.md` first; this doc is the *what* and
|
||
> *how*, grounded in file citations.
|
||
|
||
---
|
||
|
||
## 1. Routing / dispatch — `routes.el`
|
||
|
||
**Responsibility:** turn an inbound HTTP request into a handler call. One
|
||
function does it.
|
||
|
||
- **Entry point:** `handle_request(method, path, body) -> String`
|
||
(`routes.el:358-753`). Structure: branch by method (`GET` `:384`, `POST`
|
||
`:549`, `DELETE` `:726`, `PATCH` `:739`), then an ordered sequence of exact
|
||
(`str_eq`) and prefix (`str_starts_with`) tests against the cleaned path.
|
||
First match wins. There is **no route table and no `register-route`** — this is
|
||
a deliberate hand-written dispatcher.
|
||
- **Path params** are extracted manually with `str_slice`/`str_index_of`
|
||
(session id `:539-541`, typed-node type `:508-513`).
|
||
- **Query strings** stripped up front by `strip_query` (`:77-83`); the raw path
|
||
(with query) is still passed to handlers that read params.
|
||
- **Pre-dispatch middleware** (cross-cutting, inline): an activity timestamp
|
||
(`state_set("soul.last_activity_ts", …)` `:367`) and **rate limiting**
|
||
(`rate_limit_check(ip, path)` `:38-75`, `:372-378`) — a per-IP 60 req/min
|
||
sliding window, `/health` exempt, loopback skipped, returns a 429 body.
|
||
- **Auth:** none in the dispatch path. See doc 01, Divergence 3.
|
||
- **Fallbacks:** `err_404` / `err_405`.
|
||
|
||
**Collaborators:** delegates to `neuron-api.el` (`/api/neuron/*`), `sessions.el`
|
||
(`/api/sessions/*`), `chat.el` (`/api/chat`, `/dharma/recv`), the Axon Accessor
|
||
(`axon_get/post` for `/api/backlog|artifacts|projects|memories|knowledge`),
|
||
`connectd_*` (`/api/connectors*`), `studio.el` (`/`), and engram builtins for the
|
||
raw `/api/graph*` reads.
|
||
|
||
**Route surface** (grouped; full table with line numbers is in the survey notes):
|
||
|
||
| Group | Representative routes | Handler home |
|
||
|---|---|---|
|
||
| Session/context | `/api/neuron/session/begin`, `/api/neuron/ctx`, `/api/sessions*` | neuron-api, sessions.el |
|
||
| Memory | `/api/neuron/memory`, `/recall`, `/memory/{evolve,forget,delete,update}`, `/node/{create,update,delete}` | neuron-api |
|
||
| Knowledge | `/api/neuron/knowledge/{search,capture,evolve,promote}`, `/knowledge` | neuron-api |
|
||
| Graph/activation | `/api/neuron/graph`, `/graph/link`, `/api/graph*`, `/list/:type` | neuron-api + engram builtins |
|
||
| Cultivation/self | `/api/neuron/cultivate`, `/lineage`, `/imprint/*`, `/synthesize` | neuron-api, routes.el |
|
||
| Processes/config | `/api/neuron/processes{,/define}`, `/config{,/tune}` | neuron-api |
|
||
| State/consolidate | `/api/neuron/state-events`, `/consolidate` | neuron-api |
|
||
| Backlog/artifacts | `/api/backlog`, `/artifacts`, `/projects`, `/memories` | Axon (HTTP) |
|
||
| Chat/NLG | `/api/chat`, `/see`, `/elp/chat`, `/dharma*`, `/nlg*` | chat.el, elp-input.el |
|
||
| Health/UI | `/health`, `/lineage`, `/` | routes.el, studio.el |
|
||
|
||
---
|
||
|
||
## 2. The cognitive API — `neuron-api.el`
|
||
|
||
**Responsibility:** the `/api/neuron/*` handlers — the operations that read and
|
||
write the engram as *cognition* (session, memory, knowledge, graph, config,
|
||
processes, state, cultivation). The file header notes these were migrated **out
|
||
of the MCP wrapper's HTTP calls into in-process engram builtins**
|
||
(`neuron-api.el:3-9`) — so most handlers call the store directly, no HTTP
|
||
round-trip.
|
||
|
||
**Primary collaborators** are the engram builtins (`engram_node_full`,
|
||
`engram_search_json`, `engram_activate_json`, `engram_scan_nodes_json`,
|
||
`engram_scan_nodes_by_type_json`, `engram_neighbors_json`, `engram_connect`,
|
||
`engram_get_node_json`, `engram_stats_json`, `engram_save`) and `memory.el` for
|
||
tombstoning.
|
||
|
||
**Handler groups:**
|
||
|
||
- **Session / context** — `handle_api_begin_session` (`:273-301`),
|
||
`handle_api_compile_ctx` (`:305-317`). Pull `engram_stats_json`, run
|
||
spreading activation (`engram_activate_json`, depth-1 for begin, depth-2 for
|
||
ctx), scan recent `InternalStateEvent`s, then **project the result through the
|
||
compaction helpers** so the payload can't overflow the MCP client's context.
|
||
This is Engine work (axis 1) inside a Manager-shaped entry point.
|
||
|
||
- **Memory** — `handle_api_remember` (`:322-348`): maps `importance` → salience,
|
||
injects a `project:<name>` tag, writes a `Memory`/`Episodic` node, then
|
||
**read-back-verifies** persistence (`api_persisted`). Deletes are **tombstone,
|
||
never hard delete** — `node_delete` / `memory_delete` / `forget` all route
|
||
through `tombstone_node` → `mem_tombstone`. Updates/evolves are **immutable
|
||
supersede** — `node_update` (`:397-429`), `evolve_memory` (`:711-735`) create a
|
||
new node and wire `engram_connect(new, old, "supersedes")`.
|
||
|
||
- **Knowledge** — `search_knowledge` (`:458-478`, falls back to
|
||
`engram_activate_json(q,2)` when lexical search returns nothing),
|
||
`browse_knowledge`, `capture_knowledge` (`:492-504`), `evolve_knowledge`,
|
||
`promote_knowledge` (`:526-542`, writes a canonical-tier node + supersede
|
||
edge). Evolve/promote respect `is_protected_node`.
|
||
|
||
- **Graph** — `handle_api_inspect_graph` (`:778-813`): resolves a named anchor
|
||
(`self`/`neuron` → `kn-efeb4a5b…`, `values` → `kn-5b606390…`) or an explicit
|
||
id, then `engram_neighbors_json(resolved, depth, "both")`. By default this is a
|
||
plain neighbor traversal (byte-identical to the old behavior, so the studio app
|
||
is unaffected). **When called with `compact=1` (or `true`) it returns a
|
||
relevance-ranked projection** (`:804-810`): the neighborhood is ranked and the
|
||
top **`k`** neighbors (default 12) keep a UTF-8-safe content snippet (default
|
||
`snip=600`) via `api_neigh_full`, while the remainder collapse to lightweight
|
||
`{id,label,node_type,tier,edge,pointer:true}` stubs via `api_neigh_pointer`.
|
||
This bounds a high-fanout identity anchor (voice, writing-imprint, self-root)
|
||
from ~670 KB to ~25 KB so the MCP transport no longer socket-closes on
|
||
self-load. The MCP wrapper appends `&compact=1` on its inspectGraph/fetch-by-id
|
||
path; the studio app omits the flag and is unchanged.
|
||
`handle_api_link_entities` (`:818-…`) creates edges but blocks edges *into*
|
||
protected nodes.
|
||
|
||
- **Cultivation** — `handle_api_cultivate` (`:781-839`): dispatches on
|
||
`operation` (evolve_knowledge / evolve_memory / forget / link_entities) and
|
||
performs the same engram ops **but skips `is_protected_node`** — the sanctioned
|
||
identity-write path, gated by convention to Will's explicit cultivation
|
||
sessions.
|
||
|
||
- **Config / processes / state-events / consolidate** — config anchors + a
|
||
`ConfigEntry` node search (`:616-639`), `tune_config` (`:642-653`),
|
||
`browse_processes` / `define_process` (`:547-568`), state-event log/list
|
||
(`:575-610`), and `consolidate` (`:855-880`, an `engram_save` snapshot plus an
|
||
optional `SessionSummary` node).
|
||
|
||
**The projection/compaction layer** (a real, recurring concern) lives in
|
||
`api_compact_node` (`:132-148`), `api_compact_node_array` (`:152-165`),
|
||
`api_compact_activated` (`:170-189`), and `api_utf8_trunc` (`:116-127`). These
|
||
**cap array length and truncate each node to identity + a bounded UTF-8-safe
|
||
content snippet.** Their consumers are `begin_session` and `compile_ctx`.
|
||
The design principle is the important part: *the API returns a relevance-bounded
|
||
projection of the graph, not the graph.* That bounding started as
|
||
length-capping + activation-ordering; it now also includes a **relevance-ranked
|
||
neighbor projection** — `api_compact_neighbors` (`:288-317`), backed by
|
||
`api_neigh_better`/`api_neigh_rank` (relevance ordering), `api_neigh_full`
|
||
(top-K, snippet), `api_neigh_pointer` (the rest, stub), and `api_float_or`. This
|
||
is **committed fact, not an in-flight concern**: it is the `compact=1` path of
|
||
`handle_api_inspect_graph` above, and it is what makes self-load survive the MCP
|
||
transport. It is compiled into `dist/soul.c` (this PR regenerated the
|
||
amalgamation so CI ships it — see doc 05).
|
||
|
||
---
|
||
|
||
## 3. Memory & activation engine
|
||
|
||
This subsystem spans three files in this repo (`memory.el`, `awareness.el`,
|
||
`soul.el`) and one in `foundation` (`el_runtime.c`). The split matters: **the
|
||
math is in C; the El files orchestrate, persist, and instrument it.**
|
||
|
||
### 3a. Memory access — `memory.el` (the Accessor)
|
||
|
||
The single isolation point over the engram FFI. Key functions:
|
||
|
||
| Fn | Lines | Backing call | Notes |
|
||
|---|---|---|---|
|
||
| `mem_store` | `5-28` | `engram_node_full` + read-back | verified write |
|
||
| `mem_remember` | `30-32` | `mem_store` | label `soul-memory` |
|
||
| `mem_recall` | `34-36` | `engram_activate_json(query, depth)` | **spreading-activation recall** (mutates WM) |
|
||
| `mem_search` | `38-40` | `engram_search_json` | pure lexical scan (no WM side-effect) |
|
||
| `mem_strengthen` | `42-44` | `engram_strengthen` | salience bump |
|
||
| `mem_tombstone` | `52-62` | `engram_node_full` + `engram_connect` | the one canonical soft-delete |
|
||
| `mem_forget` | `70-72` | `mem_tombstone` | soft delete (no longer hard) |
|
||
| `mem_consolidate` | `92-133` | `engram_wm_top_json`, `engram_strengthen` | salience-evolution pass |
|
||
| `mem_save` / `mem_load` | `135-148` | `engram_save/load` | snapshot I/O |
|
||
|
||
Note the distinction between **recall and search**: `mem_recall` fires spreading
|
||
activation (and warms working memory as a side effect); `mem_search` is a passive
|
||
lexical lookup. Tiers here are `tier_working` / `tier_episodic` / `tier_canonical`
|
||
(`memory.el:1-3`) — see doc 03 for how these relate to the engine's tier field
|
||
and to the MCP surface vocabulary.
|
||
|
||
### 3b. Autonomous cognition — `awareness.el` (the daemon)
|
||
|
||
`awareness.el` is the **idle-cognition daemon plus observability**, not
|
||
emotional-state code. `awareness_run()` (`:1097-1284`) is the master loop,
|
||
launched last from `soul.el:627`. Each tick (`SOUL_TICK_MS`, ~200ms):
|
||
|
||
1. **`one_cycle()`** (`:1041-1095`) — the cognitive step:
|
||
`perceive()` (`:900-924`, gated on a `soul-inbox-pending` tag, then
|
||
`engram_activate_json`) → `attend()` (`:926-973`, parse trigger content into
|
||
an action verb: remember / search / activate / strengthen / forget /
|
||
consolidate / respond) → `respond()` (`:975-1029`, dispatch to the `mem_*`
|
||
fns) → `record()` (`:1031-1039`, emit an InternalStateEvent) → consume the
|
||
trigger.
|
||
2. **Heartbeat** (every 60s): `hebb_consolidate()` **then** `emit_heartbeat()`
|
||
then `mem_save` snapshot (`:1189-1197`).
|
||
3. **Curiosity scan** (every 30s when idle): `proactive_curiosity()`
|
||
(`:701-876`) rotates 4 seed-domain sets, activates a seed, strengthens the
|
||
top result **only if it changed** (novelty-gated), and derives an
|
||
autobiographical seed from the top-10 working-memory nodes with
|
||
stopword/IDF/tabu filtering.
|
||
4. **Engram sync** (every 10 min): `GET /api/sync` → `engram_load_merge` →
|
||
telemetry prune.
|
||
|
||
Two functions carry most of the file's weight and volatility:
|
||
- **`hebb_consolidate()`** (`:64-99`) — the durable-learning write-back. It drains
|
||
newly-formed co-activation edges (`engram_hebb_drain_json(64)`) and POSTs them
|
||
as one batch to `/api/edges/batch` (`:94`). The comment block (`:33-63`)
|
||
records that before this path existed the soul threw away ~1,198 learned
|
||
edges per restart — the daemon is where **essentially all co-activation
|
||
happens**, and this is how it survives.
|
||
- **`emit_heartbeat()`** (`:201-549`, ~350 lines) — assembles ~50 gauges (WM
|
||
saturation/churn, Hebbian candidate/edge counts, embedding coverage, corpus
|
||
health) into one ISE. Pure observability; a fat, churny Accessor/Utility mix.
|
||
|
||
A **threat scorer** (`:1286-1419`) is grafted onto the end — command/path/history
|
||
additive scoring, ≥70 blocks a tool call. Cross-cutting agentic-safety policy,
|
||
unrelated to memory mechanism.
|
||
|
||
### 3c. Identity & the request pipeline — `soul.el`
|
||
|
||
`soul.el` is the top-level program (`cgi "neuron-soul"`, `:12-17`) and imports
|
||
every other module (`:1-10`). It owns:
|
||
|
||
- **The identity graph.** `init_soul_edges()` (`:19-92`) hard-wires a `self_root`
|
||
node linked by `identity` edges (weight 0.95) to family/origin/value nodes,
|
||
plus a dense `co-value` mesh (weight 0.7) among 8 value nodes.
|
||
`ensure_self_canonical_bridge()` (`:101-110`) links the public traversal-root
|
||
anchor (`kn-efeb4a5b`) to the curated self node via `canonical-self` edges.
|
||
`load_identity_context()` (`:153-240`) loads intellectual-DNA / values /
|
||
memory-philosophy content into a state key for prompt injection.
|
||
- **Boot orchestration** (`:508-627`): load snapshot → optional first-boot seed
|
||
(guarded) → identity context → persona-from-env → boot-count increment →
|
||
session-start event → genesis-only edge init → `http_serve_async(port,
|
||
"handle_request")` → `awareness_run()`.
|
||
- **The request pipeline.** `layered_cycle()` (`:382-506`) — a 4-layer stack for
|
||
user input: **L1** safety screen (`safety_screen`) → **L2a** continuity/
|
||
behavioral (`steward_session_check`) → **L2b** mission alignment
|
||
(`steward_align`) → **L2c** affective-context injection → **L3**
|
||
`imprint_respond` → **L1** output validation (`safety_validate`). Hard-bell
|
||
inputs bypass the upper layers. The JSON contract between these layers is
|
||
pinned by `tests/test_layer_contract.el`.
|
||
|
||
### 3d. Where the activation math actually is
|
||
|
||
`el_runtime.c` implements the two-layer activation model
|
||
(`background_activation` via BFS fan-out, then `working_memory_weight` via an
|
||
executive filter), ACT-R base-level learning (per-node access-timestamp ring
|
||
buffer), 768-dim semantic embeddings, and Hebbian eligibility traces. Retrieval
|
||
is **spreading activation, not query**:
|
||
`strength = parent_strength × edge_weight × target_salience ×
|
||
cosine(query, target)`. The El files never compute this — they seed it
|
||
(`engram_activate_json`), harvest it (`engram_hebb_drain_json`), and persist it.
|
||
See `03-data-and-memory.md`.
|
||
|
||
---
|
||
|
||
## 4. The MCP transport chain — `mcp-proxy`, `mcp-wrapper`
|
||
|
||
The chain exists because two boundaries vary independently: the *client
|
||
transport* (stdio MCP JSON-RPC) and the *soul's protocol* (HTTP REST). Each hop
|
||
absorbs one.
|
||
|
||
- **`mcp-proxy/src/main.el`** (listens `:7779`) — a **byte-forwarder**. It
|
||
accepts the client connection, forwards to the wrapper, and adds resilience:
|
||
retry, health-gating, and a well-formed error envelope so a downstream hiccup
|
||
never surfaces to the client as a broken pipe. It holds no MCP semantics —
|
||
pure transport orchestration.
|
||
|
||
- **`mcp-wrapper/src/main.el`** (listens `:17779`) — the **protocol translator**.
|
||
It speaks MCP JSON-RPC to the client and REST to the soul (`:7770`), owns the
|
||
MCP lifecycle (`initialize`, `tools/list`, `tools/call`), and carries the
|
||
**tool catalog** (~90 tools) that clients enumerate. `dispatch_tool_call` maps
|
||
each tool to a soul REST endpoint. It also fires a **spread-activation side
|
||
effect** (`fire_activation`) — after relevant calls it issues a `/recall` to
|
||
warm related nodes, so tool use itself nudges working memory. The tool schemas
|
||
in the catalog are largely name-only stubs — flag as a place where richer
|
||
schemas could live.
|
||
|
||
- **Manifests** (`mcp-proxy/manifest.el`, `mcp-wrapper/manifest.el`) declare the
|
||
build entry and package metadata for each transport binary.
|
||
|
||
**End-to-end (one `tools/call`):** client → proxy (`:7779`, forward+retry) →
|
||
wrapper (`:17779`, JSON-RPC→REST, catalog dispatch) → soul (`:7770`,
|
||
`handle_request` → `handle_api_*`) → engram builtins → (HTTP `:8742` when in HTTP
|
||
mode). The response walks back up, and the wrapper may fire a `/recall` warm-up
|
||
on the way. The full sequence is drawn in `04-runtime-and-deployment.md`.
|
||
|
||
**VBD reading:** proxy and wrapper are Managers of transport; the wrapper is also
|
||
the Accessor that isolates the *MCP protocol* boundary from the soul (the soul
|
||
knows only HTTP). The multi-hop shape is justified: the client transport, the
|
||
protocol translation, and the cognition each change for different reasons and are
|
||
deployed/updated independently.
|