will.anderson
4bff40fa4a
High-fanout identity anchors (voice, writing-imprint, self-root) have ~670KB
neighborhoods. inspect_graph returned the full traversal, which overflowed the
MCP client's context and socket-closed the wrapper mid self-load -- the soul
could not traverse its own identity graph.
handle_api_inspect_graph gains an opt-in `compact` projection (compact=1|true):
the neighborhood is relevance-ranked, the top K (default 12) keep a UTF-8-safe
content snippet (default snip=600), and the remainder collapse to lightweight
{id,label,node_type,tier,edge,pointer:true} stubs. This bounds the voice node
from 669,799B -> 25,353B (HTTP 200, valid JSON) and the wrapper's soul-load no
longer socket-closes. New helpers: api_compact_neighbors, api_neigh_full,
api_neigh_pointer, api_neigh_rank, api_neigh_better, api_float_or.
The flag is gated: ABSENT it, the response is byte-identical to the old plain
traversal, so the studio app (which never sends it) is unaffected. The MCP
wrapper (mcp-wrapper/src/main.el) appends &compact=1 on its inspectGraph and
fetch-by-id paths.
dist/soul.c is REGENERATED so CI ships the fix: CI compiles the committed
single-TU dist/soul.c directly (running elb/elc on the Linux runner OOM-kills
it), so an .el-only change would build the OLD behavior. Regenerated and verified
on macOS -- compiles with the CI cc line (0 errors) and, on a throwaway soul over
a copy of the live snapshot, serves compact ~25KB / non-compact ~670KB. The regen
also syncs the amalgamation to this branch's .el sources, which had drifted
several self-review commits ahead of the previously-committed soul.c.
Docs: docs/architecture/00-05 added; 01/02/05 corrected so the relevance-ranked
inspect_graph projection reads as committed source, not an in-flight concern.
219 lines
13 KiB
Markdown
219 lines
13 KiB
Markdown
# Neuron — VBD Decomposition
|
|
|
|
> This is the load-bearing document. It applies Volatility-Based Decomposition
|
|
> (VBD) to the *actual* neuron code, not an idealized version of it. VBD asks one
|
|
> question — **what changes, why, and how often** — and draws component
|
|
> boundaries around the answers so that a change lands inside one component
|
|
> instead of rippling across many.
|
|
>
|
|
> VBD's component taxonomy:
|
|
> - **Managers** — stable orchestrators. They sequence use-cases and delegate;
|
|
> they change only when the *shape* of a workflow changes.
|
|
> - **Engines** — volatile business rules. The "how" that churns.
|
|
> - **Resource Accessors** — isolate an external dependency (a store, an API) so
|
|
> its volatility can't leak inward.
|
|
> - **Utilities** — cross-cutting, low-volatility helpers.
|
|
>
|
|
> Communication ideal: Managers orchestrate Engines and Accessors; Managers
|
|
> prefer async/event coupling to each other; Engines are stateless-ish and never
|
|
> reach external I/O directly; Accessors hide all I/O. We note below where neuron
|
|
> honors this and where it doesn't.
|
|
|
|
## The axes of change
|
|
|
|
Before classifying modules, name the volatility. These are the axes along which
|
|
neuron actually changes, ranked by observed churn (dated self-review comments in
|
|
the source are the evidence — the code keeps a changelog in its own margins).
|
|
|
|
### 1. Context / payload shaping — *highest churn*
|
|
How much of the graph, and in what projected form, gets returned to a
|
|
bounded MCP response. The `begin_session` / `compile_ctx` handlers and the
|
|
`api_compact_*` helpers carry dense dated review comments (2026-07-30, -31)
|
|
documenting repeated rework after unbounded payloads closed the MCP client
|
|
socket (`neuron-api.el:90-317`). This changes because the *client's* context
|
|
budget and the *shape* of "what's relevant right now" keep moving. The newest
|
|
rework in this axis is the **relevance-ranked neighbor projection**
|
|
(`api_compact_neighbors` + `api_neigh_*`) behind `inspect_graph`'s `compact=1`
|
|
path — it is what keeps *self-load* (traversing the high-fanout identity anchors)
|
|
from closing the socket. It is committed source, compiled into `dist/soul.c`.
|
|
|
|
### 2. Autonomous-cognition policy
|
|
What the idle soul chooses to think about: seed-domain selection, curiosity
|
|
rotation, novelty gating, and the inbox verb-mapping in `attend()`. The
|
|
`proactive_curiosity` / `auto_term_try_slot` machinery
|
|
(`awareness.el:590-876`) has the deepest git-archaeology in the codebase
|
|
(comments spanning 2026-05 → 2026-08). This is where the *behavior* of the
|
|
agent is tuned.
|
|
|
|
### 3. Epistemic & memory semantics
|
|
Tiers, salience mapping, promotion/consolidation, the immutability policy
|
|
(tombstone/supersede), and knowledge disposition. These evolve as the memory
|
|
*philosophy* matures — e.g. `mem_forget` becoming a soft delete
|
|
(`memory.el:70`), the salience-evolution pass in `mem_consolidate`
|
|
(`memory.el:92-133`), the supersede-edge pattern (`neuron-api.el:394-428`).
|
|
|
|
### 4. Safety & stewardship rules
|
|
Crisis bell thresholds, agentic threat scoring, mission alignment, CGI
|
|
continuity fingerprinting. `safety.el`, `stewardship.el`, and the threat
|
|
scorer grafted onto `awareness.el:1286-1419` change on behavioral/regulatory
|
|
pressure, independently of everything else.
|
|
|
|
### 5. API / route surface growth
|
|
New cognitive endpoints and their dispatch. `routes.el` grows structurally as
|
|
tools are added; the `handle_request` if/else chain (`routes.el:358-753`) is
|
|
edited on every surface change.
|
|
|
|
*(A sixth axis — the activation/Hebbian numeric math — is real and volatile but
|
|
is externalized to `el_runtime.c`. See "Divergences," point 6.)*
|
|
|
|
## The component map
|
|
|
|
Modules classified against the taxonomy, with the volatility that justifies each
|
|
placement. Paths are repo-relative unless noted `foundation/…`.
|
|
|
|
### Managers (stable orchestration)
|
|
|
|
| Module / function | File | Why a Manager |
|
|
|---|---|---|
|
|
| `handle_request` | `routes.el:358-753` | Top-level HTTP dispatcher. Pure method/path routing; delegates every body of work. Changes only when the *route surface* (axis 5) changes, not when logic changes. |
|
|
| Boot sequence | `soul.el:508-627` | Sequences load → seed → identity → serve → daemon. Highest stability; changes only on architecture shifts. |
|
|
| `layered_cycle` | `soul.el:382-506` | Request use-case pipeline: L1 safety → L2 stewardship (continuity, mission, affect) → L3 imprint → L1 output validation. Orchestrates Engines; holds no rules itself. |
|
|
| `awareness_run` / `one_cycle` | `awareness.el:1097-1284`, `1041-1095` | Daemon lifecycle + the perceive→attend→respond→record sequencer. Manager of the autonomous loop. |
|
|
| Session CRUD | `sessions.el` | Orchestrates the immutable delete-then-recreate dance for conversation sessions (chat product). Manager-flavored, but leaks store detail (see Divergences). |
|
|
| MCP proxy | `mcp-proxy/src/main.el` | Orchestrates transport: accept stdio, forward, retry, health-gate, wrap errors. |
|
|
| MCP wrapper | `mcp-wrapper/src/main.el` | Orchestrates the JSON-RPC ⇄ REST translation, tool catalog, lifecycle (`initialize`/`tools/list`/`tools/call`). |
|
|
|
|
### Engines (volatile business rules)
|
|
|
|
| Module / function | File | Volatility it absorbs |
|
|
|---|---|---|
|
|
| `api_compact_*`, `begin_session`, `compile_ctx` | `neuron-api.el:90-317` | Axis 1 — context/payload shaping. The single most-reworked logic on the API side. |
|
|
| `attend()` | `awareness.el:926-973` | Axis 2 — inbox content → action-verb ruleset. |
|
|
| `proactive_curiosity`, `auto_term_try_slot` | `awareness.el:590-876` | Axis 2 — seed selection, stopword/IDF gates, tabu ring. Textbook Engine: highest churn. |
|
|
| threat scoring | `awareness.el:1286-1419` | Axis 4 — additive command/path/history threat rules. |
|
|
| `safety.el` (crisis/harm/bell) | `safety.el` | Axis 4 — crisis screening, bell thresholds, output validation. |
|
|
| `stewardship.el` | `stewardship.el` | Axis 4 — mission alignment, CGI check, continuity fingerprint. |
|
|
| `imprint.el` | `imprint.el` | Axis 2/3 — persona response + knowledge/memory surfacing per imprint. |
|
|
| `mem_consolidate` | `memory.el:92-133` | Axis 3 — which nodes to strengthen; salience-evolution rules. |
|
|
| salience/importance mapping | `neuron-api.el` (repeated in `remember`, `node_create`, `evolve_memory`, `cultivate`) | Axis 3 — importance-enum → salience float mapping. |
|
|
| chat mode selection | `chat.el` (via `routes.el:433-440`, `597-604`) | plan / agentic / `layered_cycle` routing. |
|
|
| **activation + Hebbian math** | `foundation/.../el_runtime.c` | Axis 6 — the true cognitive Engine, externalized to C. |
|
|
|
|
### Resource Accessors (isolate external I/O)
|
|
|
|
| Accessor | File | Dependency isolated |
|
|
|---|---|---|
|
|
| `mem_*` | `memory.el` | The engram FFI/HTTP. **The** memory Accessor — clean, single isolation point; every forget routes through `mem_tombstone` (`memory.el:46`). |
|
|
| `engram_*` builtins + `server.el` | `el_runtime.c`, `foundation/el/engram/src/server.el` | The graph store over HTTP `:8742`. |
|
|
| `axon_get` / `axon_post` | `routes.el` | The Axon backend (backlog, artifacts, projects, memories, non-neuron knowledge). |
|
|
| `connectd_get` / `connectd_post` | `routes.el:303-324` | `neuron-connectd` bridge (`:7771`). |
|
|
| `llm_call_system` / `llm_call_agentic` | runtime builtins (used in `routes.el:115`, chat) | The LLM. |
|
|
| `ise_post`, `hebb_consolidate` | `awareness.el:101-148`, `64-99` | Durable engram HTTP (`/api/neuron/state-events`, `/api/edges/batch`). |
|
|
| `render_studio` | `studio.el` | The UI surface. |
|
|
|
|
### Utilities (cross-cutting, stable)
|
|
|
|
`flag_true`, `strip_query`, `err_404/405` (`routes.el:14-91`);
|
|
`api_json_escape`, `api_query_param/int`, `api_ok/err`, `api_nonempty`,
|
|
`api_utf8_trunc`, `api_persisted` (`neuron-api.el:45-201`); `idle_*`/`pulse_*`
|
|
counters, `elapsed_ms/human`, `make_action`, `embed_ok` (`awareness.el`);
|
|
`session_make_content`, `aff_try_slot`, JSON builders (`sessions.el`, `soul.el`).
|
|
Beneath all of these, the El runtime builtins (`json_*`, `http_*`, crypto, time)
|
|
are the utility substrate every module shares.
|
|
|
|
## Communication topology (as built)
|
|
|
|
```
|
|
MCP client
|
|
│ JSON-RPC
|
|
proxy ──► wrapper ──► soul.handle_request ──► neuron-api.handle_api_*
|
|
│ │
|
|
│ layered_cycle │ engram_* builtins
|
|
▼ ▼
|
|
safety / steward / imprint memory.el (Accessor)
|
|
(Engines) │
|
|
▼
|
|
el_runtime.c graph
|
|
engram HTTP :8742
|
|
|
|
awareness_run (daemon) ──perceive──► engram inbox (soul-inbox-pending tag)
|
|
──hebb_consolidate──► POST /api/edges/batch
|
|
```
|
|
|
|
Two things about coupling:
|
|
|
|
- **Manager → Engine/Accessor is in-process and synchronous** (direct El calls),
|
|
which matches VBD: rules and I/O sit behind the Managers.
|
|
- **Manager ↔ Manager is *not* the VBD async-event ideal.** It is synchronous
|
|
HTTP (soul → engram, soul → Axon) plus one genuine event-ish channel: the
|
|
**engram inbox**. The awareness daemon `perceive()`s by polling a
|
|
`soul-inbox-pending` tag and consumes trigger nodes
|
|
(`awareness.el:900-924`, `1090-1093`), and modules communicate asynchronously
|
|
by writing **InternalStateEvent** nodes. That is a partial actor/event
|
|
pattern, realized through the graph rather than a message bus.
|
|
|
|
## Where reality diverges from VBD (call it out)
|
|
|
|
Honest deviations, so no one reads this doc as a conformance certificate:
|
|
|
|
1. **No route table.** Dispatch is a hand-written if/else chain in
|
|
`handle_request` (`routes.el:358-753`); there is no `register-route`
|
|
registry. Path params are sliced by hand (`str_slice` + `str_index_of`,
|
|
`routes.el:508-513, 539-541`) — one site carries an inline offset bug-fix
|
|
comment. Acceptable for a single dispatcher, but it means the "route surface"
|
|
Manager is edited manually on every change.
|
|
|
|
2. **Store I/O leaks into Managers.** `routes.el` inlines engram export logic for
|
|
`/api/graph/edges` (`routes.el:394-422`, with a 2026-08-07 comment about a
|
|
read-route that corrupted the canonical snapshot). The `awareness_run` sync
|
|
block inlines `http_get /api/sync` + `engram_load_merge`
|
|
(`awareness.el:1219-1279`). `emit_heartbeat` (`awareness.el:201-549`, ~350
|
|
lines) mixes Utility (formatting), Accessor (HTTP/FFI reads), and Manager
|
|
(state-delta tracking) in one function. These are Accessor responsibilities
|
|
living inside orchestration — the clearest VBD smell in the codebase.
|
|
|
|
3. **No authentication.** The only access control on the HTTP surface is per-IP
|
|
rate limiting (`routes.el:38-75`) plus `is_protected_node` on 15 hardcoded
|
|
identity IDs (`neuron-api.el:20-37`). There is no bearer/token check in the
|
|
dispatch path. Security is a cross-cutting concern only partially realized;
|
|
the deployment relies on a **single-trusted-client, internal-only** boundary
|
|
assumption (the `neuron-mcp` Service is ClusterIP, no external LB — see doc 04).
|
|
|
|
4. **Immutability is enforced above the Accessor, not in it.** The engram store
|
|
itself hard-deletes (`DELETE /api/nodes/:id` → `engram_forget`,
|
|
`server.el:322`). The invariant "we never delete, we tombstone/supersede"
|
|
is a *routing policy* in `memory.el` / `neuron-api.el`, not a property of the
|
|
store. A caller that hits the raw engram HTTP bypasses it.
|
|
|
|
5. **Mutation via delete-then-recreate.** Because nodes are immutable,
|
|
`sessions.el` mutates a session by deleting and recreating the node — flagged
|
|
non-atomic in its own comments (`sessions.el:303-308`, `:456`).
|
|
|
|
6. **The volatile core is in the stable layer.** The activation, decay, and
|
|
Hebbian co-activation math — genuinely high-volatility numeric policy — lives
|
|
in `el_runtime.c`, the foundational runtime every binary links. The El files
|
|
here are a Manager+Accessor shell around it. This inverts VBD's usual
|
|
layering (volatile logic should sit *above* stable infrastructure) and is the
|
|
single most important thing to understand before changing memory behavior:
|
|
you often can't, from this repo, without touching `foundation/el`.
|
|
|
|
7. **Vocabulary mismatch across layers.** The MCP-facing memory vocabulary
|
|
(tiers `note → lesson → canonical`, disposition
|
|
`experimental → … → deprecated`, importance enum `low/normal/high/critical`)
|
|
is **not** the engine's model. The engine uses cognitive tiers
|
|
`Working / Episodic / Semantic / Canonical` (a `tier` string field) plus
|
|
continuous `salience`/`importance`/`confidence` floats, and stores epistemic
|
|
tier/disposition as **tags** (`tier:canonical`, `disposition:stable`), not as
|
|
enforced state (`neuron-api.el:533`, `server.el:519-522`). The mapping is a
|
|
convention, not a guarded state machine. See `03-data-and-memory.md`.
|
|
|
|
## Testing spiral (VBD heuristic, as observed)
|
|
|
|
VBD recommends testing Engines first (pure logic), then Accessors (mock I/O),
|
|
then Managers (integration). The repo has `tests/*.el` matching this instinct —
|
|
`test_safety.el`, `test_bell_safety.el` (Engines), `test_layer_contract.el`
|
|
(the Manager↔Engine JSON contract `layered_cycle` depends on), `test_soul_guard.el`
|
|
(the boot Manager's seed guard), `test_sessions.el`. **Flag:** CI compiles and
|
|
smoke-tests only (`dist/neuron --help`); it does **not** run these `.el` suites
|
|
(`ci.yaml`). Whether they gate merges elsewhere is unverified — see doc 05.
|