diff --git a/MEMORY_RECALL_BUG.md b/MEMORY_RECALL_BUG.md new file mode 100644 index 0000000..1e9f661 --- /dev/null +++ b/MEMORY_RECALL_BUG.md @@ -0,0 +1,184 @@ +# Memory Recall Bug — Handoff for Will + +**Reported by:** Tim (via the Neuron UI chat) +**Diagnosed by:** Claude (Claude Code session), 2026-06-05 +**Symptom:** The soul can't recall anything specific — e.g. "do you remember the jokes +from that night with Will, Tim, and April?" → it has no idea, and correctly self-reports +that either retrieval is failing or the memory was never captured. + +--- + +## TL;DR + +The memories are almost certainly **intact in the graph**. The problem is the +**retrieval layer**: `engram_search_json` and `engram_activate_json` return empty for +*every* query, so the chat falls back to two hardcoded pinned nodes and effectively +remembers nothing. Strongly looks like the **embedding / search index was never built or +isn't loaded at boot**. + +Separately: the **soul daemon on :7770 was down** at the end of the investigation (it had +been up earlier in the session — it died/stopped partway through). Restart needed before +any of this can be re-tested. + +--- + +## Evidence + +All commands run against the live services during the session. + +### Search/activate return nothing — even for guaranteed-present terms +``` +curl "http://127.0.0.1:8742/api/search?q=MUDCraft&limit=3" -H "X-API-Key: ntn-user-2026" → [] +curl "http://127.0.0.1:8742/api/search?q=neuron&limit=3" -H "X-API-Key: ntn-user-2026" → [] +curl "http://127.0.0.1:8742/api/search?q=Will&limit=3" -H "X-API-Key: ntn-user-2026" → [] +curl "http://127.0.0.1:8742/api/activate?q=jokes&depth=3" -H "X-API-Key: ntn-user-2026" → {"results":[]} + +# soul's in-process equivalents (port 7770) — also empty: +curl "http://127.0.0.1:7770/api/neuron/recall?query=neuron" → (empty) +curl "http://127.0.0.1:7770/api/neuron/knowledge/search?q=MUDCraft" → (empty) +``` + +### But the raw data is present +``` +curl "http://127.0.0.1:7770/api/graph/nodes?limit=2" +→ [{"id":"mem-30425134-...","content":"CGI ARCHITECTURE ? THREE LAYERS, MCP RETIRED ... +``` +`/api/graph/nodes` is served by `engram_scan_nodes_json(9999, 0)` (routes.el:223-224) and +returns hundreds of rich nodes. So node storage is fine — only the **search/activation +index** is dead. + +### The two standalone-engram counters +``` +curl "http://127.0.0.1:8742/api/stats" → {"node_count":0,"edge_count":0,"layer_count":5} +``` +Note: the standalone engram process on :8742 reports **0 nodes**, while the soul's +in-process engram (:7770) has the data. Worth confirming which engram instance is the +source of truth and whether they've diverged. (The `:8742` process was also showing up as +`engram --help` in `ps`, which is suspicious — may not be a real server instance.) + +--- + +## Root cause (where it breaks in code) + +`neuron/chat.el → engram_compile(intent)` (lines 15-53) builds the entire memory context +for every chat turn from exactly two sources: + +```el +let activate_json: String = engram_activate_json(intent, 5) // returns [] +let search_json: String = engram_search_json(intent, 15) // returns [] +``` + +When **both are empty**, it falls back to two hardcoded nodes by literal ID +(chat.el:29-41): + +```el +// "Fallback: when vector search returns nothing (no embeddings), fetch pinned +// high-salience nodes by their known IDs." +let family_node = engram_get_node_json("knw-35940684-abc4-42f0-b942-818f66b1f69a") +let origin_node = engram_get_node_json("knw-729fc901-8335-44c4-9f3a-b150b4aa0915") +``` + +So today the soul's *entire* recallable memory in a chat = those two nodes. That's why it +can't surface jokes, social moments, the dynamic with Tim/April, or anything else specific. + +The comment ("when vector search returns nothing (no embeddings)") is the key hint: this +fallback was written *expecting* the embedding index to sometimes be absent — and right +now it's absent **all the time**. + +Affected callers all funnel through the same two dead builtins: +- `handle_api_recall` (neuron-api.el:118) — `engram_search_json` +- `handle_api_search_knowledge` (neuron-api.el:135) — `engram_search_json` + `engram_activate_json` +- `engram_compile` (chat.el:15) — both + +Working callers use a *different* builtin (`engram_scan_nodes_json` / +`engram_scan_nodes_by_type_json`), which is why graph/list views work but recall doesn't. + +--- + +## Fix options (Will's call) + +### Option 1 — Proper fix: rebuild/restore the embedding + activation index +`engram_search_json` and `engram_activate_json` are native runtime builtins. They're +returning empty because (most likely) the vector/search index was never built or isn't +loaded at boot, even though node storage loads fine. Investigate the engram boot path: +does it build embeddings for loaded nodes? Is there an index file that's missing/stale? +Fixing this restores recall everywhere at once. **This is the real fix.** + +### Option 2 — Pragmatic EL-level fallback (no native changes) +Since `engram_scan_nodes_json()` works, `engram_compile` could do a keyword scan when the +vector path is empty: pull nodes, substring/token match the query against `content` + +`label`, rank by overlap, return the top N. Restores basic recall even with the vector +index down. ~20 lines of EL in `engram_compile`, but requires a soul rebuild + restart. +Claude offered to write this patch for your review if you want it — say the word. + +Tradeoff: keyword matching is much weaker than semantic recall (won't find "jokes" unless +the node text literally contains joke-ish words), but it's strictly better than the current +two-node fallback and needs no native/runtime work. + +--- + +## Also needs attention + +- **Soul daemon (:7770) was down** at end of session — restart and confirm it stays up. +- **Confirm the engram instance topology** — :8742 standalone shows 0 nodes while the + soul's in-process engram has the data. Make sure chat is reading the populated one and + they haven't diverged. +- **Social memory weighting** (Tim's deeper point): even once retrieval works, jokes / + interpersonal moments may not be tagged or salience-weighted to surface as "important." + Worth a look at how those get captured and scored — but that's secondary to getting + retrieval working at all. + +--- + +## Daemon lifecycle — needs a supervisor (NEW, 2026-06-06) + +The soul daemon **crashed again** the next day. It had been up earlier, then died on its +own (not from any change). When it's down, the UI's Backlog / Artifacts / Knowledge / +Graph / Memories tabs all go **blank**, because they read from `:7770/api/graph/nodes`. +The chat also stops working. This is the second unexplained death in two days. + +### How it's currently run (fragile) +- Binary: `neuron/dist/neuron-fresh` (compiled from the EL sources) +- Launched manually as a bare background process (`./neuron-fresh &`) — **no supervisor, + no auto-restart, no crash logging beyond stdout**. When it dies, it stays dead until a + human notices the blank UI and restarts it. +- Boot log only shows `[http] listening on [::]:7770` — there's no captured stack/exit + reason when it crashes, so we can't yet say *why* it's dying. + +### How I restarted it (for reference) +```sh +# snapshot lives at ~/.neuron/engram/snapshot.json (loaded on boot, ~9.7MB) +# ALWAYS back it up first — genesis boot re-saves it: +cp ~/.neuron/engram/snapshot.json ~/.neuron/engram/snapshot.backup-$(date +%Y%m%d-%H%M%S).json + +cd neuron/dist +ANTHROPIC_API_KEY='' NEURON_PORT=7770 ./neuron-fresh > /tmp/soul-restart.log 2>&1 & +# verify: +curl -s http://127.0.0.1:7770/health +# → {"status":"alive","cgi_id":"ntn-genesis","boot":2,"node_count":3660,"edge_count":14207,...} +``` +After this, data came back: 3,660 nodes / 14,207 edges; Backlog 485, Memory 493, etc. + +### Recommendations for Will +1. **Put it under a supervisor** so it auto-restarts on crash and logs exit codes: + - macOS dev: a `launchd` LaunchAgent plist (KeepAlive=true), or `brew services`, or + even a simple `while true; do ./neuron-fresh; done` wrapper with timestamped logs. + - Prod/k8s already has `entrypoint.sh` + restart policy — the gap is the **local dev** + run path. +2. **Capture crash diagnostics** — redirect stdout/stderr to a rotating logfile and, if the + EL runtime can, dump a reason on exit. Right now we're blind to the cause. +3. **Find the root cause of the crashes** — two self-deaths in two days suggests a real bug + (memory? an unhandled request? a panic in a native builtin?). The supervisor stops the + *symptom* (blank UI) but not the underlying instability. +4. **Snapshot safety** — genesis boot calls `engram_save(snapshot)` (soul.el:240,248). A + crash mid-save could corrupt the 9.7MB memory file. Consider write-to-temp + atomic + rename, and/or periodic timestamped backups, so a bad save can't lose Neuron's memory. + +--- + +## What was NOT touched +No backend EL code and no engram data were modified — the memory-recall diagnosis is +read-only. The only operational action taken was **restarting the already-existing +`neuron-fresh` daemon** (after backing up the snapshot) to bring the blank UI tabs back; +no source or data was changed by that. All UI work this session was in `neuron-ui` and is +unrelated to this bug. diff --git a/chat.el b/chat.el index 935d893..fc5829b 100644 --- a/chat.el +++ b/chat.el @@ -270,6 +270,70 @@ fn agentic_tools_with_web() -> String { return "[" + inner + ",{\"type\":\"web_search_20250305\",\"name\":\"web_search\",\"max_uses\":5}]" } +// --------------------------------------------------------------------------- +// MCP connectors. The soul consumes external MCP tools through neuron-connectd, +// the loopback bridge (Accessor) on 127.0.0.1:7771. The bridge isolates all MCP +// wire complexity (stdio framing, SSE, OAuth, server lifecycle); the soul only +// speaks flat HTTP. Spec: docs/research/mcp-connectors-adoption-spec.md. +// --------------------------------------------------------------------------- + +// Fetch the merged, namespaced tool schemas (mcp____) from the bridge. +// Short timeout + empty-array fallback: if the bridge is down, the soul runs +// exactly as before with only its built-in tools (graceful degradation). +fn connector_tools_json() -> String { + let raw: String = exec_capture("curl -s --max-time 2 http://127.0.0.1:7771/mcp/tools") + if str_eq(raw, "") { + return "[]" + } + let arr: String = json_get_raw(raw, "tools") + if str_eq(arr, "") { + return "[]" + } + return arr +} + +// Built-in tools + native web_search + every connector tool, as one tools array. +// Splices connector tools in before the closing bracket of the base array. +fn agentic_tools_all() -> String { + let base: String = agentic_tools_with_web() + let conn: String = connector_tools_json() + let conn_inner: String = str_slice(conn, 1, str_len(conn) - 1) + if str_eq(conn_inner, "") { + return base + } + let base_open: String = str_slice(base, 0, str_len(base) - 1) + return base_open + "," + conn_inner + "]" +} + +// Proxy one tool call to the bridge. The model-supplied input is written to a +// temp file and handed to curl via -d @file, so arbitrary JSON can never reach +// the shell as an argument (no injection through tool_input). +fn call_mcp_bridge(tool_name: String, tool_input: String) -> String { + let eff_input: String = if str_eq(tool_input, "") { "{}" } else { tool_input } + let body: String = "{\"name\":\"" + tool_name + "\",\"input\":" + eff_input + "}" + let tmp: String = "/tmp/neuron-mcp-call.json" + fs_write(tmp, body) + return exec_capture("curl -s --max-time 30 -X POST http://127.0.0.1:7771/mcp/call -H 'Content-Type: application/json' -d @" + tmp) +} + +// Per-connector auto-approve: true only for an mcp__* tool whose server the user has +// explicitly opted into skipping the approval card (off by default). Built-in tools are +// never auto-approved here — they keep their existing gating. Bridge down → false (safe). +fn tool_auto_approved(tool_name: String) -> Bool { + if !str_starts_with(tool_name, "mcp__") { + return false + } + let raw: String = exec_capture("curl -s --max-time 2 http://127.0.0.1:7771/mcp/auto-approved") + if str_eq(raw, "") { + return false + } + let list: String = json_get_raw(raw, "tools") + if str_eq(list, "") { + return false + } + return str_contains(list, "\"" + tool_name + "\"") +} + fn dispatch_tool(tool_name: String, tool_input: String) -> String { if str_eq(tool_name, "read_file") { let path: String = json_get(tool_input, "path") @@ -297,6 +361,21 @@ fn dispatch_tool(tool_name: String, tool_input: String) -> String { let result: String = exec_capture(cmd) return json_safe(result) } + // MCP connector tools (namespaced mcp____) are routed through + // neuron-connectd. The bridge handles all MCP wire protocol complexity. + if str_starts_with(tool_name, "mcp__") { + let out: String = call_mcp_bridge(tool_name, tool_input) + if str_eq(out, "") { + return json_safe("MCP bridge unreachable (neuron-connectd on :7771)") + } + let content: String = json_get(out, "content") + if str_eq(content, "") { + let err: String = json_get(out, "error") + let msg: String = if str_eq(err, "") { "MCP call failed" } else { "MCP error: " + err } + return json_safe(msg) + } + return json_safe(content) + } return "unknown tool: " + tool_name } diff --git a/routes.el b/routes.el index a532254..d179620 100644 --- a/routes.el +++ b/routes.el @@ -207,6 +207,59 @@ fn route_sessions() -> String { return results } +// --------------------------------------------------------------------------- +// MCP Connectors proxy — thin pass-through to neuron-connectd on :7771. +// The UI talks to ONE origin (the soul); all MCP/config complexity lives in +// the bridge. Bridge-down returns a clear error (not a panic). +// --------------------------------------------------------------------------- + +fn connectd_get(suffix: String) -> String { + let out: String = exec_capture("curl -s --max-time 5 http://127.0.0.1:7771" + suffix) + if str_eq(out, "") { + return "{\"ok\":false,\"error\":\"connector bridge unreachable (neuron-connectd on :7771)\"}" + } + return out +} + +// POST passthrough: request body is written to a temp file and passed via -d @file +// so arbitrary JSON cannot reach the shell as a command-line argument. +fn connectd_post(suffix: String, body: String) -> String { + let eff: String = if str_eq(body, "") { "{}" } else { body } + let tmp: String = "/tmp/neuron-connectors-req.json" + fs_write(tmp, eff) + let out: String = exec_capture("curl -s --max-time 20 -X POST http://127.0.0.1:7771" + suffix + " -H 'Content-Type: application/json' -d @" + tmp) + if str_eq(out, "") { + return "{\"ok\":false,\"error\":\"connector bridge unreachable (neuron-connectd on :7771)\"}" + } + return out +} + +fn handle_connectors(method: String, clean: String, body: String) -> String { + if str_eq(method, "GET") { + // /api/connectors -> each configured server with status, tools, auth, auto-approve. + return connectd_get("/mcp/servers") + } + if str_eq(clean, "/api/connectors/add") { + return connectd_post("/mcp/servers/add", body) + } + if str_eq(clean, "/api/connectors/toggle") { + return connectd_post("/mcp/servers/toggle", body) + } + if str_eq(clean, "/api/connectors/auto-approve") { + return connectd_post("/mcp/servers/auto-approve", body) + } + if str_eq(clean, "/api/connectors/remove") { + return connectd_post("/mcp/servers/remove", body) + } + if str_eq(clean, "/api/connectors/secret") { + return connectd_post("/mcp/servers/secret", body) + } + if str_eq(clean, "/api/connectors/oauth/start") { + return connectd_post("/mcp/oauth/start", body) + } + return "{\"ok\":false,\"error\":\"unknown connectors route\"}" +} + fn handle_request(method: String, path: String, body: String) -> String { let clean: String = strip_query(path) @@ -320,6 +373,9 @@ fn handle_request(method: String, path: String, body: String) -> String { if str_starts_with(clean, "/api/neuron/recall") { return handle_api_recall(method, path, body) } + if str_starts_with(clean, "/api/connectors") { + return handle_connectors(method, clean, body) + } return err_404(clean) } @@ -458,6 +514,9 @@ fn handle_request(method: String, path: String, body: String) -> String { if str_eq(clean, "/api/neuron/cultivate") { return handle_api_cultivate(body) } + if str_starts_with(clean, "/api/connectors") { + return handle_connectors(method, clean, body) + } return err_404(clean) }