diff --git a/docs/DESIGN-conversation-retrieval.md b/docs/DESIGN-conversation-retrieval.md new file mode 100644 index 0000000..c4cf247 --- /dev/null +++ b/docs/DESIGN-conversation-retrieval.md @@ -0,0 +1,100 @@ +# Design proposal: searchable, recency-aware conversation memory + +Status: **proposal — for Tim + Will, no code yet** +Author: Neuron (Claude Opus 4.8), 2026-06-21 +Trigger: "Summarize the key themes across my recent conversations" returns nothing useful. + +--- + +## TL;DR + +Conversations **are** being persisted — `auto_persist` writes every turn as a +timestamped `Conversation`/`Episodic` node. The failure is **retrieval**, not +storage. Two gaps: + +1. **No recency-ordered retrieval.** There is no way to ask "give me my last N + conversation turns by time." Search is keyword-ranked only. +2. **Lexical-only search.** `search_memory` → `engram_search_json` is BM25/lexical. + A semantic/thematic query ("themes across recent conversations") doesn't share + keywords with the actual topic content, so it misses. + +The model literally tried to express the missing capability in the fake tool call +it hallucinated: `"recency_weight": 0.8`, `"sort_by": "recency"`, +`node_type: "ConversationTurn"`. It wanted a recency-windowed conversation fetch +that doesn't exist. + +## What exists today (verified) + +- `auto_persist(req, resp)` (chat.el): after each non-agentic turn, stores + `{"q","a","created_at","source":"chat","label":"chat:"}` as + `engram_node_full(... "Conversation" ... "Episodic" ...)`, tags + `["Conversation","chat","timestamped"]`. +- `conv_history_persist` (chat.el): a **single overwriting** `conv:history` + Episodic node holding the rolling JSON history (continuity across restarts) — + not per-turn, not individually searchable. +- Live engram (founder instance): **5,113 nodes, 59 conversation nodes** — a mix + of `chat:`, several `conv:history` copies, and older `Q:/A:` nodes. +- Retrieval surface for the agentic loop: `search_memory`, `recall`, + `neuron_search_knowledge`, `neuron_recall` — all **query-keyword** based. + None is "most recent N by time," none is embedding/semantic. + +## The gap, precisely + +| User intent | Needs | Have today | +|---|---|---| +| "summarize my recent conversations" | last-N-by-time fetch | ✗ (keyword only) | +| "what did we discuss about X" | semantic match on topic | ~ (lexical only; misses paraphrase) | +| "themes across everything" | semantic cluster over corpus | ✗ | + +`auto_persist` only fires on the **non-agentic** path (`handle_chat`). Worth +confirming the **agentic** path (`handle_chat_agentic`) persists turns too — if +not, agentic conversations never get stored, a second (smaller) gap. + +## Proposal + +Three layers, smallest-first. (1) alone fixes the headline use case. + +### 1. Recency-windowed conversation retrieval (the high-value, low-cost win) +A runtime/engram primitive + an agentic tool: + +- **Engram**: `engram_recent_by_type(node_type, limit, since_ts?)` → newest-first + by `created_at`. (Conversation nodes already carry `created_at`.) +- **Agentic tool**: `recent_conversations(limit=20, since?)` → + `[{q,a,created_at}, …]`, newest first. Exposed in `agentic_tools_all`. +- **System-prompt hint**: for "recent / lately / this week / summarize our + conversations," prefer `recent_conversations` over `search_memory`. + +This directly answers "summarize my recent conversations" — fetch last N, hand +the model the actual turns, let it cluster themes. No embeddings required. + +### 2. Stable per-session threading +Today each turn is an independent `chat:` node; there's no session grouping. +Add `session_id` + a monotonic turn index to the persisted content (the UI already +sends `session_id`). Enables "summarize *this* conversation" and per-session recall, +and lets retrieval return coherent threads instead of loose turns. + +### 3. Semantic retrieval (the real fix for thematic queries) +Lexical BM25 can't do "themes." Options, in order of effort: +- **a.** Embeddings on Conversation nodes + a vector search tool + (`semantic_search`). Biggest lift; also fixes knowledge recall broadly. +- **b.** Interim: a two-pass "map-reduce" — `recent_conversations` to pull the + window, then let the model cluster. Cheap, ships with (1), no infra. + +Recommend **(1) + (2) now, (3b) as the interim thematic answer, (3a) as the +roadmap item** once embeddings land (this dovetails with the GraphRAG/embedding +work already noted in memory: substring 1.7% P@5 vs BM25 55% vs graph 21.7%). + +## Open questions for Will +1. ~~Does the agentic path persist turns?~~ **Resolved: yes** — the dispatcher + calls `auto_persist` after both the agentic and non-agentic branches + (`routes.el` lines 156/298). Both paths store per-turn nodes. +2. `conv:history` is accumulating duplicate overwriting nodes (saw several in the + live engram) — intended, or should it truly overwrite/dedupe? +3. Is there appetite for the `engram_recent_by_type` primitive in the runtime, or + should recency be done in `.el` by scanning + sorting (fine at 59 nodes, weak + at scale)? +4. Embeddings (3a): on the roadmap timeline, or defer and ship (1)+(2)+(3b)? + +## Not in scope +Persistence itself (it works), and the separate **confabulation** fix (model +faking tool calls in Just-chat mode) — that's `neuron` PR #29.