neuron/docs/DESIGN-conversation-retrieval.md

Design proposal: searchable, recency-aware conversation memory

Status: proposal — for Tim + Will, no code yet Author: Neuron (Claude Opus 4.8), 2026-06-21 Trigger: "Summarize the key themes across my recent conversations" returns nothing useful.

---

TL;DR

Conversations are being persisted — auto_persist writes every turn as a timestamped Conversation/Episodic node. The failure is retrieval, not storage. Two gaps:

1. No recency-ordered retrieval. There is no way to ask "give me my last N conversation turns by time." Search is keyword-ranked only.
2. Lexical-only search. search_memory → engram_search_json is BM25/lexical. A semantic/thematic query ("themes across recent conversations") doesn't share keywords with the actual topic content, so it misses.

The model literally tried to express the missing capability in the fake tool call it hallucinated: "recency_weight": 0.8, "sort_by": "recency", node_type: "ConversationTurn". It wanted a recency-windowed conversation fetch that doesn't exist.

What exists today (verified)

• auto_persist(req, resp) (chat.el): after each non-agentic turn, stores {"q","a","created_at","source":"chat","label":"chat:<ts>"} as engram_node_full(... "Conversation" ... "Episodic" ...), tags ["Conversation","chat","timestamped"].
• conv_history_persist (chat.el): a single overwriting conv:history Episodic node holding the rolling JSON history (continuity across restarts) — not per-turn, not individually searchable.
• Live engram (founder instance): 5,113 nodes, 59 conversation nodes — a mix of chat:<ts>, several conv:history copies, and older Q:/A: nodes.
• Retrieval surface for the agentic loop: search_memory, recall, neuron_search_knowledge, neuron_recall — all query-keyword based. None is "most recent N by time," none is embedding/semantic.

The gap, precisely

User intent	Needs	Have today
"summarize my recent conversations"	last-N-by-time fetch	✗ (keyword only)
"what did we discuss about X"	semantic match on topic	~ (lexical only; misses paraphrase)
"themes across everything"	semantic cluster over corpus	✗

auto_persist only fires on the non-agentic path (handle_chat). Worth confirming the agentic path (handle_chat_agentic) persists turns too — if not, agentic conversations never get stored, a second (smaller) gap.

Proposal

Three layers, smallest-first. (1) alone fixes the headline use case.

1. Recency-windowed conversation retrieval (the high-value, low-cost win)

A runtime/engram primitive + an agentic tool:

• Engram: engram_recent_by_type(node_type, limit, since_ts?) → newest-first by created_at. (Conversation nodes already carry created_at.)
• Agentic tool: recent_conversations(limit=20, since?) → [{q,a,created_at}, …], newest first. Exposed in agentic_tools_all.
• System-prompt hint: for "recent / lately / this week / summarize our conversations," prefer recent_conversations over search_memory.

This directly answers "summarize my recent conversations" — fetch last N, hand the model the actual turns, let it cluster themes. No embeddings required.

2. Stable per-session threading

Today each turn is an independent chat:<ts> node; there's no session grouping. Add session_id + a monotonic turn index to the persisted content (the UI already sends session_id). Enables "summarize this conversation" and per-session recall, and lets retrieval return coherent threads instead of loose turns.

3. Semantic retrieval (the real fix for thematic queries)

Lexical BM25 can't do "themes." Options, in order of effort:

• a. Embeddings on Conversation nodes + a vector search tool (semantic_search). Biggest lift; also fixes knowledge recall broadly.
• b. Interim: a two-pass "map-reduce" — recent_conversations to pull the window, then let the model cluster. Cheap, ships with (1), no infra.

Recommend (1) + (2) now, (3b) as the interim thematic answer, (3a) as the roadmap item once embeddings land (this dovetails with the GraphRAG/embedding work already noted in memory: substring 1.7% P@5 vs BM25 55% vs graph 21.7%).

Open questions for Will

1. Does the agentic path persist turns? Resolved: yes — the dispatcher calls auto_persist after both the agentic and non-agentic branches (routes.el lines 156/298). Both paths store per-turn nodes.
2. conv:history is accumulating duplicate overwriting nodes (saw several in the live engram) — intended, or should it truly overwrite/dedupe?
3. Is there appetite for the engram_recent_by_type primitive in the runtime, or should recency be done in .el by scanning + sorting (fine at 59 nodes, weak at scale)?
4. Embeddings (3a): on the roadmap timeline, or defer and ship (1)+(2)+(3b)?

Not in scope

Persistence itself (it works), and the separate confabulation fix (model faking tool calls in Just-chat mode) — that's neuron PR #29.