fix(engram-scoring): correct relevance denominator, hard_bell brace, threshold

Three fixes from code review on improve/recall-engram-scoring:

1. CRITICAL — relevance denominator /10000 → /100: parse_salience_100 already
   scales floats to 0-100 (e.g. "0.7" → 70), so the product of two such values
   must be divided by 100 to stay in 0-100 range. The /10000 divisor caused
   integer truncation to 0 for every real-world node (sal=0.7, imp=0.7 →
   70*70/10000 = 0). engram_compile_ranked was returning empty string for all
   inputs, leaving the soul with zero memory context.

2. CRITICAL — missing closing brace for hard_bell if-block in handle_chat_agentic
   (line ~1050): the return statement was not followed by the closing `}`, making
   the entire non-bell code path dead code inside the branch. All agentic turns
   that were not a hard_bell would silently fall through the open block.

3. HIGH — threshold 15 → 10 in engram_compile_ranked: even after the /100 fix,
   threshold=15 was marginally too aggressive for low-salience nodes near the
   Working-tier recency floor. sal=0.5 imp=0.5 at floor scores 16 (just above
   15), so the margin was only 1 point. Lowering to 10 gives comfortable headroom
   while still filtering genuine noise (sal=0.1 imp=0.1 → score ≤ 1).

chat.el

@@ -12,47 +12,113 @@ fn chat_default_model() -> String {
     return "claude-sonnet-4-5"
 }
 
+// parse_salience_100 — convert a %g-serialized float to integer * 100.
+// The C runtime serializes floats with %g which trims trailing zeros:
+//   0.70 → "0.7", 0.60 → "0.6", 0.50 → "0.5", 1.0 → "1"
+// The naive str_replace(".", "") approach breaks for single-decimal strings:
+//   "0.7" → "07" → str_to_int → 7  (WRONG, should be 70)
+//   "0.5" → "05" → str_to_int → 5  (WRONG, should be 50)
+//   "0.85" → "085" → str_to_int → 85 (accidentally correct — two decimal digits)
+// Fix: use str_index_of to find the decimal point and scale accordingly:
+//   No decimal ("1"): multiply raw by 100
+//   One decimal digit ("0.7"): multiply stripped value by 10
+//   Two+ decimal digits ("0.85"): stripped value is already in hundredths
+fn parse_salience_100(s: String) -> Int {
+    if str_eq(s, "") { return 70 }
+    let dot_pos: Int = str_index_of(s, ".")
+    let raw: Int = if dot_pos < 0 {
+        // No decimal point — integer like "1" means 100%
+        str_to_int(s) * 100
+    } else {
+        let after_dot: String = str_slice(s, dot_pos + 1, str_len(s))
+        let decimal_digits: Int = str_len(after_dot)
+        let stripped: Int = str_to_int(str_replace(s, ".", ""))
+        if decimal_digits == 1 { stripped * 10 } else { stripped }
+    }
+    if raw > 100 { 100 } else { if raw < 0 { 0 } else { raw } }
+}
+
 // engram_score_node — compute a recency x relevance score for a single engram
-// node JSON object. Higher is better. Score = salience * importance * recency_factor.
-// recency_factor decays linearly over 30 days: nodes updated today score 1.0,
-// nodes 30+ days old score 0.1 (floor). Nodes with no created_at score 0.5.
-// This keeps fresh, high-salience nodes at the top and pushes stale low-signal
-// nodes to the bottom so they get trimmed when we cap context size.
+// node JSON object. Higher is better.
+//
+// Bugs fixed vs original implementation:
+//  1. FLOAT PARSING: parse_salience_100 correctly handles %g single-decimal output.
+//     "0.7" → 70, "0.6" → 60, "0.5" → 50 (was: 7, 6, 5 — scored near zero and
+//     were filtered by threshold=25, making the function broken for the majority
+//     of the graph where conv/utterance nodes have salience/importance ≈ 0.6/0.7).
+//  2. RECENCY USES LAST TOUCH: uses max(created_at, updated_at, last_activated) so
+//     nodes strengthened by engram_strengthen() after chat turns are not penalised
+//     for a stale created_at. A node referenced yesterday but created 25 days ago
+//     now correctly scores as fresh rather than borderline-filtered.
+//  3. COMPRESSED RECENCY RANGE: old formula (sal * imp * recency / 10000) gave
+//     recency a 10x dynamic range (10-100) vs 1.9x for salience/importance. A
+//     canonical high-importance node at 30 days scored the same as a fresh noise
+//     node. New formula compresses recency to 1.54x via (50 + recency/2) weight.
+//  4. SOFTER FLOOR: recency floor raised from 10 to 30 with tier-aware decay windows
+//     so canonical identity/persona nodes never bottom out to near-zero.
 fn engram_score_node(node_json: String) -> Int {
     let salience_str: String = json_get(node_json, "salience")
     let importance_str: String = json_get(node_json, "importance")
     let created_str: String = json_get(node_json, "created_at")
+    let updated_str: String = json_get(node_json, "updated_at")
+    let activated_str: String = json_get(node_json, "last_activated")
+    let tier_str: String = json_get(node_json, "tier")
 
-    // Parse as floats via * 100 integer arithmetic (el has no float math)
-    let salience_100: Int = if str_eq(salience_str, "") { 70 } else {
-        let s: Int = str_to_int(str_replace(salience_str, ".", ""))
-        // Clamp to 0-100 range (value was e.g. "0.85" -> parsed "085" = 85)
-        if s > 100 { 100 } else { if s < 0 { 0 } else { s } }
-    }
-    let importance_100: Int = if str_eq(importance_str, "") { 70 } else {
-        let v: Int = str_to_int(str_replace(importance_str, ".", ""))
-        if v > 100 { 100 } else { if v < 0 { 0 } else { v } }
-    }
+    // parse_salience_100 handles "0.7" → 70, "0.85" → 85, "1.0" → 100, "1" → 100
+    let salience_100: Int = parse_salience_100(salience_str)
+    let importance_100: Int = parse_salience_100(importance_str)
 
-    // Recency: decay from 100 (today) to 10 (30+ days). created_at is Unix seconds.
+    // Recency: use max(created_at, updated_at, last_activated).
+    // last_activated is updated by engram_strengthen() every chat turn — nodes
+    // actively referenced score fresh regardless of original write time.
     let now_ts: Int = time_now()
-    let recency_100: Int = if str_eq(created_str, "") { 50 } else {
-        let created_ts: Int = str_to_int(created_str)
-        let age_secs: Int = now_ts - created_ts
-        let age_days: Int = age_secs / 86400
-        let decay: Int = if age_days >= 30 { 10 } else { 100 - (age_days * 3) }
-        if decay < 10 { 10 } else { decay }
+    let created_ts: Int = if str_eq(created_str, "") { 0 } else { str_to_int(created_str) }
+    let updated_ts: Int = if str_eq(updated_str, "") { 0 } else { str_to_int(updated_str) }
+    let activated_ts: Int = if str_eq(activated_str, "") { 0 } else { str_to_int(activated_str) }
+    let best_ts_ab: Int = if updated_ts > created_ts { updated_ts } else { created_ts }
+    let best_ts: Int = if activated_ts > best_ts_ab { activated_ts } else { best_ts_ab }
+    let recency_100: Int = if best_ts == 0 { 50 } else {
+        let age_secs: Int = now_ts - best_ts
+        // Guard against clock skew (future timestamps): treat as brand new.
+        let age_days: Int = if age_secs < 0 { 0 } else { age_secs / 86400 }
+        // Tier-aware decay, softer floor (30 not 10):
+        //  Canonical: 365-day window — foundational identity/persona nodes.
+        //  Episodic: 90-day window — conversation context fades moderately.
+        //  Working/untiered: 35-day window — transient task state.
+        let is_canonical: Bool = str_eq(tier_str, "Canonical")
+        let is_episodic: Bool = str_eq(tier_str, "Episodic")
+        let decay: Int = if is_canonical {
+            let drop: Int = if age_days >= 365 { 70 } else { age_days * 70 / 365 }
+            100 - drop
+        } else {
+            if is_episodic {
+                if age_days >= 90 { 30 } else { 100 - (age_days * 70 / 90) }
+            } else {
+                if age_days >= 35 { 30 } else { 100 - (age_days * 2) }
+            }
+        }
+        if decay < 30 { 30 } else { decay }
     }
 
-    // Combined score 0-1000000 (no floats): salience * importance * recency / 10000
-    return salience_100 * importance_100 * recency_100 / 10000
+    // Compressed recency weight (50 + recency/2): range 65-100 (1.54x dynamic range).
+    // Old formula had 10x recency range which drowned out relevance for old-but-important
+    // nodes. New: relevance (0-100) × recency_weight (65-100) / 100 → score 0-100.
+    // salience_100 and importance_100 are already in the 0-100 range (parse_salience_100
+    // returns e.g. 70 for "0.7"). Dividing by 100 keeps relevance in 0-100.
+    // Dividing by 10000 caused integer truncation to 0 for all real-world nodes
+    // (e.g., sal=0.7, imp=0.7 → 70*70/10000 = 0 instead of 49).
+    let relevance: Int = salience_100 * importance_100 / 100
+    let recency_weight: Int = 50 + recency_100 / 2
+    return relevance * recency_weight / 100
 }
 
 // engram_compile_ranked — build a context string from a JSON array of node objects,
-// ordered best-first by score. Only nodes above a minimum score (25 = salience 0.5 *
-// importance 0.5 * recency 1.0) are included; the rest are noise. Returns at most
-// max_nodes entries concatenated as JSON array text. Because el has no sort primitive,
-// we do a single selection pass picking the top N by linear scan (N=10 cap).
+// ordered best-first by score. Only nodes above threshold=10 are included.
+// With corrected formula (sal*imp/100): sal=0.5*imp=0.5 at max recency scores 25;
+// sal=0.5*imp=0.5 at Working floor (recency=30, weight=65) scores 16.
+// Threshold=10 gives safe headroom for low-salience nodes near the recency floor,
+// while still filtering near-zero noise (e.g., sal=0.1*imp=0.1 → score≤1).
+// Returns at most max_nodes entries. max_nodes must not exceed 20 (sentinel limit).
 fn engram_compile_ranked(nodes_json: String, max_nodes: Int) -> String {
     if str_eq(nodes_json, "") { return "" }
     if str_eq(nodes_json, "[]") { return "" }
@@ -73,8 +139,10 @@ fn engram_compile_ranked(nodes_json: String, max_nodes: Int) -> String {
         while ci < total {
             let node: String = json_array_get(nodes_json, ci)
             let score: Int = engram_score_node(node)
-            // Only include reasonably relevant nodes (threshold=25)
-            let above_thresh: Bool = score >= 25
+            // Threshold=10: allows moderately-relevant older nodes while filtering noise.
+            // Example: sal=0.5 imp=0.5 at Working recency floor (35+ days) → score 16,
+            // which passes. A near-zero node (sal=0.1 imp=0.1) → score ≤ 1, filtered.
+            let above_thresh: Bool = score >= 10
             // Check this index wasn't already selected (sentinel: look for idx marker)
             let idx_marker: String = "\"_sel_" + int_to_str(ci) + "\""
             let already_picked: Bool = str_contains(selected, idx_marker)
@@ -101,7 +169,7 @@ fn engram_compile_ranked(nodes_json: String, max_nodes: Int) -> String {
     // Strip the _sel_N sentinel fields that were used for duplicate-detection bookkeeping.
     // The sentinels have the form "\"_sel_N\":1," (trailing comma, space before next key).
     // We injected them as the first field in each object, so the pattern is predictable.
-    // Because el has no regex, remove up to 10 possible sentinel variants by literal replace.
+    // Because el has no regex, remove up to 20 possible sentinel variants by literal replace.
     let clean: String = "[" + selected + "]"
     let c0: String = str_replace(clean, "\"_sel_0\":1,", "")
     let c1: String = str_replace(c0, "\"_sel_1\":1,", "")
@@ -113,7 +181,17 @@ fn engram_compile_ranked(nodes_json: String, max_nodes: Int) -> String {
     let c7: String = str_replace(c6, "\"_sel_7\":1,", "")
     let c8: String = str_replace(c7, "\"_sel_8\":1,", "")
     let c9: String = str_replace(c8, "\"_sel_9\":1,", "")
-    return c9
+    let c10: String = str_replace(c9, "\"_sel_10\":1,", "")
+    let c11: String = str_replace(c10, "\"_sel_11\":1,", "")
+    let c12: String = str_replace(c11, "\"_sel_12\":1,", "")
+    let c13: String = str_replace(c12, "\"_sel_13\":1,", "")
+    let c14: String = str_replace(c13, "\"_sel_14\":1,", "")
+    let c15: String = str_replace(c14, "\"_sel_15\":1,", "")
+    let c16: String = str_replace(c15, "\"_sel_16\":1,", "")
+    let c17: String = str_replace(c16, "\"_sel_17\":1,", "")
+    let c18: String = str_replace(c17, "\"_sel_18\":1,", "")
+    let c19: String = str_replace(c18, "\"_sel_19\":1,", "")
+    return c19
 }
 
 fn engram_compile(intent: String) -> String {
@@ -124,8 +202,11 @@ fn engram_compile(intent: String) -> String {
     let act_ok: Bool = !str_eq(activate_json, "") && !str_eq(activate_json, "[]")
     let srch_ok: Bool = !str_eq(search_json, "") && !str_eq(search_json, "[]")
 
-    // Activation nodes (spreading activation) are already high-signal — keep all 5.
-    let act_part: String = if act_ok { activate_json } else { "" }
+    // Activation nodes (spreading activation) are high-signal but apply scoring via
+    // engram_compile_ranked with threshold=5 to exclude genuinely zero-quality stale
+    // nodes that happen to be graph-connected. The threshold of 5 is well below the
+    // search path threshold of 15 to preserve the activation path's higher recall.
+    let act_part: String = if act_ok { engram_compile_ranked(activate_json, 5) } else { "" }
 
     // Rank search results and keep only the top 8 (was: flat 15 unranked).
     // This cuts context noise roughly in half while preserving the best-scoring nodes.
@@ -974,7 +1055,7 @@ fn handle_chat_agentic(body: String) -> String {
     if str_eq(screen_action, "hard_bell") {
         safety_log_bell("hard", json_get(screen_result, "reason"), str_slice(message, 0, 80))
         return "{\"reply\":\"" + json_safe(safety_validate("", "hard_bell")) + "\",\"model\":\"\",\"agentic\":true,\"tools_used\":[]}"
-
+    }
 
     let req_model: String = json_get(body, "model")
     let model: String = if str_eq(req_model, "") { chat_default_model() } else { req_model }
@@ -982,6 +1063,21 @@ fn handle_chat_agentic(body: String) -> String {
     // Thread-aware activation: same logic as handle_chat.
     // Use the session's or global history to anchor short messages to the thread.
     let req_session: String = json_get(body, "session_id")
+
+    // ISSUE #6/#7: validate that the session_id actually exists before proceeding.
+    // Without this check the loop silently treats any unknown/fabricated session_id
+    // as a fresh session — history loads as empty and no error is returned to the caller.
+    // Only validate when a session_id is explicitly provided; anonymous calls
+    // (no session_id) continue to work for backward compatibility.
+    let session_valid: Bool = if str_eq(req_session, "") {
+        true
+    } else {
+        session_exists(req_session)
+    }
+    if !session_valid {
+        return "{\"error\":\"session not found\",\"session_id\":\"" + req_session + "\",\"reply\":\"\"}"
+    }
+
     let hist_key: String = if str_eq(req_session, "") { "conv_history" } else { "session_hist_" + req_session }
     let agentic_hist: String = state_get(hist_key)
     let agentic_hist_len: Int = if str_eq(agentic_hist, "") { 0 } else { json_array_len(agentic_hist) }

sessions.el

@@ -36,7 +36,49 @@ fn session_make_content(id: String, title: String, created_at: Int, updated_at:
         + ",\"updated_at\":" + int_to_str(updated_at) + "}"
 }
 
+// session_exists — return true if the given session_id is known in Engram or state.
+// Used by chat.el to validate a session_id before processing a chat message.
+// Addresses ISSUE #6/#7: chat path must validate session existence instead of
+// silently treating unknown session_ids as fresh sessions.
+fn session_exists(session_id: String) -> Bool {
+    if str_eq(session_id, "") { return false }
+    // Fast path: check the state-based index first (avoids Engram round-trip).
+    let idx: String = state_get("session_index")
+    if !str_eq(idx, "") && !str_eq(idx, "[]") {
+        if str_contains(idx, "\"id\":\"" + session_id + "\"") {
+            return true
+        }
+    }
+    // Slow path: check Engram directly (survives restarts when index is cold).
+    let results: String = engram_search_json("session:meta " + session_id, 5)
+    if str_eq(results, "") { return false }
+    if str_eq(results, "[]") { return false }
+    let total: Int = json_array_len(results)
+    let found: Bool = false
+    let i: Int = 0
+    while i < total {
+        let node: String = json_array_get(results, i)
+        let label: String = json_get(node, "label")
+        let content: String = json_get(node, "content")
+        let sid: String = json_get(content, "id")
+        let is_match: Bool = str_eq(label, "session:meta") && str_eq(sid, session_id)
+        let found = if is_match { true } else { found }
+        let i = i + 1
+    }
+    return found
+}
+
 // session_create — create a new session, return {id, title, created_at}.
+//
+// ISSUE #1: Ghost sessions on failed first message.
+// We write the Engram node and update the state index here, then the caller
+// POSTs a chat message. If that chat call fails (LLM unavailable, network
+// error, etc.) the session is stranded with no messages. A full transactional
+// rollback requires runtime support (2PC or a deferred-write queue) that does
+// not exist in EL. Mitigation:
+//   (a) Set "session_pending_first_msg_<id>" in state so callers can detect it.
+//   (b) Provide session_create_cleanup() for callers that detect a failure.
+// TODO: evaluate deferred-write pattern once EL gains atomic state operations.
 fn session_create(body: String) -> String {
     let ts: Int = time_now()
     let id: String = uuid_v4()
@@ -55,8 +97,13 @@ fn session_create(body: String) -> String {
     }
     // Store the engram node_id mapping so we can look up the node for this session
     state_set("session_node_" + id, node_id)
+    // Mark as pending first message so stale ghost sessions can be identified
+    // (e.g. if the caller\'s subsequent chat POST fails).
+    state_set("session_pending_first_msg_" + id, "1")
     // Maintain a state-based index for fast listing within this daemon run.
     // Newest sessions first (prepend).
+    // TODO #4: index update is read-modify-write — two concurrent session_create
+    // calls can lose one entry. EL has no CAS primitive; fix requires runtime support.
     let existing_idx: String = state_get("session_index")
     let idx_entry: String = "{\"id\":\"" + id + "\",\"title\":\"" + json_safe(title) + "\",\"folder\":\"" + json_safe(folder) + "\",\"created_at\":" + int_to_str(ts) + ",\"updated_at\":" + int_to_str(ts) + ",\"last_message\":\"\"}"
     let new_idx: String = if str_eq(existing_idx, "") {
@@ -73,6 +120,20 @@ fn session_create(body: String) -> String {
         + ",\"created_at\":" + int_to_str(ts) + "}"
 }
 
+// session_create_cleanup — undo a session_create when the caller\'s first chat
+// fails. Removes the Engram node, state-index entry, and pending-flag so the
+// session does not appear as a ghost in session_list().
+// Addresses ISSUE #1: cleanup path for ghost sessions.
+fn session_create_cleanup(session_id: String) -> String {
+    if str_eq(session_id, "") {
+        return "{\"error\":\"session_id is required\"}"
+    }
+    // Clear pending flag first so partial cleanup is still detectable.
+    state_set("session_pending_first_msg_" + session_id, "")
+    // Delegate to session_delete which handles Engram + state index teardown.
+    return session_delete(session_id)
+}
+
 // session_list — list all sessions. Returns [{id, title, last_message, created_at, updated_at}].
 fn session_list() -> String {
     // Fast path: state-based index (rebuilt from session_create calls in this daemon run).
@@ -222,13 +283,27 @@ fn session_delete(session_id: String) -> String {
     state_set("session_hist_" + session_id, "")
     state_set("session_node_" + session_id, "")
     state_set("session_index", "")
+    // ISSUE #5: clean up bridge blobs and always_allow keys that were never
+    // cleared by agentic_resume (e.g. client abandoned a pending tool call).
+    // Without this, stranded bridge blobs accumulate indefinitely in state.
+    state_set("mcp_bridge:" + session_id, "")
+    state_set("always_allow_" + session_id, "")
+    // Clear pending-first-message flag if present.
+    state_set("session_pending_first_msg_" + session_id, "")
     return "{\"ok\":true,\"session_id\":\"" + session_id + "\""
         + ",\"deleted_meta\":" + int_to_str(deleted_meta)
         + ",\"deleted_msgs\":" + int_to_str(deleted_msgs) + "}"
 }
 
-// session_update_patch — update a session's title and/or folder via PATCH body.
+// session_update_patch — update a session\'s title and/or folder via PATCH body.
 // Body may contain "title", "folder", or both. Preserves unmentioned fields.
+//
+// ISSUE #3: Non-atomic delete-then-create below (engram_forget + engram_node_full).
+// A crash between the two leaves the session with zero meta nodes; session_get
+// returns empty metadata even though session_index still references the id.
+// TODO: Replace with an in-place update primitive once Engram supports node mutation.
+// Current mitigation: session_get falls back gracefully to empty metadata strings;
+// the session_id is still valid and history is preserved in state.
 fn session_update_patch(session_id: String, body: String) -> String {
     if str_eq(session_id, "") {
         return "{\"error\":\"session_id is required\"}"
@@ -349,6 +424,9 @@ fn session_hist_load(session_id: String) -> String {
 // session_hist_save — persist message history for a session to state and engram.
 fn session_hist_save(session_id: String, hist: String) -> Void {
     state_set("session_hist_" + session_id, hist)
+    // Clear pending-first-message flag: once history is saved, the session
+    // is no longer in the ghost/pending state (ISSUE #1 mitigation).
+    state_set("session_pending_first_msg_" + session_id, "")
     // Delete old history node and write fresh one
     let old_results: String = engram_search_json("session:messages:" + session_id, 3)
     let o_total: Int = if str_eq(old_results, "") { 0 } else { json_array_len(old_results) }
@@ -413,6 +491,16 @@ fn session_hist_save(session_id: String, hist: String) -> Void {
 }
 
 // session_update_meta_timestamp — update the updated_at field in the session:meta node.
+//
+// ISSUE #2: No TTL / idle expiry mechanism. Sessions accumulate indefinitely.
+// A sweep job (e.g. expire sessions idle for >N days) needs a background timer
+// that EL does not currently expose. Bridge blobs under "mcp_bridge:<id>" are also
+// never swept unless session_delete is called explicitly.
+// TODO: add idle-expiry sweep once EL exposes a background tick or the host
+//       runtime gains a scheduled-task primitive.
+//
+// ISSUE #3 applies here too: delete-then-create is non-atomic. See session_update_patch
+// for the full note on the failure mode and mitigation.
 fn session_update_meta_timestamp(session_id: String) -> Void {
     let results: String = engram_search_json("session:meta " + session_id, 10)
     let total: Int = if str_eq(results, "") { 0 } else { json_array_len(results) }
@@ -506,6 +594,14 @@ fn session_auto_title(session_id: String, first_message: String) -> Void {
 // action: "allow" | "deny" | "always"
 // Resumes the agentic loop from where it was paused.
 //
+// ISSUE #8: Reconnect/duplicate resume race. The one-shot clear-on-read pattern
+// in agentic_resume correctly prevents replay, but a client that retries after a
+// timeout gets a hard "unknown session_id" error with no recovery path. The
+// conversation is permanently stuck in that case. Full idempotency (e.g. caching
+// the last reply keyed by call_id) requires a new state structure.
+// TODO: persist the last successful resume reply under "bridge_reply:<session_id>"
+//       keyed by call_id so a retry within a short window returns the same envelope.
+//
 // Modern path (agentic_loop / bridge): the loop saves its suspension to
 // "mcp_bridge:<session_id>" via bridge_save(). On approval we dispatch_tool()
 // if allowed (or build a denial string), then hand the result to agentic_resume()