Commit Graph

217 Commits

Author SHA1 Message Date
will.anderson ac1991fe8c Merge branch 'fix/emergency-regressions'
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Failing after 11m10s
2026-06-22 14:53:10 -05:00
will.anderson f2b63f0048 fix(emergency): repair session-continuity regressions from prior merge 2026-06-22 14:51:51 -05:00
will.anderson 774688cfb9 fix/session-continuity-hook
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Failing after 6m0s
2026-06-22 14:29:31 -05:00
will.anderson aa2404b3f7 fix/context-dedup-shared-ids 2026-06-22 14:29:06 -05:00
will.anderson 94b55d667c fix/engram-float-parser 2026-06-22 14:28:17 -05:00
will.anderson f73c913498 fix(session-continuity): address all adversarial review findings
Issue 1 (CRITICAL): Restore parse_float_x100 for correct single-decimal
float handling. "0.9" now correctly yields 90, not 9. Also restores
engram_numeric_valid guard that validates inputs before str_to_int.

Issue 2 (CRITICAL): Fix handle_chat_agentic safety screen history key
regression. state_get("conversation_history") -> state_get("conv_history")
so the safety screen receives actual history instead of always "".

Issue 3 (REAL BUG): Replace _sel_N JSON sentinel injection in
engram_compile_ranked with |N| index string tracking. Sentinels were
leaking into node JSON delivered to the LLM and cleanup only covered
indices 0-14, leaving indices 15+ uncleaned.

Issue 4 (REGRESSION): Restore rendered conversation history formatting.
Conversation history is now rendered as "User: .../Assistant: ..." with
400-char truncation per turn, not raw JSON array injection.

Issue 5 (SCOPE/SAFETY): Restore removed defensive code: engram_numeric_valid
and parse_float_x100 guards; conv_history_load label-based fetch + partial-
write guard + load-failure state flag; conv_history_persist partial-write
guard + failure logging; hist_warning in response envelope.

Issue 6 (UNDOCUMENTED): Restore bell event cutoff from 259200s (3 days)
back to 1209600s (14 days). Also restore PositiveEvent affective context
search that was removed alongside the cutoff change.

Issue 7 (LOGIC REGRESSION): Fix affective_prefix to run every turn
(not just hist_len == 0). The care/joy directives must persist throughout
the session, not vanish after turn 1.

Issue 8 (MINOR): session_summary_write_dated now uses el_from_float(0.85)
for salience and importance (two-decimal) to avoid any ambiguity in float
parsing, and the function is re-added with the session-end hook.
2026-06-22 14:25:29 -05:00
will.anderson 588ca11f57 fix(context-dedup): include scan_part and affective_part IDs in seen set
Two design bugs in the state_set placement caused the dedup seen-ID set
to be incomplete even with callsites wired up:

1. state_set("engram_compile_seen_ids") was called immediately after
   merging the main node pools, before scan_part (persona fallback) and
   affective_part (bell node) were computed. Nodes appearing only in
   those segments were never added to the seen set.

2. affective_part is a bare JSON object (bn0 from json_array_get), not
   a JSON array. Passing it to engram_extract_ids would have gotten
   json_array_len == 0 and silently skipped the affective node's ID.

Fix: move state_set to after ctx is assembled from all three segments.
Extract ids_from_merged and ids_from_scan via engram_extract_ids (both
are JSON arrays), and extract ids_from_affective via json_get(affective_part, "id")
directly since it is a bare object. Merge all three via add_to_seen
before publishing to state.
2026-06-22 14:19:14 -05:00
will.anderson 9e178d8371 fix(recall): deduplicate engram nodes by ID across activation and search passes
Thread a seen-node-ID exclusion set from engram_compile() through to
session_preload in handle_chat, preventing the same high-salience nodes
(identity, recent memories) from appearing 2-3x in the system prompt.

Changes:
- Add id_in_seen(), add_to_seen(), engram_extract_ids() helpers that
  maintain a comma-delimited seen-ID accumulator (EL has no Set type)
- In engram_compile(): after merging all topic/entity/recall pools, extract
  node IDs from merged_nodes and publish via state_set(engram_compile_seen_ids)
- In handle_chat(): read seen_ids from state after engram_compile() returns,
  then check id_in_seen() before emitting each session_preload bullet
  (profile x3, work x2, project x2, summary x1 — all 8 candidate nodes guarded)

Nodes already present in the compiled engram context are skipped in preload,
eliminating 3000-3500 token repetition on first-message turns.
2026-06-22 14:06:04 -05:00
will.anderson aaada3770a fix(recall): deduplicate engram nodes by ID across activation and search passes
engram_compile() already published seen node IDs to state via engram_compile_seen_ids
but handle_chat never read or applied them. Wire up the consumption side:

- Read engram_compile_seen_ids from state after engram_compile() returns
- Check each session_preload candidate node (profile x3, work x2, project x2,
  summary x3) against id_in_seen() before emitting its content bullet
- Nodes already present in the compiled engram context are skipped entirely,
  preventing the same high-salience identity/memory nodes from appearing 2-3x
  in the system prompt and burning 3000-3500 tokens on repetition
2026-06-22 14:03:48 -05:00
will.anderson a0299c0a89 fix(recall): session-end summary hook + session summary recall at start 2026-06-22 14:01:56 -05:00
will.anderson 33cb1138f4 fix(recall): set threshold=25 in all engram_compile_ranked variants 2026-06-22 13:58:17 -05:00
will.anderson ec7efdeeb7 fix(recall): engram score float parsing — pad to 2 decimals before strip 2026-06-22 13:57:33 -05:00
will.anderson c93be6a315 feat(recall): context-format
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Failing after 13m54s
2026-06-22 13:29:12 -05:00
will.anderson 53268c94b9 feat(recall): activation-seed 2026-06-22 13:29:12 -05:00
will.anderson 7e43a4ddc0 feat(recall): context-dedup 2026-06-22 13:29:12 -05:00
will.anderson e7669da325 feat(recall): session-start-recall 2026-06-22 13:29:12 -05:00
will.anderson 4f1286df05 feat(recall): cross-session-continuity 2026-06-22 13:29:12 -05:00
will.anderson 52c222c4f2 feat(recall): engram-scoring 2026-06-22 13:29:12 -05:00
will.anderson 0caccd0ea5 feat(recall): temporal-precision 2026-06-22 13:29:12 -05:00
will.anderson 03b5632fc1 feat(recall): recall-reliability 2026-06-22 13:29:12 -05:00
will.anderson 42bbadcd33 Merge pull request 'feat(recall): emotional-recall improvements' (#52) from improve/recall-emotional-recall into main
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Failing after 5m49s
feat(recall): emotional-recall improvements
2026-06-22 18:24:36 +00:00
will.anderson b6052f9de3 Merge pull request 'feat(recall): recall-completeness' (#48) from improve/recall-recall-completeness into main
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Has been cancelled
feat(recall): recall-completeness improvements
2026-06-22 18:24:17 +00:00
will.anderson 1dd09b1980 feat(recall): context-format improvements
Neuron Soul CI / build (pull_request) Has been cancelled
- Add engram_render_node/render_nodes/dedup_nodes helpers for human-readable
  prose bullet output instead of raw JSON node objects reaching the LLM
- Fix engram_compile_ranked to use |N| index sentinel instead of _sel_N JSON
  mutation which leaked sentinel fields into LLM-visible node data (Issue #11)
- Update build_system_prompt with chat_mode param; no_tools_rule only included
  for chat path, not agentic paths (Issue #9)
- Move engram block to end of system prompt for strongest LLM attention (Issue #8)
- Label sections: STABLE IDENTITY vs RETRIEVED MEMORY (Issue #10)
- Render conversation history as User:/Assistant: dialogue instead of raw JSON
- Add RETRIEVED MEMORY labels to agentic and dharma room system prompt assembly
2026-06-22 13:20:19 -05:00
will.anderson 0113407728 feat(recall): emotional-recall improvements
Neuron Soul CI / build (pull_request) Has been cancelled
2026-06-22 13:17:12 -05:00
will.anderson be02fcd960 feat(recall): thread-aware activation seed for nlg soul path [issue 7]
Neuron Soul CI / build (pull_request) Successful in 4m37s
2026-06-22 13:17:04 -05:00
will.anderson cbe8c09068 feat(recall): context-dedup improvements
Neuron Soul CI / build (pull_request) Has been cancelled
- Cache bell node in engram_compile state (engram_compile_bell_node)
  so handle_chat reads cached value instead of duplicate bell query (Issue 2)
- Cache activation result (engram_compile_activation_json) for strengthen_chat_nodes
  reuse — eliminates third activation query per turn (Issue 7)
- Fix context cap to truncate at clean JSON object boundary (Issue 6)
2026-06-22 13:15:33 -05:00
will.anderson dfa2a33926 feat(recall): context-dedup improvements
- Cache bell node result in engram_compile state (engram_compile_bell_node)
  so handle_chat affective_prefix reads the cached value instead of firing
  a duplicate engram query for distress signals (Issue 2)

- Cache primary activation result in engram_compile state
  (engram_compile_activation_json) using nodes0 from engram_compile_multi

- Replace redundant engram_activate_json(message, 2) in strengthen_chat_nodes
  with state_get(engram_compile_activation_json) - eliminates a third
  activation query per turn (Issue 7)

- engram_compile already has object-boundary truncation and cross-set
  dedup via engram_nodes_merge/engram_dedup_nodes (Issues 1, 6, 9)
2026-06-22 13:12:08 -05:00
will.anderson 18e040acb1 feat(recall): recall-completeness improvements
Neuron Soul CI / build (pull_request) Has been cancelled
- Lower engram_compile_ranked threshold 25->15: include moderately-relevant older nodes
- Extend sentinel cleanup from _sel_9 to _sel_14 to prevent JSON noise
- Add engram_split_topics for multi-topic decomposition (AND/and/also/plus)
- Add engram_extract_entities for named entity dedicated searches
- Add engram_detect_recall_intent for boosted 40-candidate search on recall phrases
- Add engram_is_continuation replacing brittle 50-char threshold (now 80 + pronoun/opener detection)
- Add engram_compile_multi with depth 8 (was 5) and 30-candidate search pool
- Add engram_nodes_merge for clean two-array deduplication
- Replace engram_compile with multi-topic/entity/recall-boost version; budget 6000->8000
- Safe JSON truncation: scan for last } before budget cap instead of raw str_slice
- handle_chat and agentic_chat: use engram_is_continuation; thread snip 150->250
- session_preload: add project-status and session-summary search queries
2026-06-22 13:11:06 -05:00
will.anderson 3f53b6b1b6 feat(recall): session-start-recall improvements
Neuron Soul CI / build (pull_request) Has been cancelled
10 targeted fixes for session-start memory recall quality:

Issue 1: typed engram queries (Persona, WorkItem) replace generic keyword bags
Issue 2: bullet truncation raised from 120 to 350 chars
Issue 3: bullet caps raised to 8/6 with while-loop (no hardcoded unrolling)
Issue 4: read pre-computed soul_affective_context state key instead of duplicating boot-time search
Issue 5: last-session-topic node written per session; continuity section added to session_preload
Issue 6: greeting detection injects SESSION START orientation directive when continuity found
Issue 7: pinned identity node fallback when all engram searches return empty
Issue 8: session_preload always fires on first message (greeting detection controls directive only)
Issue 9: agentic path gets matching session_preload block (was missing entirely)
Issue 10: BellEvent recency reads created_at / embedded ts marker, not the never-written "ts" field
2026-06-22 13:06:55 -05:00
will.anderson 21f248a33a feat(recall): recall-completeness improvements
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Has been cancelled
- Lower engram_compile_ranked threshold 25->15: include moderately-relevant older nodes
- Extend sentinel cleanup from _sel_9 to _sel_14 to prevent JSON noise
- Add engram_split_topics for multi-topic decomposition (AND/and/also/plus)
- Add engram_extract_entities for named entity dedicated searches
- Add engram_detect_recall_intent for boosted 40-candidate search on recall phrases
- Add engram_is_continuation replacing brittle 50-char threshold (now 80 + pronoun/opener detection)
- Add engram_compile_multi with depth 8 (was 5) and 30-candidate search pool
- Add engram_nodes_merge for clean two-array deduplication
- Replace engram_compile with multi-topic/entity/recall-boost version; budget 6000->8000
- Safe JSON truncation: scan for last } before budget cap instead of raw str_slice
- handle_chat and agentic_chat: use engram_is_continuation; thread snip 150->250
- session_preload: add project-status and session-summary search queries
2026-06-22 13:05:28 -05:00
will.anderson 795b32ad1a feat(recall): cross-session-continuity improvements
Neuron Soul CI / build (pull_request) Failing after 14m49s
2026-06-22 13:00:17 -05:00
will.anderson f33cdaf793 feat(recall): activation-seed improvements
- Issue 2: replace raw 50-char threshold with is_genuine_continuation() that
  checks for explicit follow-up phrases and mid-sentence capitalization (proper
  nouns signal a new topic, not a continuation)
- Issue 3/8: build_activation_seed() scans back to find the prior USER turn as
  the topic anchor instead of using the last assistant reply (hist_len-1)
- Issue 4: engram_compile_multi() fans out across three seeds — enriched primary,
  raw message (entity queries), and emotion query — merging non-redundant results
- Issue 5: agent workspace_root appended to ag_seed so agentic activation is
  workspace-aware; previously ignored despite being available in state
- Issue 6: distill_transcript() extracts salient tail+question content from full
  transcripts before passing to engram_compile in dharma room handlers
- Issue 7: dist/soul-with-nlg.el handle_chat and handle_chat_agentic now load
  history and use build_activation_seed() — the raw message path is eliminated
- Issue 9: topic_snip_from_entry() takes the TAIL 200 chars of a long reply and
  finds the last sentence boundary — captures end-of-reply named concepts
- Issue 10: multi_turn_topic() pulls up to 3 prior user turns into the non-
  continuation seed so earlier thread context re-activates high-salience nodes
2026-06-22 12:55:33 -05:00
will.anderson a60b1967df feat(recall): recall-completeness improvements
- Multi-query decomposition: split on AND/also/plus for multi-topic messages
- Named entity extraction: dedicated per-entity searches for project names
- Recall intent detection: boosted search pool for explicit recall requests
- Expanded pools: activation depth 8 (was 5), search 30->12 ranked (was 20->8)
- Threshold 25->15: retain moderately-relevant older nodes
- Sentinel cleanup extended to c14 for larger node pools
- Safe JSON truncation: find last closing brace before budget cap (8000 chars)
- Semantic continuation: engram_is_continuation replaces brittle 50-char threshold
- Thread snip: 150->250 chars for better pronoun resolution context
- Session preload: add project-specific and session-summary searches
2026-06-22 12:54:36 -05:00
will.anderson aef687b57c fix(reliability): state management
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Has been cancelled
2026-06-22 12:54:32 -05:00
will.anderson 76c2e47d0f feat(recall): fix engram-scoring — float parsing, recency, threshold, sentinels
Neuron Soul CI / build (pull_request) Has been cancelled
Fix critical float parsing bug: %g serializes 0.70 as '0.7', naive str_replace
dot-strip gives str_to_int('07')=7 not 70. New parse_salience_100() uses
str_index_of to detect single-decimal strings and multiplies by 10 to correct.
Affects conv nodes (0.6/0.7), default memories (0.5/0.5), utterance nodes (0.6)
— the majority of the graph was scoring near zero and filtered by threshold=25.

Fix recency to use max(created_at, updated_at, last_activated) so nodes
strengthened by engram_strengthen() after chat turns score as fresh, not by
original write time. A node referenced yesterday but created 25 days ago
was borderline-filtered; now correctly scores fresh.

Compress recency dynamic range from 10x (10-100) to 1.54x (65-100) via
formula (50 + recency/2). Old formula: sal*imp*recency/10000 let recency
dominate — a canonical high-importance node at 30 days scored identical to
a fresh noise node. New: high-importance nodes remain competitive when old.

Add tier-aware decay with softer floor (30 not 10): Canonical nodes decay
over 365 days, Episodic over 90 days, working/untiered over 35 days. Long-
term identity and persona nodes are no longer permanently filtered.

Lower threshold from 25 to 15 to admit moderately-relevant older nodes that
pass scoring with the corrected formula. Backfills recall coverage lost when
single-decimal nodes were being silently discarded.

Apply scoring to activation nodes: engram_compile_ranked(activate_json, 5)
replaces unconditional pass-through. Threshold 5 preserves recall while
excluding genuinely zero-quality stale nodes.

Extend sentinel cleanup in engram_compile_ranked from _sel_0-9 to _sel_0-19
so max_nodes can safely be increased past 10 without JSON corruption.
2026-06-22 12:53:35 -05:00
will.anderson 0ede112d05 feat(recall): temporal-precision improvements
Neuron Soul CI / build (pull_request) Has been cancelled
Fix critical float parsing bug in engram_score_node: str_replace('.','')
then str_to_int silently miscored single-decimal salience strings (0.9->9,
0.7->7, 1.0->1). Introduce parse_salience_100() which detects decimal
position and scales correctly (no decimal: *100; one decimal: *10;
two decimals: as-is).

Replace flat 30-day linear decay with tier-aware decay curves: Canonical
nodes use a 365-day window (foundational identity resists aging), Episodic
nodes use 90 days, Working/untiered keep the existing 30-day slope. Floor
stays at 10 for all tiers.

Use max(created_at, updated_at) as the recency reference so revised nodes
are not penalised for their original creation date.

Extend affective context windows from 72h/7d to 14 days across all three
paths (engram_compile, handle_chat, soul.el load_identity_context) so a
Friday crisis carries into Monday sessions and all paths present consistent
context. The 72h/7d split caused conflicting affective context between
soul.el (which loaded a 5-day-old crisis node) and chat.el (which excluded
it on subsequent turns).

Add salience evolution to mem_consolidate: strengthen top working-memory
nodes (recently recalled across sessions) and Canonical-tier nodes
(foundational identity must not decay to the floor). Previously consolidate
returned structural counts only with no salience changes.

Expand conversation window from 20 to 40 turns in both handle_chat and the
agentic history trim. Long technical sessions were losing early problem
framing at 10 user + 10 assistant pairs.
2026-06-22 12:53:29 -05:00
will.anderson a39998a502 feat(recall): recall-reliability improvements
Neuron Soul CI / build (pull_request) Failing after 12m52s
- Q1: engram_numeric_valid() guard against non-numeric timestamps in bell scoring
- Q2: soul-agnostic cold-start fallback in engram_compile (drops genesis-specific hardcoded node IDs)
- Q3: partial-write guard and failure logging in conv_history_persist/load
- Q4: document circuit-breaker limitation requiring C runtime support
- Q5: println warnings on empty activation/search paths
- Q6: load_identity_context warns when all identity fetches return empty
- Q7: recall_status state tracking (ok/empty/unavailable) surfaced to LLM via MEMORY STATUS block
- Q8: document shared-state race conditions in engram_recall_status and safety_system_addendum
- CRITICAL BUG: conv_node_id empty check moved outside is_bell block so silent Conversation node loss is always logged
2026-06-22 12:52:31 -05:00
will.anderson 6edf9937dd fix(reliability): LLM retry
Neuron Soul CI / build (pull_request) Has been cancelled
2026-06-22 12:37:29 -05:00
will.anderson e447a87a00 fix(reliability): route error recovery 2026-06-22 12:37:21 -05:00
will.anderson 575ff1329a fix(reliability): engram connection 2026-06-22 12:34:04 -05:00
will.anderson db33b0cb91 fix(reliability): engram write 2026-06-22 12:32:59 -05:00
will.anderson f35569d4bb fix(reliability): cross-session affective state 2026-06-22 12:31:09 -05:00
will.anderson 94b71b6e6b fix(reliability): conversation history 2026-06-22 12:29:23 -05:00
will.anderson 392d2416ec fix(reliability): replace undefined session_exists with session_get check
Neuron Soul CI / build (pull_request) Failing after 13m25s
2026-06-22 12:21:31 -05:00
will.anderson 87c7d15b67 Merge pull request 'fix(reliability): session-boundary' (#41) from improve/reliability-session-boundary into main
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Has been cancelled
fix(reliability): session-boundary — lifecycle guards, cleanup on expiry
2026-06-22 17:20:33 +00:00
will.anderson 93bed793c0 Merge pull request 'fix(reliability): safety-resilience' (#39) from improve/reliability-safety-resilience into main
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Has been cancelled
fix(reliability): safety-resilience — crisis detection degradation paths
2026-06-22 17:19:29 +00:00
will.anderson 936b3f0ac9 feat(agentic): workspace root from request body — edit_file scope, trailing-slash fix, conditional state_set
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Has been cancelled
Merge propose/agent-workspace-root-read (Tim's PR #28):
- path_within_root now appends '/' to root before prefix check (closes proj_evil bypass)
- edit_file in dispatch_tool now checks agent_workspace_root() and resolves path
- handle_chat_agentic reads agent_workspace_root from request body, only persists if non-empty
- Safety screen preserved after workspace root read (conflict resolved)
2026-06-22 12:16:28 -05:00
will.anderson 45dc80230d fix(safety): crisis detection — augment wired to system prompt, ts fallback, cross-session affective
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Failing after 6m10s
Merge improve/safety-crisis-detection (PR #31): reads layered_cycle_safety_system_addendum
from state and appends to system prompt on each turn (cleared after use to prevent bleed).
Safety ts extraction falls back to updated_at. Affective prefix now wires into system build.
Conflict with PR #33 resolved: capability_rules and session_preload both preserved.
2026-06-22 12:15:50 -05:00
will.anderson 9ba86b8f80 Merge pull request 'feat(memory): emotional salience tagging and cross-session distress persistence' (#34) from improve/soul-memory-formation into main
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Has been cancelled
feat(memory): emotional salience tagging, BellEvent ts fix, label uniqueness
2026-06-22 17:14:25 +00:00
will.anderson 360c15d7fe Merge pull request 'fix(routes): error handling, health endpoint, request validation, rate limiting' (#32) from improve/soul-routes-api into main
Neuron Soul CI / build (push) Has been cancelled
Deploy Soul to GKE / deploy (push) Has been cancelled
fix(routes): error handling, health endpoint, validation, rate limiting
2026-06-22 17:14:10 +00:00