proposal: prevent engram corruption at the source (boot-seeding upsert + UTF-8 + read-back audit)

Soul-core proposal for Will's review (untested El). Root cause: boot-time writes re-insert instead of
update (dup spam), no UTF-8 validation on write. Full spec in docs repo CORRUPTION-PREVENTION-SPEC.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

PREVENTION-FIXES-FOR-REVIEW.md

@@ -0,0 +1,52 @@
+# Prevention fixes for review (engram corruption at the source) — for Will
+
+**Context:** a scan of Tim's engram found 21% of nodes corrupt. Root cause is **boot-time writes that
+re-insert instead of update**, plus no UTF-8 validation. New users start clean but would rot the same way.
+Full spec: `docs/research-archive/p0-prototypes/CORRUPTION-PREVENTION-SPEC.md`. These are **soul core**, so
+this is a proposal for your review/build/test — nothing applied. Prepared by Neuron-in-the-CLI (untested El;
+needs your engram-internals knowledge to finalize).
+
+## The one question that unblocks everything
+**Does `engram_node_full(content, type, label, …)` upsert by label, or always insert a new node?**
+- `conv_history_persist` (chat.el:786) reuses label `"conv:history"` and is described as "upsert by label",
+  and does NOT show up as heavy duplicates.
+- `mem_boot_count_inc` (memory.el:127) reuses label `"soul:boot_count"` but **accumulated ~120 copies**.
+- Both call `engram_node_full` with a fixed label + changing content. If it upserts by label, boot_count
+  shouldn't accumulate; since it does, either it inserts, or conv:history avoids dups another way.
+- **Your answer decides the fix:** (a) if there's an upsert/update-by-label primitive, the fixes are
+  one-line swaps; (b) if not, we add a `find-by-label → update-or-insert` helper and use it everywhere.
+
+`engram_node_full` / `engram_get_node_by_label` / any update primitive live in the engram repo — couldn't
+inspect their semantics from the soul repo.
+
+## FIX 1 — Idempotent boot seeding (biggest cause, ~75% of corruption)
+- **`mem_boot_count_inc` (memory.el:127-140)** — the code comment admits it: *"Each boot creates a new
+  'soul:boot_count:N' node. Old ones accumulate as history."* → change to **update the single
+  `soul:boot_count` node** (find-by-label → set content to new count), not create a new one.
+- **Identity/safety belief seeding** (the `safety:*-boundary`, `safety:anti-hallucination` beliefs that hit
+  ~81 copies each) — wherever these are seeded on boot, make them **upsert by label** so re-seeding updates
+  the one node instead of adding a copy. (Reuse the `chat.el` "upsert by label" approach.)
+- **Test:** boot the clean profile (:7798) 5×; each `safety:*-boundary` belief and `soul:boot_count` exists
+  exactly **once**; counter shows the latest value.
+
+## FIX 2 — UTF-8 validation/sanitization on every engram write
+- No UTF-8 validation found on the write path; invalid bytes got persisted (garbled nodes).
+- **Fix:** validate/normalize to valid UTF-8 before `engram_node_full` persists (reject or sanitize).
+- **Test:** write a node with invalid bytes → stored clean (or rejected); snapshot parses with zero
+  replacement characters.
+
+## FIX 3 — Confirm read-back-verify covers ALL write paths
+- Already present: `api_persisted` ("read-back-after-write guard", neuron-api.el:90) + safety.el:410. Good.
+- **Review the deliberate exception** at neuron-api.el:198 ("NOT read-back-verify here … can return a STALE
+  hit for a just-written node") and close it safely so every write path verifies.
+- **Test:** save → read back → matches; force a failed write → returns `api_not_persisted`, not false success.
+
+## FIX 4 — Cap/prune time-series events (housekeeping, NOT corruption)
+- The ~120 `session-start` InternalStateEvent nodes (soul.el:294) are **legitimate per-boot history** — do
+  **not** dedup them. But keep them bounded (keep last N / summarize older) so the engram doesn't grow forever.
+- **Test:** after many boots, event count stays bounded; older history still summarized.
+
+## Sequence
+Confirm the upsert question → implement Fix 1 (biggest win) → Fix 2 → Fix 3 → Fix 4 → build + test on the
+clean profile (:7798) before prod. Legacy cleanup of existing corrupt data is a **separate, secondary**
+safety net (and its dedup must be time-series-aware + merge edges, not blind-delete).

dist/elp-c-decls.h

@@ -2,6 +2,7 @@
 #include "el_runtime.h"
 
 el_val_t add_punct(el_val_t s, el_val_t intent);
+el_val_t add_to_seen(el_val_t seen, el_val_t node_id);
 el_val_t aff_try_slot(el_val_t slot_json, el_val_t aff_7d_ts, el_val_t acc_key);
 el_val_t agent_number(el_val_t agent);
 el_val_t agent_person(el_val_t agent);
@@ -93,8 +94,10 @@ el_val_t api_err(el_val_t msg);
 el_val_t api_err_protected(el_val_t id);
 el_val_t api_json_escape(el_val_t s);
 el_val_t api_nonempty(el_val_t s);
+el_val_t api_not_persisted(el_val_t id);
 el_val_t api_ok(el_val_t extra);
 el_val_t api_or_empty(el_val_t s);
+el_val_t api_persisted(el_val_t id);
 el_val_t api_query_int(el_val_t path, el_val_t key, el_val_t default_val);
 el_val_t api_query_param(el_val_t path, el_val_t key);
 el_val_t append_tool_log(el_val_t log, el_val_t name);
@@ -138,6 +141,7 @@ el_val_t build_np(el_val_t referent, el_val_t slots);
 el_val_t build_pp(el_val_t loc);
 el_val_t build_rules(void);
 el_val_t build_system_prompt(el_val_t ctx);
+el_val_t build_system_prompt(el_val_t ctx, el_val_t chat_mode);
 el_val_t build_vocab(void);
 el_val_t build_vp_body(el_val_t slots);
 el_val_t build_vp_from_slots(el_val_t slots);
@@ -254,6 +258,19 @@ el_val_t en_verb_form(el_val_t base, el_val_t tense, el_val_t person, el_val_t n
 el_val_t en_verb_gerund(el_val_t base);
 el_val_t en_verb_past(el_val_t base);
 el_val_t engram_compile(el_val_t intent);
+el_val_t engram_compile_multi(el_val_t topic);
+el_val_t engram_compile_ranked(el_val_t nodes_json, el_val_t max_nodes);
+el_val_t engram_dedup_nodes(el_val_t nodes_json);
+el_val_t engram_detect_recall_intent(el_val_t message);
+el_val_t engram_extract_entities(el_val_t message);
+el_val_t engram_extract_ids(el_val_t nodes_json);
+el_val_t engram_is_continuation(el_val_t message, el_val_t hist_len);
+el_val_t engram_nodes_merge(el_val_t a, el_val_t b);
+el_val_t engram_numeric_valid(el_val_t s);
+el_val_t engram_render_node(el_val_t node_json);
+el_val_t engram_render_nodes(el_val_t nodes_json);
+el_val_t engram_score_node(el_val_t node_json);
+el_val_t engram_split_topics(el_val_t message);
 el_val_t enm_been_past(el_val_t slot);
 el_val_t enm_been_present(el_val_t slot);
 el_val_t enm_comen_past(el_val_t slot);
@@ -567,6 +584,9 @@ el_val_t handle_api_list_typed(el_val_t node_type, el_val_t path, el_val_t body)
 el_val_t handle_api_log_state_event(el_val_t body);
 el_val_t handle_api_memory_delete(el_val_t body);
 el_val_t handle_api_memory_update(el_val_t body);
+el_val_t handle_api_node_create(el_val_t body);
+el_val_t handle_api_node_delete(el_val_t body);
+el_val_t handle_api_node_update(el_val_t body);
 el_val_t handle_api_promote_knowledge(el_val_t body);
 el_val_t handle_api_recall(el_val_t method, el_val_t path, el_val_t body);
 el_val_t handle_api_remember(el_val_t body);
@@ -648,6 +668,8 @@ el_val_t hi_verb_stem(el_val_t infinitive);
 el_val_t hi_verb_stem_clean(el_val_t infinitive);
 el_val_t hist_append(el_val_t hist, el_val_t role, el_val_t content);
 el_val_t hist_trim(el_val_t hist);
+el_val_t hist_trim_with_bell_guard(el_val_t hist);
+el_val_t id_in_seen(el_val_t node_id, el_val_t seen);
 el_val_t idle_count(void);
 el_val_t idle_inc(void);
 el_val_t idle_reset(void);
@@ -836,6 +858,7 @@ el_val_t non_vera_present(el_val_t slot);
 el_val_t non_weak_past(el_val_t stem, el_val_t slot);
 el_val_t non_weak_present(el_val_t stem, el_val_t slot);
 el_val_t one_cycle(void);
+el_val_t parse_float_x100(el_val_t s);
 el_val_t parse_session_id_from_path(el_val_t path);
 el_val_t parse_session_subpath(el_val_t path);
 el_val_t path_within_root(el_val_t path, el_val_t root);
@@ -970,6 +993,7 @@ el_val_t safety_classify_hard_bell(el_val_t message);
 el_val_t safety_contact_path(void);
 el_val_t safety_count_match(el_val_t text, el_val_t phrases_json);
 el_val_t safety_detect_bell_level(el_val_t message);
+el_val_t safety_detect_positive_level(el_val_t message);
 el_val_t safety_general_hard_phrases(void);
 el_val_t safety_hard_directive(el_val_t hard_type);
 el_val_t safety_log_bell(el_val_t level, el_val_t reason, el_val_t input_summary);
@@ -1007,13 +1031,19 @@ el_val_t sem_to_spec(el_val_t frame);
 el_val_t sem_to_spec_full(el_val_t frame, el_val_t verb, el_val_t tense, el_val_t aspect);
 el_val_t session_auto_title(el_val_t session_id, el_val_t first_message);
 el_val_t session_create(el_val_t body);
+el_val_t session_create_cleanup(el_val_t session_id);
 el_val_t session_delete(el_val_t session_id);
+el_val_t session_exists(el_val_t session_id);
 el_val_t session_get(el_val_t session_id);
 el_val_t session_hist_load(el_val_t session_id);
 el_val_t session_hist_save(el_val_t session_id, el_val_t hist);
 el_val_t session_list(void);
 el_val_t session_make_content(el_val_t id, el_val_t title, el_val_t created_at, el_val_t updated_at, el_val_t folder);
+el_val_t session_preload_bullets(el_val_t nodes, el_val_t max_bullets, el_val_t snip_len);
 el_val_t session_search(el_val_t query);
+el_val_t session_summary_autogenerate(el_val_t hist);
+el_val_t session_summary_write(el_val_t summary_text);
+el_val_t session_summary_write_dated(el_val_t summary_text, el_val_t label);
 el_val_t session_title_from_message(el_val_t message);
 el_val_t session_update_meta_timestamp(el_val_t session_id);
 el_val_t session_update_patch(el_val_t session_id, el_val_t body);