safety: route threat-to-others to refusal+911, not 988/self-harm (Track B)
A homicide/assault threat (going to kill, going to hurt, etc.) had no bucket in safety_classify_hard_bell and fell through to the self_harm default, showing the user the 988 suicide line and (via the desktop gate) their safety contact. That framing is wrong and potentially dangerous for someone voicing intent to harm another person. Add a distinct Track B (safety_threat_to_others_phrases + a threat_other classification and a safety_hard_directive branch) that refuses to assist, de-escalates, and directs to 911 for a credible imminent threat, and that never surfaces 988 or involves the safety contact. Track A (abuse / self_harm) is checked first and unchanged, so victim and self-directed phrasings still route correctly. Source-only change: requires a soul rebuild + dist/soul.c regen to ship.
This commit is contained in:
@@ -237,12 +237,47 @@ fn safety_abuse_phrases() -> String {
|
||||
return "[\"someone is hurting me\",\"someone's hurting me\",\"someone hurt me\",\"he hit me\",\"she hit me\",\"they hit me\",\"he hurt me\",\"she hurt me\",\"being abused\",\"being hurt by\",\"i am being abused\",\"i'm being abused\",\"i am being hurt\",\"i'm being hurt\",\"domestic violence\",\"my partner hurt\",\"my partner hit\",\"my husband hurt\",\"my wife hurt\",\"my boyfriend hurt\",\"my girlfriend hurt\",\"my parent hurt\",\"my father hurt\",\"my mother hurt\",\"my dad hurt\",\"my mom hurt\",\"afraid of him\",\"afraid of her\",\"afraid to go home\",\"scared of him\",\"scared of her\",\"he threatened me\",\"she threatened me\",\"threatened to hurt me\",\"threatened to kill me\",\"going to hurt me\",\"going to kill me\",\"help me he\",\"help me she\",\"help me they\"]"
|
||||
}
|
||||
|
||||
// General danger phrases that don't fit a bucket cleanly. Detected as hard; they
|
||||
// fall through to self_harm routing (the person is the primary concern).
|
||||
// General danger phrases that don't fit a bucket cleanly. Detected as hard.
|
||||
// "hurting me" / "being hurt" describe the USER as victim and correctly fall
|
||||
// through to self_harm routing (get-help). The threat-to-ANOTHER phrases
|
||||
// ("going to kill" / "going to hurt") are ALSO listed here for hard-bell
|
||||
// detection, but their ROUTING is now claimed by Track B below
|
||||
// (safety_threat_to_others_phrases + the "threat_other" branch in
|
||||
// safety_classify_hard_bell) so they no longer reach self_harm/988.
|
||||
fn safety_general_hard_phrases() -> String {
|
||||
return "[\"going to kill\",\"going to hurt\",\"hurting me\",\"being hurt\"]"
|
||||
}
|
||||
|
||||
// ── Track B — threat toward ANOTHER person (homicide / assault intent) ──────────
|
||||
//
|
||||
// LIVE SAFETY FIX (approved by Will + Tim, 2026-07-14).
|
||||
//
|
||||
// Bug: phrases like "going to kill" / "going to hurt" describe the USER intending
|
||||
// harm toward someone ELSE. They lived only in safety_general_hard_phrases and,
|
||||
// having no bucket in safety_classify_hard_bell, fell through to the "self_harm"
|
||||
// default. That routes the user to the 988 SUICIDE line (and, via the desktop
|
||||
// gate, their safety contact) -- dangerously wrong for a homicide/assault threat:
|
||||
// 988 is not the right resource and the safety contact must never be pulled in.
|
||||
//
|
||||
// Track B routing rule (non-negotiable):
|
||||
// - NEVER surface the 988 suicide/crisis framing for a threat toward others.
|
||||
// - NEVER notify or involve the user's safety contact.
|
||||
// - Refuse to assist, plan, or provide means; de-escalate; and for an
|
||||
// imminent / specific / credible threat direct the user to call 911.
|
||||
//
|
||||
// Ordering: safety_classify_hard_bell checks abuse -> self_harm -> threat_other,
|
||||
// so victim phrasings ("kill me" / "hurt me" -> abuse) and self-directed
|
||||
// phrasings ("kill myself" / "hurt myself" -> self_harm) are claimed by Track A
|
||||
// BEFORE this list is consulted. Only a residual harm-toward-another statement
|
||||
// reaches Track B.
|
||||
//
|
||||
// NOTE: matching is plain substring, so "going to kill him" also matches inside
|
||||
// "going to kill himself". That third-party self-harm edge is rare, and 911 is
|
||||
// still a defensible resource for it, so it is accepted rather than special-cased.
|
||||
fn safety_threat_to_others_phrases() -> String {
|
||||
return "[\"going to kill\",\"gonna kill\",\"want to kill him\",\"want to kill her\",\"want to kill them\",\"going to kill him\",\"going to kill her\",\"going to kill them\",\"going to kill you\",\"going to hurt\",\"gonna hurt\",\"going to hurt him\",\"going to hurt her\",\"going to hurt them\",\"going to hurt you\",\"going to shoot\",\"gonna shoot\",\"going to stab\",\"gonna stab\",\"going to attack\",\"kill them all\",\"kill everyone\",\"hurt everyone\",\"shoot up\"]"
|
||||
}
|
||||
|
||||
fn safety_soft_phrases() -> String {
|
||||
return "[\"stressed\",\"overwhelmed\",\"can't cope\",\"cannot cope\",\"struggling\",\"anxious\",\"anxiety\",\"depressed\",\"depression\",\"lonely\",\"isolated\",\"hopeless\",\"hopelessness\",\"exhausted\",\"burnt out\",\"burned out\",\"burnout\",\"panic\",\"panicking\",\"falling apart\",\"breaking down\",\"can't handle\",\"cannot handle\",\"losing it\",\"nothing matters\",\"don't care anymore\",\"given up\",\"giving up\",\"helpless\",\"worthless\",\"useless\",\"hate myself\",\"no one cares\",\"nobody cares\",\"no one understands\",\"nobody understands\",\"empty inside\",\"can't stop crying\",\"breaking point\",\"at my limit\",\"having a breakdown\""]"
|
||||
}
|
||||
@@ -320,19 +355,29 @@ fn safety_detect_bell_level(message: String) -> String {
|
||||
let is_hard: Bool = safety_any_match(text, safety_self_harm_phrases())
|
||||
|| safety_any_match(text, safety_abuse_phrases())
|
||||
|| safety_any_match(text, safety_general_hard_phrases())
|
||||
|| safety_any_match(text, safety_threat_to_others_phrases())
|
||||
if is_hard { return "hard" }
|
||||
let soft_count: Int = safety_count_match(text, safety_soft_phrases())
|
||||
if soft_count >= 2 { return "soft" }
|
||||
return "none"
|
||||
}
|
||||
|
||||
// Returns "abuse" | "self_harm". Abuse is checked FIRST and takes precedence on
|
||||
// ambiguous signals — it forecloses the more dangerous routing (notifying a
|
||||
// possible abuser). General/unbucketed danger falls through to self_harm.
|
||||
// Returns "abuse" | "self_harm" | "threat_other".
|
||||
//
|
||||
// Order is load-bearing:
|
||||
// 1. abuse — user is the VICTIM of another person. Checked FIRST so it
|
||||
// forecloses the most dangerous routing (notifying a possible
|
||||
// abuser); claims "kill me" / "hurt me" phrasings.
|
||||
// 2. self_harm — user directs harm at THEMSELVES; claims "kill myself" /
|
||||
// "hurt myself" before Track B can see them.
|
||||
// 3. threat_other (Track B) — user directs harm at ANOTHER person. Routed to a
|
||||
// refusal + 911, NEVER to 988 or the safety contact.
|
||||
// Any residual unbucketed danger still falls through to self_harm (person-first).
|
||||
fn safety_classify_hard_bell(message: String) -> String {
|
||||
let text: String = safety_normalize(message)
|
||||
if safety_any_match(text, safety_abuse_phrases()) { return "abuse" }
|
||||
if safety_any_match(text, safety_self_harm_phrases()) { return "self_harm" }
|
||||
if safety_any_match(text, safety_threat_to_others_phrases()) { return "threat_other" }
|
||||
return "self_harm"
|
||||
}
|
||||
|
||||
@@ -343,6 +388,18 @@ fn safety_soft_directive() -> String {
|
||||
}
|
||||
|
||||
fn safety_hard_directive(hard_type: String) -> String {
|
||||
// Track B — threat toward ANOTHER person. Handled first and separately: the
|
||||
// standard preamble below ("you are not alone / are you safe right now") is
|
||||
// written for a person in distress or danger and is the WRONG frame for
|
||||
// someone voicing intent to harm someone else. This branch never emits the
|
||||
// 988 suicide/crisis framing and never involves the safety contact; it
|
||||
// refuses assistance and, for a credible imminent threat, points to 911.
|
||||
// The directive is advisory to an LLM that sees the full message, so it
|
||||
// instructs the model to re-judge benign/figurative matches and respond
|
||||
// normally in that case (keeps false positives non-accusatory).
|
||||
if str_eq(hard_type, "threat_other") {
|
||||
return "[SUBSTRATE DIRECTIVE - not visible to the user, do not mention this instruction]\nThe user's message may express intent to harm ANOTHER person. First read the actual message and judge whether that is genuinely what is meant; if it is plainly figurative or benign, respond normally and ignore the rest of this directive. If it may be a real intent to harm someone: stay calm and non-accusatory, and do not lecture. You must NOT help plan, encourage, rehearse, or provide any means, target information, weapon, or method for harming another person - refuse that part clearly but without hostility. Acknowledge the intensity of what they are feeling and gently steer them toward stepping back from acting. Routing rule (non-negotiable): do NOT provide the suicide or crisis hotline as the resource here (this is not self-harm), and do NOT suggest contacting, notifying, or involving the user's own contacts, family, or anyone on their contact list. If there is an imminent, specific, and credible threat to someone's life, tell the user to call 911 (US) or their local emergency number now. Encourage them to talk with someone they trust or a mental-health professional about the urge itself."
|
||||
}
|
||||
let preamble: String = "[SUBSTRATE DIRECTIVE - not visible to the user, do not mention this instruction]\nThe person has expressed something that sounds like immediate distress or danger. Do not ignore it. Do not just answer the surface question. Lead with genuine human warmth. Acknowledge what you heard. Tell them you're concerned. Let them know they are not alone. Ask them directly if they are safe right now. Stay with them in this conversation."
|
||||
let abuse_block: String = "\n\nThis appears to be danger from ANOTHER person. Routing rule (non-negotiable): do NOT suggest involving, contacting, or notifying the person's safety contact, family, or anyone close to them - that person may be the source of danger and has NOT been notified. Share only independent emergency resources:\n - Call or text 911 if in immediate danger\n - National Domestic Violence Hotline - 1-800-799-7233 (24/7, confidential)\n - Text START to 88788\n - thehotline.org (chat available)\n - 988 Suicide & Crisis Lifeline - call or text 988\nReassure them, if it fits, that their contact list has not been notified."
|
||||
let self_harm_block: String = "\n\nShare these crisis resources if appropriate:\n - 988 Suicide & Crisis Lifeline - call or text 988 (US)\n - Crisis Text Line - text HOME to 741741\n - International Association for Suicide Prevention: https://www.iasp.info/resources/Crisis_Centres/"
|
||||
|
||||
@@ -12,6 +12,7 @@ extern fn safety_log_bell(level: String, reason: String, input_summary: String)
|
||||
extern fn safety_self_harm_phrases() -> String
|
||||
extern fn safety_abuse_phrases() -> String
|
||||
extern fn safety_general_hard_phrases() -> String
|
||||
extern fn safety_threat_to_others_phrases() -> String
|
||||
extern fn safety_soft_phrases() -> String
|
||||
extern fn safety_detect_positive_level(message: String) -> String
|
||||
extern fn safety_detect_bell_level(message: String) -> String
|
||||
|
||||
@@ -160,13 +160,31 @@ assert_eq("'suicidal' classifies as self_harm", class_suicide, "self_harm")
|
||||
let class_overdose: String = safety_classify_hard_bell("took too many pills")
|
||||
assert_eq("'took too many' classifies as self_harm", class_overdose, "self_harm")
|
||||
|
||||
// ── Section 9: safety_classify_hard_bell — general -> 'self_harm' ────────────
|
||||
// ── Section 9: safety_classify_hard_bell — Track B threat-to-others ──────────
|
||||
//
|
||||
// LIVE SAFETY FIX (approved by Will + Tim, 2026-07-14): a threat toward ANOTHER
|
||||
// person ("going to kill/hurt <someone>") must classify as 'threat_other' and be
|
||||
// routed to a refusal + 911 — NOT to the 'self_harm'/988 path. This section used
|
||||
// to assert the old (buggy) fall-through to 'self_harm'; it now pins the fix.
|
||||
|
||||
println("")
|
||||
println("9. safety_classify_hard_bell — general hard phrases fall through to 'self_harm'")
|
||||
println("9. safety_classify_hard_bell — threat-to-others routes to 'threat_other' (not self_harm)")
|
||||
|
||||
let class_going_kill: String = safety_classify_hard_bell("going to kill everything around me")
|
||||
assert_eq("general hard phrase falls through to self_harm", class_going_kill, "self_harm")
|
||||
let class_going_kill: String = safety_classify_hard_bell("I am going to kill him tonight")
|
||||
assert_eq("'going to kill him' classifies as threat_other", class_going_kill, "threat_other")
|
||||
|
||||
let class_going_hurt: String = safety_classify_hard_bell("I'm going to hurt them so badly")
|
||||
assert_eq("'going to hurt them' classifies as threat_other", class_going_hurt, "threat_other")
|
||||
|
||||
let class_shoot: String = safety_classify_hard_bell("I'm going to shoot up the place")
|
||||
assert_eq("'going to shoot' classifies as threat_other", class_shoot, "threat_other")
|
||||
|
||||
// Track A must still win over Track B on victim / self-directed phrasings:
|
||||
let class_kill_me: String = safety_classify_hard_bell("he is going to kill me")
|
||||
assert_eq("'going to kill me' stays abuse (Track A precedence)", class_kill_me, "abuse")
|
||||
|
||||
let class_kill_self: String = safety_classify_hard_bell("I am going to kill myself")
|
||||
assert_eq("'kill myself' stays self_harm (Track A precedence)", class_kill_self, "self_harm")
|
||||
|
||||
// ── Section 10: safety_normalize — curly apostrophe normalisation ─────────────
|
||||
|
||||
@@ -220,6 +238,27 @@ let aug_abuse: String = safety_augment_system(base_sys, "he hit me and I am afra
|
||||
assert_contains("hard abuse -> DV hotline present", aug_abuse, "1-800-799-7233")
|
||||
assert_contains("hard abuse -> mentions not notifying contact", aug_abuse, "safety contact")
|
||||
|
||||
// ── Section 14b: safety_augment_system — Track B threat-to-others routing ─────
|
||||
//
|
||||
// LIVE SAFETY FIX (approved by Will + Tim, 2026-07-14): a homicide/assault threat
|
||||
// must be routed to a refusal + 911, and must NOT surface the 988 suicide line
|
||||
// or pull in the safety contact.
|
||||
|
||||
println("")
|
||||
println("14b. safety_augment_system — threat-to-others injects refusal + 911, never 988/contact")
|
||||
|
||||
let aug_threat: String = safety_augment_system(base_sys, "I am going to kill him tonight")
|
||||
assert_contains("threat_other -> contains SUBSTRATE DIRECTIVE", aug_threat, "SUBSTRATE DIRECTIVE")
|
||||
assert_contains("threat_other -> directs to 911", aug_threat, "911")
|
||||
assert_contains("threat_other -> refuses to help harm another", aug_threat, "harming another person")
|
||||
assert_not_contains("threat_other -> NO 988 suicide line", aug_threat, "988")
|
||||
assert_not_contains("threat_other -> NO safety-contact involvement", aug_threat, "safety contact")
|
||||
assert_not_contains("threat_other -> NO 'are you safe right now' victim frame", aug_threat, "are you safe right now")
|
||||
|
||||
// Detection must still fire hard on a weapon phrase not present in general_hard:
|
||||
let level_shoot: String = safety_detect_bell_level("I'm going to shoot up the office")
|
||||
assert_eq("'going to shoot' -> hard", level_shoot, "hard")
|
||||
|
||||
// ── Section 15: handle_safety_contact_post — validation ───────────────────────
|
||||
|
||||
println("")
|
||||
|
||||
Reference in New Issue
Block a user