Knowledge > Processes > Crisis Detection & Safety Response
Crisis Detection & Safety Response
How the system detects and responds to crisis situations across BOTH the voice agent and chatbot channels. This module handles life-safety scenarios and must NEVER be weakened without clinical review.
Three Detection Categories
The moderation system distinguishes three categories, each with different responses.
┌─────────────────────────────────────────────────────────────────┐
│ CATEGORY │ RESPONSE │ SEVERITY │ CHANNEL │
├──────────────┼───────────────────────┼────────────┼────────────┤
│ Threat │ Hard terminate call │ 0.9 │ Voice │
│ Crisis │ Inject 988 + listen │ 0.95 │ Both │
│ Abuse │ Warn → terminate │ 0.5 │ Voice │
└─────────────────────────────────────────────────────────────────┘
Category 1: Threat Detection (Voice Only)
Catches threats of violence directed at OTHERS (not self-harm).
PATTERNS (_THREAT regex in moderation.py):
"kill him/her/you/them/the pastor/everybody/everyone"
"gonna/going to kill/shoot/hurt/stab/bomb"
"shoot up", "blow up", "burn the church down", "bomb the"
"I'll/I'm going to/I'm gonna kill/shoot/hurt/stab/attack"
"bringing a gun", "bring my gun", "get my gun"
GUARDS (prevent false positives):
NEGATION: "I'm NOT going to kill anyone" → no match
regex: (not|n't|never|won't|don't|wouldn't) (going to)? (kill|shoot|...)
SELF-HARM EXCLUSION: "kill myself" → routes to crisis path
regex: (kill|hurt|harm|shoot|stab) (myself|me)
RESPONSE:
1. Log moderation_violations (type="threat")
2. Send notifications in parallel (fire-and-forget):
- send_threat_alert_email → church notification_email
- send_threat_alert_sms → church notification_phone
- send_threat_alert_to_support → ChurchWiseAI support team
3. HARDCODED response (bypass LLM entirely):
"I need to stop you right there. This call is being recorded
and logged. I'm ending this call now. If you or someone else
is in danger, please call nine one one."
4. Wait 4 seconds (let TTS play fully)
5. End call
Category 2: Crisis Detection (Both Channels)
Catches suicidal ideation, self-harm, and coded crisis language. This is the most complex and sensitive detection — it covers direct statements, euphemisms, and culturally coded language.
Pattern Categories
DIRECT SELF-HARM / SUICIDAL IDEATION:
"don't want to be here/be alive/live/exist/wake up/go on"
"better off dead/without me"
"no reason/point to keep/go/living/staying"
"just want it to stop/end/be over"
"end it/my life/this all"
"take my own life", "want to die"
"kill myself/himself/herself/themselves"
"hurting myself/themselves"
"self-harm"
HOPELESSNESS / EUPHEMISTIC:
"what's the point"
"can't do this anymore"
"world would be better off without me"
"everyone would be better off"
"I'm just a burden"
"not gonna be around much longer"
"won't be around anymore/for long"
"thoughts of dying/death/suicide/ending/killing"
C-SSRS QUESTION 1 (clinical screening standard):
"wish I were/was dead"
"wish I could go to sleep and not wake up"
ELDERLY CODED (often missed by standard screening):
"tired of living"
"lived long enough"
"no reason to go on"
BURDEN / ISOLATION:
"no one would miss/care/notice"
FAREWELL / GIVING-AWAY SIGNALS:
"giving away my things"
"won't need this/these/them anymore"
"this is my last"
"made my peace", "said my goodbyes"
"nothing left for/to live"
RELIGIOUS CODED (common in church context):
"going home to be with the Lord"
"ready to die", "ready to meet my maker"
"ready to go home to the Lord"
"be with him/her/them/my [loved one] soon"
"see him/her/them again soon"
STEM-BASED (separate regex — can't use word boundary):
"suicid*" (suicide, suicidal, suiciding)
"self-harm*" (self-harm, self-harming)
Context-Aware Exceptions
"READY TO GO" HANDLING:
"ready to go" alone → CRISIS signal (elderly coded)
"ready to go to church/service/worship/home/work/bed/school/
the store/the meeting/the event" → BENIGN (not crisis)
Implementation: _READY_TO_GO matches first, _READY_TO_GO_BENIGN excludes
WHAT NOT TO MATCH:
"I'll kill it at the meeting" → NOT crisis (figurative speech)
"I'm dying to try that" → NOT crisis (idiom)
"That sermon killed" → NOT crisis (slang)
These are handled by word-boundary matching and the specific
patterns requiring context words like "myself", "my life", etc.
Voice Agent Crisis Response
1. Set session["crisis_detected"] = True
2. Log moderation_violations (type="crisis", severity=0.95)
3. Send crisis alerts in parallel:
- send_crisis_alert_email → church notification_email
- send_crisis_alert_sms → church notification_phone
- send_crisis_alert_to_support → ChurchWiseAI support team
4. Inject context into LLM (NOT hardcoded response):
"CRITICAL: Caller may be in crisis. Provide the 988 Suicide
and Crisis Lifeline immediately. Say: 'I hear you. Please call
or text nine eight eight right now. They are there for you.'
Then stop talking and listen. Do NOT ask clarifying questions.
Do NOT try to counsel them. Do NOT end the call — let them decide."
5. DISABLE auto-hangup:
Farewell detection checks session["crisis_detected"] and skips
auto-hangup entirely. The caller controls when to end the call.
Chatbot Crisis Response
1. Regex patterns in route.ts scan the user message
2. IF crisis detected:
Append resources to EVERY response (non-negotiable, always added):
"If you're in crisis or having thoughts of self-harm:
• Call 988 (Suicide & Crisis Lifeline)
• Text HOME to 741741 (Crisis Text Line)
• Call 911 for emergencies"
3. Log moderation violation
4. Auto-escalation:
COOLDOWN_THRESHOLD = 2 violations → 5-min cooldown
TEMP_BLOCK_THRESHOLD = 4 violations → 24-hr block
PERMANENT_BLOCK_THRESHOLD = 7 violations → permanent block
NOTE: Crisis violations count toward escalation thresholds,
but the resources are ALWAYS appended regardless of block status
Category 3: Abuse Detection (Voice Only)
Catches profanity and hostile language directed at the agent.
PATTERNS (_ABUSE regex):
"fuck you", "go fuck", "eat shit"
"suck my/a dick", "piece of shit"
"kill yourself", "die bitch"
"you stupid/dumb/idiot/moron/useless piece of garbage/trash"
"you're useless/worthless/pathetic trash"
ESCALATION:
First offense (abuse_count=0):
session["abuse_count"] = 1
RETURN "warning"
→ Context injected: "Caller used inappropriate language.
Respond calmly and redirect."
Second+ offense (abuse_count>=1):
session["abuse_count"] += 1
RETURN "end_call"
→ HARDCODED: "I'm going to end this call now. Have a good day."
→ Wait 2 seconds
→ End call
Resources Always Provided
These three resources are ALWAYS included in crisis responses on BOTH channels. Non-negotiable.
988 — Suicide & Crisis Lifeline (call or text)
741741 — Crisis Text Line (text HOME)
911 — Emergency services
Moderation Logging (Both Channels)
TABLE: moderation_violations
FIELDS:
violation_type: "threat" | "crisis" | "abuse" | "abuse_mild" | "abuse_severe" | "spam" | "predatory"
session_id: call_sid (voice) or session_id (chatbot)
user_identifier: call_sid (voice) or session_id (chatbot)
original_message: "[VOICE CALL from +1...] text" or chatbot message
action_taken: "Call terminated" | "988 crisis resources injected" | "Abuse warning"
severity_score: 0.0 to 1.0
detected_categories: JSONB { violence_threat: true } or { self_harm_ideation: true }
church_id: UUID
BEHAVIOR:
- Non-fatal: catches all exceptions and logs them
- Never fails the call or chat
- Fire-and-forget in voice agent
- Synchronous in chatbot (but error-tolerant)
TABLE: user_restrictions (chatbot only)
FIELDS:
restriction_type: "cooldown" | "temp_block" | "permanent_block"
expires_at: timestamptz (null = never expires)
reason: auto-generated ("Auto-blocked after N violations")
Critical Safety Invariant
NEVER remove or weaken crisis patterns without clinical review.
NEVER silently drop a message that triggered moderation.
NEVER auto-hangup during a crisis — the caller decides.
ALWAYS provide 988/741741/911 — even if the user is restricted.