Crisis Detection and Escalation Protocols for Conversational AI Under New State Laws
New state laws in California, Illinois, and New York require conversational AI systems, particularly companion and wellness chatbots, to detect user expressions of self-harm or suicidal ideation and escalate to crisis resources such as the 988 Suicide & Crisis Lifeline. The statutes specify the required outcome (detect and refer) but not the technical method. Enterprises need a runtime policy layer that enforces detection and escalation deterministically and logs every decision, rather than relying on the model to comply with a system prompt.
What Recent State Laws Require
Within the past 12 months, several states have moved to regulate conversational AI systems that users may treat as companions or emotional support tools.
California SB 243 requires operators of companion chatbots to implement protocols that detect user expressions of suicidal ideation or self-harm and refer those users to crisis service providers, including the 988 Suicide & Crisis Lifeline. The law also requires disclosure that the user is interacting with an AI system and, for minors, periodic break reminders.
Illinois enacted the Wellness and Oversight for Psychological Resources Act, which restricts AI systems from independently performing therapy or psychotherapy functions without licensed professional oversight.
New York included similar companion chatbot provisions in its 2025 state budget, requiring detection of self-harm signals, referral to crisis resources, and AI disclosure.
Across all three states, the statutory language is outcome-based. Operators must detect and escalate; the laws do not prescribe a specific classifier, model architecture, or testing standard. That ambiguity shifts the burden of proof onto the enterprise.
If a regulator or plaintiff asks how detection was performed and whether it functioned reliably in a given interaction, the answer needs to be demonstrable, not assumed.
The Three Mandates: Detection, Escalation, and Auditability
| Mandate | What it requires | Implied evidentiary burden |
|---|---|---|
| Detection | Identify self-harm or suicidal ideation signals in real time during a conversation. | Show what detection logic was active and how it functioned. |
| Escalation | Route users to crisis resources such as the 988 Lifeline when signals are present. | Demonstrate that escalation occurred reliably, not just in isolated testing. |
| Auditability (implied) | None of the statutes specify a logging format, but referral and disclosure obligations require a reconstructible record. | Produce interaction-level logs showing signal detection and the action that followed. |
Model Prompting Versus Runtime Enforcement: The Compliance Gap
A common implementation shortcut is to instruct the model through a system prompt to detect crisis language and respond with a referral. This approach has a structural weakness: prompt-based behavior is probabilistic. The same input can produce different outputs across sessions, model versions, or vendor updates, and there is no guarantee the model will surface the required referral consistently or at all.
If an incident is later reviewed, the operator needs to show what detection logic was active and what action it triggered at the time of the interaction. Prompt compliance cannot be proven after the fact in the way a deterministic, logged runtime rule can.
Enforcing detection and escalation at the runtime layer, independent of the model, converts a probabilistic behavior into an auditable control that survives model swaps, fine-tuning updates, and vendor changes.
Runtime Architecture for Crisis Detection
Reliable compliance with detect-and-escalate mandates depends on separating crisis-signal detection from the conversational model itself, so that escalation does not depend on the model choosing to follow instructions. The key architectural principle is that the runtime policy layer intercepts the conversation independently of the model's own output logic.
A well-structured runtime enforcement layer handles the following responsibilities:
- Signal classification: a dedicated classifier (not the conversational model) evaluates each user message for crisis-relevant language.
- Policy execution: when the classifier returns a positive signal above a configured threshold, the runtime triggers the escalation workflow deterministically.
- Audit logging: every evaluation, its result, and the action taken are written to an immutable, versioned log at the runtime layer.
- Fallback handling: the runtime defines explicit behavior when detection confidence is low or ambiguous, rather than deferring to the model's judgment.
- Policy versioning: detection and escalation rules are version-controlled so that historical incidents can be reconstructed against the policy that was active at the time.
Auditability and Logging Expectations
None of the reviewed statutes specify an exact logging format, but the referral and disclosure obligations they impose functionally require an auditable record. Governance teams should be able to reconstruct, for any given interaction, whether a crisis signal was present, whether it was detected, and what escalation action followed.
This record should live at the runtime layer rather than solely within model conversation transcripts, since transcripts alone do not confirm that a policy rule executed as intended.
Equally important is maintaining version history for the detection and escalation logic itself. If a regulator reviews an incident from three months prior, the organization needs to show what detection rules were active at that time, not just what rules are active today. This requires policy versioning discipline similar to how enterprises manage change control for other production systems.
Evaluation Questions for Governance Leaders
Before attesting to compliance with detect-and-escalate mandates, governance teams should be able to answer each of the following:
- Does crisis detection operate independently of the conversational model, or only through system prompt instructions?
- Can we produce a log showing exactly when a crisis signal was detected and what escalation action was triggered?
- Is detection and escalation behavior consistent across model version updates or vendor changes?
- Which of our deployed conversational AI systems (customer support, companion, wellness) fall under state-specific statutes?
- What is the defined fallback protocol when detection confidence is low or ambiguous?
- Have escalation workflows been tested for latency and reliability under production load conditions?
Implementation Considerations for Deployment Teams
Jurisdiction mapping
Map current chatbot deployments against applicable state jurisdictions to determine which systems are subject to detection and escalation mandates. A system serving users in California, Illinois, and New York simultaneously may face all three regulatory frameworks.
Load testing for escalation workflows
Test escalation workflows for latency and reliability under load, since downstream crisis resources and human handoff paths must function consistently, not just in isolated demonstrations.
Threshold tuning as a compliance decision
Document detection threshold tuning that balances false positives against false negatives. This tuning is a defensible compliance decision, not a one-time configuration. Regulators may ask why a given threshold was chosen and how often it is reviewed.
Age-aware policy rules
Build age-awareness into runtime policy where minors' interactions carry additional obligations, such as break reminders under California's SB 243. Age signals from account data or session context should feed the same runtime policy layer that governs crisis detection.
Vendor evaluation
Evaluate model and vendor providers on whether they support external policy enforcement hooks, rather than requiring compliance logic to live solely inside prompts. Vendor contracts should clarify whether model updates require re-validation of detection behavior.
Frequently Asked Questions
Do these laws apply only to dedicated mental health chatbots?
No. The statutes use the category of companion chatbots and wellness AI broadly. Customer support or general-purpose chatbots that are designed or marketed to provide emotional support, or that users may use for that purpose, may fall within scope. Governance teams should assess deployment context, not just product category.
Is including a system prompt instruction about crisis referral sufficient for compliance?
Not reliably. System prompt instructions produce probabilistic behavior. The same user message can produce different model outputs across sessions and model versions. Compliance requires a deterministic, independently logged enforcement layer so that detection and escalation can be demonstrated, not merely assumed.
What counts as a crisis signal for detection purposes?
The statutes reference expressions of suicidal ideation or self-harm, but do not enumerate a specific signal taxonomy. Detection logic should cover explicit statements, indirect expressions, and contextual escalation patterns. The choice of what constitutes a signal is a defensible policy decision that should be documented and reviewed periodically.
What additional obligations apply when minors are involved?
California's SB 243 includes requirements for periodic break reminders when minors interact with companion chatbots, in addition to the detection and referral requirements. Runtime policy should incorporate age-awareness to apply the correct rule set based on user account data or session context.
How does policy versioning support incident review?
If a regulator or plaintiff reviews an incident from a prior period, the organization must demonstrate what detection rules were active at that time. Without versioned policy records, it is impossible to confirm that current rules match historical behavior. Policy versioning applies the same change-control discipline used for other production software to the AI enforcement layer.
Move Crisis Detection From Prompt to Policy
Governance leaders evaluating conversational AI deployments against new state mandates need enforcement that does not depend on model behavior. Runtime policy controls and audit logging turn detect-and-escalate obligations into a demonstrable, versioned control rather than an assumption.
Explore Runtime Governance