Crisis Detection and Escalation Protocols for Conversational AI Under New State Laws

What Recent State Laws Require

Within the past twelve months, several states have moved to regulate conversational AI systems that users may treat as companions or emotional support tools. The statutes differ in scope but share a common core: detect signals of self-harm and route the user to help.

Across all three, the statutory language is outcome-based. Operators must detect and escalate; the laws do not prescribe a specific classifier, model architecture, or testing standard. That ambiguity shifts the burden of proof onto the enterprise: if a regulator or plaintiff asks how detection was performed and whether it functioned reliably in a given interaction, the answer needs to be demonstrable, not assumed.

Runtime Architecture for Crisis Detection and Escalation

Reliable compliance with detect-and-escalate mandates depends on separating crisis-signal detection from the conversational model itself, so that escalation does not depend on the model choosing to follow instructions.

Model Prompting Versus Runtime Enforcement: The Compliance Gap

A common implementation shortcut is to instruct the model, through a system prompt, to detect crisis language and respond with a referral. This approach has a structural weakness: prompt-based behavior is probabilistic. The same input can produce different outputs across sessions, model versions, or vendor updates, and there is no guarantee the model will surface the required referral consistently, or at all.

This matters for compliance specifically because these laws imply an evidentiary standard. If an incident is later reviewed, the operator needs to show what detection logic was active and what action it triggered at the time of the interaction. Prompt compliance cannot be proven after the fact in the way a deterministic, logged runtime rule can. Enforcing detection and escalation at the runtime layer, independent of the model, converts a probabilistic behavior into an auditable control that survives model swaps, fine-tuning updates, and vendor changes.

Evaluation Questions for Governance Leaders

• Does crisis detection operate independently of the conversational model, or only through system prompt instructions?
• Can we produce a log showing exactly when a crisis signal was detected and what escalation action was triggered?
• Is detection and escalation behavior consistent across model version updates or vendor changes?
• Which of our deployed conversational AI systems (customer support, companion, wellness) fall under state-specific statutes?
• What is the defined fallback protocol when detection confidence is low or ambiguous?

Auditability and Logging Expectations

None of the reviewed statutes specify an exact logging format, but the referral and disclosure obligations they impose functionally require an auditable record. Governance teams should be able to reconstruct, for any given interaction, whether a crisis signal was present, whether it was detected, and what escalation action followed. This record should live at the runtime layer rather than solely within model conversation transcripts, since transcripts alone do not confirm that a policy rule executed as intended.

Equally important is maintaining version history for the detection and escalation logic itself. If a regulator reviews an incident from three months prior, the organization needs to show what detection rules were active at that time, not just what rules are active today. This requires policy versioning discipline similar to how enterprises manage change control for other production systems.

Implementation Considerations for Deployment Teams

• Map current chatbot deployments against applicable state jurisdictions to determine which systems are subject to detection and escalation mandates.
• Test escalation workflows for latency and reliability under load, since downstream crisis resources and human handoff paths must function consistently, not just in isolated demos.
• Document detection threshold tuning that balances false positives against false negatives, since this tuning is a defensible compliance decision, not a one-time setting.
• Build age-awareness into runtime policy where minors' interactions carry additional obligations, such as break reminders under California's SB 243.
• Evaluate model and vendor providers on whether they support external policy enforcement hooks, rather than requiring compliance logic to live solely inside prompts.

Detect, Escalate, Prove It