There's a version of crisis detection that works like this. A user types something into a chat. The system runs the message through a list of flagged phrases. If a phrase matches, the system escalates.
It's a reasonable starting point — and for some classes of harm, a keyword pass adds real value as one layer of a broader system. But on its own, in a mental health context, it isn't enough. We'll explain why, and what we built alongside it.
Why keyword filters break
Keyword detection has a few problems that aren't fixable by improving the list.
The first is that distress doesn't always announce itself with the words on your list. A user describing exhaustion, hopelessness, and a feeling that nothing will change has surfaced a serious clinical signal. None of the words in that sentence will be on a crisis keyword list.
The second is the inverse: the words on your list will appear in conversations that aren't about crisis. Someone discussing a book about depression. Someone joking about how they're 'going to die' if they don't get coffee. Someone talking about a friend's situation in the third person. A keyword filter cannot tell these apart from a real disclosure, so it either over-escalates (and trains everyone, including the clinicians, to ignore alerts) or under-escalates (and misses real ones by being tuned down).
The third is that the most clinically meaningful disclosures are often the ones constructed carefully. A user who has thought a lot about what they're feeling will choose their words. They will use language that lets them say what they mean without saying it directly. 'I don't see the point anymore' is one of those sentences. A keyword filter does not understand it.
Context, not match
Our crisis detection is LLM-based and operates on the conversation, not on individual messages. It looks at what's being said, what's been said before, and the emotional shape of the whole exchange — and it's calibrated to assess context and pattern, not just match keywords.
This matters in three specific ways.
It catches indirect disclosures. A user expressing hopelessness without explicit language is identified as an ambiguous signal — not as nothing. A user describing escalating distress across a few messages is recognised as a pattern, not as individual lines that didn't trigger anything alone.
It reduces over-escalation. Keyword lists generate enormous amounts of noise. Context-aware detection doesn't fire on a quote, a third-person reference, or a joke. The signal is cleaner, which means when it does fire, the clinical team can take it seriously.
And it allows the system to ask before escalating. When a signal is ambiguous, the system has the option to surface a clarifying question in the conversation itself — 'can you tell me more about what you mean by that?' — and re-evaluate based on the response. This is something a keyword filter cannot do. It either fires or it doesn't.
Detection is not decision
Worth being clear here, because it matters.
LLM-based detection is what surfaces a potential signal. It does not decide what happens next. Every detected signal is reviewed by a human clinician, who makes the call on whether outreach is warranted and what kind. The AI's job is to flag with high recall and reasonable precision. The clinician's job is to apply judgment.
The reason we say context-aware detection is better than keyword detection is not that it's a better decision-maker. It's that it produces a better feed for the human decision-maker. Fewer false positives in the feed means clinicians can pay real attention to the alerts they get. More true positives means fewer real signals get missed.
This is the part that doesn't fit into a product comparison chart. It's not a feature, it's a property of how the system is built.
The bar we set for ourselves
When we built our crisis detection, the question we kept asking wasn't 'will this fire on the obvious cases?' Most systems will. The question was 'how does this handle the message that contains no crisis keywords but still conveys risk, and how does it handle the message that contains crisis keywords but in a non-risk context?'
Those two cases are where context-aware detection earns its place. A user saying 'I don't see the point anymore' without any flagged words. A user quoting a passage from a book they're reading that's full of them. A keyword filter cannot reliably tell these apart. Our system has to.
If you're evaluating mental health AI for your workforce, those are also the cases worth asking about specifically. Not 'do you detect crisis?' — every vendor says yes. The useful question is how a detection system handles signal in the absence of expected words, and absence of signal in the presence of them.
Distress doesn't always come on a keyword list. The safety system has to understand that, or it isn't doing the work the category needs it to do.