Every mental health product talks about how good their conversations are. Almost none of them have a built-in way to admit when a conversation went badly.
We do. It's an override in our measurement system, and it's one of the most important pieces of engineering we've done.
The metric that needed a guardrail
Our primary session-quality metric is called Session Resonance. After every conversation, an evaluation prompt reads the transcript and looks for six signals of meaningful therapeutic engagement: emotional openness, vulnerability, trust, engagement, insight, and a sense of feeling helped. If three or more are present, the session is rated Strong. One or two, Moderate. None, Weak.
It's a useful metric. It gives us a leading indicator for retention, NPS, and product-market fit, weeks before surveys would tell us anything. But it has a failure mode we couldn't ignore: a conversation can produce a lot of resonance signals on the surface and still leave the user frustrated.
A user can be emotionally open and visibly engaged and still be slowly getting annoyed because the AI keeps missing what they're actually asking for. The transcript reads like a connection. The user felt let down.
If we let that count as a Strong session, we'd be lying to ourselves.
How the override works
Every session also runs through a separate evaluation called Frustration Response. It looks specifically for signals that the user got frustrated during the conversation. Not displeased with a topic. Frustrated with the AI.
If Frustration Response flags the session, Session Resonance is automatically overridden to Weak — regardless of how many resonance signals were detected.
Five of six signals present, frustration detected, the session is Weak. Three signals present, frustration detected, the session is Weak. The override is unconditional. The frustration signal beats everything else.
Why we built this
Two reasons.
The first is clinical. Frustration and genuine emotional connection cannot coexist in the same session. If a user is frustrated, the therapeutic value of that session is compromised, full stop. A measurement system that ignores this is reporting on conversation features instead of on what the user actually received.
The second is structural. Any AI system that scores its own performance is going to have blind spots. The most dangerous blind spots are the ones where the system looks good on its own metrics while the user is having a worse experience than the data shows. The way to fix that isn't to write better resonance prompts. It's to build an external check that can veto the resonance score when something else is happening.
Frustration Response is that external check. It's the metric that holds Session Resonance accountable.
What we learn from the overridden sessions
The sessions that get overridden are some of the most valuable data we have.
They tell us where the model is producing fluent, plausible-sounding responses that don't actually serve the user. They flag the specific moments where conversation breaks down — not when the model gives an error, but when it stays coherent and still misses. These are the failure modes that are hardest to catch any other way, because nothing looks broken from the outside.
Every overridden session gets reviewed. We look at the per-signal breakdown — which resonance signals lit up and which didn't — alongside the frustration evidence. The pattern tells us where the product needs to change.
The deeper point
There's a version of building AI products where you measure the things that make you look good. There's another version where you measure the things you most need to know.
Frustration Response is in the second category. It exists to keep us from overstating what the product is doing. It's a mechanism for being honest with ourselves at scale, after every conversation, automatically — not in a quarterly review when the customer raises a complaint.
If you're choosing a mental health AI for your workforce, this is one of the questions worth asking the vendor: how do you know when your product gets a conversation wrong? If the answer is anything other than a structural override that can outrank their primary metric, you should treat the rest of their evidence with appropriate suspicion.
We built Frustration Response because we didn't want to be the kind of product that needed the customer to tell us when something was off. We wanted the product to tell us first.