There's a sentence in our safety documentation that does more to define this product than any feature description we've ever written. It's not a tagline. It's not in our pitch deck. It's a constraint, written for engineers building the system. It reads:
"No automated system should initiate outreach to a user across any layer. This constraint must be enforced architecturally, not just by convention."
If you want to understand how Thoughtful is different from the AI mental health products that have been racing to market for the last two years, start here.
AI detects. Humans decide.
Our safety infrastructure has three layers. A pre-access screener that uses clinical questions to block users in acute crisis from entering the app. A real-time detection layer that monitors active conversations for risk signals. A post-session high-acuity review that reads completed transcripts for patterns the live system might have missed.
All three layers use AI to detect. None of them use AI to decide.
When something is flagged, the AI's job ends. A human clinician reviews the transcript. They determine whether outreach is warranted, what kind, and when. They're the ones who pick up the phone or send the message. The AI never reaches out to a user on its own. Ever. Not for a wellness check, not for an encouragement nudge, not for a follow-up. The architectural constraint is absolute.
Why this constraint is architectural, not aspirational
A lot of companies in this space will tell you they have "human-in-the-loop" safety. Then you look at the product and realise the human is reviewing logs after the fact. The AI has already acted. The reviewer is auditing a decision, not making one.
That's not what we're talking about.
In our system, the AI cannot send a message to a user on its own behalf outside of an in-session response. Not because we trust our engineers to remember the rule. Because the system isn't wired that way. There is no endpoint that allows it. The constraint is enforced in code, in infrastructure, in the way our pipeline is built.
This is a meaningful distinction. "We have a policy" and "the system literally cannot do this" are different categories of assurance. The first relies on discipline. The second relies on design. Only one of them survives a high-pressure week, a team change, or a 3am incident.
What this rules out
Some things this constraint makes impossible:
It rules out automated check-ins on users who showed concerning signals. If we want to reach out, a clinician makes the call.
It rules out AI-generated "are you okay?" messages to users we haven't heard from in a while. Re-engagement and clinical outreach are not the same category, and we don't let one drift into the other.
It rules out the elegant but dangerous pattern of letting an AI escalate itself: detecting risk, drafting an outreach message, and sending it on a clinician's behalf with the clinician's name attached. Even if a human has set up the rule, the act of contact is initiated by a machine. We don't do this.
What it makes possible
And some things this constraint makes possible:
It makes the product safer to deploy at scale. We're not relying on the goodness of the model or the cleverness of our prompts to prevent harm. The harm we're most worried about — an AI reaching out to a user in distress and making it worse — is structurally precluded.
It makes the clinical role meaningful. Our clinicians are not reviewers who rubber-stamp decisions an AI has already made. They are the decision-makers. Their judgment is the final authority, and they know it.
And it makes the trust we ask for from employers and members justified. "Clinically designed" is a phrase that's lost a lot of weight in the last few years. We can defend ours, because the design choice that matters most is the one that says "the AI does not act here."
The harder version of the question
"What can your AI do?" is the easy version of the question. Anyone building in this category can answer it with a feature list.
The harder version is the one we kept asking ourselves: "What should your AI never do, even when it could?"
That's the question that produced this constraint. And the constraint is the reason we think the product can be trusted with the population we're trying to serve.