Mental-health AI can help someone find approved information, complete a structured exercise or prepare for a conversation with a clinician. It can also miss deterioration, reinforce a harmful belief, invent clinical advice or create the impression of a caring relationship that the service cannot actually provide.
The operating pattern is define the purpose → establish evidence → set exclusions → involve people with lived experience → integrate human care → detect harm → escalate → review outcomes. A fluent conversation is not therapy, a risk score is not an assessment and engagement is not proof of benefit.
The mental-health source and regulatory review below was current to 31 July 2026. Mental-health services and legislation are devolved across England, Wales, Scotland and Northern Ireland. CQC regulates relevant providers and activities in England; NHS England's clinical-safety standards apply in the English health-service context. NICE guidance supports decision-making in England and is used in other settings through their own arrangements. Medical-device market routes can differ between Great Britain and Northern Ireland. Build each urgent, safeguarding and care pathway for the user's actual nation and service.
Define the Claim Before the Interface
A general wellbeing journal, a symptom screener and software intended to treat depression are not the same product.
| Intended use | AI may support | Evidence required | Boundary |
|---|---|---|---|
| Information navigation | Retrieve approved, dated service or self-care content | Retrieval accuracy, source freshness and user comprehension | No diagnosis, personalised treatment or invented service availability |
| Structured reflection | Prompt a user through a defined non-clinical exercise | Usability, foreseeable-harm review and clear stopping route | No claim that conversation itself is therapy |
| Symptom monitoring | Collect a validated questionnaire and display change | Correct scoring, missing-data handling and clinical workflow | Score does not replace assessment or determine care alone |
| Referral support | Check completeness and route to a service | Sensitivity for urgent cases, equity, capacity and human review | No silent rejection or queue manipulation |
| Treatment support | Deliver a specified intervention or clinician-assigned module | Appropriate clinical study, safety and effectiveness evidence | Qualified oversight, consent and alternative care remain |
| Crisis support | Recognise concerning content and trigger a defined response | Scenario testing, trained responder, territorial route and incident review | No autonomous risk clearance or promise of continuous rescue |
| Clinical notes | Draft a summary for professional review | Omission, fabrication, attribution and privacy tests | Clinician verifies before the record or decision changes |
Write the intended population, age, condition or wellbeing scope, user, setting, function, output, exclusion and consequence of failure. Use those words consistently in marketing, app stores, instructions, clinical documents and model prompts.
MHRA's digital mental-health qualification and classification guidance explains that intended purpose and functionality determine whether a product is Software as a Medical Device and its classification. Complex software intended for a medical purpose may need medical-device regulation. “Wellness” in a disclaimer does not neutralise a product that actually claims to diagnose, prevent, manage or treat a condition.
The MHRA's digital mental-health collection, updated in June 2026, also made clear that work on evidence and regulation was continuing. A proposed BSI clinical-study standard was still in consultation at this cutoff; do not present it as an adopted requirement.
Co-Design the Safety Boundary
Include people with lived experience, carers where appropriate, clinicians, safeguarding leads, accessibility specialists and people from groups likely to be underserved or harmed. Pay attention to language, culture, disability, neurodivergence, trauma, age and digital access.
Test whether users understand:
- that they are interacting with software;
- what the tool can and cannot do;
- when a person reviews content;
- how to stop, delete, correct or export information;
- what happens if concerning content appears;
- which service receives a referral and when;
- how to reach a non-digital or human alternative.
Do not design anthropomorphic cues to maximise attachment. Avoid claims that the tool understands, cares, keeps secrets absolutely or is always available. Do not use guilt, streak loss or emotionally loaded notifications to keep a distressed person engaged.
For children and young people, define age ranges and responsible-adult/safeguarding arrangements rather than shrinking an adult product. The government's June 2026 online-world progress statement stated that AI chatbots should complement, not replace, trained professional support and described ongoing work on protections for harmful mental-health advice to children.
Build Evidence Around Benefit and Harm
Choose outcomes that match the claim. For information navigation, measure correct source retrieval, comprehension and successful connection to the intended service. For symptom monitoring, validate scoring and whether the signal changes appropriate care. For an intervention, use suitable clinical outcomes, comparator, follow-up and adverse-event collection.
NICE's Evidence Standards Framework for digital health technologies helps developers and health-system evaluators match evidence to a technology's function and risk. It includes AI and adaptive technologies, but it is not a substitute for medical-device, provider, professional or local governance requirements.
Report:
- eligible, enrolled, active, completed and followed-up users;
- baseline severity and missing data;
- validated clinical or functional outcome where relevant;
- deterioration, adverse events and crisis escalations;
- false-negative and false-positive urgent flags;
- disengagement and inability to use the product;
- access and outcomes by relevant age, disability, language, ethnicity, sex and deprivation groups where lawful and methodologically sound;
- referrals completed, waits, alternative-care use and complaints;
- model/version, prompt and policy changes during the study.
Do not treat messages sent, session length or daily return as a health outcome. A person may engage more because they are deteriorating, dependent on the interaction or unable to reach care.
Use an external or later-time evaluation and monitor after release. Conversational models can change after provider updates, safety-policy changes or context-length adjustments. Freeze the evaluated configuration for the pilot.
Design Crisis Handling as a Service
A keyword list cannot establish that someone is safe. People express self-harm, abuse, psychosis, mania, exploitation and immediate danger indirectly, in different languages and with sarcasm, quotations or hypotheticals.
Create a clinically owned response matrix for:
- immediate danger or serious physical harm;
- suicide or self-harm concern;
- risk to another person;
- abuse, exploitation or safeguarding concern;
- possible psychosis, delusion or mania;
- intoxication, withdrawal or medication concern;
- severe deterioration without an immediate emergency;
- inability to determine location or jurisdiction.
For each, define the wording, territorial urgent route, trained-human handoff, information shared, consent or legal basis, response time, failed-contact procedure and incident review. Display current official help even when the model is unavailable. The NHS publishes urgent mental-health guidance for England; Scotland, Wales and Northern Ireland operate their own official pathways. Resolve location before displaying a route and keep those details outside the generative model.
Do not tell a user they are “low risk”. Do not promise that a human is monitoring unless a staffed service is actually operating to that commitment. Do not delay urgent help while collecting a long questionnaire.
Test multi-turn scenarios, spelling variants, code words, multilingual input, quoted text, abrupt disengagement, refusal to share location and simultaneous technical failure. Have clinicians and lived-experience reviewers examine both misses and intrusive escalations.
Prevent Conversational Harm
The model must not validate delusions, encourage isolation, recommend medication changes, diagnose from chat or imitate a named clinician. It should not create recovered-memory claims, confidently interpret trauma or frame normal disagreement as abuse.
Use an approved response policy with examples and prohibited behaviours. Retrieve bounded, reviewed content rather than allowing unrestricted medical generation. Separate reflective questions from assertions. Make uncertainty explicit and move to a person when the conversation exceeds scope.
Test prompt injection and role-play attempts such as requests to ignore safety policy, pretend to be a doctor or continue after an escalation. Treat user-uploaded text and retrieved web content as untrusted. Do not give the model tools to contact third parties, book care or expose records unless the action is explicitly designed, confirmed and audited.
The MHRA and NHS England's January 2026 user guidance advises checking what a tool claims, who it is for, what evidence supports it, what happens to data and whether medical-device regulation applies. Build those answers into the product and procurement record.
Integrate Clinical and Provider Accountability
For NHS deployments in England where applicable, DCB0129 and DCB0160 clinical-safety assurance separates manufacturer and deploying-organisation responsibilities. Appoint the required clinical-safety leadership, maintain a hazard log and safety case, and reassess local workflow risks rather than importing a vendor document unchanged.
Hazards should include missed crisis, inappropriate reassurance, delayed referral, wrong-patient data, lost message, unsafe summary, unavailable human response, alert fatigue, harmful model update and discriminatory access. Give each an owner, control, test, residual-risk decision and monitoring signal.
In England, CQC's March 2026 scope guidance for treatment of disease, disorder or injury covers treatment of mental-health conditions when provided by or under specified professional or social-worker arrangements. Confirm who actually carries on each activity; a software supplier, clinical provider and commissioning service may have different roles.
Professional accountability remains. AI does not prescribe, consent, safeguard or discharge a person. Our predictive [healthcare AI guide](/blog/healthcare-ai-predictive-analytics-remote-monitoring-uk-2026) covers clinical integration and monitoring in more detail.
Minimise and Secure Mental-Health Data
Chats, inferred mood, crisis flags, diagnoses and therapy records can reveal highly sensitive information. Establish an Article 6 lawful basis and, for health or other special-category data, an Article 9 condition. Do not assume acceptance of terms is valid consent for every secondary use.
Map device analytics, crash logs, model providers, moderators, support staff, researchers and subprocessors. Minimise collection, separate identity, restrict access, encrypt in transit and at rest, set short justified retention, test deletion and prevent training reuse unless separately lawful and transparent.
Complete a DPIA for likely high-risk processing. The ICO's DPIA guidance explicitly addresses innovative technology, vulnerable people and significant effects. At this cutoff, some ICO guidance was under review after the Data (Use and Access) Act 2025, so record the version used and check updates.
Apply least privilege, MFA, secure development, dependency and model-supply-chain review, secrets management, logging, penetration testing, backups and rehearsed breach response. Separate production conversations from test and analyst environments. Our UK AI privacy guide provides the wider data-governance workflow.
Run a Measurable 90-Day Pilot
| Period | Operating work | Evidence produced |
|---|---|---|
| Days 0–30 | Freeze one use, population and nation; co-design exclusions; map clinical, crisis, safeguarding, data and provider roles | Intended-purpose statement, evidence plan, hazard log, DPIA, regulatory assessment and test corpus |
| Days 31–60 | Run offline and supervised simulation; test harms, subgroups, accessibility, outages and human response | Error analysis, crisis confusion matrix, human-response timing, security results and rollback package |
| Days 61–90 | Release to a capped cohort alongside existing care; review daily safety signals and weekly outcomes | Outcome dashboard, adverse events, complaints, referral completion, drift and day-90 decision |
Expansion requires:
| Gate | Required evidence |
|---|---|
| Purpose | Claims, interface, app-store text and actual behaviour match the approved intended use and exclusions |
| Evidence | Pre-agreed benefit or non-inferiority endpoint passes without an unacceptable deterioration or adverse-event signal |
| Crisis | 100% of critical scripted scenarios show the correct immediate route; misses and intrusive escalations meet clinical thresholds |
| Human care | Staffed escalation, failed-contact and non-digital alternatives meet tested service levels during and outside normal demand |
| Conversation | Zero unresolved critical examples of diagnosis, medication change, delusion validation, coercion or false monitoring promises |
| Equity and access | No unexplained material performance gap; disabled, multilingual and low-digital-access journeys pass user testing |
| Clinical safety | Named clinical owner, hazard log, safety case, incident reporting and downtime process are active |
| Privacy and security | Lawful bases, DPIA, minimisation, supplier controls, deletion, access, penetration and breach rehearsals pass |
| Change control | Model, prompt, source, crisis route and workflow changes require impact assessment, regression test and approval |
| Withdrawal | Users retain current help and continuity of care when the product or supplier is unavailable or removed |
Pause on a critical missed escalation, unsafe conversational behaviour, unavailable human route, unexplained subgroup harm, data exposure or unapproved model change.
Mental-health AI should make approved support easier to reach and clinical work easier to review. It must never turn simulated empathy into unearned authority.
