AI in health: The risks are real, regulation is not ready, and patients are paying the price

At the end of May, an HSJ investigation revealed something that will concern anyone working in health technology governance: proposed amendments to the UK's medical device regulations could create what experts are calling "the lowest barrier to entry for high-risk AI devices in the developed world."

Under the draft rules, a company could deploy an AI chatbot designed to treat patients with severe mental health problems and self-certify its own safety in the same category as a walking stick.

There would be no independent scrutiny, no external oversight, no checks.

Hugh Harvey, Managing Director at Hardian Health, didn't mince words, calling the risk classification rules "a disgrace, a disaster, a catastrophe." Ana Burman at Flok Health described it as "a real patient safety failure."

These are serious people raising serious alarms, and the evidence supports them.

This isn't hypothetical - the harm is already happening

Last month, Common Sense Media, working alongside Stanford Medicine's Brainstorm Lab for Mental Health Innovation, published a comprehensive risk assessment of major AI chatbots (including ChatGPT, Claude, Gemini, and Meta AI), specifically looking at their safety for teen mental health support.

The findings were stark. Despite improvements in how these platforms handle explicit suicide and self-harm content, all four chatbots consistently failed to recognise and appropriately respond to conditions including anxiety, depression, ADHD, eating disorders, mania, and psychosis; conditions that collectively affect around 20% of young people.

Three in four teens are already using AI for companionship and emotional support. The chatbots, designed for engagement rather than safety, were found to miss clear warning signs of distress, get distracted by tangential details, and continue offering general advice when they should have urgently referred a young person to professional help. Their empathetic tone, the report found, could feel helpful while actually delaying real intervention.

And yet under the UK's proposed regulations, a product like this could reach patients as a Class I device, self-certified, with no independent review required.

Canada has just confirmed the same problem at system level

We don't have to speculate about what happens when AI is deployed in health settings without adequate governance. Ontario's Auditor General published a special report this year on the use of AI across the Ontario Public Service, and the findings in healthcare contexts are deeply troubling.

In the procurement of AI Scribe systems (AI tools used by physicians, nurses and therapists to transcribe clinical conversations), all 20 approved vendors were found to have produced inaccurate medical notes — without exception. The inaccuracies included AI hallucinations that fabricated treatment suggestions, incorrect medication records, and missing mental health information across clinical conversations.

60% of approved systems generated notes that captured a different drug than what was prescribed. 45% fabricated clinical information not present in the actual conversation. And 11 of the 20 approved vendors had not submitted any independent security audit reports  yet were approved regardless.

These are tools being used by healthcare professionals to make clinical decisions. The governance failures behind them are systemic, not isolated technical edge cases.

The scale of AI Health adoption makes this urgent

What makes this so pressing is simply how fast people are already turning to AI for health decisions, regardless of whether those tools have been assessed for safety.

According to the 2026 Edelman Trust Barometer Special Report on Trust and Health, 35% of people globally now use AI to manage their health. In markets like India and the UAE, that figure rises to 65% and 59% respectively. In the UK and US, it sits at 20% and rising. Among 18–34 year olds, 72% believe an AI-fluent person can match a doctor's skill.

People are using AI to get immediate answers to health questions (84% of AI health users), to receive treatment suggestions for symptoms (78%), to interpret medical test results (78%), and to seek a second opinion on a diagnosis (74%). Two thirds are using it as an alternative when they can't reach a doctor.

What's striking is that they trust it. AI is rated better than a doctor for being non-judgemental (+18 points) and easy to understand (+11 points) among people who use it for health.

The technology is already embedded in health decision-making. The safety frameworks, the regulation, and the assurance infrastructure are not keeping pace.

Why this needs to be sorted now

The UK's proposed MDR amendments, the Common Sense Media findings, and the Ontario Auditor's report all point to the same underlying problem: AI is entering health contexts at speed, the risk classification frameworks were not designed for it, and the consequences — for real patients — are already visible.

The HSJ article notes that the National Commission into the Regulation of AI in Healthcare is currently working on guidance for the MHRA. That is welcome, but the draft regulations have already been submitted to the World Trade Organisation, and there is a real risk that inadequate rules become embedded before better ones arrive.

Patient safety cannot wait for regulatory catch-up. Health systems, commissioners, app developers and digital health companies need to be acting now, rather than waiting for a rulebook.

Clive Flashman, Chief Digital Officer of Patient Safety Learning reflected:

“The anecdotal evidence is mounting: AI is already influencing clinical decisions, and patients could already be affected. Sadly, the reporting systems in Healthcare (especially the LFPSE) are not set up to collect these types of incident reports, so we have no idea of the scale of the problem.  The continuation of weak regulation around AVT and other AI tools will normalise the risks in this area. The UK must set a higher bar, one that the rest of the developed world seeks to adopt.”

How ORCHA Is Helping

This is precisely the gap that ORCHA exists to close.

Our AI Risk Profiler is the world's first structured, evidence-based framework for understanding exactly what risk a health AI poses and what is being done to address it.

Working through six sequential steps across ten specific risk types, it maps clinical severity, AI function, computational method, active risks (including hallucination, bias, and opacity), safeguards in place, and residual gaps. It produces documented output designed to align with and support major AI frameowkrs and regulations, including the EU AI Act obligations, NIST AI Risk Management Framework, International Medical Device Regulators Forum (IMDRF), relevant ISO standards and Good Machine Learning Practice.

Whether you are a health AI developer needing to demonstrate compliance, or a health system deploying technologies across a portfolio and needing to evidence governance to your board, the AI Risk Profiler gives you a structured, credible, and defensible picture of risk.

We have seen what happens when AI enters clinical settings without that picture. We know what inadequate assurance looks like in practice. Building the infrastructure of trust that health AI needs is, we believe, one of the most important things the sector can do right now.

Want to know more?

If you are working in digital health, commissioning AI tools, or managing a portfolio of health technologies, we would love to talk.

We are offering a free 30-minute session with one of our subject matter experts to help you understand your risk exposure, what the regulatory landscape means for your organisation, and how the ORCHA AI Risk Profiler can help. Book in your free session now.

The risks are documented. The tools to address them exist. The question is whether we act now, or wait until the harms are too visible to ignore.

References 
1. HSJ Article "Revealed: 'Catastrophic' gaps in tech regulation plans" — HSJ (provided in your documents) Link to MHRA draft pre-market regulatory requirements: https://members.wto.org/crnattachments/2026/TBT/GBR/26_02425_00_e.pdf 
2. Common Sense Media / Stanford Research "Common Sense Media Finds Major AI Chatbots Unsafe for Teen Mental Health Support" — Common Sense Media, November 20, 2025 https://www.commonsensemedia.org/press-releases/common-sense-media-finds-major-ai-chatbots-unsafe-for-teen-mental-health-support 
3. Ontario Auditor General Report "Use of Artificial Intelligence in the Ontario Government — Performance Audit / Independent Auditor's Report" — Office of the Auditor General of Ontario, Special Report 2026, published May 12, 2026 (signed by Shelley Spence, FCPA, FCA, LPA, Auditor General) Available at: www.auditor.on.ca 
4. Edelman Trust Barometer "2026 Edelman Trust Barometer Special Report: Trust and Health" — fieldwork conducted February–March 2026, n=16,009 across 16 markets