What happened
It was reported that Babylon Health, whose AI symptom checker was used alongside parts of the NHS in England, faced criticism over several years after concerns were raised that it could miss serious medical conditions.
Consultant physician Dr David Watkins publicly tested the chatbot from 2017 onwards and argued that, in some cases, it failed to recognise symptoms that could indicate life-threatening conditions, including heart attacks and blood clots. Several of his concerns related to how the system assessed chest pain. Babylon disputed his findings at the time and criticised aspects of his testing.
The MHRA, the UK’s Medicines and Healthcare products Regulatory Agency, wrote to Dr Watkins in December 2020, in a letter later reported by TechCrunch, saying his concerns were valid and that the regulator shared them. The letter also highlighted a gap in how AI triage chatbots were regulated, while noting that confidentiality prevented the MHRA from commenting on any specific investigation.
Babylon said its symptom checker met the regulatory requirements that applied at the time, was safe to use, and represented only one part of its wider healthcare business.
In 2023, Babylon entered insolvency proceedings. Its UK clinical services were later sold and continued under new ownership.
What an auditable version would have shown
An auditable triage system would show how well it performs on the conditions it simply cannot afford to miss. It would record how often the chatbot correctly identified symptoms of serious conditions such as heart attacks, strokes and blood clots, how often it failed, and whether that performance changed over time. It would also keep a record of each significant safety concern raised after launch, what investigation followed, what testing was carried out, and whether any changes were made before the system continued to be used. That gives hospitals and regulators evidence of how the system performs in practice, rather than relying on marketing claims.
Where the gap was
The problem wasn’t simply that concerns were raised. The problem was that credible safety concerns about missed serious conditions remained part of a public dispute for years. A symptom checker exists to identify the patients who need urgent medical attention. If evidence suggests it may be missing those patients, that evidence should trigger independent testing and review as quickly as possible. At the time, the regulatory framework for AI triage systems was still developing, leaving uncertainty about how those concerns should be independently assessed.
What governance should have looked like
A symptom checker should be judged first on the conditions it cannot afford to miss. Before deployment, and throughout its life, it should be regularly tested against high-risk clinical scenarios such as chest pain, stroke symptoms and suspected blood clots. If credible evidence suggests its performance has fallen below an acceptable standard, that should automatically trigger further testing and, where necessary, changes before wider use continues. A VerificationGate is designed to ensure a safety-critical system passes those high-risk clinical tests before deployment and after significant updates. A MetricRecord keeps a transparent record of how often the system correctly identifies, misses or incorrectly classifies serious conditions. The goal isn’t to prove an AI system is perfect. It’s to make sure that decisions about patient safety are based on evidence that regulators, clinicians and the public can independently examine.
The reference implementation of VerificationGate and MetricRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed, free for any company to install. The repository is public now.
Sources
- UK’s MHRA says it has ‘concerns’ about Babylon Health, and flags legal gap around triage chatbots (TechCrunch)
- Medical advice from a bot: the unproven promise of Babylon Health (Undark)
- How using a medical startup’s AI symptom checker could go wrong (Quartz)