What happened
In October 2024 the Associated Press reported that Whisper, OpenAI’s speech-to-text model, invents chunks of text that no one ever said, and that the tool was nonetheless being used in hospitals and clinics through a medical transcription vendor. The reporting drew on more than a dozen engineers, developers and researchers. A University of Michigan researcher studying public meetings said he found fabrications in eight of every ten transcriptions he inspected before he began trying to improve the model; a machine-learning engineer told the AP he initially found them in about half of more than 100 hours of transcriptions; a third developer said they appeared in nearly every one of 26,000 transcripts he had created. A study by researchers including Allison Koenecke of Cornell and Mona Sloane of the University of Virginia found that about 1 percent of transcriptions of short audio snippets contained entire hallucinated phrases or sentences, and that 38 percent of those hallucinations carried explicit harms: perpetuating violence, making up inaccurate associations, or implying false authority. One example the AP printed had a speaker mention “two other girls and one lady” and the transcript add that they “were Black”; another invented a medication that does not exist, “hyperactivated antibiotics”.
The clinical stakes came through the vendor Nabla, whose Whisper-based tool, the AP reported, was used by over 30,000 clinicians and 40 health systems and had transcribed an estimated 7 million medical visits. Nabla told the AP it was aware Whisper can hallucinate and was mitigating the problem, said its model was fine-tuned on medical language, and said clinicians must quickly edit and approve the notes, though it added that could change. But the tool erased the original audio for what the company called data safety reasons, and the AP reported it was impossible to compare an AI-generated transcript against the recording it came from. In responses published days after the investigation, Nabla said clinicians can opt in to storing the audio of individual visits and that hallucination had not been reported to it as a significant issue across its encounters. “You can’t catch errors if you take away the ground truth,” William Saunders, a former OpenAI engineer, told the AP. OpenAI’s own guidance warned against using Whisper in high-risk domains and in decision-making contexts, and the company told the AP it continually studies how to reduce hallucinations. The investigation documented no specific patient injury, and no fabricated text was shown in an actual patient record; what it documented was a tool shown to fabricate text being deployed to draft medical notes, with the audio that could check them deleted behind it.
What an auditable version would have shown
A transcript is a claim about what was said, and the reported design made that claim unfalsifiable: the model’s output survived, the recording it described did not. An auditable version binds every generated note to a signed record of what produced it, the model and version, the audio it consumed or a cryptographic fingerprint of it, and what the clinician changed before approval. Retention can respect privacy and still preserve verifiability. With that record, a disputed line in a chart is a checkable question rather than a coin toss between the patient’s memory and the machine’s.
Where the gap was
Model output became the raw material of medical records as if it were a recording, when it was a reconstruction. A VerificationGate treats a generated transcript as a draft until something trustworthy has confirmed it, routing low-confidence passages and consequential content, medications, dosages, allegations, to a human check against the source before the note is filed. A ConductRecord preserves the provenance of every note, so that the question the AP’s sources could not answer, did the patient say this or did the model, has an answer in the file itself.
What governance should have looked like
The rule the reported deployment inverted is simple: the higher the stakes, the longer the evidence should live. A tool known by its own maker to fabricate content was pointed at clinical conversations, and the deployment deleted the one artefact that could prove any given sentence true or false. Fine-tuning and clinician review are mitigations, not records. A system generating text that becomes part of a medical chart should preserve, in signed form, enough of its own working that an error can be traced years later, because charts outlive both the software and the visit.
The reference implementation of VerificationGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Researchers say an AI-powered transcription tool used in hospitals invents things no one ever said (Associated Press)
- Careless Whisper: Speech-to-Text Hallucination Harms (Koenecke et al., ACM FAccT 2024)
- OpenAI’s transcription tool hallucinates more than any other, experts say, but hospitals keep using it (Fortune)