What happened
IBM sold Watson for Oncology as artificial intelligence that would change cancer treatment, a system that could read the medical literature and a patient’s records and recommend the best therapy. It was marketed worldwide and taken up by hospitals on several continents. IBM told customers the advice drew on data from real patients.
It did not. Internal IBM documents from 2017, obtained by the health-news organisation STAT, showed that Watson for Oncology had been trained on a small number of “synthetic” cases, hypothetical patients devised by a handful of doctors at Memorial Sloan Kettering Cancer Center, rather than on real patient records and outcomes. The recommendations reflected the treatment preferences of those few doctors, not an analysis of how actual patients had fared. The number of training cases for each cancer had been set without statistical input, ranging from 635 for lung cancer down to 106 for ovarian.
The same documents recorded that IBM’s own specialists and some customers had found “multiple examples of unsafe and incorrect treatment recommendations.” In one example cited internally, the system recommended that a 65-year-old man with lung cancer and severe bleeding be given a drug, bevacizumab, that carries an explicit warning against use in patients with serious bleeding because it can cause fatal haemorrhage. IBM and Memorial Sloan Kettering said this recommendation had arisen in testing and was never given to a real patient, and no patient was reported to have been harmed. A doctor at Jupiter Hospital in Florida told IBM executives: “We bought it for marketing and with hopes that you would achieve the vision. We can’t use it for most cases.”
The gap between the promise and the product showed up in dollars as well. MD Anderson Cancer Center in Texas spent about 62 million US dollars on a Watson-powered project that, after a University of Texas audit, was shelved around 2016 and 2017 without ever being used to treat a patient. IBM sold Watson Health to private equity firm Francisco Partners in January 2022.
What an auditable version would have shown
The decisive fact about Watson for Oncology was the provenance of its advice: what it had actually learned from. IBM said real patient data; the documents said a handful of synthetic cases reflecting a few doctors’ preferences. A recommendation that records what it was based on would have made that claim checkable. An auditable version writes, with each recommendation, the model version and the evidence and cases it drew on, and routes the clinical question to a trusted external source, real outcomes and published guidelines, rather than back to a model trained to echo its authors. With that, the difference between “based on thousands of real patients” and “based on a hundred invented ones” is not a marketing claim a customer must take on trust but a recorded fact, and a recommendation that contradicts the evidence base is caught before it reaches a clinician.
Where the gap was
The failure was a system deployed far beyond what it had been shown to do, with nothing checking its output against reality or recording what that output rested on. A VerificationGate is the control on the first: a clinical recommendation is checked against a trusted source, real evidence and guidelines, not accepted because the model produced it. A ConductRecord is the control on the second: each recommendation carries its provenance, the model version and the cases and evidence behind it, so the claim that the advice comes from real patient data can be verified rather than asserted. The toolkit cannot make a system more capable than it is. What it can do is make a system’s actual basis legible, which is precisely what would have stopped “trained on synthetic cases” from being sold as “trained on real patients.”
What governance should have looked like
Selling a tool into cancer wards carries an obligation to be honest about what it can do and what it knows, and Watson for Oncology was sold ahead of both. The lesson is that capability claims about a high-stakes system have to be backed by something verifiable: recommendations checked against real evidence rather than the model’s own training, a recorded provenance for every piece of advice, and marketing that cannot outrun what the record can support. The technology was not useless, but it was deployed and described as something it was not, and only internal documents, surfaced by reporters, showed the distance between the two.
The reference implementation of VerificationGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- IBM’s Watson supercomputer recommended ‘unsafe and incorrect’ cancer treatments, internal documents show (STAT, PDF)
- IBM pitched its Watson supercomputer as a revolution in cancer care. It’s nowhere close (STAT)
- How IBM Watson Overpromised and Underdelivered on AI Health Care (IEEE Spectrum)
- IBM’s Watson ‘TKO’d by cancer center woes’ after damning audit (The Register)