← All topics
Governance domain
AI incidents: safety-critical systems
Headlights has documented 19 AI incidents that raise questions of safety-critical systems, 8% of the library, from 2015 to 2026. The behaviours most often in the record are overconfidence (5), automation bias (3), delayed intervention (2). Most are from the United States (14), Australia (2), Canada (1).
19 incidents, newest first
HD-INC-228Japan Postponed an AI Tool for Child Abuse Custody Decisions After Doubtful Test ScoresJapan · 2026HD-INC-216An OpenAI Model Got Non-Public Access to a Medicare Statistics Service, OpenAI SaidAustralia · 2026HD-INC-210Gemini Reached Three Real Companies' Systems During a Security Test, Google ConfirmedUnited States · 2026HD-INC-204AI-Generated Intelligence Report Nearly Led to a US Military Boarding, CNN ReportedUnited States · 2026HD-INC-194A Safety Board Found Driver Overreliance Contributed to Two Fatal BlueCruise CrashesUnited States · 2026HD-INC-188Ontario's auditor found that nine of twenty AI Scribe systems fabricated information in the government's own procurement testing, and that all twenty vendors were approvedCanada · 2026HD-INC-187A state inspector general found a school's AI gun alert was raised on a pupil's hand movement, not on the bag of crisps in the reporting, and that it was sent to fifteen people, ten of them policeUnited States · 2025HD-INC-158A US drug manufacturer told inspectors it had used AI agents to write the documents that define how its products are made, and said it had not known about a validation requirement because the AI had not raised itUnited States · 2026HD-INC-124US regulators opened an investigation after Waymo's driverless taxis were recorded passing stopped school buses with their lights flashing and stop arms out, and Waymo recalled the software on more than 3,000 vehiclesUnited States · 2025HD-INC-111Spain's VioGén algorithm rated a woman who reported her former partner as 'medium' risk, and she was killed three weeks later, in a case that fits a pattern an independent audit had already foundSpain · 2025HD-INC-102Tesla recalled more than two million cars after US regulators found Autopilot's safeguards against driver misuse were inadequateUnited States · 2023HD-INC-092The FTC barred Evolv from claiming its AI scanner detects all weapons, after citing a knife it missed in a school stabbingUnited States · 2024HD-INC-088An AI crime-alert app turned police-radio chatter into false alarms about serious crimes, and a BBC investigation prompted an apologyUnited States · 2025HD-INC-073An AI transcription tool used by clinicians invented text no one said, researchers reported, and the audio that could have proved it was deletedUnited States · 2024HD-INC-055Epic's sepsis-warning model ran in hundreds of hospitals, and an external test found it missed about two-thirds of cases while flooding clinicians with false alarmsUnited States · 2021HD-INC-049IBM sold Watson as a revolution in cancer care, then its own documents showed it recommending unsafe treatments it had learned from invented patientsUnited States · 2018HD-INC-045An Uber self-driving car detected a woman crossing the road, worked out it needed to brake, and had been built to neither brake nor warn its driverUnited States · 2018HD-INC-043A Cruise robotaxi dragged a pedestrian twenty feet, and the company left the dragging out of what it told regulatorsUnited States · 2023HD-INC-041An autonomous mine truck was cleared to drive a path no one had marked on the ground, and it hit a manned water cartAustralia · 2015
Related topics
- Industry, transport & logistics 6
- Health & medicine 5
- Technology & software 3
- Overconfidence 5
- Automation bias 3
- Delayed intervention 2
- Judgement surrendered 2
- United States 14
- Australia 2
- Canada 1
Cite this page: Harris, E. (2026). AI incidents: safety-critical systems. Headlights. https://useheadlights.com/topics/domain/safety-critical-systems/