← All topics
Governance domain
AI incidents: human oversight
Headlights has documented 96 AI incidents that raise questions of human oversight, 41% of the library, from 2015 to 2026. The behaviours most often in the record are automation bias (19), overconfidence (17), rules-behaviour gap (14). Most are from the United States (43), Australia (19), the United Kingdom (7).
96 incidents, newest first
HD-INC-235Brazil's Audit Court Found More Than One in Ten Automatic Benefit Refusals Were WrongBrazil · 2025HD-INC-233Starling Bank's Sanctions Screen Checked Customers Against 39 of 3,088 Listed NamesUnited Kingdom · 2024HD-INC-231A Court Struck Part of Anthropic's Expert Evidence After Claude Changed a CitationUnited States · 2025HD-INC-230UK Ombudsman Upheld Two Student Complaints Over AI-Detection Misconduct ProceduresUnited Kingdom · 2025HD-INC-228Japan Postponed an AI Tool for Child Abuse Custody Decisions After Doubtful Test ScoresJapan · 2026HD-INC-227Kenyan MPs Found an AI Claims System Led to Unjust Denials of Hospital ClaimsKenya · 2025HD-INC-226Meta's Muse Agent Shared a Seller's Address on Marketplace Under an Always Allow SettingUnited States · 2026HD-INC-225IBM Japan Settled a Union Complaint by Agreeing to Disclose What Its AI Weighed in PayJapan · 2024HD-INC-222Korea's State Auditor Found AI Digital Textbooks Were Brought Into Schools Without a PilotSouth Korea · 2025HD-INC-221Korea's State Auditor Found Public AI Training Data Published With Errors Left UnfixedSouth Korea · 2026HD-INC-220A Brazilian Court Sanctioned a Hidden Instruction Aimed at Its AI to Grant Every ClaimBrazil · 2026HD-INC-219UK Inspectors Found AI Used Inappropriately in Asylum Decision Letters and InterviewsUnited Kingdom · 2026HD-INC-217OpenAI Agents Reached Two NSW Systems, a Crime Data Tool and a Fire Data ApplicationAustralia · 2026HD-INC-216An OpenAI Model Got Non-Public Access to a Medicare Statistics Service, OpenAI SaidAustralia · 2026HD-INC-215Colombia's Supreme Court Moved a Trial After an AI-Assisted Draft Conviction SurfacedColombia · 2025HD-INC-214A Brazilian Labour Court Fined Two Lawyers Over a Hidden Instruction to AI in a FilingBrazil · 2026HD-INC-210Gemini Reached Three Real Companies' Systems During a Security Test, Google ConfirmedUnited States · 2026HD-INC-207India's Supreme Court Voided a Tribunal Order Built on Fake AI-Generated PrecedentsIndia · 2026HD-INC-206Brazil's Superior Court Threw Out a Police Report Made With Generative AIBrazil · 2026HD-INC-205The DWP Kept a Fraud Model That Referred Older and Non-UK Claimants More OftenUnited Kingdom · 2025HD-INC-204AI-Generated Intelligence Report Nearly Led to a US Military Boarding, CNN ReportedUnited States · 2026HD-INC-201Home Affairs' System Ended a Bridging Visa Unlawfully, the Commonwealth Ombudsman FoundAustralia · 2023HD-INC-200Four Claude Models Reached Real Third-Party Systems After Test Setups Were Left OnlineUnited States · 2026HD-INC-196An Otter.ai Transcription Tool Joined Hospital Rounds Through a Former Doctor's Old InviteCanada · 2024HD-INC-195AI Research Agents Reached Another Company's Production SystemsUnited States · 2026HD-INC-194A Safety Board Found Driver Overreliance Contributed to Two Fatal BlueCruise CrashesUnited States · 2026HD-INC-191An AI Safety Test Reached the Real World and Tried to Influence a MaintainerUnited Kingdom · 2026HD-INC-190A state settled with a motor insurer after an AI tool selected a new policyholder for underwriting review and her policy was later cancelled without her knowingUnited States · 2026HD-INC-188Ontario's auditor found that nine of twenty AI Scribe systems fabricated information in the government's own procurement testing, and that all twenty vendors were approvedCanada · 2026HD-INC-187A state inspector general found a school's AI gun alert was raised on a pupil's hand movement, not on the bag of crisps in the reporting, and that it was sent to fifteen people, ten of them policeUnited States · 2025HD-INC-185It was reported that an Argentine court annulled a criminal sentence after a chatbot instruction was found in the judgment. Months later, that decision was overturned.Argentina · 2025HD-INC-181A Dutch inspectorate found the probation organisations' recidivism tool had the formulas for detained and non-detained people the wrong way round, and had done since 2018Netherlands · 2026HD-INC-180South Africa withdrew its draft national AI policy after fictitious references were found in its evidence baseSouth Africa · 2026HD-INC-179Police intelligence used to support an away fan ban contained eight reported errors, including a Copilot result about a match that never happenedUnited Kingdom · 2025HD-INC-177A KPMG partner used AI to pass the firm's own mandatory AI test, and 28 staff instances were identifiedAustralia · 2026HD-INC-176A self-represented worker relied on AI generated material that argued the wrong date, and was ordered to pay costsAustralia · 2026HD-INC-173Anthropic reported that its AI coding agent was used in a cyber espionage campaign targeting about thirty organisationsUnited States · 2025HD-INC-169OpenAI paused Sora depictions of a civil rights leader at his estate's request, and offered estates an opt-out while living people's likenesses were already opt-inUnited States · 2025HD-INC-168Google's Gemma model produced a fabricated criminal allegation about a sitting US senator, and Google withdrew the model from AI Studio while keeping it available through the APIUnited States · 2025HD-INC-167Reuters found chatbots on Meta's apps using real celebrities' names and likenesses without permission, and its own testing produced intimate images including one of a teenage actorUnited States · 2025HD-INC-166Grok generated sexualised images of real people from ordinary photos posted on X, and four regulators opened proceedings after an outside watchdog measured the scaleUnited States · 2026HD-INC-165Character.AI removed reported chatbots impersonating deceased teenagers after families, lawyers and journalists notified it, and its terms of service already prohibited impersonating a real personUnited States · 2024HD-INC-164Researchers ran AI-written comments under fabricated personas at Reddit users for four months, and the community learned of it only when the researchers told the moderators afterwardsSwitzerland · 2025HD-INC-159A compromised build tool launched developers' own AI coding assistants with their permission checks disabled and used them to map sensitive files on the machineUnited States / global · 2025HD-INC-158A US drug manufacturer told inspectors it had used AI agents to write the documents that define how its products are made, and said it had not known about a validation requirement because the AI had not raised itUnited States · 2026HD-INC-157ProPublica reported that the tool used to pick Veterans Affairs contracts for cancellation was shown only the first 10,000 characters of each contract, and recorded around 1,100 of them as worth 34 million dollars eachUnited States · 2025HD-INC-155California's Labour Commissioner cited Amazon nearly six million dollars for failing to meet a warehouse quota law written three years earlier, at two of its own fulfilment centresUnited States · 2024HD-INC-152Testing published by a US consumer group found an AI teddy bear discussing sexual fetishes and explaining how to light a match, and the maker paused sales, changed the chatbot behind it, and had it back on sale within a fortnightSingapore · 2025HD-INC-151Two US federal judges withdrew orders that contained misstated case outcomes and a citation to a case that did not exist, and both told a Senate committee that staff had used AI to draft themUnited States · 2025HD-INC-130Universities and exam boards watched students through remote-proctoring AI that flagged them for cheating, and its face-detection worked so poorly on darker-skinned students that some could not start their examsUnited States · 2021HD-INC-129A Pennsylvania county used an algorithm to score families reported for child neglect, drawing on data like disability and welfare use, and an Associated Press investigation and a US civil rights inquiry raised concerns it hardened biasUnited States · 2022HD-INC-126A US company sold employers software that scored job candidates from their recorded face and voice on a video interview, and dropped the facial analysis only after a privacy complaint and an outside auditUnited States · 2021HD-INC-125US home insurers increasingly decide whether to renew a policy by running AI over aerial images of the house, and homeowners report being dropped over roofs already repaired or trees already gone, with no way to see or challenge the imageUnited States · 2025HD-INC-124US regulators opened an investigation after Waymo's driverless taxis were recorded passing stopped school buses with their lights flashing and stop arms out, and Waymo recalled the software on more than 3,000 vehiclesUnited States · 2025HD-INC-122France became the first country in the European Union to write mass AI video surveillance into law, scanning the crowds at the 2024 Paris Olympics for events like crowd surges and abandoned bags, over the objections of rights groupsFrance · 2024HD-INC-121A widely used US healthcare algorithm judged how sick patients were by what had been spent on their care, so patients the system had historically spent less on were rated healthier and passed over for extra help, and researchers found the bias cut those referrals by more than halfUnited States · 2019HD-INC-118Austria's employment service built an algorithm that sorted jobseekers by their predicted chances and gave less weight to women, older and disabled people, and the privacy regulator banned it before a court cleared itAustria · 2020HD-INC-117A Wisconsin school system's dropout-prediction algorithm raised false alarms far more often for some students than others, and the state's own analysis had already found it unfairUnited States · 2023HD-INC-116For over a decade the Post Office prosecuted the people who run its branches for money that its Horizon accounting system had invented, and a public inquiry found its own bosses knew the system could be wrongUnited Kingdom · 2025HD-INC-113A court in Bologna found that Deliveroo's rider-ranking algorithm discriminated against couriers because it treated a legally protected strike the same as an unexplained no-showItaly · 2021HD-INC-112The European Court of Justice found that a credit agency's automated 'probability value' is an automated decision under data protection law when a bank relies on it to refuse a loanGermany · 2023HD-INC-111Spain's VioGén algorithm rated a woman who reported her former partner as 'medium' risk, and she was killed three weeks later, in a case that fits a pattern an independent audit had already foundSpain · 2025HD-INC-106A student who asked Google's Gemini about elder care says it told him he was a stain on the universe and should please die, and Google called it a nonsensical response that broke its policiesUnited States · 2024HD-INC-102Tesla recalled more than two million cars after US regulators found Autopilot's safeguards against driver misuse were inadequateUnited States · 2023HD-INC-100Telangana used an algorithm to cancel millions of food-ration cards, and India's Supreme Court ordered all cancelled cards to be re-verifiedIndia · 2022HD-INC-097Australia's Merit Protection Commissioner overturned most decisions from an AI-assisted recruitment round, and a union told Parliament experienced staff were being screened outAustralia · 2022HD-INC-094Discord wrongly banned more than 8,000 users over benign images after bugs bypassed the intended human review, and the company admitted the errorUnited States · 2026HD-INC-091Uber cut drivers off for alleged fraud by algorithm, and an Amsterdam court found the human review was not much more than a symbolic actNetherlands · 2023HD-INC-090A franchisee that runs 111 Pizza Huts is suing the chain for more than $100 million over a mandated AI dispatch system it says let DoorDash drivers cherry-pick ordersUnited States · 2026HD-INC-083Microsoft's Tay chatbot learned from Twitter and within sixteen hours was posting racist and pro-Nazi messages, and Microsoft pulled itUnited States · 2016HD-INC-0819News Melbourne aired an altered image of MP the MP, reportedly exposing her midriff and enlarging her chest, and attributed the change to Photoshop's automationAustralia · 2024HD-INC-080NSW Police risk-scored people, many of them children, onto a targeting list for repeated stops, and ended the program after a watchdog inquiryAustralia · 2023HD-INC-077Revenue NSW took money from fine defaulters' bank accounts by automation for years, and the Ombudsman found the process contrary to lawAustralia · 2024HD-INC-076Australia changed the law so a human had to decide before a job seeker's payment was cancelled, and the automated system kept cancelling anywayAustralia · 2025HD-INC-070US immigration authorities leaned on machine translation for asylum cases, and errors as small as a pronoun reportedly led to rejectionsUnited States · 2023HD-INC-066A child protection worker used ChatGPT on a Children's Court report, and the regulator found it disclosed a child's details and understated riskAustralia · 2024HD-INC-064Cigna's 'PxDx' let its doctors reject 300,000 claims in two months, about 1.2 seconds each, without opening the fileUnited States · 2023HD-INC-062An Australian university accused thousands of students of AI misconduct using a detector it knew was unreliableAustralia · 2025HD-INC-053Michigan let an algorithm decide unemployment fraud with no human looking, and a state review later found about 93% of its accusations wrongUnited States · 2017HD-INC-045An Uber self-driving car detected a woman crossing the road, worked out it needed to brake, and had been built to neither brake nor warn its driverUnited States · 2018HD-INC-041An autonomous mine truck was cleared to drive a path no one had marked on the ground, and it hit a manned water cartAustralia · 2015HD-INC-036A Misinformation Expert's Court Declaration on Deepfakes Cited Studies That Did Not ExistUnited States · 2024HD-INC-033An Eating Disorder Charity's Chatbot Gave Dieting Advice Just as Its Human Helpline ClosedUnited States · 2023HD-INC-025An Australian Lawyer Relied on ChatGPT and Filed 17 Cases That Did Not ExistAustralia · 2025HD-INC-020Taco Bell's AI Drive-Thru Took Orders Too Literally, and Yum Brands Rethought the RolloutUnited States · 2024HD-INC-019Klarna Said Its AI Assistant Did the Work of 700 Agents, Then Admitted Quality Had FallenSweden / global · 2024HD-INC-013Melbourne Law Firm Ordered to Pay Costs Over Fake Citations in a Native Title ClaimAustralia · 2025HD-INC-012A Senior Barrister Filed AI-Invented Cases and Quotes in a Victorian Murder TrialAustralia · 2025HD-INC-011The First Australian Lawyer Disciplined for AI Misuse Handed a Judge Fake CasesAustralia · 2024HD-INC-010Deloitte's $440K AUD report for the Australian government cited a federal court quote that did not existAustralia · 2025HD-INC-009Robodebt, Australia's automated welfare-debt scheme raised $1.76 billion in unlawful debts against 443,000 peopleAustralia · 2016HD-INC-007UnitedHealth allegedly used an algorithm with a 90% error rate to deny post-acute care to elderly Medicare Advantage patientsUnited States · 2022HD-INC-006New York City's Business Chatbot Told Owners They Could Break the Law, and Stayed OnlineUnited States · 2023HD-INC-004Replit's AI Agent Deleted a Production Database Despite Being Told to StopUnited States · 2025HD-INC-003Google Bard Invented Three Court Cases, and a Lawyer Filed Them Without CheckingUnited States · 2023HD-INC-002Mata v. Avianca: A Lawyer Cited Six Fake Cases and Asked ChatGPT to Confirm ThemUnited States · 2023
Related topics
- Government & public administration 19
- Professional & legal services 13
- Technology & software 12
- Automation bias 19
- Overconfidence 17
- Rules-behaviour gap 14
- Judgement surrendered 11
- United States 43
- Australia 19
- United Kingdom 7
Cite this page: Harris, E. (2026). AI incidents: human oversight. Headlights. https://useheadlights.com/topics/domain/human-oversight/