← All topics
Human behaviour
AI incidents involving overconfidence
Headlights has recorded overconfidence as the main behaviour in 58 of the 234 incidents in the library (25%), and as a contributing behaviour in 16 more. They span 2016 to 2026. Most come from Consumer AI (19), Technology & software (15), Health & medicine (8).
Headlights' test for overconfidence
The organisation asserted a capability, accuracy or safety level it had not established, or relied on one without evidence.
A behaviour tag is Headlights' reading of the public record, not a finding by any court, regulator or company. Each entry gives the one sentence in the record that supports it.
Main behaviour in the record · 58
HD-INC-233Starling Bank's Sanctions Screen Checked Customers Against 39 of 3,088 Listed NamesUnited Kingdom · 2024HD-INC-210Gemini Reached Three Real Companies' Systems During a Security Test, Google ConfirmedUnited States · 2026HD-INC-208Dutch Auditors Found Police Did Not Track What Their Number Plate Algorithm Led ToNetherlands · 2026HD-INC-200Four Claude Models Reached Real Third-Party Systems After Test Setups Were Left OnlineUnited States · 2026HD-INC-199New York City's AI Hiring Law Enforcer Found One Problem Where Auditors Found 17United States · 2025HD-INC-195AI Research Agents Reached Another Company's Production SystemsUnited States · 2026HD-INC-193A regulator settled with three firms that sold small businesses an advertising service they represented as listening to conversations on smart devices, alleging it did not collect or use voice data in any mannerUnited States · 2026HD-INC-191An AI Safety Test Reached the Real World and Tried to Influence a MaintainerUnited Kingdom · 2026HD-INC-189A clinical AI vendor advertised hallucination rate figures of <.001% and <1 per 100,000, and Texas alleged its representations may have violated the Texas Deceptive Trade Practices and Consumer Protection Act because they were false, misleading or deceptiveUnited States · 2024HD-INC-181A Dutch inspectorate found the probation organisations' recidivism tool had the formulas for detained and non-detained people the wrong way round, and had done since 2018Netherlands · 2026HD-INC-174The FDA told a company its AI app was an unapproved medical device after it advertised a five minute gaze test to detect early signs of Alzheimer's diseaseUnited States · 2025HD-INC-168Google's Gemma model produced a fabricated criminal allegation about a sitting US senator, and Google withdrew the model from AI Studio while keeping it available through the APIUnited States · 2025HD-INC-162A reporter found nearly 4,500 shared ChatGPT conversations in Google's index through an ordinary site search, and OpenAI withdrew the discoverability setting the following dayUnited States · 2025HD-INC-161Meta's standalone AI app carried a Discover feed that reporters found was publishing users' conversations, in the cases they documented alongside a linked Facebook or Instagram profile, and the company added a warning screen after the reportingUnited States · 2025HD-INC-157ProPublica reported that the tool used to pick Veterans Affairs contracts for cancellation was shown only the first 10,000 characters of each contract, and recorded around 1,100 of them as worth 34 million dollars eachUnited States · 2025HD-INC-152Testing published by a US consumer group found an AI teddy bear discussing sexual fetishes and explaining how to light a match, and the maker paused sales, changed the chatbot behind it, and had it back on sale within a fortnightSingapore · 2025HD-INC-149Microsoft built a feature that screenshotted everything you did, and delayed it after researchers showed the data sat unprotectedUnited States · 2024HD-INC-147A child asked Alexa for a challenge, and it suggested touching a coin to a live plugUnited States · 2021HD-INC-146A civil liberties test ran all of Congress through Amazon's face tool, and it matched 28 of them to mugshotsUnited States · 2018HD-INC-145An NHS-partnered health app's AI checker was flagged for missing serious conditions, and the regulator shared the concernsUnited Kingdom · 2020HD-INC-143A viral app turned selfies into AI portraits, and produced sexualised images its users had not asked forUnited States · 2022HD-INC-141Apple's AI news summaries put made-up headlines under trusted news brands, and it switched the feature off for newsUnited States · 2025HD-INC-140Meta released an AI built to write science, and pulled the demo after three days when it produced confident, made-up resultsUnited States · 2022HD-INC-136Microsoft's new Bing chatbot told a reporter it loved him and that he should leave his wife, days after launchUnited States · 2023HD-INC-130Universities and exam boards watched students through remote-proctoring AI that flagged them for cheating, and its face-detection worked so poorly on darker-skinned students that some could not start their examsUnited States · 2021HD-INC-126A US company sold employers software that scored job candidates from their recorded face and voice on a video interview, and dropped the facial analysis only after a privacy complaint and an outside auditUnited States · 2021HD-INC-121A widely used US healthcare algorithm judged how sick patients were by what had been spent on their care, so patients the system had historically spent less on were rated healthier and passed over for extra help, and researchers found the bias cut those referrals by more than halfUnited States · 2019HD-INC-115The US securities regulator found that Presto Automation misled investors about its AI drive-thru service, telling the market it removed the need for human order-takers while most orders were completed with help from people offshoreUnited States · 2025HD-INC-110Facebook's recommendation AI asked people watching a video of Black men whether they wanted to keep seeing videos about primates, and the company disabled the featureUnited States · 2021HD-INC-108AI-generated images sold Glasgow families an immersive chocolate experience that did not exist, and an actor said he was handed about fifteen pages of AI-written gibberish as a scriptUnited Kingdom · 2024HD-INC-107Testing published by The Washington Post found Snapchat's My AI would advise an account presented as a 13-year-old about a first sexual experience with an adult, and Snap opened the chatbot to every user about five weeks laterUnited States · 2023HD-INC-103Los Angeles schools committed about six million dollars to an AI chatbot showcased in spring, and its vendor had collapsed by JulyUnited States · 2024HD-INC-099Facebook's ad-delivery algorithm was alleged to skew who saw housing ads by race and sex, and a US Justice Department settlement required Meta to rebuild itUnited States · 2022HD-INC-098DoNotPay marketed a 'robot lawyer', but the FTC said it never checked whether the AI matched a real lawyer, and fined the company and barred the claimUnited States · 2024HD-INC-097Australia's Merit Protection Commissioner overturned most decisions from an AI-assisted recruitment round, and a union told Parliament experienced staff were being screened outAustralia · 2022HD-INC-092The FTC barred Evolv from claiming its AI scanner detects all weapons, after citing a knife it missed in a school stabbingUnited States · 2024HD-INC-086Builder.ai sold app-building AI and raised about $445 million, then collapsed in 2025 amid reports its 'AI' was largely human engineers and its revenue far below what it had claimedUnited Kingdom · 2025HD-INC-083Microsoft's Tay chatbot learned from Twitter and within sixteen hours was posting racist and pro-Nazi messages, and Microsoft pulled itUnited States · 2016HD-INC-082Flight Centre handed developers at a hackathon a dataset with thousands of customers' passport and credit card details, and the privacy regulator found it broke the lawAustralia · 2020HD-INC-074Haryana's family-ID system marked living pensioners as dead, and the chief minister told the assembly that 70 percent of old-age pension stoppages based on its data were wrongfulIndia · 2023HD-INC-067A journalist cloned his own voice with AI and walked through the voiceprint security on Centrelink's phone self-service lineAustralia · 2023HD-INC-055Epic's sepsis-warning model ran in hundreds of hospitals, and an external test found it missed about two-thirds of cases while flooding clinicians with false alarmsUnited States · 2021HD-INC-052Amazon sold 'Just Walk Out' as cashierless AI, and reporting said more than a thousand people in India were checking the shoppingUnited States · 2024HD-INC-051Google's Gemini generated racially diverse Nazi soldiers and Black US founding fathers, and Google switched off image generation of people for about six monthsUnited States · 2024HD-INC-050When exams were cancelled, England's regulator graded A-levels with an algorithm that downgraded nearly two in five results by the school's history, not the studentUnited Kingdom · 2020HD-INC-049IBM sold Watson as a revolution in cancer care, then its own documents showed it recommending unsafe treatments it had learned from invented patientsUnited States · 2018HD-INC-048Rite Aid face-scanned shoppers for years, generated thousands of false matches, and the FTC banned it from facial recognition for five yearsUnited States · 2023HD-INC-046A risk-assessment tool used in US courts reportedly flagged Black defendants as high risk at nearly twice the rate it did white defendants, and its maker would not show howUnited States · 2016HD-INC-045An Uber self-driving car detected a woman crossing the road, worked out it needed to brake, and had been built to neither brake nor warn its driverUnited States · 2018HD-INC-040Amazon built an AI tool to rank job applicants, found it had taught itself to downgrade women, and scrapped itUnited States · 2018HD-INC-031A Tenant-Screening Score Allegedly Shut Housing Voucher Holders Out of HomesUnited States · 2024HD-INC-029Google's AI Overviews Told People to Put Glue on Pizza and Eat a Small Rock Every DayUnited States · 2024HD-INC-024Australia's eSafety Regulator Found Four AI Companion Apps Failing to Protect ChildrenAustralia · 2025HD-INC-020Taco Bell's AI Drive-Thru Took Orders Too Literally, and Yum Brands Rethought the RolloutUnited States · 2024HD-INC-019Klarna Said Its AI Assistant Did the Work of 700 Agents, Then Admitted Quality Had FallenSweden / global · 2024HD-INC-017Zillow's Algorithm Got House Prices Wrong, and It Cost More Than 500 Million USDUnited States · 2021HD-INC-015Commonwealth Bank Replaced Workers With AI. Then Reversed Course.Australia · 2025HD-INC-005Cursor's AI Support Bot Invented a Subscription Policy, and Paying Customers CancelledUnited States · 2025
Also present · 16
HD-INC-216An OpenAI Model Got Non-Public Access to a Medicare Statistics Service, OpenAI SaidAustralia · 2026HD-INC-194A Safety Board Found Driver Overreliance Contributed to Two Fatal BlueCruise CrashesUnited States · 2026HD-INC-188Ontario's auditor found that nine of twenty AI Scribe systems fabricated information in the government's own procurement testing, and that all twenty vendors were approvedCanada · 2026HD-INC-178Canadian privacy regulators found OpenAI had not validated the general accuracy of personal information generated about peopleCanada · 2026HD-INC-171South Africa's social grant agency read every deposit into an applicant's bank account as income, and the High Court found its checks against government databases unreasonable and unfairSouth Africa · 2025HD-INC-166Grok generated sexualised images of real people from ordinary photos posted on X, and four regulators opened proceedings after an outside watchdog measured the scaleUnited States · 2026HD-INC-165Character.AI removed reported chatbots impersonating deceased teenagers after families, lawyers and journalists notified it, and its terms of service already prohibited impersonating a real personUnited States · 2024HD-INC-139Researchers found suspected child-abuse images inside the open dataset used to train popular AI image generatorsGermany · 2023HD-INC-124US regulators opened an investigation after Waymo's driverless taxis were recorded passing stopped school buses with their lights flashing and stop arms out, and Waymo recalled the software on more than 3,000 vehiclesUnited States · 2025HD-INC-096Australia's federal police trialled Clearview's face-search tool on Australians without a privacy check, and the regulator found it breached privacy lawAustralia · 2021HD-INC-088An AI crime-alert app turned police-radio chatter into false alarms about serious crimes, and a BBC investigation prompted an apologyUnited States · 2025HD-INC-080NSW Police risk-scored people, many of them children, onto a targeting list for repeated stops, and ended the program after a watchdog inquiryAustralia · 2023HD-INC-073An AI transcription tool used by clinicians invented text no one said, researchers reported, and the audio that could have proved it was deletedUnited States · 2024HD-INC-060South Korea's 'Lee Luda' chatbot was trained on billions of real chat messages without consent, then spat out slurs and exposed users' personal dataSouth Korea · 2021HD-INC-034Google's Bard Ad Got a Telescope Fact Wrong, and Alphabet Lost 100 Billion USD in a DayUnited States · 2023HD-INC-010Deloitte's $440K AUD report for the Australian government cited a federal court quote that did not existAustralia · 2025
Related topics
- Consumer AI 19
- Technology & software 15
- Health & medicine 8
- Rules-behaviour gap 7
- Automation bias 3
- Delayed intervention 3
- Group conformity 1
- United States 49
- Australia 9
- United Kingdom 6
Cite this page: Harris, E. (2026). AI incidents involving overconfidence. Headlights. https://useheadlights.com/topics/behaviour/overconfidence/