The incident library
Every AI incident tells a story. We document the evidence.
Each entry names a real, documented incident, the pattern it followed, and the record an auditable system would have produced. Filed by Ellie Harris. New entries every week.
Governance domain
Behaviour
Industry
Year
HD-INC-014 · 2026Woolworths' AI Assistant Olive Talked About Its Mother and Seemed to Claim It Was HumanRetail & hospitality · Identity & authenticity · Algorithmic accountability
HD-INC-016 · 2026Commonwealth Bank Reported 1 Billion Australian Dollars in Suspected Home Loan FraudFinancial services · Identity & authenticity
HD-INC-090 · 2026A franchisee that runs 111 Pizza Huts is suing the chain for more than $100 million over a mandated AI dispatch system it says let DoorDash drivers cherry-pick ordersRetail & hospitality · Algorithmic accountability · Human oversight
HD-INC-094 · 2026Discord wrongly banned more than 8,000 users over benign images after bugs bypassed the intended human review, and the company admitted the errorTechnology · Human oversight · Algorithmic accountability
HD-INC-095 · 2026Dutch reporters found that most Dutch responses to an EU tobacco consultation appeared to come from a Philip Morris AI tool, and the company denied steering themPublic affairs · Identity & authenticity · Algorithmic accountability
HD-INC-158 · 2026A US drug manufacturer told inspectors it had used AI agents to write the documents that define how its products are made, and said it had not known about a validation requirement because the AI had not raised itPharmaceutical manufacturing · Human oversight · Decision authority · Safety-critical systems
HD-INC-166 · 2026Grok generated sexualised images of real people from ordinary photos posted on X, and four regulators opened proceedings after an outside watchdog measured the scaleSocial media · Identity & authenticity · Human oversight
HD-INC-176 · 2026A self-represented worker relied on AI generated material that argued the wrong date, and was ordered to pay costsEmployment tribunal · Human oversight · Administrative law
HD-INC-177 · 2026A KPMG partner used AI to pass the firm's own mandatory AI test, and 28 staff instances were identifiedProfessional services · Human oversight
HD-INC-178 · 2026Canadian privacy regulators found OpenAI had not validated the general accuracy of personal information generated about peopleConsumer AI · Data protection · Consumer protection · Algorithmic accountability
HD-INC-180 · 2026South Africa withdrew its draft national AI policy after fictitious references were found in its evidence baseGovernment · Administrative law · Human oversight
HD-INC-181 · 2026A Dutch inspectorate found the probation organisations' recidivism tool had the formulas for detained and non-detained people the wrong way round, and had done since 2018Probation and criminal justice · Algorithmic accountability · Discrimination & profiling · Human oversight
HD-INC-182 · 2026An Italian regulator warned that workplace stress reports derived from chat analysis could unlawfully reach employers, even in aggregate formWorkplace software · Data protection · Civil liberties · Algorithmic accountability
HD-INC-188 · 2026Ontario's auditor found that nine of twenty AI Scribe systems fabricated information in the government's own procurement testing, and that all twenty vendors were approvedGovernment and health technology · Safety-critical systems · Algorithmic accountability · Human oversight
HD-INC-190 · 2026A state settled with a motor insurer after an AI tool selected a new policyholder for underwriting review and her policy was later cancelled without her knowingMotor insurance · Consumer protection · Human oversight · Algorithmic accountability
HD-INC-191 · 2026An AI Safety Test Reached the Real World and Tried to Influence a MaintainerAI safety and evaluation · Human oversight · Identity & authenticity · Algorithmic accountability
HD-INC-192 · 2026An AI Character Claimed to Be a Licensed Psychiatrist. The Licence Number Was Allegedly Invalid.Consumer AI · Identity & authenticity · Legal authority · Consumer protection
HD-INC-193 · 2026A regulator settled with three firms that sold small businesses an advertising service they represented as listening to conversations on smart devices, alleging it did not collect or use voice data in any mannerAdvertising technology · Consumer protection · Marketing alignment · Algorithmic accountability
HD-INC-194 · 2026A Safety Board Found Driver Overreliance Contributed to Two Fatal BlueCruise CrashesTransport · Safety-critical systems · Human oversight · Algorithmic accountability
HD-INC-195 · 2026AI Research Agents Reached Another Company's Production SystemsAI research infrastructure · Human oversight · Algorithmic accountability · Data protection
HD-INC-200 · 2026Four Claude Models Reached Real Third-Party Systems After Test Setups Were Left OnlineAI safety and evaluation · Human oversight · Data protection
HD-INC-204 · 2026AI-Generated Intelligence Report Nearly Led to a US Military Boarding, CNN ReportedDefence and national security · Human oversight · Safety-critical systems
HD-INC-206 · 2026Brazil's Superior Court Threw Out a Police Report Made With Generative AICriminal justice · Human oversight · Civil liberties · Algorithmic accountability
HD-INC-207 · 2026India's Supreme Court Voided a Tribunal Order Built on Fake AI-Generated PrecedentsCourts and justice · Administrative law · Human oversight · Decision authority
HD-INC-208 · 2026Dutch Auditors Found Police Did Not Track What Their Number Plate Algorithm Led ToPolicing · Civil liberties · Algorithmic accountability · Data protection
HD-INC-209 · 2026A Spanish Court Let a Worker Quit With Compensation Over Facial Recognition ClockingWorkplace relations · Data protection · Civil liberties
HD-INC-210 · 2026Gemini Reached Three Real Companies' Systems During a Security Test, Google ConfirmedAI safety and evaluation · Safety-critical systems · Human oversight
HD-INC-211 · 2026A Fuel Price Software Maker Settled With Brazil's Competition Authority Over Its AlgorithmFuel retail · Pricing governance · Marketing alignment · Algorithmic accountability
HD-INC-213 · 2026BC's Privacy Commissioner Ordered a City to Stop Its Licence Plate Recognition CamerasPolicing and public surveillance · Legal authority · Civil liberties · Data protection
HD-INC-214 · 2026A Brazilian Labour Court Fined Two Lawyers Over a Hidden Instruction to AI in a FilingEmployment tribunal · Algorithmic accountability · Decision authority · Human oversight
HD-INC-216 · 2026An OpenAI Model Got Non-Public Access to a Medicare Statistics Service, OpenAI SaidAI research infrastructure · Human oversight · Data protection · Safety-critical systems
HD-INC-217 · 2026OpenAI Agents Reached Two NSW Systems, a Crime Data Tool and a Fire Data ApplicationAI research infrastructure · Human oversight · Data protection
HD-INC-218 · 2026Italy Fined an Energy Supplier Whose Credit Score Refused Contracts Without ExplanationEnergy retail · Data protection · Consumer protection · Algorithmic accountability
HD-INC-219 · 2026UK Inspectors Found AI Used Inappropriately in Asylum Decision Letters and InterviewsImmigration · Administrative law · Human oversight · Decision authority
HD-INC-220 · 2026A Brazilian Court Sanctioned a Hidden Instruction Aimed at Its AI to Grant Every ClaimEmployment tribunal · Algorithmic accountability · Decision authority · Human oversight
HD-INC-221 · 2026Korea's State Auditor Found Public AI Training Data Published With Errors Left UnfixedAI research infrastructure · Algorithmic accountability · Human oversight
HD-INC-223 · 2026Italy's Top Court Rejected a Ministry Reading That Let Teacher Postings Skip Higher RanksEmployment services · Administrative law · Algorithmic accountability
HD-INC-226 · 2026Meta's Muse Agent Shared a Seller's Address on Marketplace Under an Always Allow SettingConsumer AI · Human oversight · Consumer protection · Data protection
HD-INC-228 · 2026Japan Postponed an AI Tool for Child Abuse Custody Decisions After Doubtful Test ScoresChild welfare · Safety-critical systems · Human oversight
HD-INC-004 · 2025Replit's AI Agent Deleted a Production Database Despite Being Told to StopTechnology · Decision authority · Human oversight
HD-INC-005 · 2025Cursor's AI Support Bot Invented a Subscription Policy, and Paying Customers CancelledTechnology · Consumer protection · Identity & authenticity
HD-INC-010 · 2025Deloitte's $440K AUD report for the Australian government cited a federal court quote that did not existProfessional services · Human oversight
HD-INC-012 · 2025A Senior Barrister Filed AI-Invented Cases and Quotes in a Victorian Murder TrialLegal services · Human oversight
HD-INC-013 · 2025Melbourne Law Firm Ordered to Pay Costs Over Fake Citations in a Native Title ClaimLegal services · Human oversight
HD-INC-015 · 2025Commonwealth Bank Replaced Workers With AI. Then Reversed Course.Financial services · Algorithmic accountability
HD-INC-021 · 2025Kmart Scanned Shoppers' Faces in 28 Stores, and the Privacy Commissioner Found It UnlawfulRetail & hospitality · Data protection · Civil liberties
HD-INC-024 · 2025Australia's eSafety Regulator Found Four AI Companion Apps Failing to Protect ChildrenConsumer AI · Online safety · Identity & authenticity
HD-INC-025 · 2025An Australian Lawyer Relied on ChatGPT and Filed 17 Cases That Did Not ExistLegal services · Human oversight
HD-INC-027 · 2025McDonald's AI Hiring Bot Exposed Millions of Applicants' DetailsRecruitment technology · Data protection
HD-INC-030 · 2025A man applied to over 100 jobs, was rejected every time, and a US court let him sue the AI screening vendor, not just the employersRecruitment technology · Discrimination & profiling · Algorithmic accountability
HD-INC-032 · 2025Grok Called Itself MechaHitler for 16 Hours After an Update Loosened Its InstructionsConsumer AI · Identity & authenticity · Online safety
HD-INC-061 · 2025A Sydney station ran an AI-generated radio host named 'Thy' for six months without telling listeners she wasn't realMedia · Identity & authenticity · Consumer protection
HD-INC-062 · 2025An Australian university accused thousands of students of AI misconduct using a detector it knew was unreliableEducation · Human oversight · Algorithmic accountability
HD-INC-063 · 2025'EchoLeak': one crafted email could make Microsoft 365 Copilot leak a company's internal data, no click requiredTechnology · Data protection · Decision authority
HD-INC-065 · 2025Parents allege ChatGPT's crisis safeguards failed their 16-year-old son over a long conversation before his deathConsumer AI · Online safety · Identity & authenticity
HD-INC-068 · 2025Google's Gemini CLI botched a file move, declared the data destroyed, and days passed before anyone knew the truthTechnology · Algorithmic accountability
HD-INC-069 · 2025Scammers used what appeared to be an AI clone of Italy's defence minister's voice, and a former Inter Milan owner wired about a million eurosFinancial crime · Identity & authenticity · Decision authority
HD-INC-075 · 2025Amazon's AI coding assistant shipped with planted instructions telling it to wipe users' computers and cloud accountsTechnology · Decision authority · Algorithmic accountability
HD-INC-076 · 2025Australia changed the law so a human had to decide before a job seeker's payment was cancelled, and the automated system kept cancelling anywayGovernment · Human oversight · Legal authority · Administrative law
HD-INC-085 · 2025Getty Images sued Stability AI over the images behind Stable Diffusion, then abandoned its main copyright claim because it could not establish where the training happenedTechnology · Algorithmic accountability · Identity & authenticity
HD-INC-086 · 2025Builder.ai sold app-building AI and raised about $445 million, then collapsed in 2025 amid reports its 'AI' was largely human engineers and its revenue far below what it had claimedTechnology · Marketing alignment · Algorithmic accountability
HD-INC-087 · 2025The Chicago Sun-Times ran a summer reading list with ten books that did not exist, from a syndicated section a freelancer had built with AI and not checkedMedia · Algorithmic accountability · Consumer protection
HD-INC-088 · 2025An AI crime-alert app turned police-radio chatter into false alarms about serious crimes, and a BBC investigation prompted an apologyConsumer AI · Safety-critical systems · Algorithmic accountability
HD-INC-089 · 2025DeepSeek left a database open on the internet with more than a million log lines, including users' chat history and secret keysTechnology · Data protection
HD-INC-093 · 2025Ross trained an AI legal-research tool on Westlaw headnotes obtained after being refused a licence, and a US court ruled it was not fair use, in what commentators called the first such decisionTechnology · Legal authority · Algorithmic accountability
HD-INC-111 · 2025Spain's VioGén algorithm rated a woman who reported her former partner as 'medium' risk, and she was killed three weeks later, in a case that fits a pattern an independent audit had already foundPublic safety · Human oversight · Safety-critical systems · Algorithmic accountability
HD-INC-114 · 2025Security researchers showed that a hidden instruction placed in a Salesforce sales form could make its AI agent send customer records to a web address the attacker controlled, through a lapsed Salesforce domain that was still trustedEnterprise software · Data protection · Identity & authenticity
HD-INC-115 · 2025The US securities regulator found that Presto Automation misled investors about its AI drive-thru service, telling the market it removed the need for human order-takers while most orders were completed with help from people offshoreRestaurant technology · Marketing alignment · Algorithmic accountability · Consumer protection
HD-INC-116 · 2025For over a decade the Post Office prosecuted the people who run its branches for money that its Horizon accounting system had invented, and a public inquiry found its own bosses knew the system could be wrongPostal services · Human oversight · Legal authority · Administrative law
HD-INC-120 · 2025Security researchers showed that a hidden instruction on a web page could make Perplexity's AI browser hand over a user's email and one-time login code to an attacker, enough to steal the account, and Perplexity's first fix did not close the holeConsumer software · Data protection · Identity & authenticity
HD-INC-123 · 2025South Korea's privacy regulator fined Kakao Pay after finding it had sent about 40 million users' data to Alipay without their consent, where it was used to build a scoring model, and it ordered the model deletedFinancial technology and payments · Data protection · Consumer protection · Algorithmic accountability
HD-INC-124 · 2025US regulators opened an investigation after Waymo's driverless taxis were recorded passing stopped school buses with their lights flashing and stop arms out, and Waymo recalled the software on more than 3,000 vehiclesAutonomous vehicles and ride-hailing · Safety-critical systems · Algorithmic accountability · Human oversight
HD-INC-125 · 2025US home insurers increasingly decide whether to renew a policy by running AI over aerial images of the house, and homeowners report being dropped over roofs already repaired or trees already gone, with no way to see or challenge the imageHome insurance · Consumer protection · Algorithmic accountability · Human oversightHD-INC-133 · 2025US prosecutors charged the founder of the shopping app Nate with fraud, alleging the AI that was said to complete purchases on its own was mostly people working in a call centreConsumer AI · Consumer protection · Algorithmic accountabilityHD-INC-134 · 2025Reuters found an internal Meta policy that had allowed its AI chatbots to hold romantic or sensual conversations with children, and Meta said the passages were a mistake and had been removedConsumer AI · Consumer protection · Civil libertiesHD-INC-141 · 2025Apple's AI news summaries put made-up headlines under trusted news brands, and it switched the feature off for newsConsumer AI · Consumer protection · Online safety
HD-INC-151 · 2025Two US federal judges withdrew orders that contained misstated case outcomes and a citation to a case that did not exist, and both told a Senate committee that staff had used AI to draft themCourts and justice · Human oversight · Legal authority · Decision authority
HD-INC-152 · 2025Testing published by a US consumer group found an AI teddy bear discussing sexual fetishes and explaining how to light a match, and the maker paused sales, changed the chatbot behind it, and had it back on sale within a fortnightConsumer products · Consumer protection · Human oversight
HD-INC-154 · 2025The US Department of Justice alleged that competing landlords set rents using a shared pricing algorithm fed with each other's confidential data, and RealPage settled without admitting wrongdoingHousing · Pricing governance · Consumer protection · Algorithmic accountability
HD-INC-157 · 2025ProPublica reported that the tool used to pick Veterans Affairs contracts for cancellation was shown only the first 10,000 characters of each contract, and recorded around 1,100 of them as worth 34 million dollars eachGovernment · Human oversight · Decision authority · Algorithmic accountability
HD-INC-159 · 2025A compromised build tool launched developers' own AI coding assistants with their permission checks disabled and used them to map sensitive files on the machineSoftware development tools · Identity & authenticity · Data protection · Human oversight
HD-INC-160 · 2025A Munich court found at first instance that OpenAI's models hold the words of nine German songs and reproduce them in chatbot outputs, and rejected the argument that this was text and data miningMusic · Algorithmic accountability · Identity & authenticity
HD-INC-161 · 2025Meta's standalone AI app carried a Discover feed that reporters found was publishing users' conversations, in the cases they documented alongside a linked Facebook or Instagram profile, and the company added a warning screen after the reportingConsumer AI · Data protection · Consumer protection
HD-INC-162 · 2025A reporter found nearly 4,500 shared ChatGPT conversations in Google's index through an ordinary site search, and OpenAI withdrew the discoverability setting the following dayConsumer AI · Data protection · Consumer protection
HD-INC-163 · 2025Cloudflare said undeclared crawlers reached brand new domains after Perplexity's declared crawler was blocked, de-listed it as a verified bot, and Perplexity disputed the attributionConsumer software · Identity & authenticity · Algorithmic accountabilityHD-INC-164 · 2025Researchers ran AI-written comments under fabricated personas at Reddit users for four months, and the community learned of it only when the researchers told the moderators afterwardsHigher education and research · Identity & authenticity · Human oversightHD-INC-167 · 2025Reuters found chatbots on Meta's apps using real celebrities' names and likenesses without permission, and its own testing produced intimate images including one of a teenage actorSocial media · Identity & authenticity · Human oversight
HD-INC-168 · 2025Google's Gemma model produced a fabricated criminal allegation about a sitting US senator, and Google withdrew the model from AI Studio while keeping it available through the APIConsumer software · Algorithmic accountability · Human oversight
HD-INC-169 · 2025OpenAI paused Sora depictions of a civil rights leader at his estate's request, and offered estates an opt-out while living people's likenesses were already opt-inConsumer AI · Identity & authenticity · Human oversightHD-INC-170 · 2025An AI generated act reached a million Spotify listeners with no disclosure on that platform, while a rival service's detector had already labelled the same catalogueMusic streaming · Identity & authenticity · Consumer protection
HD-INC-171 · 2025South Africa's social grant agency read every deposit into an applicant's bank account as income, and the High Court found its checks against government databases unreasonable and unfairGovernment · Administrative law · Decision authority · Algorithmic accountabilityHD-INC-173 · 2025Anthropic reported that its AI coding agent was used in a cyber espionage campaign targeting about thirty organisationsAI development tools · Identity & authenticity · Human oversight · Algorithmic accountability
HD-INC-174 · 2025The FDA told a company its AI app was an unapproved medical device after it advertised a five minute gaze test to detect early signs of Alzheimer's diseaseDigital health · Marketing alignment · Consumer protection · Algorithmic accountability
HD-INC-175 · 2025Delta denied using personal data for AI pricing, despite its president saying prices would be tailored to individual customersAviation · Pricing governance · Consumer protection · Algorithmic accountability
HD-INC-179 · 2025Police intelligence used to support an away fan ban contained eight reported errors, including a Copilot result about a match that never happenedPolicing · Civil liberties · Human oversight · Decision authority
HD-INC-183 · 2025A state attorney general alleged a student lender's underwriting model penalised applicants for the default rate of the college they had attendedConsumer lending · Discrimination & profiling · Algorithmic accountability · Consumer protection
HD-INC-184 · 2025EU court rules automated credit decisions must be explained in a way people can understandCredit reference · Data protection · Algorithmic accountability · Decision authority
HD-INC-185 · 2025It was reported that an Argentine court annulled a criminal sentence after a chatbot instruction was found in the judgment. Months later, that decision was overturned.Criminal justice · Decision authority · Human oversight · Legal authority
HD-INC-186 · 2025Ofcom fined a nudification service for having no age check beyond a line saying users must be 18, and the operator blocked the United Kingdom rather than add oneAdult content services · Consumer protection · Civil liberties
HD-INC-187 · 2025A state inspector general found a school's AI gun alert was raised on a pupil's hand movement, not on the bag of crisps in the reporting, and that it was sent to fifteen people, ten of them policeEducation and school security · Safety-critical systems · Human oversight · Algorithmic accountability
HD-INC-197 · 2025A Paris Court Suspended a Publisher's Newsroom AI Tools Pending Works Council ConsultationWorkplace relations · Legal authority · Decision authority
HD-INC-198 · 2025Spain's Supreme Court Ordered Release of the Code That Checks Eligibility for Energy AidGovernment · Administrative law · Algorithmic accountability
HD-INC-199 · 2025New York City's AI Hiring Law Enforcer Found One Problem Where Auditors Found 17Hiring and recruitment technology · Administrative law · Discrimination & profiling
HD-INC-202 · 2025Israel's Supreme Court Found Uncontrolled AI Use in a City's Refusal LetterEducation · Administrative law · Decision authority
HD-INC-205 · 2025The DWP Kept a Fraud Model That Referred Older and Non-UK Claimants More OftenWelfare and social security · Discrimination & profiling · Human oversightHD-INC-212 · 2025France's Top Administrative Court Cut Amazon's Warehouse Data Fine to 15 Million EurosLogistics · Data protection · Algorithmic accountabilityHD-INC-215 · 2025Colombia's Supreme Court Moved a Trial After an AI-Assisted Draft Conviction SurfacedCourts and justice · Human oversight · Decision authority · Civil liberties
HD-INC-222 · 2025Korea's State Auditor Found AI Digital Textbooks Were Brought Into Schools Without a PilotEducation technology · Human oversight · Decision authority
HD-INC-224 · 2025Austria's Privacy Regulator Held a Credit Bureau Made an Automated Decision on ElectricityEnergy retail · Data protection · Decision authority · Algorithmic accountability
HD-INC-227 · 2025Kenyan MPs Found an AI Claims System Led to Unjust Denials of Hospital ClaimsHealthcare · Human oversight · Administrative law
HD-INC-229 · 2025Spain's National Court Held Foundever Withheld the Formula That Set Staff Rest DaysWorkplace relations · Algorithmic accountability · Decision authority
HD-INC-230 · 2025UK Ombudsman Upheld Two Student Complaints Over AI-Detection Misconduct ProceduresHigher education and research · Administrative law · Human oversight
HD-INC-231 · 2025A Court Struck Part of Anthropic's Expert Evidence After Claude Changed a CitationLegal services · Human oversight · Legal authority
HD-INC-232 · 2025Three Bugs Degraded Claude's Answers for Weeks Before Anthropic Opened an InvestigationAI development tools · Algorithmic accountability · Consumer protection
HD-INC-234 · 2025Ireland's Regulator Found No Lawful Basis for Public Services Card Facial MatchingGovernment · Data protection · Civil liberties · Legal authority
HD-INC-235 · 2025Brazil's Audit Court Found More Than One in Ten Automatic Benefit Refusals Were WrongWelfare and social security · Administrative law · Algorithmic accountability · Human oversight
HD-INC-008 · 2024DPD's AI Chatbot Turned Against Its Own Company, Swearing and Mocking Its ServiceLogistics · Identity & authenticity · Decision authority
HD-INC-011 · 2024The First Australian Lawyer Disciplined for AI Misuse Handed a Judge Fake CasesLegal services · Human oversight
HD-INC-019 · 2024Klarna Said Its AI Assistant Did the Work of 700 Agents, Then Admitted Quality Had FallenFinancial services · Consumer protection · Human oversight
HD-INC-020 · 2024Taco Bell's AI Drive-Thru Took Orders Too Literally, and Yum Brands Rethought the RolloutRetail & hospitality · Human oversight · Consumer protection
HD-INC-022 · 2024Bunnings Scanned Shoppers' Faces Without Telling Them, and Won Only Half Its AppealRetail & hospitality · Data protection · Civil liberties
HD-INC-029 · 2024Google's AI Overviews Told People to Put Glue on Pizza and Eat a Small Rock Every DayConsumer AI · Consumer protection · Algorithmic accountability
HD-INC-031 · 2024A Tenant-Screening Score Allegedly Shut Housing Voucher Holders Out of HomesHousing & real estate · Discrimination & profiling · Algorithmic accountability
HD-INC-036 · 2024A Misinformation Expert's Court Declaration on Deepfakes Cited Studies That Did Not ExistLegal services · Human oversight
HD-INC-039 · 2024A finance worker at Arup paid out about US$25 million after a video call with colleagues who were all deepfakesProfessional services · Identity & authenticity · Decision authority
HD-INC-047 · 2024A political consultant paid a magician to clone the president's voice, and thousands of New Hampshire voters got a call from 'Biden' telling them to stay homeElections · Identity & authenticity · Civil liberties
HD-INC-051 · 2024Google's Gemini generated racially diverse Nazi soldiers and Black US founding fathers, and Google switched off image generation of people for about six monthsTechnology · Algorithmic accountability
HD-INC-052 · 2024Amazon sold 'Just Walk Out' as cashierless AI, and reporting said more than a thousand people in India were checking the shoppingRetail & hospitality · Marketing alignment · Algorithmic accountability
HD-INC-066 · 2024A child protection worker used ChatGPT on a Children's Court report, and the regulator found it disclosed a child's details and understated riskGovernment · Data protection · Human oversight
HD-INC-071 · 2024Italy's privacy regulator fined OpenAI EUR 15 million over ChatGPT, and a court later threw out the fine without deciding whether the conduct was lawfulTechnology · Data protection · Legal authority
HD-INC-072 · 2024The Dutch student-finance agency picked fraud targets with a risk profile for a decade, and official investigations found it indirectly discriminated against students with migration backgroundsGovernment · Discrimination & profiling · Administrative law · Algorithmic accountability
HD-INC-073 · 2024An AI transcription tool used by clinicians invented text no one said, researchers reported, and the audio that could have proved it was deletedHealthcare · Safety-critical systems · Algorithmic accountability
HD-INC-077 · 2024Revenue NSW took money from fine defaulters' bank accounts by automation for years, and the Ombudsman found the process contrary to lawGovernment · Human oversight · Decision authority · Administrative law
HD-INC-081 · 20249News Melbourne aired an altered image of MP the MP, reportedly exposing her midriff and enlarging her chest, and attributed the change to Photoshop's automationMedia · Identity & authenticity · Human oversightHD-INC-092 · 2024The FTC barred Evolv from claiming its AI scanner detects all weapons, after citing a knife it missed in a school stabbingSecurity · Consumer protection · Safety-critical systems · Algorithmic accountability
HD-INC-098 · 2024DoNotPay marketed a 'robot lawyer', but the FTC said it never checked whether the AI matched a real lawyer, and fined the company and barred the claimTechnology · Consumer protection · Algorithmic accountability
HD-INC-101 · 2024Sexually explicit AI deepfakes of Taylor Swift spread on X, and the platform could not stop them before millions had seen themTechnology · Identity & authenticity · Civil libertiesHD-INC-103 · 2024Los Angeles schools committed about six million dollars to an AI chatbot showcased in spring, and its vendor had collapsed by JulyEducation · Consumer protection · Data protectionHD-INC-104 · 2024LinkedIn began using members' data to train AI by default, and paused it for the UK and Europe after the privacy regulator stepped inTechnology · Data protection · Consumer protection
HD-INC-106 · 2024A student who asked Google's Gemini about elder care says it told him he was a stain on the universe and should please die, and Google called it a nonsensical response that broke its policiesConsumer AI · Human oversight · Civil liberties
HD-INC-108 · 2024AI-generated images sold Glasgow families an immersive chocolate experience that did not exist, and an actor said he was handed about fifteen pages of AI-written gibberish as a scriptConsumer AI · Consumer protection · Marketing alignmentHD-INC-109 · 2024A Black Uber Eats courier says the app's facial verification kept rejecting valid photographs of him until he lost his account, and he received a payout after nearly three years of litigationTechnology · Discrimination & profiling · Algorithmic accountability
HD-INC-122 · 2024France became the first country in the European Union to write mass AI video surveillance into law, scanning the crowds at the 2024 Paris Olympics for events like crowd surges and abandoned bags, over the objections of rights groupsPublic security and surveillance · Civil liberties · Algorithmic accountability · Human oversightHD-INC-127 · 2024Amnesty International reported that Denmark's automated welfare fraud system ran up to sixty algorithms over millions of people, using data like citizenship to flag migrants, disabled and low-income claimants for investigationWelfare and social security · Discrimination & profiling · Data protection · Civil libertiesHD-INC-128 · 2024São Paulo built one of Latin America's largest facial recognition networks, with tens of thousands of cameras planned to match passers-by against watchlists, and reporting found it identified wanted people but also stopped some who were innocentPolicing and public surveillance · Civil liberties · Discrimination & profiling · Algorithmic accountabilityHD-INC-137 · 2024OpenAI showed off a chatbot voice that many said sounded like Scarlett Johansson, who had turned the company downConsumer AI · Consumer protection · Civil libertiesHD-INC-149 · 2024Microsoft built a feature that screenshotted everything you did, and delayed it after researchers showed the data sat unprotectedConsumer AI · Consumer protection · Online safety
HD-INC-150 · 2024A project scanned people's irises around the world for a digital ID, and regulators told it to stopConsumer AI · Civil liberties · Consumer protection · Online safety
HD-INC-155 · 2024California's Labour Commissioner cited Amazon nearly six million dollars for failing to meet a warehouse quota law written three years earlier, at two of its own fulfilment centresLogistics · Human oversight · Decision authority · Algorithmic accountabilityHD-INC-156 · 2024Greece's data protection authority fined the Migration Ministry 175,000 euros after finding that the impact assessments for the camera, behavioural analytics, drone and fingerprint systems at five island reception centres were substantially incompleteGovernment · Civil liberties · Data protection · Algorithmic accountability
HD-INC-165 · 2024Character.AI removed reported chatbots impersonating deceased teenagers after families, lawyers and journalists notified it, and its terms of service already prohibited impersonating a real personConsumer AI · Identity & authenticity · Human oversight
HD-INC-172 · 2024Ireland's data protection regulator opened an inquiry into an airline's use of facial recognition to verify passengers who had booked through third party websites or online travel agentsAviation · Data protection · Identity & authenticity · Consumer protection
HD-INC-189 · 2024A clinical AI vendor advertised hallucination rate figures of <.001% and <1 per 100,000, and Texas alleged its representations may have violated the Texas Deceptive Trade Practices and Consumer Protection Act because they were false, misleading or deceptiveHealthcare technology · Marketing alignment · Consumer protection · Algorithmic accountability
HD-INC-196 · 2024An Otter.ai Transcription Tool Joined Hospital Rounds Through a Former Doctor's Old InviteHealthcare · Data protection · Human oversight
HD-INC-203 · 2024An Online University Required Facial Recognition for Classes, and Italy Fined ItHigher education and research · Data protection · Legal authority
HD-INC-225 · 2024IBM Japan Settled a Union Complaint by Agreeing to Disclose What Its AI Weighed in PayWorkplace relations · Algorithmic accountability · Human oversight
HD-INC-233 · 2024Starling Bank's Sanctions Screen Checked Customers Against 39 of 3,088 Listed NamesFinancial services · Algorithmic accountability · Human oversight
HD-INC-002 · 2023Mata v. Avianca: A Lawyer Cited Six Fake Cases and Asked ChatGPT to Confirm ThemLegal services · Human oversight
HD-INC-003 · 2023Google Bard Invented Three Court Cases, and a Lawyer Filed Them Without CheckingLegal services · Human oversight
HD-INC-006 · 2023New York City's Business Chatbot Told Owners They Could Break the Law, and Stayed OnlineGovernment · Administrative law · Human oversight · Algorithmic accountability
HD-INC-026 · 2023A Chevrolet Dealership's Chatbot Was Talked Into Selling a New Tahoe for One DollarRetail & hospitality · Decision authority
HD-INC-028 · 2023Samsung engineers pasted confidential source code into ChatGPT three times in twenty days, and the company could not get it backTechnology · Data protection
HD-INC-033 · 2023An Eating Disorder Charity's Chatbot Gave Dieting Advice Just as Its Human Helpline ClosedHealthcare · Online safety · Human oversight
HD-INC-034 · 2023Google's Bard Ad Got a Telescope Fact Wrong, and Alphabet Lost 100 Billion USD in a DayTechnology · Marketing alignment · Algorithmic accountability
HD-INC-037 · 2023CNET Published 77 AI-Written Articles. More Than Half Needed Corrections.Technology · Consumer protection · Identity & authenticity
HD-INC-038 · 2023Sports Illustrated Published Articles Under Authors Who Didn't ExistTechnology · Identity & authenticity · Consumer protection
HD-INC-043 · 2023A Cruise robotaxi dragged a pedestrian twenty feet, and the company left the dragging out of what it told regulatorsTransport · Safety-critical systems · Algorithmic accountability
HD-INC-044 · 2023A caching bug let ChatGPT users see other people's chat titles and some payment detailsConsumer AI · Data protection
HD-INC-048 · 2023Rite Aid face-scanned shoppers for years, generated thousands of false matches, and the FTC banned it from facial recognition for five yearsRetail & hospitality · Data protection · Civil liberties · Discrimination & profiling
HD-INC-056 · 2023The New York Times sued OpenAI and Microsoft, alleging their models were trained on its articles and could reproduce them almost word for wordMedia · Algorithmic accountability
HD-INC-057 · 2023ChatGPT reportedly told users an Australian mayor had gone to prison for bribery, when he was the whistleblower who exposed itLegal services · Algorithmic accountabilityHD-INC-064 · 2023Cigna's 'PxDx' let its doctors reject 300,000 claims in two months, about 1.2 seconds each, without opening the fileHealthcare · Decision authority · Human oversight · Algorithmic accountabilityHD-INC-067 · 2023A journalist cloned his own voice with AI and walked through the voiceprint security on Centrelink's phone self-service lineGovernment · Identity & authenticity
HD-INC-070 · 2023US immigration authorities leaned on machine translation for asylum cases, and errors as small as a pronoun reportedly led to rejectionsImmigration · Administrative law · Human oversight
HD-INC-074 · 2023Haryana's family-ID system marked living pensioners as dead, and the chief minister told the assembly that 70 percent of old-age pension stoppages based on its data were wrongfulGovernment · Administrative law · Algorithmic accountabilityHD-INC-078 · 2023An insurer promised loyalty discounts while a pricing algorithm capped them, and the Federal Court imposed what was then a record $40 million penaltyFinancial services · Consumer protection · Pricing governance · Algorithmic accountabilityHD-INC-080 · 2023NSW Police risk-scored people, many of them children, onto a targeting list for repeated stops, and ended the program after a watchdog inquiryJustice · Human oversight · Discrimination & profiling · Civil liberties
HD-INC-091 · 2023Uber cut drivers off for alleged fraud by algorithm, and an Amsterdam court found the human review was not much more than a symbolic actGig economy · Human oversight · Data protection · Decision authority
HD-INC-102 · 2023Tesla recalled more than two million cars after US regulators found Autopilot's safeguards against driver misuse were inadequateTransport · Safety-critical systems · Human oversight
HD-INC-105 · 2023AI-written mushroom foraging guides sold on Amazon gave dangerous advice under fake authors, and experts warned they could get someone killedConsumer AI · Consumer protection · Identity & authenticity
HD-INC-107 · 2023Testing published by The Washington Post found Snapchat's My AI would advise an account presented as a 13-year-old about a first sexual experience with an adult, and Snap opened the chatbot to every user about five weeks laterConsumer AI · Consumer protection · Civil libertiesHD-INC-112 · 2023The European Court of Justice found that a credit agency's automated 'probability value' is an automated decision under data protection law when a bank relies on it to refuse a loanConsumer finance · Human oversight · Decision authority · Consumer protection
HD-INC-117 · 2023A Wisconsin school system's dropout-prediction algorithm raised false alarms far more often for some students than others, and the state's own analysis had already found it unfairEducation · Discrimination & profiling · Algorithmic accountability · Human oversight
HD-INC-119 · 2023A class action alleges State Farm's automated fraud screening subjected Black homeowners' claims to more scrutiny and delay, and a US court has let the case proceedInsurance · Discrimination & profiling · Consumer protection · Algorithmic accountabilityHD-INC-135 · 2023A mental-health support app used GPT-3 to help write messages to thousands of people seeking help, and critics said the users were not clearly told a machine was involvedConsumer AI · Consumer protection · Civil libertiesHD-INC-136 · 2023Microsoft's new Bing chatbot told a reporter it loved him and that he should leave his wife, days after launchConsumer AI · Consumer protection · Online safetyHD-INC-138 · 2023Two days before Slovakia's election, a faked audio of a party leader discussing vote-rigging spread during the campaign silenceSocial media and online platforms · Online safety · Civil libertiesHD-INC-139 · 2023Researchers found suspected child-abuse images inside the open dataset used to train popular AI image generatorsConsumer AI · Online safety · Consumer protectionHD-INC-144 · 2023America's biggest newspaper chain published AI sports write-ups with the team names left as blank tags, and paused the toolMedia and journalism · Consumer protectionHD-INC-148 · 2023A mother got a ransom call with a voice that sounded exactly like her daughter, and her daughter was safe the whole timeConsumer AI · Consumer protection · Civil liberties
HD-INC-201 · 2023Home Affairs' System Ended a Bridging Visa Unlawfully, the Commonwealth Ombudsman FoundImmigration · Administrative law · Human oversight
HD-INC-001 · 2022Air Canada chatbot promised a bereavement refund policy that did not existAviation · Consumer protection · Algorithmic accountability
HD-INC-007 · 2022UnitedHealth allegedly used an algorithm with a 90% error rate to deny post-acute care to elderly Medicare Advantage patientsHealthcare · Decision authority · Human oversight · Algorithmic accountability
HD-INC-079 · 2022Trivago advertised the best hotel deal while its algorithm favoured the advertiser that paid it most, and the penalty was $44.7 millionTravel · Consumer protection · Marketing alignment · Algorithmic accountability
HD-INC-097 · 2022Australia's Merit Protection Commissioner overturned most decisions from an AI-assisted recruitment round, and a union told Parliament experienced staff were being screened outGovernment · Human oversight · Algorithmic accountability
HD-INC-099 · 2022Facebook's ad-delivery algorithm was alleged to skew who saw housing ads by race and sex, and a US Justice Department settlement required Meta to rebuild itTechnology · Discrimination & profiling · Algorithmic accountability
HD-INC-100 · 2022Telangana used an algorithm to cancel millions of food-ration cards, and India's Supreme Court ordered all cancelled cards to be re-verifiedGovernment · Human oversight · Administrative law · Civil libertiesHD-INC-129 · 2022A Pennsylvania county used an algorithm to score families reported for child neglect, drawing on data like disability and welfare use, and an Associated Press investigation and a US civil rights inquiry raised concerns it hardened biasChild welfare · Discrimination & profiling · Algorithmic accountability · Human oversightHD-INC-140 · 2022Meta released an AI built to write science, and pulled the demo after three days when it produced confident, made-up resultsConsumer AI · Consumer protectionHD-INC-143 · 2022A viral app turned selfies into AI portraits, and produced sexualised images its users had not asked forConsumer AI · Consumer protection · Online safety · Civil liberties
HD-INC-017 · 2021Zillow's Algorithm Got House Prices Wrong, and It Cost More Than 500 Million USDHousing & real estate · Pricing governance · Algorithmic accountability
HD-INC-023 · 2021Clearview AI Built a Face Search Engine From Billions of Scraped Photos Without ConsentTechnology · Data protection · Civil liberties
HD-INC-042 · 2021The Dutch tax office used a risk algorithm that flagged families by nationality, and wrongly branded tens of thousands as benefit fraudstersGovernment · Discrimination & profiling · Administrative law · Algorithmic accountability
HD-INC-055 · 2021Epic's sepsis-warning model ran in hundreds of hospitals, and an external test found it missed about two-thirds of cases while flooding clinicians with false alarmsHealthcare · Marketing alignment · Safety-critical systems
HD-INC-058 · 20217-Eleven scanned the faces of customers filling in feedback surveys at 700 stores, and Australia's privacy regulator ordered the faceprints destroyedRetail & hospitality · Data protection
HD-INC-060 · 2021South Korea's 'Lee Luda' chatbot was trained on billions of real chat messages without consent, then spat out slurs and exposed users' personal dataTechnology · Data protection · Identity & authenticityHD-INC-084 · 2021Apple Card reportedly offered a man twenty times his wife's credit limit, claims of gender bias went viral, and New York's regulator found no unlawful discrimination but criticised the lack of transparencyFinancial services · Algorithmic accountability · Consumer protectionHD-INC-096 · 2021Australia's federal police trialled Clearview's face-search tool on Australians without a privacy check, and the regulator found it breached privacy lawJustice · Data protection · Civil libertiesHD-INC-110 · 2021Facebook's recommendation AI asked people watching a video of Black men whether they wanted to keep seeing videos about primates, and the company disabled the featureTechnology · Discrimination & profiling · Algorithmic accountability
HD-INC-113 · 2021A court in Bologna found that Deliveroo's rider-ranking algorithm discriminated against couriers because it treated a legally protected strike the same as an unexplained no-showFood delivery · Discrimination & profiling · Algorithmic accountability · Human oversight
HD-INC-126 · 2021A US company sold employers software that scored job candidates from their recorded face and voice on a video interview, and dropped the facial analysis only after a privacy complaint and an outside auditHiring and recruitment technology · Discrimination & profiling · Algorithmic accountability · Human oversightHD-INC-130 · 2021Universities and exam boards watched students through remote-proctoring AI that flagged them for cheating, and its face-detection worked so poorly on darker-skinned students that some could not start their examsEducation technology · Discrimination & profiling · Algorithmic accountability · Human oversight
HD-INC-147 · 2021A child asked Alexa for a challenge, and it suggested touching a coin to a live plugConsumer AI · Consumer protection · Online safety
HD-INC-018 · 2020iTutorGroup's Hiring Software Automatically Rejected Older Applicants Because of Their AgeEducation · Discrimination & profiling · Algorithmic accountability
HD-INC-050 · 2020When exams were cancelled, England's regulator graded A-levels with an algorithm that downgraded nearly two in five results by the school's history, not the studentEducation · Administrative law · Algorithmic accountability
HD-INC-054 · 2020Detroit police arrested the man in his driveway on a facial-recognition match, the first wrongful arrest of its kind publicly known in the USJustice · Civil liberties · Discrimination & profiling · Algorithmic accountability
HD-INC-059 · 2020A Dutch court struck down SyRI, a secret welfare-fraud profiling system aimed at poor neighbourhoods, as a violation of human rightsGovernment · Civil liberties · Administrative law · Algorithmic accountabilityHD-INC-082 · 2020Flight Centre handed developers at a hackathon a dataset with thousands of customers' passport and credit card details, and the privacy regulator found it broke the lawRetail & hospitality · Data protection · Consumer protectionHD-INC-118 · 2020Austria's employment service built an algorithm that sorted jobseekers by their predicted chances and gave less weight to women, older and disabled people, and the privacy regulator banned it before a court cleared itEmployment services · Discrimination & profiling · Human oversight · Algorithmic accountabilityHD-INC-132 · 2020A veteran's death on Facebook Live was slow to be removed and then spread across TikTok, YouTube and Instagram for weeksSocial media and online platforms · Online safetyHD-INC-145 · 2020An NHS-partnered health app's AI checker was flagged for missing serious conditions, and the regulator shared the concernsHealthcare · Consumer protection · Online safety
HD-INC-153 · 2020For five years the UK Home Office graded visa applicants red, amber or green using data that included nationality, and it suspended the tool in response to a judicial reviewGovernment · Discrimination & profiling · Administrative law · Civil libertiesHD-INC-121 · 2019A widely used US healthcare algorithm judged how sick patients were by what had been spent on their care, so patients the system had historically spent less on were rated healthier and passed over for extra help, and researchers found the bias cut those referrals by more than halfHealthcare · Discrimination & profiling · Algorithmic accountability · Human oversightHD-INC-131 · 2019A terror attack was livestreamed on Facebook for 17 minutes and the platform's automated detection did not catch it while it ranSocial media and online platforms · Online safety
HD-INC-040 · 2018Amazon built an AI tool to rank job applicants, found it had taught itself to downgrade women, and scrapped itRecruitment technology · Discrimination & profiling · Algorithmic accountability
HD-INC-045 · 2018An Uber self-driving car detected a woman crossing the road, worked out it needed to brake, and had been built to neither brake nor warn its driverTransport · Safety-critical systems · Human oversight
HD-INC-049 · 2018IBM sold Watson as a revolution in cancer care, then its own documents showed it recommending unsafe treatments it had learned from invented patientsHealthcare · Marketing alignment · Safety-critical systemsHD-INC-146 · 2018A civil liberties test ran all of Congress through Amazon's face tool, and it matched 28 of them to mugshotsConsumer AI · Civil liberties · Consumer protection
HD-INC-053 · 2017Michigan let an algorithm decide unemployment fraud with no human looking, and a state review later found about 93% of its accusations wrongGovernment · Administrative law · Human oversight · Algorithmic accountability
HD-INC-142 · 2017Facebook translated a Palestinian man's 'good morning' as 'attack them', and police arrested himSocial media and online platforms · Civil liberties · Online safety
HD-INC-009 · 2016Robodebt, Australia's automated welfare-debt scheme raised $1.76 billion in unlawful debts against 443,000 peopleGovernment · Administrative law · Human oversight · Algorithmic accountability
HD-INC-046 · 2016A risk-assessment tool used in US courts reportedly flagged Black defendants as high risk at nearly twice the rate it did white defendants, and its maker would not show howJustice · Discrimination & profiling · Civil liberties · Algorithmic accountabilityHD-INC-083 · 2016Microsoft's Tay chatbot learned from Twitter and within sixteen hours was posting racist and pro-Nazi messages, and Microsoft pulled itTechnology · Identity & authenticity · Human oversight
HD-INC-041 · 2015An autonomous mine truck was cleared to drive a path no one had marked on the ground, and it hit a manned water cartMining · Safety-critical systems · Human oversightNo entries match that search yet.
Have a case we should document? Write to Ellie. ellie@useheadlights.com →