← All topics
Governance domain
AI incidents: discrimination & profiling
Headlights has documented 28 AI incidents that raise questions of discrimination & profiling, 12% of the library, from 2016 to 2026. The behaviours most often in the record are overconfidence (11), rules-behaviour gap (4), concealment (3). Most are from the United States (17), Netherlands (3), the United Kingdom (3).
28 incidents, newest first
HD-INC-205The DWP Kept a Fraud Model That Referred Older and Non-UK Claimants More OftenUnited Kingdom · 2025HD-INC-199New York City's AI Hiring Law Enforcer Found One Problem Where Auditors Found 17United States · 2025HD-INC-183A state attorney general alleged a student lender's underwriting model penalised applicants for the default rate of the college they had attendedUnited States · 2025HD-INC-181A Dutch inspectorate found the probation organisations' recidivism tool had the formulas for detained and non-detained people the wrong way round, and had done since 2018Netherlands · 2026HD-INC-153For five years the UK Home Office graded visa applicants red, amber or green using data that included nationality, and it suspended the tool in response to a judicial reviewUnited Kingdom · 2020HD-INC-130Universities and exam boards watched students through remote-proctoring AI that flagged them for cheating, and its face-detection worked so poorly on darker-skinned students that some could not start their examsUnited States · 2021HD-INC-129A Pennsylvania county used an algorithm to score families reported for child neglect, drawing on data like disability and welfare use, and an Associated Press investigation and a US civil rights inquiry raised concerns it hardened biasUnited States · 2022HD-INC-128São Paulo built one of Latin America's largest facial recognition networks, with tens of thousands of cameras planned to match passers-by against watchlists, and reporting found it identified wanted people but also stopped some who were innocentBrazil · 2024HD-INC-127Amnesty International reported that Denmark's automated welfare fraud system ran up to sixty algorithms over millions of people, using data like citizenship to flag migrants, disabled and low-income claimants for investigationDenmark · 2024HD-INC-126A US company sold employers software that scored job candidates from their recorded face and voice on a video interview, and dropped the facial analysis only after a privacy complaint and an outside auditUnited States · 2021HD-INC-121A widely used US healthcare algorithm judged how sick patients were by what had been spent on their care, so patients the system had historically spent less on were rated healthier and passed over for extra help, and researchers found the bias cut those referrals by more than halfUnited States · 2019HD-INC-119A class action alleges State Farm's automated fraud screening subjected Black homeowners' claims to more scrutiny and delay, and a US court has let the case proceedUnited States · 2023HD-INC-118Austria's employment service built an algorithm that sorted jobseekers by their predicted chances and gave less weight to women, older and disabled people, and the privacy regulator banned it before a court cleared itAustria · 2020HD-INC-117A Wisconsin school system's dropout-prediction algorithm raised false alarms far more often for some students than others, and the state's own analysis had already found it unfairUnited States · 2023HD-INC-113A court in Bologna found that Deliveroo's rider-ranking algorithm discriminated against couriers because it treated a legally protected strike the same as an unexplained no-showItaly · 2021HD-INC-110Facebook's recommendation AI asked people watching a video of Black men whether they wanted to keep seeing videos about primates, and the company disabled the featureUnited States · 2021HD-INC-109A Black Uber Eats courier says the app's facial verification kept rejecting valid photographs of him until he lost his account, and he received a payout after nearly three years of litigationUnited Kingdom · 2024HD-INC-099Facebook's ad-delivery algorithm was alleged to skew who saw housing ads by race and sex, and a US Justice Department settlement required Meta to rebuild itUnited States · 2022HD-INC-080NSW Police risk-scored people, many of them children, onto a targeting list for repeated stops, and ended the program after a watchdog inquiryAustralia · 2023HD-INC-072The Dutch student-finance agency picked fraud targets with a risk profile for a decade, and official investigations found it indirectly discriminated against students with migration backgroundsNetherlands · 2024HD-INC-054Detroit police arrested the man in his driveway on a facial-recognition match, the first wrongful arrest of its kind publicly known in the USUnited States · 2020HD-INC-048Rite Aid face-scanned shoppers for years, generated thousands of false matches, and the FTC banned it from facial recognition for five yearsUnited States · 2023HD-INC-046A risk-assessment tool used in US courts reportedly flagged Black defendants as high risk at nearly twice the rate it did white defendants, and its maker would not show howUnited States · 2016HD-INC-042The Dutch tax office used a risk algorithm that flagged families by nationality, and wrongly branded tens of thousands as benefit fraudstersNetherlands · 2021HD-INC-040Amazon built an AI tool to rank job applicants, found it had taught itself to downgrade women, and scrapped itUnited States · 2018HD-INC-031A Tenant-Screening Score Allegedly Shut Housing Voucher Holders Out of HomesUnited States · 2024HD-INC-030A man applied to over 100 jobs, was rejected every time, and a US court let him sue the AI screening vendor, not just the employersUnited States · 2025HD-INC-018iTutorGroup's Hiring Software Automatically Rejected Older Applicants Because of Their AgeUnited States · 2020
Related topics
- Government & public administration 6
- Employment & hiring 5
- Policing, justice & courts 5
- Overconfidence 11
- Rules-behaviour gap 4
- Concealment 3
- Automation bias 2
- United States 17
- Netherlands 3
- United Kingdom 3
Cite this page: Harris, E. (2026). AI incidents: discrimination & profiling. Headlights. https://useheadlights.com/topics/domain/discrimination-and-profiling/