90 incidents on record · 2026 Headlights Incident reports by Ellie Harris · Melbourne
10 new this week Library last updated 13 July 2026
← The incident library
HD-INC-072
Government · Netherlands · 2024 · Algorithmic discrimination

The Dutch student-finance agency picked fraud targets with a risk profile for a decade, and official investigations found it indirectly discriminated against students with migration backgrounds

By Ellie Harris · Filed Risk profile in use from 2012 to June 2023; investigation published June 2023

Alleged: Dienst Uitvoering Onderwijs (DUO) developed or deployed the AI system implicated in this incident. Details are drawn from public reports; parties are presumed innocent of any wrongdoing not established by an official finding.

The Dutch student-finance agency picked fraud targets with a risk profile for a decade, and official investigations found it indirectly discriminated against students with migration backgrounds

What happened

In June 2023, the investigative platform Investico, NOS op 3 and the Hoger Onderwijs Persbureau reported that students with migration backgrounds were strikingly overrepresented among those accused by DUO, the Dutch student-finance agency, of misusing the grant for students living away from their parents. Of 376 objection and appeal cases handled by 32 surveyed lawyers since 2012, the journalists found, 97 percent involved a student with a migration background. Behind the pattern was a risk profile DUO had used since 2012 to decide whom to check. It scored students on three apparently neutral criteria: their type of education, their age, and how close they lived to their parents. Staff then manually selected students from the scored lists for unannounced home visits, and the government’s later account records that this manual step added bias of its own. By the government’s own later account, the profile lacked a proper underpinning and was not well validated. Over the decade to 2023, about 26,800 students received home visits, and roughly 10,000 were fined or ordered to repay their grants. The minister suspended the risk-based selection within days of the reporting; checks since have used random sampling.

The official findings arrived in stages, each worse than the last. A PwC investigation delivered in March 2024 found the method amounted to indirect discrimination, though ethnicity itself was never an input, and education minister Robbert Dijkgraaf apologised on behalf of the whole cabinet, saying this should never have been allowed to happen. A follow-up study by the foundation Algorithm Audit in May 2024 found the bias was stronger than PwC’s work had suggested: students with a non-European migration background were selected as high risk about twice as often in the cohorts analysed, and across the years sampled roughly 80 to 85 percent of students who appealed a fraud decision had a non-European migration background. In November 2024 the Dutch privacy regulator, the Autoriteit Persoonsgegevens, found the processing unlawful because the three criteria lacked objective justification, and a court had found the evidence from the checks unlawfully obtained. The new education minister, Eppo Bruins, announced that every fine and clawback from the algorithm era would be reversed, reserving EUR 61 million. In April 2026 his successor announced a further damages scheme of about EUR 80 million, including payments to some 12,000 more students who were visited but never fined. The state accepted that even students who had actually misused the grant would be repaid, because the evidence against them was tainted.

What an auditable version would have shown

It took three separate investigations, spread across the year and a half after the journalists’ exposé, to establish what the system had been doing, because the discrimination lived in the aggregate and the aggregate was never measured. Worse, according to PwC’s findings, signals that bias was a risk were known inside the organisation as early as 2010 and were not adequately acted on. A signed metric record computed over the selection decisions, who was scored, who was selected, who was visited, and how the outcomes distributed across the population, would have surfaced the skew in its first year, not more than a decade in. And a conduct record for each selection, preserving the score, the criteria that produced it, and the manual choice that followed, would have separated the algorithm’s contribution from the human one, the question PwC and Algorithm Audit each had to reconstruct from fragments a decade later.

Where the gap was

A scoring system ran for eleven years with no measurement of what it was doing to whom. A MetricRecord binds signed aggregate statistics to the underlying decisions, so that a skew of twice the selection rate for one group is a number the agency’s own board sees in quarter one, rather than a finding a foundation publishes more than a decade later. A ConductRecord preserves each selection as it happened, score, criteria and human override together, which is also what makes remediation precise: the Dutch state is now repaying everyone, including actual fraud cases, because no record can distinguish a sound accusation from a tainted one.

What governance should have looked like

The reported failure is not that a risk model existed; agencies must choose whom to check. It is that, by the official findings, the choosing ran for more than a decade without ever being evaluated, that internal signals of bias were not acted on, and that a manual selection layer amplified the skew the scoring began. A selection system pointed at citizens should carry the burden of proving, continuously and from its own records, that its choices are justifiable, because when the proof is absent the eventual remedy is indiscriminate: every decision reversed, EUR 141 million reserved, and a settlement operation expected to run for years.

The reference implementation of MetricRecord and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.

Sources

The mailing list

Fresh incident reports every week. One email to match.

We add new incidents to the library regularly, and send a single short email each week with what's new. The library stays free and open; this is just how you keep up with it.

No tracking. Unsubscribe in one click.

The record

An auditable system would have produced a signed, tamper-evident record the moment this happened: what the system did, the version that did it, the basis it acted on, and the action taken, and Dienst Uitvoering Onderwijs (DUO) could have produced it on demand.

This is the record the system as deployed did not produce in a signed, auditable form.

What this teaches
Capture what happened when it happens
What the system did, the version that did it, the basis it acted on, and the action taken, recorded at the moment, not reconstructed after.
Sign it, so no one has to trust the record-keeper
A tamper-evident entry. Edit it later and the signature breaks. The record does not ask for the benefit of the doubt.
Make it verifiable by anyone
A court, a regulator, a customer's lawyer can check the record themselves, without taking the company, or us, at our word.

Headlights summarises publicly reported AI incidents. All summaries are independently written, attributed to their original sources, and intended for research and educational purposes. Allegations are identified as such until established through official findings.

Last reviewed June 2026. This report is based on the sources listed above and reflects information available at the time of review; later developments may not be captured. Where a person is described as charged with or alleged to have done something, that allegation is unproven unless a conviction or a court or regulatory finding is stated. Headlights publishes journalism and commentary, not legal advice.

Want to write back?

Direct to my inbox.

ellie@useheadlights.com →