What happened
SyRI, short for Systeem Risico Indicatie, was a tool the Dutch state used to hunt for fraud in welfare, benefits and tax. It worked by pulling together personal data that government bodies had previously kept in separate silos, employment, housing, benefits, debts and more, and running it through a risk model to flag people who, by the model’s reckoning, were likely to be cheating. The flagged were singled out for investigation. The model itself was secret: neither the public nor, in practice, the courts could see how a risk indication was reached, which data weighed in it, or why one person was flagged and another not.
SyRI was not deployed evenly across the country. It was used in selected neighbourhoods, and those neighbourhoods were disproportionately poor, with higher shares of low-income and minority residents. Entire areas could be profiled without any individual suspicion, on the logic that fraud was more likely to be found where people had less. The effect was to turn living in a particular postcode into grounds for algorithmic scrutiny.
A coalition of civil-society organisations and citizens took the Dutch state to court, joined by the writers Tommy Wieringa and Maxim Februari and the trade union federation FNV. The case drew an unusual intervention: Philip Alston, then the UN Special Rapporteur on extreme poverty and human rights, filed a brief warning of the dangers of a “digital welfare state.” On 5 February 2020 the District Court of The Hague ruled the SyRI legislation unlawful. It found that the system breached Article 8 of the European Convention on Human Rights, the right to respect for private and family life, because it failed to strike the fair balance the Convention requires: the interference with people’s privacy was disproportionate to the aim, and the system suffered from a serious lack of transparency, with insufficient safeguards against the risks it created. Alston called the decision “a clear victory for all those who are justifiably concerned about the serious threats digital welfare systems pose for human rights,” and noted it was “one of the first times a court anywhere has stopped the use of digital technologies and abundant digital information by welfare authorities on human rights grounds.” SyRI was halted.
What an auditable version would have shown
The court’s objection was not only that SyRI was intrusive; it was that SyRI was opaque, and could not be checked. No one outside the system could see how it scored people, and so no one could test whether it was proportionate or whether it discriminated. The secrecy was not incidental to the harm; it was central to why the court could not let it stand. A system that profiles citizens and will not show its working asks to be trusted exactly where trust is least warranted.
An auditable version is built so that its functioning can be examined. It records each risk indication and the basis on which it was reached, and it computes a standing measure of how its flags are distributed across neighbourhoods and groups, so that disproportionate targeting of poorer areas is a visible, recomputable figure rather than a hidden property. With those records, the proportionality question the court had to resolve, does the privacy intrusion justify the aim, and does the system fall unevenly on the vulnerable, becomes answerable from evidence, by an auditor or a court, instead of being shielded behind a model no one is allowed to inspect.
Where the gap was
The gap was that a government profiling system operated without transparency or auditability: there was no legible record of how it reached its decisions and no measurement of how its scrutiny was distributed. That opacity is what made the intrusion impossible to justify and the discrimination impossible to rule out.
A ConductRecord is the control on the first: a record of each risk indication and the data behind it, so that a system profiling citizens can be inspected, contested and held to the law rather than operating as a black box. A MetricRecord is the control on the second: a signed, recomputable measure of how flags fall across areas and groups, so that disproportionate targeting shows up as monitoring an oversight body can verify, not as a suspicion advocates must prove from the outside.
What governance should have looked like
The pattern, seen here and in the automated-decision scandals elsewhere in this library, is that a government system which cannot show its working cannot demonstrate that it is proportionate or fair, and a court is left with no choice but to stop it. Transparency is not a courtesy a profiling system extends; for a public algorithm it is the precondition of being allowed to run at all.
The reference implementation of ConductRecord and MetricRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Landmark ruling by Dutch court stops government attempts to spy on the poor (UN OHCHR)
- Profiling and SyRI (PILP-NJCM, litigants’ case dossier)
- Netherlands District Court Rules Benefits Fraud Detection Tool Violates Human Rights (American Society of International Law)
- Digital welfare fraud detection and the Dutch SyRI judgment (van Bekkum & Zuiderveen Borgesius, 2021)