What happened
It was reported that in November 2024 Amnesty International published a study, Coded Injustice, into how Denmark hunts for welfare fraud. It found that the agency Udbetaling Danmark, and ATP, the body that runs its systems, used up to sixty algorithms to score claimants and decide who to investigate, drawing on a wide sweep of data that included residency status, citizenship, place of birth, family relationships, travel, and health. Amnesty, which was given partial access to four of the models, said the sheer reach of the data amounted to mass surveillance.
It was reported that Amnesty found the system risked treating some groups unfairly. One model, it said, flagged people for having what the system judged to be strong ties to countries outside the European Economic Area, using citizenship as a factor; another flagged living arrangements it treated as unusual, which Amnesty said could catch disabled couples who lived apart or migrant households of several generations. Amnesty said the effect fell hardest on migrants, disabled people and those on low incomes. Udbetaling Danmark and ATP disputed the findings, said that using citizenship in this way was not the processing of sensitive data, and declined Amnesty’s request to open the models to a full independent audit. Amnesty said the people the system scored were not told why they had been flagged.
What an auditable version would have shown
A person pulled in for a fraud investigation could not see what had singled them out: which data, which rule, which score. An auditable version keeps a record for each person it flags, showing what triggered the flag and why, and makes it reachable to the person and to an oversight body, so a flag can be explained and, if it is wrong or unlawful, challenged. It would also show the bigger pattern: how the flags land across groups like citizenship, disability and income, so a system that falls hardest on one group is something the agency can measure and a regulator can read, not something an outside body has to guess at from partial access.
Where the gap was
An automated system decided who among millions of claimants to investigate, on data that reached into citizenship, family and health, and neither the flagged person nor an independent auditor could see how it worked. A ConductRecord keeps each flag with the data and rule behind it, so a person can be told why they were investigated and can contest it. A MetricRecord counts how the flags land across groups, so a system that targets migrants or disabled people shows up as a number that can be checked, not a claim the agency can simply deny. When it refused a full audit, this is what the agency held back: a record open enough for someone outside to test whether the system was fair and lawful.
What governance should have looked like
Where a state runs an automated system to decide who to investigate, the data and rules it uses have to be lawful, a person has to be able to see why they were flagged and to challenge it, and the effect across groups has to be measured and open to an independent check. Best practice would be for the agency to record each flag and its basis, to give a flagged person a real explanation, and to publish, or open to an auditor, how the flags fall across citizenship, disability and income. Amnesty could only estimate the effect because it was given partial access to four models out of sixty. A record built to be checked would let that question be answered in full, not in part.
Failure Pattern: an automated fraud system scored welfare claimants on data including citizenship and living arrangements to decide who to investigate, and the people it flagged could not see or challenge how they had been judged.
Governance Principle: where an automated system flags people for investigation, the data and rules it uses must be lawful and open to challenge, a person must be able to see why they were flagged, and the system’s effect across groups must be measured and independently checkable.
The reference implementation of ConductRecord and MetricRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Denmark: AI-powered welfare system fuels mass surveillance and risks discriminating against marginalized groups (Amnesty International)
- Coded Injustice: Surveillance and discrimination in Denmark’s automated welfare state (Amnesty International, full report)
- In a quest to optimize welfare management, Denmark built a surveillance behemoth (AlgorithmWatch)