What happened
It was reported that a person in Germany, known in the case only as OQ, was turned down for a loan because of a SCHUFA score, a number the country’s main credit agency puts on how likely someone is to keep up their repayments. When they asked SCHUFA how the figure was reached, the company refused, calling its scoring a trade secret, and the data protection regulator sided with it, so the fight went to court. On 7 December 2023 the Court of Justice of the European Union ruled that, in a case like this, the score itself counts as the decision under Article 22 of the GDPR, which says a decision this important should not be left to an algorithm alone. SCHUFA had argued that it only hands over a number and the bank makes the real choice. The court disagreed: if a low score means the bank says no almost every time, then it is the score that decides, and the person has a right to have someone look at it and to be told how it was worked out. SCHUFA said its scores are usually not the sole factor in a lender’s decision. The case went back to the German court, and because the ruling comes from the EU’s top court, there is no appeal.
What an auditable version would have shown
The dispute turned on a question no one outside the parties could answer from the record: how far the bank’s refusal actually rested on the score, and whether a person looked at the case in any way that could change the outcome. An auditable version would keep a signed record of the decision, the score received, whether a human considered anything beyond it, and the reasons given, so that the difference between a genuine human decision and a figure passed straight through is a matter of record rather than inference. The reasons a person is entitled to are only as good as the record kept at the moment the decision was made.
Where the gap was
A score generated by one company functioned as the decision at another, and the person it was about could neither see how it was produced nor identify a human who had weighed their case. A ConductRecord preserves the decision and its basis, the inputs relied on and any human step, so that whether the outcome was a real judgement or an automated pass-through can be checked afterwards. A MetricRecord holds the aggregate behaviour of the scoring, so a regulator or the affected person can see how far outcomes track the score across many cases. The Court did not decide whether OQ was treated unfairly; it decided that the scoring is a decision the law reaches, which is the point at which a record of how it was used, and a human who can account for it, has to exist.
What governance should have looked like
Where an automated score determines a consequential outcome for a person, best practice is for a human to make an identifiable decision that can be reviewed, and for the person to be able to obtain a meaningful account of how the outcome was reached, in terms that do not require the release of a proprietary formula. None of this means handing over SCHUFA’s formula. A company can tell someone which facts about them pushed the score down, and show that a person actually looked at the case, without giving away how the model works. What the case settled is that this duty falls on whoever makes the score, not only on the bank that acts on it.
Failure Pattern: an automated score produced by one company became the effective decision at another, with no meaningful human judgement in between and no explanation the affected person could obtain, until a court had to rule that the score was a decision at all.
Governance Principle: where an automated score determines a consequential outcome for a person, there must be a real human decision that can be identified and reviewed, and the person must be able to obtain a meaningful explanation of how the outcome was reached.
The reference implementation of ConductRecord and MetricRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Judgment in Case C-634/21, OQ v Land Hessen (SCHUFA Holding) (Court of Justice of the European Union, EUR-Lex)
- Court of Justice press release No 186/23 on the SCHUFA scoring judgment (Court of Justice of the European Union)